I'm actually quite a fan of Office 2004. Basically, it fixed the things with Office v.X that I didn't like. Disclaimer: I paid $5 for Office 2004 from my school.
First off, the programs boot up very quickly, possibly faster than their PC counterparts. Second, Word is faster overall - again, on par with its PC counterpart - as are the other programs. The reason I state these two pieces of information immediately and explicitly is because so many people have stated the opposite. The fact is, though, that I am very impressed with both the startup time and the responsiveness/speed/efficiency of each application. For comparison, I'm using my impressions of Office 97, 2000, XP (2002), and 2003 on PCs of .266 PII, .900 Duron, .933 PIII, 2.0 Celeron, 2.4 P4, and 3.0 P4 Prescott HT GHz speed. Fact is, with a program like office, there comes a point when no more speed can be gained. I'd say the startup time of Office XP and 2003 reaches its peak on the 2.0 Celeron and remains the same all the way up. Again, these are just my impressions.
Office v.X I used on an iBook G4 12" 800 MHz. I wasn't terribly impressed; there were strange font spacing issues, Quartz anti aliasing wasn't enabled on early versions, and the program was noticeably unresponsive/slow/inefficient. However, it got the job done, albeit slower than its PC counterparts.
Office 2004 I used on the iBook and now on a PowerBook G4 12" 1.33 GHz. On the iBook, I found Office 2004 an overall better implementation of the program, but given that I used an unlicensed and cracked version, I found the program was not amazing. The same holds true when I used it on the PowerBook. However, after uninstalling that installation and installing the licensed CD version from my school, I found a great performance gain. I'm not sure why I would notice a difference to be honest, but I do.
The main reason you would want to upgrade to Office 2004 is if you wish to resolve the annoyances you experienced with Office v.X. There are some, but not many, collaboration enhancements; these, however, are nowhere near their PC counterparts (read: if you're in an Exchange Server environment, don't expect to have the collaboration bells-and-whistles your Office XP and 2003 colleagues enjoy). The notebook view in Word is interesting; having beta tested OneNote on the PC, I was excited to see a Mac iteration of the program. Though not as robust as OneNote, I expect to use the notebook view quite a bit at school. In conclusion, if you have to spend over $100 to upgrade to Office 2004, you might want to consider just how annoyed with Office v.X you are. If you are annoyed to the extent that you will pay $150-300, by all means upgrade; you'll be glad you did. However, if you want to upgrade just because you like to have the latest version of everything (like me), you may want to wait for the next version. If you can get Office 2004 from a school for $0-20, absolutely upgrade.
Mike LaRiviere
Addendum
As much as everyone (hyperbole) on these forums hates Microsoft, step back a moment and consider all the Microsoft products you've bought/"shared"/used. All the ones I've used were pretty good. We're talking Word 6.0 for Mac in the early 90's, Office 97, 2000, XP, and 2003; Publisher, OneNote, and some other Microsoft innovations; MS-DOS (yes, I know Microsoft did not invent this OS), Windows 3.1, 95, 98, ME (preinstalled on a pre-XP Dell), 2000, XP Home, XP Pro, XP Pro SP2 Beta; PocketPC. I realize and agree with the fact that OS X 10.3 runs better than Windows XP SP2, but that doesn't mean Microsoft is a horrible company out to steal my money. Don't want to pay for a Microsoft product because you don't like the company? Don't pay for one then. You can always "share" a copy of Office with peer-to-peer, or you can use OpenOffice or Star Office. Don't like Windows? Don't use it then. Use Mac or some other Unix flavor. Microsoft's not a bad company in my opinion, though; they make good products that are used everywhere. Blue screen of death? Happened to me sometimes in ME, rarely in XP, but I've had many kernel panics in OS X 10.3. So yes, given the choice, I would and do use a Mac. But I also use Windows occasionally, but I don't feel the need to criticize Microsoft at every opportunity just because I've found a better product. Bill Gates has never wronged me, nor has Microsoft. Steve Jobs is not my buddy, he's just a more charismatic CEO than Gates. If you've switched, that's fine, but there's no need to insult Microsoft and Intel at every opportunity.