I agree it's partly to do with the memory in the iPods/iPhones being slow but the USB side of things is crappy. Apple should have kept the FireWire side of things. Most pc's have FireWire onboard now. A nice fw800 setup with a decent speed memory would see sync times drop alot.
I have three iPhones, a 2G, 3G and a 3GS. The 3GS can load up songs three times as fast as the two previous iPhones. I've loaded 2500 songs onto the 3GS in under 30 minutes. Most of the songs are 256-320Kbps.