Cell phones sync to the cell system, not the Internet, per se.
These are really the first WiFi-only devices (if you have 3G off on the 3G...), and I don't remember anything about time sync anywhere. My WiFi seems to creep somewhat, but nothing like what we've seen in this thread. You would hope that Apple would update the clock if you are connected to the Internet (which most of these are, most of the time)....
I'm watching the clock on my MBP (synced more or less perfectly to time.apple.com, I hope), and my 3G iPad (with 3G off) and it's not perfect. Off by about 10 seconds. It was iTunes-synced Friday night, and not rebooted/turned off since. (Sleep, of course.)
So, don't know where the pattern is or what is causing the issue with the ones that aren't keeping good time.
Adding... how do/how are iPod touches in keeping time?
Also, there is a 'Set Automatically' switch in Settings. Does this mean anything like it should (to time.apple.com), or just when you iTunes sync?