My 6 year old son has had his own 16gb iPod touch since he was in pre-school.
I got it off Craiglist about two years ago and he has taken very good care of it.
It's my iTunes pass word that I won't be trusting him with any time soon.
Sure it's a toy to him but it's also one of his favorite toys, it's loaded with games, movies, and a few music videos he likes. He uses it all the time and from what I can see most kids his age have no problem figuring out how to use it and would love to have one of there own if only to watch episodes of the Backyardagins and let's not forget Netflix streaming.
As for gaming, many of the games he likes have been free or quite cheep compared to the price of some of the cartridges you buy for Nintendo DS or a PSP - and I don't have to ever worry about him keeping up with games cartridges.
Now when it comes to my iPad he enjoys it also and since I keep it in a protective case I have no problem letting him handle it, it's not any bigger or much heavier than some of his books or other toys. I'll probably give him mine in few years when I decide to upgrade.
I was really glad to learn that the new iPod Touch is not all glass like the iP4 because I will definitely be buying one for my son so that he can enjoy the new camera and face time.
So the way I see it, if you can afford it buy it because your child will never get bored with it no matter how old they get they can choose a zillion games, sync different movies, do face time, Netflix, read books, listen to music and get into photography all from a pocket size device that's made well enough to survive being handled by most kids.