When I go into computers and select my iTunes library, it will say Accessing or Loading xxxxx's Library, and give me the spinning gear for about 5-7 seconds I would say. This is with about 2,500 songs, and 1000 photos, one music video, and 3 episodes of Heroes. I haven't decided if I want to take the time to handbrake my DVDs yet, so my library is much smaller than most. So, it is possible it may be related to how large your library is. Although, if your library is similar in size to mine, then maybe something else is going on.
I will also add that when trying to watch Movie Trailers and Podcasts on my ATV, it does spin alot longer than it does directly on my mac. Movie trailers start nearly instantly in Front Row on both my MacBook and my wife's iMac (both wireless). I hope that Apple fixes this. If our Macs can start streaming near-instantly, then so should the ATV. I know it's a new unit, and so Apple will just have to put in time to squash a few bugs. Hopefully it will all be working like a well-oiled machine soon.