I gave up on locally streaming from my apple TV. it used to work fine in the past but somehow it turned to crap. like I can stream stuff from Netflix which actually has to download first just fine but as soon as I try to locally stream something from my iTunes library it buffers forever than the bar suddenly jumps to 30% loaded before it takes forever to buffer again.
Hmm. I have two AppleTVs with a library on fast USB 3 disk on a Mac Mini that is dedicated as an iTunes server. Everything is connected with gigabit ethernet. Movies load quickly and are very responsive when I scrub through them. I have tested file copies from the Mini on another Mac with gigabit ethernet and I get over 100MB/sec.
Sounds like you have some kind of network issues...