I am taking a stab in the dark but it seems possible that the h.264 encoded videos are larger and more bandwidth hungry than regular you tube videos. I also notice that in my home (where there are no less than seven wifi devices) the number of devices turned on and the number transferring data over the network can have a huge impact on the iPod. I have tested on a few networks and found when there are few or no other wifi devices then the performance is pretty good. Although I find every now and then it will lock up and look like its bufffering and the scrubber bar will not move then after what seems like ages it will just fill up a whole chunk and continue playing.
Remember using security on your wireless network eats up your bandwidth so use mac address filtering and dont broadcast your ssid for the best performance/security balance.