And I'm not the only one. I tech multimedia at several Universities over here and for as long as I can recall Quicktime streaming has never "just worked" the way it does with other streaming technologies and I've tried it from a host of different machines from ye old eMacs, G5s, Mac Pros, iMacs, PCs of every kind. Of course most of that was trying to use streams from other countries, but that shouldn't matter. I'd go as far to compare it to Windows Media Format streaming which falls on it's face just as often as Apple's technology seems to.
The only time I've ever had problems with Quicktime streaming was with respect to port numbers. Otherwise it was fine. It really sounds the "that shouldn't matter" comment is what is actually causing the problem. If the stream isn't compressed enough and you're trying to stream from say, Japan to America, then yeah you're going to have a heck of a time getting it to buffer well.