quicktime will normally load the entire ( i think ) video into the RAM on your computer. if there is no RAM or little ram avaliable maybe it puts it into VM. im not exactly sure maybe someone can verify>?
Which version of Perian do you have? I remember when I updated my Qt to version 7 it started to load really slow. Then I updated my Perian to version 1 and all was well. The only thing I know is that, it loads flv's completely before playing
Is this on a Mac or PC? I noticed on a PC, it takes longer using Quicktime to load a video. On a Mac, it takes longer only if it is using the Windows media format with the plugin. I think that the Quicktime player provides a way to scrub video back and forth very easy, so perhaps it takes a couple of extra seconds up front to perform that magic. It seems to be that way by design.
In contrast, the Windows player assumes the user will just play the video in real-time and not really scrub the video for what they are interested in.