Is this on a Mac or PC? I noticed on a PC, it takes longer using Quicktime to load a video. On a Mac, it takes longer only if it is using the Windows media format with the plugin. I think that the Quicktime player provides a way to scrub video back and forth very easy, so perhaps it takes a couple of extra seconds up front to perform that magic. It seems to be that way by design.
In contrast, the Windows player assumes the user will just play the video in real-time and not really scrub the video for what they are interested in.