Compressing video files

I'm going to be presumptuous and guess that you might be having a problem I did when I first starting using iMovie with digicam movies. My camera takes nice video (Canon S3 IS) but when I input that video into iMovie there was an immediate degradation in quality that really disturbed me. Really significant degradation. Nothing in the instructions mentioned this and online I only found pithy responses like "it's meant for film not digital cameras." The trick to maintain quality, at least with M-JPG video like my Canon shoots, is to creat an "iSight project" in iMovie. That's one of the settings available when you create a new project. That's the only one that will not degrade the quality of your video immediately on import.

Your question is different, pertaining to quality on export, and that I can't help you much with. At least my advice above will allow you to start with the best quality video instead of reduced-quality. I agree it's difficult to get the file size down without sacrificing a lot of quality. When I'm ready to export a movie I made I usually make five or six versions of it with different "Expert" settings. I play with 24 vs. 30 fps, and the maximum data rate, and the quality of the audio, and the codec used. I have to do that because I just don't understand all the nuances of the video compression. So make five or six different versions and then I compare them with each other, noting the filesize of each. Of course the movies with large file size look good, but if you tweak the settings enough you can usually create a video that looks almost as good but is significantly smaller. If you play it next to the uncompressed video you can see the difference, but otherwise you don't know what you're missing.

So that's my not-so-helpful advice: time-consuming experimentation.

Keep in mind that YouTube recompresses your video again, a lot, with significant degradation in quality. You might as compress it that much to start with if you need to get your video down to 10 megs (which is I think what YouTube requires), because YouTube will automatically recompress again anyway.