Check out Perian. If you are looking to encode in a particular format, you might try macupdate for encoding plug-ins or conversion applications (search for "encode iMovie" for starters).
Perian is definitely what you're looking for. I use it and haven't found a single file yet that it won't play (but I don't download that much video, really).
Edit - NM - I thought you were only interested in playback, and I have no idea about encoding with Perian....oops.