Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Look at the settings in the preference pane. You set the highest resolution you want, and if it isn't available the player steps down to the next available one.

Only that didn't work for me. There was a video I tried to view and all it done was kept cycling because the vid wasn't in 1080, or 720. So until I unchecked it, it wouldn't play.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.