Though normally a Tiger-User, I lately installed Leopard and Handbrake. The x.264-mp4 files (created from iMovie-DV-files) could be played by QT under Leopard, but not in QT and VLC under 10.4. Also, my mediaplayer (from 2011, supporting h.264 level 5) on my TV wasn't able to play it.
But maybe this is different for avis made from DVDs.
I find Handbrake's built in x.264 encoder does produce very good quality at low bitrates compared to the available freeware x.264 encoders one can install for Tiger and use with mpegstreamclip.