On your iMac, use Quicktime on one of the problematic movies and choose "show movie inspector". Then compare some of that information to the specs of what the

TV can play, OR write them into a reply here and we'll quickly tell you what's wrong.
A few tips as guesses:
- Are you using Handbrake with the standard
TV preset or some kind of customized preset (the latter is likely your problem)?
- FPS needs to be less than or equal to 30, and less than or equal to 25 for 720p HD content
- Datarate is challenged at anything above about 6K, with 5K being the Apple spec
You can't think something is wrong based on if it will play on your iMac or in iTunes on that iMac. Both are using full versions of Quicktime on much higher quality underlying hardware.

TV is minimally speced hardware with significantly limited Quicktime playback options out of the box.
Your problem is very likely going to show up when you use movie inspector and compare it to the specs on Apple's site for what

TV can play. One or more of those specs- probably the bulleted ones above- will probably be higher than what

TV can handle. If not, share those details, and we'll try some other possibilities.