i'll check out subler (yes im on a mac) - never heard of it. same deal as handbrake - but better?
Not exactly. Subler is a remuxing program instead of a re-encoding program. This difference is that remuxing passes through the video file (copies it exactly) and just changes the container (from MKV to MP4). The video file is already in a h.264 format from a Blu Ray so it doesn't actually need to be re-encoded. Most people just re-encode to reduce file sizes because the loss of quality isnt as important to them.
The Apple TV is not necessarily rated to play such high bitrate files, but if you have a strong high speed network you should be fine. Keep in mind with remuxing that you can never pass through Audio because MP4 containers dont do well with DTS audio, and the Apple TV does not support it.
There are many posts on here on how to use the different programs, and it is up to you which one you want to use.
Personally I use a combination of both. I encode with Handbrake to reduce file sizes and then remux with Subler to add iTunes metadata. Try using handbrake with these settings and see how you like it:
1. Start with the ATV3 preset
2. Change the Video settings from the CQ slider to Average Bitrate
3. Check the 2-pass option
4. Change the bitrate to 9600
5. Check the option for "Web Optomized"
This is what I use for my Blu Ray encodes and the loss of quality for me is almost nonexistent, yet it reduces the file size from 35+GB to about 10-15GB. This will run perfectly with the Apple TV 3 over wired networking or wireless-N networking.
This type of encoding process will take 2x as long to re-encode, but it will work universally with all movies (in my experience).