ChadH, what settings are you using? That's quite the impressive collection!
Most of my tv shows are downloaded from iTunes. But most of my movies come from my own collection and others. I use the standard apple tv setting in handbrake and only do one pass. Others are done on iSquint and highest settings. Fixing to add 5 more movies and 8gb of music as we speak.