I have a late 2012 Mac Mini 2.3 (or 2.6, I can't remember) i5. When I do a conversion in handbrake from a large .mkv file (20-30GBs) to ATV3, it will literally take 3-5 hours per movie. I'm only averaging around 9-10 fps per movie. Other than a new processor/motherboard, what can I do to speed this up? Sometimes I will have 7-9 movies in the queue for handbrake and it will run for a couple of days before it is finished.
I have a 2.6Ghz 2012. Your times are about right. You need to remember the fps on a bluray rip is slow because of the size of the frames being transcoded. You run an old 600mb avi and it transcodes in a few minutes.
You can't do anything to speed that up. The CPU will be running at 100% capacity in a HandBrake transcode, particularly a bluray rip. 100% is 100%. Although if the OS is throttling the CPU because it is running hot then cooling the mini down could give you back some of the CPU that is being throttled. In otherwords, if the CPU is running at 90% because the mini is at its temp limit, then by cooling the mini down the CPU is able to run at 100%. That is only an issue if it is being throttled. iStat Menus is a good app to see if this is the case. Although having said that I think it is unlikely. My mini transcodes in Handbrake at 100% and sits on about 98-99C. It's limit is 105C so it is pushing at full CPU capacity. I just can't ring anything more out if it!
You can try and limit CPU usage with other apps but that is likely to have minimal in effect if it is running standard background applications.
RAM will have minimal effect (8 or 16), although if you have 4GB you might want to get more.