How do you mean DVD looks fine on my iMac so does 720p video and it even plays a handful of 1080p movies fine...
To both of those, get a Windows Vista system with a modern dedicated GPU or the nVidia 8000 series IGP or ATI 3200 IGP.
Not only is quality a night and day difference, but so is CPU use and heat.
If I play a blu-ray disc on my HP with GeForce 8400M GS, my CPU use hangs out around 5% CPU use.
If I play a 720p H.264 video on my MacBook, a single core hangs out around 60%.
2.16GHz C2D on the MacBook, 2GHz C2D on the HP.
Mac OS X doesn't have system wide hardware acceleration for video. EVERYTHING is done in software. That means a couple of things. First, the quality looks terrible compared to what GPUs can do, and it means the CPU use is ridiculously high because its doing all the work.
Honestly... 1080p stuff looks FANTASTIC on my 24" iMac. M2TS and MKV play with no problem and look amazing. I think the hardware could do BR w/out a problem. Has anyone tried hooking up an external BR player to an iMac or other machine and see what is possible? Naturally software will need to know how to play a BR, but I would imagine the Mac will be able to see the folders/file structure of the disc without any problem....
It's all being done in software, not hardware, because OS X, unlike Windows, has no hardware acceleration for video playback. Illegally downloaded MKVs are much different than the 1080p ~50Mbps video you get off blu-ray.
Also, because of OS X's lack of modern video support, the quality isn't anywhere near as good as it is on Vista with a modern dedicated GPU.
The converted 1080p files from the discs are h.264 at 15 Mbps. Playing these files using VLC is near impossible on my 2.16 GHz iMac (18 months old). I would think that a new iMac should be able to play Blu-ray but it might struggle.
Install Vista. Your iMac has the X1600, right? That has ATI's video technology built-in. You should be able to play them without a sweat in Vista. I mean install Vista via Boot Camp so it has full hardware acceleration.
