Yeah Apple really messed up their design when they flipped the logic board around. I don't think there is much you can do(as much as VGA coolers and such) since there's not enough room between the graphics card and memory/heatsink.
You're best bet is to install one or 2 PCI slot exhaust fans to get rid of the heat trapped in the case since the MDDs were so poorly designed.
Had the same problem with my MDD. Since a slot cooler only puts the air out behind the card it doesn't do much.
What you can do: Locate the spot on the perforated wall between CPU and your Video Card / AGP slot. Its the corner there. There you can place an additional fan that sucks out the hot air just above the Ram and the GPU heat sink.
Its a very tight spot so either you take a small fan (50mm) one or you need some "dremel" work to fit in a 60 mm flat fan in. I used the flat one from Zalman.