I know that the 9600 is fanless and supposedly the 9800 has a really loud fan. I think they both support quartz extreme and core image, right? Is there and advantage to either one other than that?
9800 will be much faster. Especially in games and software which uses Core Image (like iMaginator or Aperture). I had few 9800 Pro (Mac and flashed ones) and its fan wasn't very loud. You can always change cooler to quieter or passive one (be sure that it will fit in MDD's enclosure).
If you aren't very demanding of graphics performance, you'll be satisfied with 9600XT. One important thing: if you'll choose 9600, get dual DVI version from G5. This is the only one worth it. http://www.tomshardware.com/reviews/facelift,695-8.html - this will be true in Mac world too.
Alternatively, if you want to do some flashing job, you can grab 9700 from Hercules http://www.guru3d.com/review/hercules/prophet9700/. Best of the best of 9700 cards IMO.
9700 has one major advantage over 9800: it needs less power and runs cooler. Performance wise it's way better than 9600XT and bit slower than 9800. I had one of these and, in my usage, performance differences between 9700 and 9800 Pro were unnoticeable.
I know that the 9600 is fanless and supposedly the 9800 has a really loud fan. I think they both support quartz extreme and core image, right? Is there and advantage to either one other than that?
No it isn't but it should be the most cost effective. I believe the fastest card available for this computer, via rom flashing, is the Nvidia 7800gs with the G70 chip, but those seem hard to come by. Personally, I went for an ATI 9800xt, which is a little faster than the standard 9800 Pro and it worked perfectly. I have a post over on the strangedogs forums which details my success with that card: http://bit.ly/dyZdC9
It's second fastest flashable AGP PC card, but it's GS, not GT. GS has 16/6 Pixel/Vertex shaders, GT has 20/7 Pixel/Vertex shaders. It makes the difference. First one is Gainward 7800GS 512MB which is real 7800GT with AGP interface.