9800 will be much faster. Especially in games and software which uses Core Image (like iMaginator or Aperture). I had few 9800 Pro (Mac and flashed ones) and its fan wasn't very loud. You can always change cooler to quieter or passive one (be sure that it will fit in MDD's enclosure).
If you aren't very demanding of graphics performance, you'll be satisfied with 9600XT. One important thing: if you'll choose 9600, get dual DVI version from G5. This is the only one worth it.
http://www.tomshardware.com/reviews/facelift,695-8.html - this will be true in Mac world too.
Alternatively, if you want to do some flashing job, you can grab 9700 from Hercules
http://www.guru3d.com/review/hercules/prophet9700/. Best of the best of 9700 cards IMO.
9700 has one major advantage over 9800: it needs less power and runs cooler. Performance wise it's way better than 9600XT and bit slower than 9800. I had one of these and, in my usage, performance differences between 9700 and 9800 Pro were unnoticeable.