Look for either ATI Displays software, or something called ATI Rom Xtender. I remember that I needed the latter extension when running the retail 9800 on my Mystic, or else it had the same general issues that you're having. I think that the Displays software supplies this extension.
I found some info here- https://discussions.apple.com/thread/1206116
It's for the 9600, but it also applies to the 9800.
Already gave that a try, too. Still no luck