Probably the best and the cheapest way to upgrade your Mac's GPU is to flash a compatible PC card. I suggest you looked for these:
7800GTX 512MB (very rare, but reportedly the most powerful option when flashed with Quadro ROM. Ignore the common 256MB version)
- Quadro FX4500 512MB (must have a silver heatsink on the bottom side, the version w/out it may not work)
- Radeon X1900XT / GT 256MB (I suggest getting XT because it's noticeably quieter. Probably the 2nd best thing for typical day to day usage if you can't get 7800GTX or just don't feel like spending too much money)
- Radeon X1950XT / GT 256MB (basically the same as above, take whatever is better accessible for you. Good thing is that you can get these fairly cheap.)
All of these will perform great under 10.5 Leopard as it has all the drivers pre-loaded. The rest of the cards mentioned in the article linked by MisterKeeks is trash not and worth the money. I personally use a flashed X1950XT in my G5 and can't complain, however the choice is up to you.
All you need to flash a card is a PC and a suitable ROM for the card. Google it.
P.S. 9800GT won't work under any circumstances.
I hope it answers your question.