@reddrag0n, from what people have been saying, there's no known modified ROM of the 256Mb version of the 7800 GTX. When I dug up that sort of stuff as much as I could, I couldn't find anything. Same goes for my question regarding the 512Mb version of the ATI X1950 XT (called ATI X1950 XTX, if I recall correctly, which today I "happen" to own). And same goes for the 7900 GTX.
But this whole thing is not over yet. Answering my own question, "are there theoretically better cards that are technically-compatible, but lack a modified ROM", the answer is "yes, a fu!kton of them". Basically, any PCIe card is a potential candidate, as they all DO work on the G5 even today, but under GNU/Linux (and maybe also regular BSDs?) instead, without any flashing required, because there are drivers for those cards. We can see one such instance in this video from Luigi Burdo on YouTube, equipped with an AMD Radeon 6570, which has 2Gb GDR5 VRAM! (According to him in the credits, anyway. Seems like at least one Mac-OS-X-compatible card needs to be inserted alongside in another PCIe slot, perhaps the x16 slot, however, but still.)
The catch, though, of course, is that since the late G5s have PCIe 1.0, the highest-end cards, all of which are PCIe 3.0 today (and soon PCIe 4.0), won't work at their full potential. But you can keep pushing the machine up all you want with them. The returns ARE there, although diminishing.
Theoretically, it should be possible to port the Linux kernel drivers to work with Tiger and Leopard on the G5 (assuming the drivers are even open-source, because Linux is full of closed-source binary blobs, too). Both the endianness and the archictecture (PPC) have been addressed in those drivers already, but there's more to porting them to Mac OS X. Now if you ask me "how do we go on about porting those drivers?", my answer is a near-absolute "I don't know". I have zero knowledge for most things low-level (hardware interfacing), and both graphics cards and video processing as a whole (other than that they are awesome for SIMD processing). But if anyone is serious about this, it's possible. It can be done.