i wouldent push the memory any more unless you want to take your imac apart and check what it;s rated at (you have to take the G5 sink off and read off what is written on the video memory surrounding the gpu) the core should go to 400MHz, put it to that and then run a game, if there are no artifacts then it'll be ok.
the g5 makes 90% of the heat i wouldent bother, just wear some headphones or pump the sound up.
You can underclock too. Should do no harm, give it a go and then report back to see whether the fans are effected.
To the original poster:
I used that program overclock my 9000 pro and you know you are in trouble when you start seeing distortion in the monitor. Games start having random pixels which show the memory is overheating, giving you enough time to turn off the computer and give it time to cool off.