Did you try the latest official driver from nvidia?
I'm getting 5.9 for graphics on my early '08 MBP (Win7 x64 RTM). I'm using Rivatuner to force the 8600M to "performance 3D". It's not overclocking though. It's just forcing the card to operate at the highest standard clock all the time. I'm not concerned about the slight heat increase and battery performance hit because my MBP is plugged in most of the time. In spite of that it still runs cooler in Win7 than it ever did in XP. I mainly chose to force the performance mode because it has been shown that audio dropouts can occur when the card switches modes automatically. I use my MBP for audio recording so I can't accept any interruptions (I also kill the wireless and latency-riddled bootcamp service while I'm working).
I don't recommend using Rivatuner to overclock beyond the standard levels. There simply isn't enough heat-sinking available. I also caution against using Rivatuner in general unless you know exactly what you are doing.
That said, 5.3 for graphics is not a bad score for a laptop. Unless you are doing some serious gaming, you are not likely to notice a difference of a few points. I was getting 4.9 while I was still on Win7 RC. It jumped to 5.9 when I installed RTM. It seems a little snappier, but that could be the fresh install or my imagination.