I haven't had the chance do any more testing but I can give some more general observations.
Even after resetting the SMC and PRAM the fans didn't seem to ramp up quickly under load. I thought I'd be seeing the same temps under load but with faster fan speeds as the CPU speed increased. Instead the fans seemed to act about the same and the faster CPUs just ran hotter. You can force the fans to go faster using smcFanControl or Fan Control but I didn't experiment with them too much. Perhaps I need to go back and do that. I wouldn't want the Mac Pro fans running full blast all the time, but compared to some of the older systems I'm used to its really not that loud. You can use the two fan control programs I mentioned to get an idea of how loud your's would be at various speeds. That would tell you how loud it will be if you up the fan speeds after a CPU swap.
I figured my problem was poor thermal compound application or not installing the heatsinks correctly. I eventually ended up using a sort of two step process to applying the compound 1) wearing a glove or covering your finger with a plastic bag, spread a layer of compound so thin that you can almost still read the printing on the CPU. I masked off the edges of the integrated heatsink and could barely see a difference in the color of the metal that was covered vs the metal that wasn't once I removed the tape. 2) Make a thin X of compound in the center of the processor. I do it so that the lines are half the length of the distance from one corner to the other. I could run the processors without step 2 (making the X) but doing that helped. Using this method my temps were lower than what I got after multiple tries at the standard 'place a drop at the center' method.
I began swapping out CPUs without using washers as spacers to account for the increased height of the normal CPUs with integrated heatsinks vs the stock lidless CPUs. I did it multiple times and really tightened down on the screws with no problems. i did sometimes have to loosen them up a bit in order to get the Mac to boot but I mean no problems as far as bent pins. I picked up some washers thinking maybe that would help but it doesn't seem to make a difference and I haven't been using them.
I've considered lapping the heatsinks and CPUs. I'd never done it before but just tried it on a HP XW9400 dual quad-core Opteron system with a stock water cooler. I just lapped the water blocks and temps dropped by 10 degrees. But...i went to far on the one and it seems to have developed a slow leak so that wasn't an overall success. Though no damage to the computer and a Mac Pro heatsink won't leak. It would be a pain to do with that lip on the aluminum part of the heatsink though. And the HP waterblocks looked and felt bad, the Apple heatsinks feel much smoother.
Finally, this has got me thinking about a water-cooled hackintosh with an EVGA SR-2 motherboard or a water-cooled Mac Pro. I don't think there's room in the Mac Pro for internal radiators so if I were to do it I'd drill holes in the metal that's above the CPUs and below the PCI-E slots and then make two more holes in the back of the Mac Pro next to the PCI-E slots. It'd get hooked to a ready-to-use, off-the-shelf external pump/radiator system or routed into an empty G5 case housing the pump and radiator. The former would be easier but I kind of like the dual tower idea - like a pair of linked SGI Origin 200s
. The G5 would also be a great place to house disks connected to a RAID controller in the Mac Pro. Oh, and if you're going to water cool the CPUs might as well do the GPU too.
Tutor, I haven't seen any details of your water-cooled Mac Pro anywhere. Care to share?