What you say is correct, but a CPU run at its thermal design limit often is going to shorten its lifespan. IMHO the default SMC settings in the mini are too slow to compensate for heat load and ramp up the fan, which cause unnecessary jump.
With modern CPUs like the sandy bridge, with advanced power management features, the thermal load of the CPU can change drastically from one instant to the next. For example, you are browsing the web and the CPU has shut down half or more of its cores, and lowered the voltage to an economical setting. Then you start up handbrake to encode videos and BAM- the CPU engages all cores, and jumps the voltage to the normal operating level.
In cases like these, the CPU can increase in temp by 30-40C in a matter of seconds (you can watch it happen with iStat nano/pro), particularly when they are coupled to low-mass coolers like are seen in compact systems like the mac mini.
I have run tests where starting a 1080p video playback or starting encoding causes the CPU temp to quickly jump from normal temps of 45-50C up to 95C in just a few seconds and the fan will take 10-15 seconds to fully ramp up to the maximum of 5500rpm. There it cools the CPU down to about 70C and the fan lowers way down (say 3000rpm), but the continuous load on the CPU means the temp is soon back up to 95C and again the fan is too slow to respond, so it overdrives to 5500rpm and cools it down, and the cycle repeats.
Whereas if you preset the fan to a higher level like 4000rpm, the CPU only jumps to about 80C and the swings in temperature are not as extreme, because it's not constantly "chasing its tail" so to speak. If the feedback loop was more aggressive in detecting rising temps, it would not swing around nearly as much.
Because of the very fast temperature swings of the CPU, the automatic fan control also seems to operate almost in a bi-modal state. Either it is "cool" and at or near the default speed of 2000rpm, or the CPU has jumped to 95C and the fan ramps to 5500rpm to cool. There is not much time spent in between.
I don't do anything special for my usage since most of the time my mini is in a near-idle state, but I do wish the thermal feedback loop that controls the CPU fan was a bit more aggressive with increasing heat load. If you do plan on starting a high-load job, maybe run SMCFanControl and set a higher default fan speed. You would experience fewer wild swings in fan speed (and resulting noise) and the CPU temps on the mini will be more stable over time.