Cooling occurs due to heat exchange, in this case cooler air moving over hotter components/heatsink. The case only has an effect in the sense of allowing air to move freely over all parts, how easily cool air can get in and warm air out and if there is ventilation holes to allow natural convection to occur or if it is restricted. Unless components are in direct contact with the case or a heatspreader is attached to it, the case plays almost no part in the cooling of the processor or components. However, the internal heat warms up the air and that heats up the case too, which in turn, once the heat source (power supply & processor load) is reduced, then the internal air flow has to cool the components and the case as it is acting as a heat source now too. The ambient air surrounding the case aids to cool the case down somewhat. This means for a while after a heavy load the case will heat the internal air reducing its cooling capacity.
Old Mac mini total volume 1382.98cm2
New Mac mini total volume = 806.45cm2
The larger mini takes longer to warm up as it has a larger thermal capacity, in turn it will eventually hold more heat and take longer to cool. It might also cool slightly faster due to its larger surface area. The smaller mini exposed to the same amount of energy will reach the same temperature quicker as it has a lower thermal capacity, it will hold less heat overall requiring less air movement to cool. It might cool slightly slower due to its smaller surface area. Pros and cons both ways and net result is not much in it.
In a nutshell, the case size has much less of an effect on the cooling performance than the actual case design. Could the design be better from a thermal efficiency perspective... absolutely. Is it form over function, it is both.
The new mini has a more powerful chip and a higher thermal output, so more heat, but over a shorter period. This means it needs more immediate cooling capacity, hence fans running sooner and/or harder but it won't be needed as long. This would also be true if the logic board was sitting on a desk in the open, so zero case influence, as it is a result of the higher thermal output of the processor.
This is based on direct comparison of the Mac mini M2 pro vs M4 pro chips. From the only measured results I can find, off the same (multithreaded) handbrake MP4 encoding test, the runs drew an average of 36.2 watts and 43.1 watts respectively. Making the M4 pro higher in power use, but when taking the processing time into account to calculate total energy used we get 6.05WHr vs 3.85WHr respectively, making the M4 pro more efficient. In comparison the base M4 is Apple's most efficient chip to date using only 2.39WHr by comparison.
So without speculation the new minis do have more powerful chips that put out more heat faster due to their processing power, but they do it for less time, so less overall. Is Apple's redesigned copper cooler in the M4 pro enough to tame this power? Initial indications say no but I don't think there's enough "real" evidence yet. Tweaking of power curves in updates to come from feedback will likely occur and we do not know how aggressive the curves have been set yet to keep chips cool.