Because you have no setting to adjust the headroom on the limiter, sometimes it may engage sooner then you would like it to, and can cause your audio to be quieter than you would like it to be. It can reduce your dynamic range much like todays modern mastering, everything is slammed, the soft acoustic guitars are as loud as the heavy guitars, shortening that dynamic range between the two. The limiter will remove some of that dynamic range.Since the limiter only compresses sounds above the threshhold, which would be distorted otherwise, I'm still not getting why you wouldn't just leave it on.
The limiter reduces your distance between your soft and your loud by lowering your overall loud, compressing the range. You might want that FULL dynamic rage in your audio, therefore needing to turn off the limiter.
I hope that all makes some sort of sense. haha