Some will say, yes, they notice. Others will say no.
Imo, probably will not notice, as encryption is pretty much done in hardware these days. That's what the T2 is for. And how things are done in iOS devices, where everything is being encrypted all the time (if passcode is set).
On the T2 Macs the data is already encrypted even if you do not turn on FV. FV is just adding a password to the encryption. So in theory there should be no speed difference.
I would guess that small amount in the post @NoBoMac linked is just a test to test variance.