Your RAM is overflowing into swap, the less RAM you have, the more this happens. The more this happens, the more you wear out your SSD. You don't want to wear out your SSD.
The more RAM you have, the less that happens. This is why people "obsess" over having more RAM. You're not "feeling" it because macOS does a good job of memory compression, but that's not enough forever.
People also worry about RAM because, unlike how it was in 2010, 8GB isn't a large amount of RAM anymore and on today's Macs, you can't upgrade them after the fact. The 2020 27" iMac, the 2019 Mac Pro, and the 2018 Mac mini are the last Macs sold where the RAM was at all user-upgradable. On the Apple Silicon side of this transition, none of them will be. That sort of puts pressure on one to not necessarily cheap out up front. Then again, I have an army of base model M1 MacBook Airs and a base model M1 iMac, all at 8GB of RAM. But I also have a single M1 13" MacBook Pro with 16GB of RAM and all of those other M1 Macs are secondary and used for light browsing and app usage as well as IT testing (for which 8GB of RAM, 7 GPU cores, and 256GB SSDs are more than enough). Were it my main Mac, there'd be no way in hell that I'd want 8GB of RAM in 2022.