I thought the same thing, until my phone started getting many scratches by simply sitting in my jeans pocket. The phone is always in there by itself, so there was no reason to suspect there would be any reason for concern. Perhaps there were some gravel/sand particles on my pockets. After that happened I’ve always have a glass protector on. It seems like a good trade off for the protection it offers.
That definitely could be an issue where debris that you can’t see or feel inside your pocket can scratch the display. Sometimes it’s actually scratching the oleophobic coating and not the display itself. One thing I can recommend for those who do not use screen protectors or carry the phone inside their pocket, is turn your pocket inside out and use a lint roller which can attract any dirt or debris to help potentially avoid any issues.