I had a 15 Plus and I quickly realized it was too big and heavy for me so I switched to a 16 Pro. I adjusted very quickly going from 6.7" to 6.3". Much better size but still too heavy. Now I'm seriously considering the 17e next year, just waiting to see the specs. That will be another downsize from 6.3" to 6.1" but I desperately want a very light phone. If the 17e either doesn't get released or somehow comes out heavier I will probably switch back to Samsung as their small S series model is usually under 6 ounces. That's my new cutoff, 6 ounces.
At least for me, downsizing screen size never effected my ability to use a phone as phones really are pretty big nowadays even the "small" ones. I still have an iphone 7 at home that I use on wifi and I keep kicking myself for not buying an SE3 when it came out. It's small, super light, and most importantly had rounded sides so it was comfortable to hold and use unlike today's phones.