It's a great phone.
I personally prefer the 11 pro and 12/12 pro (OLED, Face ID, edge-to-edge, battery, night mode) but i was able to get an SE 2020 for free, and sell my 11 pro and 12 for approximately 1 grand (CAD) each. The free phone plus the just over 2 grand in the pocket has made it entirely palatable to hold on to
I am more than happy to stick with the SE 2020 now as my main phone. It's fast, I don't use the camera as much as I thought I did, and when I do it takes pictures that look just fine when I blow them up on my PC screen or stream on apple TV 4k
I haven't found myself missing the 11 pro or 12 as much as I thought I would, which is a bit surprising. I consumed a fair bit of media on those phones but I found after about a week I adjusted to the SE 2020. The main advantage for me is that it is simply just as fast as the other two "higher end" phones.
I agree completely with you OP including the drawbacks (although I don't use a case or screen protector, I use it au naturale)
For me, the touch ID Has been a surprising benefit. I wear a mask exclusively at work, and the touch ID is super helpful. For extra battery, I managed to snag an official apple iPhone 7 battery case for $40 which extends the life a fair bit when needed.
So far though, no concerns.