A used XS would be a fantastic upgrade for you. They’re going for like $450 used, just be really careful about who you buy it from - you don’t want a locked one, a stolen one, a financed one, or one that’s had the screen replaced by a 3rd party shop and therefore doesn’t have TrueTone working anymore.
The swipe UI is awesome, FaceID is awesome unless you’re wearing a mask, the taller screen is really fun. The XS has the same extended dynamic range 4K 30fps video feature as the new SE. The SE has a better standard wide camera, but the XS is still decent. The processor speed difference is negligible.