I have this also in dev b4. I’m starting to think this has to do with other devices, an Apple Watch ⌚️ in my case. When I go into messages on the watch and read them, the badge goes away on the phone.
The badges do seem to eventually clear if left alone. Not sure if the watch and phone aren’t syncing correctly or what specifically is causing it.
Edited to correct which beta I’m currently on.