This is happening but with my wife. The HomePod is under my account and it recognize me perfectly, but not my wife. In the end everything is playing under my account and it's a little bit messy. I have to try this approach.
So, you removed everyone, but you and turned off voice recognition. Added everyone back and turn on voice recognition again?