I've not had the issue, but then I've turned it off, because I frequently have my iPhone, iPad and HomePod, and so saying Hey Siri wakes all of them up. I can't say Hey Siri call Ma in my home office because my HomePod and iPad will complain they cannot.
when I had it on, it didn't seem to wake up on its own