I am also having this concern while music is playing, I have owned HomePod apprx 2 weeks.
I typically listen to music / NPR between 70-80% volume and when I fist got HomePod i was impressed that I could activate Siri at normal or slightly above normal speaking volume.
I noted after a week that i was having to speak louder for the speaker to respond and a few times I had to say "Hey Siri" more than once. My concern has happened frequently with both music and NPR Fresh Air Podcast.
I think I have remedied my concern, or atleast for the time being.
After purchase I sat HomePod on a nightstand and have not moved, nor touched it. After noticing and verifying the activation concern I picked up the speaker, turned it sideways so it would recalibrate and placed it back in the same spot. It has worked properly for the last few days.