It is a problem.
And it kind of baffles me in a way. I think they made a mistake by getting caught in Amazon's game. Amazon launched speakers as a way into the home simply because that's the only option that they had. I think Apple could have sidestepped that by just realizing that there's no objective reason that home automation has to be tied to a hifi speaker. HomePod buyers are practically guaranteed to have an iPhone within audible distance at any moment. Or a watch or iPad or whatever. In retrospect, tying HomePod to HomeKit has been a huge headache for Apple and users alike.
I'd rather see Apple have a simple HomeKit device that's cheap and all it does is act as a controller. Put one in every room and yeah it's got a speaker for communication but it's not trying to replace your stereo. Then have a separate speaker lineup if they want, to compete with Sonos, but make it focused (like Sonos) on audio and not HomeKit.
I won't hold my breath, since I think if Apple could sell a single device that morphed into a phone, iPad, Mac, speaker, TV, car, and floor wax, they'd do it.