Your phone is also always on, and always listening for the activation phrase. I suspect that the speaker, like the phone, handles that task locally on the device, so there isn't an army of spies in listening posts waiting for you to say "Hey Siri". So they're also not likely to overhear you say "Come on over. My wife just left. We have 45 minutes."I'm not one of those that are hyper-sensitive to privacy issues... but there's a difference between taking reasonable precautions and inviting spying technology into one's home. The primary purpose of these smart speakers is to "spy" on you... always on, listening for the activation phrase.
The surprise party you're planning for your wife is safe from getting spoiled by Siri spilling the beans.