As you well know, nothing goes anywhere until you enable them by speaking their particular local wakeup phrase.
That said, I can visualize a future where everything IS listened to, so that the device can keep context. For example, you could suddenly ask the air, "Hey Jeeves, what did my wife tell me to do this morning?" and other intelligent assistant duties.
Hmm. It could be that everyone actually needs what we used to think we'd have back in the 1960s concept of the future: a secure personal house computer bolted down in the basement, where all our data resided, and only accessible by its owners. Nothing stored out in the cloud, unless perhaps it's heavily encrypted for backup purposes.