I don't think SIRI's constantly listening. Apple had made it clear, on the Keynote, that's the reason why the remote has dedicated button for SIRI, so it'll listen only when you want it to.
If you want that function then Apple TV (or Apple products in general) is not for you.