And how does it know you've said the words "Hey, Siri"?
I do think Apple's privacy record is better than most, but there's something inherently unsettling about a device who's purpose is to listen to everything in the room and make sense out of it. Yes, one concern is that it will report my private conversation up to the cloud, but there's other more mundane concerns: how often will it interrupt my conversation with "I'm sorry, I didn't get that", will it accidentally execute commands because I'm talking to someone and say "sometimes I worry Siri might delete my entire library out of spite", can anyone in my living room get access to my personal information by just asking the room for it?
I'm uncomfortable with a black box listening in on my private life. I may get more comfortable with it over time and as it becomes more reliable and useful, but for now I'm wary.
While i understand your concerns Apple has a good track record with this type of thing, also they have explicitly explained that they don't want a users data.
"Starting in IOS 10 the software built in a safeguarding engine, dubbed differential privacy that scrubs and adds noise to a users data so that company can't tell where it's coming from but can still provide the useful feedback to the user. It means that Apple can analyse each queries for trends and words (Hey Siri" without being able to identify any particular customer." - http://www.zdnet.com/article/apple-airpod-siri-speaker-assistant-wwdc-2017/