Maybe it's paranoia, maybe some people just like privacy. Google can now know when a person is awake, asleep, at home, or away with their thermostat, house cams, and smoke detectors. They can also tell where a person is going with either Google Maps or Waze in the car. A smart device that has the ability to listen, learn, and possibly record (yeah, I know, it doesn't really record), phone conversations is just one more leak of privacy. It's not unreasonable to believe that some people still value the right to privacy.
Just my $.02.