Privacy issues concern me. Not a lot, but enough. My Sony TV, which the Apple TV is connected to, has a camera mounted on top. You can swivel it upward, towards the ceiling. But I still don’t have the TV connected to the internet because of that camera. The built in Skype app on the TV never worked even when it was brand new.
I think those are valid concerns. Apple's privacy record isn't spotless but it is mostly security gaffs and poor management choices, not a "feature" or revenue stream so I would trust it much more than something built into a smart TV which have pretty terrible track records for privacy invasion.
My concern would be if the camera isn't built into the device, which I don't think it would be unless the Apple TV morphs into the Apple Soundbar or something like that, it is likely to be developed by a third party, like Logitech and the trackpad keyboards for the non-Pro iPads. Logitech I trust, but the knocks off that will follow if this takes off, not so much.
I do wonder after Facebook's Portal failed, whether the issue is that people don't trust the companies selling the product (personally why I never considered Portal) or whether they just really don't want the dedicated device.