Personally, I'm not. They're not so expensive that if I got unlucky and one died prematurely outside of the warranty I couldn't replace one. On the flipside, I view the likelihood of that as pretty small. This is my third AppleTV, and the original one (a model 3 I think) is still going at a family member's house. The old 4 will probably be going to a friend or family member as well and I only replaced it for the 4K.
For me, I'll buy AppleCare on anything that moves around a lot or has a lot of moving parts that can wear out, etc. For a laptop or tablet for example, it might be worth it. But for something that stays put, I consider the risk minimal and would rather save the money.
Just my opinion though. It's nice to have it IF something goes wrong - it's just a matter of amount of risk you're willing to take.