Honestly, it isn't as silly as it sounds. The M-chip is also squeezed in the iPad Pro (and we may argue, already in the iPhone), maybe not the pro version of the chip but nevertheless, the same chip. If I remember correctly, the iPad Pro M1 was clocked/benchmarked in the same range as the (at that time) new MacBook Air M1. An Apple TV in full body aluminium for cooling would still give more cooling than the iPad because of the screen that doesn't dissipate heat but generates it.If they could pack the current Mac mini into an Apple TV sized puck, it would already be that size. Ridiculous question.
Obviously you mean something incredibly simplistic like "the chip" without the slightest concern for thermals or I/O.
An Apple TV sized M-device as entry level thing (Mac mini Air) could be feasible without thermal pro/max/ultra sensitivities. Honestly, if it could exist for a reasonable price, it would buy it straight away for the living room.
Ok, 32 GB RAM and 4 TB SSD would be asking a but too much. Just my take on things.