I tried it with mine.
The big issue?
No HDMI output. Which means whatever you are using to play audio needs to have a (single) line-in.
If you're outputting to a hifi setup this may not be a problem, but you're only going to get 2 channel sound.
Maybe the port does TOSLINK (?) to get around this, but i have never seen anyone using it and this assumes your receiver does also.
Also, when working hard, it makes noise and its a bit borderline for playing high def content. It's also about 6 years old now and may not be reliable.
What I would do (and what I do) is keep the AppleTV as a player, and use the mini or other machine as an itunes library (or file store if you jailbreak the ATV2) that you can access from the ATV2.
I second the notion for a later model Mini - you want one with HDMI really (which will also have an i5 or better in it).