Mac mini server always on. How can I make it sleep when not in use?

Hi,
I have a 2006 Mac mini (see my signature) which I use as a server running Plex, AFP file sharing and Time Machine for two MacBooks that connect wirelessly through an Airport Extreme 3. gen. The mini is hardwired.

This setup would be perfect for me (fast enough, cheap and convenient) however the mini barely sleeps with the MacBooks connected basically all the time.
I have a 2 TB drive connected through Firewire 400. The sleep timer is set to 5 minutes. I use Time Machine scheduler to set the backups to run in the evening.

How can I make sure the server sleeps when I don't copy files, watch a movie or a Time Machine backup is running?

I really need some help here as this setup only makes sense if the mini goes to sleep when not in use. Otherwise the power consumption is too high.

Edit: forgot to say it's running 10.6