It may be that the DISPLAY is not "waking up" (instead of the Mini itself).
Next time this happens, don't reboot the Mini.
Instead, try disconnecting the display (tv) and re-connecting it.
I suggest you do this "break-make" at the tv end (to avoid putting strain on the Mini's HDMI port).
Does the display "come alive" again?
Can you tell us exactly WHAT you do to "wake up" the Mini?
Press a key?
Use the mouse?
Do something else?
Another suggestion:
DON'T put the Mini "to sleep".
Instead, put the display to sleep (with a screensaver) and leave the Mini "awake, but idling".
The difference in power consumption between "asleep" and "awake, but idling" is next-to-nothing.
But if the Mini doesn't sleep, then hopefully you won't have problems trying to "wake it up".
Another approach:
Have the Mini set NOT to sleep.
When you're going to be away for a little while, just TURN OFF the tv.