Two iMacs with Target Display Mode

I recently purchased another 27 inch iMac (Late 2011) to go along with my 27 inch iMac I already had. I connected the second iMac to the first one with target display mode to use it as an external monitor. However, when sleeping the first iMac either by pressing the power button or clicking the sleep button, the second iMac becomes disconnected from the first iMac! I have to enter Target Display Mode every time I wake up the first iMac so they will be connected again! Is there any way around this? Or am I doing something wrong?

 

Well now when I let the primary iMac sleep instead of putting it to sleep, upon wake the second display will turn on. Not sure why this only works this way, but its somewhat annoying.

Edit: Apparently this is only true if you wake the Mac within 30 seconds of it going to sleep. Afterwards, the second iMac will not wake. However, the first iMac still shows an external display connected in the Displays section and you can still move your mouse and windows over like there was a second display. Also, audio can still be played through the second iMac even though the video display is dark.

 

Also, if I switch around which iMac is the main iMac, both displays work. Why would it work one way and not the other?