If you want Remote to be instantly available, go to the companion iPhone app > General > Activate on Wrist Raise > Resume Previous Activity. Then if you want to pause your TV you can just raise your wrist and tap the screen, without looking even.
Some people might not like how the last app appears instead of the clock face when raising the wrist, but most apps show the time in the top right anyway. What is really annoying is that to access glances or notifications, you have to press the home button three times to go to the home screen, center the home screen, launch clock, then swipe for glances.