You're correct that it's hard to get this right. Whether Apple is up to the task is yet to be seen.This stuff is hard to get right, and Apple isn't up to the task. A system like this has to be 100% reliable. Apple can't even get the Remote app to reliably work with the Apple TV. Sometimes the Apple TV chokes and needs to be powered down and rebooted. That's acceptable for a set top box but not when you want to turn the lights on.
If they can't get the simple interaction between an iPhone and an Apple TV on a WiFi network to be bulletproof, I'm not expecting great things for HomeKit.
Much of what Apple gets right is also hard to get right.