Since getting rid of the cable box, the ATV remote is the only one we need and the experience of using it is very much better now.
Naturally, the first thing we do with the ATV remote is to turn on the TV (pushing the home button) and by the time the TV, AV receiver and ATV are all on, the remote is fully connected. Any lag is a complete non-issue now (lost in the few seconds start up time), whereas before (when we had multiple remotes and usually used the TV remote to turn things on) we encountered the lag as mentioned above. At first I thought the rapid response to the 'turn on' command was because it was sent from the ATV remote by IR, but a quick experiment shows it isn't (it works through walls). It seems that when the ATV is off the remote/ATV respond quickly to turn on the ATV and connected devices but the ATV/remote system lags in other actions (possibly the lag is the ATV box getting up to speed and ready to receive other commands).
Short version: no noticeable lag if you use the ATV (and its remote) for everything.
I do of course appreciate that this is completely useless information for most people as for most the ATV is just one of several sources, but hopefully it might be useful to fellow ATV minimalists 🙂