It'd be cool. Difficult to pull off, as it seems the AppleTV already lags a lot if the content isn't fully loaded (at least, this is the case for my previous-gen model). Clicking forward and back on the remote (after clicking up or down for smooth ff/rwd or skip by segments) helps account for the lag.
I'd be happy with using the crown to scroll up and down lists. I've tried it purely out of habit many times.