I haven't been too enamoured of my ATV4 and had decided to return it. I plugged my ATV3 back in last night - it seemed quite slow, old fashioned and clunky compared to the ATV4.
The remote is cool and yes it is fab that it can control your TV too but as to replacing your tv remote... I don't see how? As it stands, at least in the UK, you couldn't really use the ATV4 as your primary device as the apps just aren't there yet. You can obv sit and watch iTunes movies all night but for typical family viewing you're going to be flipping back to your tv / Sky / whatever remote all night.
Anyhoo, back on topic, flipping between the two models I'd say my favourite thing is the overall smoothness of operation. Personally I find the other raved-about stuff pretty meh - screensavers are cool for 20 minutes, apps are missing in action and the games, for me, are pretty juvenile. For my purposes the 4 doesn't actually do anything more than my 3.
I think it'll get better but we are still some way from using an ATV as our only box connected to a tv. Maybe the ATV 5 or 6...