I use a universal (learning) remote for a "one remote to rule them all" experience. Yes, that means I lose Siri and using it as a game controller but the universal has real buttons for a variety of common tasks and they are organized in such a way that they can be used at night on touch alone. The new Siri remote is kept nearby should I want to use any of those functions but I rarely do.
I'm also going from "3" to "4" and the key difference is that the scrolling through the list of self-owned content is not as smooth. If one has a list of- say- 300 movies and holds down the scroll button trying to run to something far down the list, about every 15-20 movies or so will result in a near (scroll) stop... as if
TV4 needs to "catch up" and/or verify that it's user wants to continue scrolling beyond the current group of 15-20. With the "3," hold down the "scroll down" button and it would simply rapidly scroll through the list until a bit after you let go of the "scroll down" button.
I'm guessing that since scrolling is now hooked to swiping, the programming has some kind of "braking" in it to prevent a fast swipe up or down from quickly scrolling to the top or bottom of such a list. And this "braking" carries over to the traditional approach of holding down a "scroll down" or "scroll up" button to scroll through a list.
I have not found any good way around this issue within the native "4" app itself. As others have suggested, perhaps a third party app like Plex or similar may be the way to go with our own content? I know when the "4" first came out, all such bugginess and issues were spun as "just one software upgrade away" and now here we are- a very long time and several software upgrades beyond- still waiting for stuff like this and even smooth fast forward and rewind to work well. I consider such functions fundamental for any video playback device. Hopefully, Apple will "fix" all such fundamental functions soon.