When you say "freezes", do you think it is the OS that is freezing or a momentary lag from the remote input?
Does the "freeze" happen when you are using the TV remote as well? Could be a few things that I can think of.
If you are like me and bought your ATV4K1 in 2017, it is been running for a long while and the fan may need cleaning. This may explain why you just started experiencing the issue only after switching to a 4K TV, you started pushing the HW harder, causing more heat, but the dirty fan may not be able to keep up. Maybe do a gentle cleaning of the fan via the vents?
Another issue could be the Match Dynamic Range settings, if this is the case, then it is working as intended. If you have your ATV settings to match the format playing, and for example, the ATV is constantly switching from HDR to non-HDR content, there is a slight delay when switching:
Learn how to turn on frame rate and dynamic range matching.
support.apple.com
If the "Freeze" doesn't happen with the TV remote over CEC, then I would suspect it is a connection issue with the Siri Remote. Consider re-pairing the remote, or move the ATV4K1 to a different location, line of sight is ideal (although, not required).
I remember a long while ago, I bought a new TV and with it a mounted the ATV4 behind the TV, and then started having intermittent, bad input lag on the remote. I think the TV mount was blocking or interfering with the RF.