Anyone losing sync with Apple remote and Macs?

[The.316]

Whether it be the older white Apple remotes, or the newer silver ones, after a few hours, my remote just stops working with my iMac, on El Capitan. I have to go into the security setting/advanced, and check/uncheck "Disable remote control IR receiver." I have never had issues like this before El Capitan, but now I am getting it all the time, and its driving me crazy. Anyone else have this problem?

 

[The.316]

Am I the only one having this problem? I would have thought that with El Capitan being a new OS, there might be a few more people with the same issue.