Logically you are correct. The Patriots have also lost some real heartbreakers since their last SB victory (2006 AFCCG, 2007 SB, 2009 WC, 2011 SB), which people have really enjoyed. It's not like it's been all glory and champagne. But they basically have a realistic shot at the Super Bowl every year as long as Brady is QB and Belichick is coach. This will be their 6th Super Bowl appearance in 14 seasons. They've had a stranglehold on their division forever. The QB is a handsome millionaire who married a supermodel who is even richer than he is. Our coach is a surly bastard who pushes the limit, ignores the unwritten (and sometimes written) rules and has no problem expressing his disdain for dumb questions and a large portion of the media. He's also smarter and better than almost all the other coaches. There's Spygate, obviously. There's the Patriots using every kind of play and advantage to win games (hurry up offense, quick snaps, unusual formations). There's the fact that they never let up on an opponent until the clock strikes 0.
The fans are from a city that went from being joked on (Red Sox, 1918, Who's Your Daddy, etc.) and irrelevant for the last 25 years (Celtics post-1986, Patriots always, Bruins) to winning championships consistently across all the sports. It's created its fair share of obnoxious, entitled fans that seem to stand out loudest among the crowd. Not exactly a recipe for everyone to love the Patriots, you understand. To an extent I really do get it.
If you're a fan of the team, you just have to accept it and realize it's just noise and it comes with the territory of utterly unprecedented winning over a long period of time. Some people are never going to like the Patriots, and they'll latch on to whatever far-fetched reason to discredit or disrespect them. If you give me the option of the Patriots success of the last 15 years and being hated vs. the Browns last 15 years and being lovable losers...well, the choice is simple.