Where is the limit?
If creating a backdoor meant avoiding another 9/11, what would be the right thing to do?
I think the mania for "privacy" has gone way beyond common sense: until 10 years ago we were all happily storing our personal info in our houses, in paper, and no one was seriously worrying about someone sneaking in our houses to look at our family pictures, love letters or -oh my God!- our weight.
We too often forget that in 99.99% of the cases NO ONE COULD CARE LESS about our oh-so-precious pictures, messages, etc. There is quite simply nothing to protect, our personal info are valuable only to ourselves.
"Just" give me a common password for all my info/website, and if someones steals my device, big deal, let me make a call and block all access. End of story.
There is a huge difference between privacy of your house and privacy of your digital life. If someone broke into your house, you would know, and there would be enough physical evidence of some sort to know that it happened. At most, the burglar could rummage through and take photos of things, and you could have purchased a hidden safe to put your private belongings in - those things aren't new.
In the digital age, backdoors mean being able to - without a trace - download all of the information from your device. It's not a 1-off grainy photo of it or an old-timey note jotted down. It's the actual thing that they can rummage through for all time. Privacy may not have been as big of a deal before, but that's because we never had to trust so many people with a possible key to our front door before. With so many possible keys and doors, it would be impossible to tell when someone enters or exits. Many trust companies like Apple to do the right thing and hold on to the keys responsibly.
Before the digital age, you could only walk out with so many rolls of film or so many notes jotted down on your notepad before that, and you could really only process all of that manually. With the price of data storage being virtually close to free, the amount of information that you can store is unlimited. Scripting a way to analyze the information is a lot easier now as well. Where before it would be hard for a rogue government employee to track down someone - hire an agent that they don't have the right to order around to follow their ex and leave a paper trail - now they could easily do it with access to software and their target's GPS data that they pulled from their phone from the comfort of their office.
Just as you can't separate the terrorists from the law-abiding citizens, you also wouldn't be able to separate the police force and domestic and foreign leaders from terrorists online. Decreasing our security threatens to undermine our national and international security.
The amount of information that exists now that simply didn't exist before is a major part of the privacy concern as well, and a big part of why "privacy mania" wasn't as big of a deal before. It was harder to access, not all of it could be taken at once, it couldn't be taken without leaving evidence, and there has never before been such an overwhelming amount of data.
Also - "privacy mania" is nothing new. You can look at cases like Lotus Notes to know that it's been going on for a while, and there are many things that preceded it.
EDIT: edited to correct fat-thumbed iPhone typing and to add this:
In a post Patriot Act world where people can be arrested on suspicion - guilty until proven innocent - I believe that backdoors and open access to personal information are incompatible with the idea of a free people.