So, let's walk through a hypothetical scenario.
You and a coworker are on a big -- super big -- project. Maybe you even work off shifts. You work first shift, and she works second shift. Because of the nature of the project you text each other a lot. This happens. I text most of my coworkers about work-related issues after hours. So, other than your spouse, she's the one you might text the most --, or depending on things, you may text her MORE than your spouse. If she's a close friend, maybe you share some of your personal life with her: the good and the bad. Maybe you complained about something your spouse insisted you do with them. Just random personal things.
Your coworker falls on some terrorist watch list, or, sadly, is actually one. Since you text her a lot, and during what people perceive to be "non-work"hours, you must be a terrorist also. The obtain a search warrant for your phone.
By your statements, you'd be perfectly fine handing them your cell phone. The FBI seeing every picture on your phone. Every text to your spouse. Every email. Every place you've been. Because even if you haven't done anything wrong, maybe the FBI can spin what you told your co-worker as criminal intent. Your phone places you in the same place as the coworker, so they are able to convince a grand jury you abetted her. Maybe you were *more* than friends. As a result of this you are fired from your job. You can't get a job because the media portrayed you as an accomplice who was also cheating on your relationship with her. Maybe you aren't really dating someone, but they still play up the sex angle. Because that's what the media does.
This happens more than you think. A lot of court cases are tried on nothing more than circumstantial evidence. If you think that someone you know or trust isn't a criminal or a bad person, you're going to be like that neighbor on the news saying, "He was a quiet person. Seemed nice. Liked animals." We don't know what our friends are really like.
We all have secrets. We all have things we don't want other people to see. I'm ok with handing my GF my phone because there's nothing on there I want to hide from her. But some conversations with her I may not want my friends to see. There are photos I may not want my boss to see. Nothing dirty, but my personal life is my personal life.
What I absolutely don't want to happen is someone to rifle through my digital life because of a chance occurrence with a criminal.
I stand a better chance of being hit by a drunk driver than a terrorist bomb, but I don't see everyone crying for breathalyzers in every car.