Yeah, if you're paranoid as all hell.
I'm certain 9/10 men will feel the same way if you put out a survey using OP's story.
We are human. Even if our mouth and rational thought say "I trust you 100%". There is always a little voice behind our head that casts doubt.
On a side note, I used to frequently a GPS-tracking software forum because of business needs (tracking fleet of trucks). I read threads where users asked how to track their SO using GPS mobile phone because they are suspicious. Many times the OP will report back confirming his/her initial doubt such as wife saying she's out with the girlfriends when in fact she is meeting some dude in a hotel.