I mean, people get arrested without having broken the law all the time
This seems to be the point that many are missing. You can be arrested for a crime and detained while the police gather evidence against you.
Also, cases are built on circumstantial evidence all the time, often circumstances are coincidental. Ex: someone says they saw you commit a crime, and your phone shows that you were in the area. Eyewitness statements are notoriously inaccurate more often that you may think. But even if there isn't an actual connection the prosecution may use coincidental information to build a case.
Any information in your phone 'can, and will, be used against you', including if you're falsely arrested.