It is worryingly easy for even someone with very scant knowledge of such matters to recover data from drives, such as myself.
I accidentally deleted some photos from an SD card, and with software easily found on Google, was able to recover them all. It seems that when data is "deleted", it is only encrypted, not actually destroyed so it is easily recovered. If I'm correct (I'm sure someone knows better than me on this) but even a formatted disk can be recovered with the right tools.
Of course, you should keep sensitive things such as card details, logins etc in an encrypted database and not in for example a note. However if an experienced hacker wants your data, they are gonna get it, no matter what you do. Hell they don't even need physical access to your device, unless it is totally cut off from the world with no radio/Internet access. It's the same as if you lock your front door. No matter how strong it is, how many locks there are, it will stop most people, but it won't stop the totally determined.
My advice would be simply a factory reset, do it twice, perhaps and then carry on with your life. If a hacker wants your data, he's gonna get it whether he has your phone or not.