If you never gave anyone direct access to your Mac, it wasn't hacked. In over 6 years of reading "my Mac was hacked!" claims in this forum and elsewhere, not a single one ever was.
Get a program called little snitch and that will report all incoming/outgoing traffic and you'll be able to determine if something suspicious is going on.
As long as you've taken the precautions that you mention, you should be safe, but for peace of mind, Little Snitch is a handy utility.