For a few years, all jailbreaks required that you connect your iDevice to a computer, convince it that the computer can be trusted, and then running software on the computer which does exploit things. So yes, security exploits, but none that a hacker can do from a distance, and if you have a passcode, none that a hacker can do without your passcode. So basically, Apple should protect you from taking a gun and shooting yourself in the foot, but they don't.
In the past, there had been one jailbreak that worked by visiting a website, and that is obviously a very dangerous security hole and was fixed very quickly. So figuratively, that would be a stranger shooting you in the foot because you asked them, but that stranger could obviously shoot you if you don't ask to be shot as well.
Good point about the physical access.
But just because all that have been done require you to ask permissions, doesn't mean that a method without asking couldn't be found (though history would imply this isn't possible, because we haven't seen it in the wild yet).