OP you don't mention if your opener is integrated into HomeKit. If it is, nobody should be able to use a HomePod to open a door - it should tell the person issuing the command to authenticate on their iPhone.
If the opener instead has its own app and is commanded via an iOS Siri Shortcut on your phone, I believe those require that "Personal Requests" be enabled on the HomePod or they won't run. You can turn Personal Requests off on individual HomePods, or roll the dice and hope voice recognition works and strangers can't run shortcuts on your iPhone. In that case, I like iStorm's answer as a safeguard as well. Hope this helps.