I believe HomePod audio handoff works the same way handoff between Mac and iPhone does with safari, documents etc...which requires both devices to be on the same Apple ID.
Hand off tasks between your iPhone and your other devices
Use Handoff on iPhone to continue working on one device where you left off on another.support.apple.com
That makes sense. I guess I figured with multi-user support also came support for handoff with multiple users.