how are you playing the music?
on your phone and airplaying to the HomePod?
on your hompod?
they look and sound almost the same, but they're handled differently.
also where is your phone when it transfers, if it's close to the hompod, it might be using handoff on it's own. Try leaving the phone at least a couple feet from the hompod.
it could be a network issue, especially since it drops out. Have you rebooted your router? or maybe try moving the router or hompod if it's in a place with poor wifi.
With AirPlay, you can stream music, podcasts, and more to your Apple TV, HomePod, or other AirPlay-enabled devices.
support.apple.com
You can hand off audio by bringing your iPhone near your HomePod. HomePod can also play audio sent from an iPhone or iPad using AirPlay and Siri.
support.apple.com