how are you playing the music?
on your phone and airplaying to the HomePod?
on your hompod?
they look and sound almost the same, but they're handled differently.
also where is your phone when it transfers, if it's close to the hompod, it might be using handoff on it's own. Try leaving the phone at least a couple feet from the hompod.
it could be a network issue, especially since it drops out. Have you rebooted your router? or maybe try moving the router or hompod if it's in a place with poor wifi.
Stream music, podcasts, and other audio to Apple TV, HomePod, or AirPlay-compatible speakers or smart TVs. Play audio across multiple speakers throughout your home in perfect sync. Or play something different in every room—all with AirPlay.
support.apple.com
Bring your iPhone near HomePod to hand off audio. HomePod can also play audio sent from an iPhone or iPad using AirPlay and Siri.
support.apple.com