Yep! I have three HomePods and an Apple TV so it's always a bit of a struggle. Once something on Apple Music or Podcasts gets AirPlayed to a HomePod, it seems to then stream there directly and kind of "detach" from the iPhone -- which just gets confusing and weird, especially when there are multiple HomePods. Sometimes the volume button on the phone controls the speaker volume, sometimes not ¯\_(ツ)_/¯Agreed. I wish they'd come up with a better way to manage the Apple TV and HomePod controls compared to local/on-device controls.
All of us see inconsistent behavior where sometimes somebody else watching TV on an Apple TV will trigger the remote to show up on our phones. Other times, the mini playback controls show up automatically on the Lock Screen but the TV remote doesn't come up on its own. Sometimes I want the TV remote to stay up (iPhone 15 Pro AOD) and it refuses to. It's just very frustratingly inconsistent, unreliable, and there's just no clear logic as to why it chooses to do each behavior some of the time.