Maybe it's your receiver but when I'm switching between channels on my AppleTV I'm adjusting volume.
I live in a condo as my primary residence so I'm pretty anal about volume as to no disturb neighbors.
Now maybe Chromecast is worse (I don't know) but AppleTV isn't exactly perfect either.
Agreed atv is not perfect, but what I've seen from chromecast is far, far worse.