That's not a bug. The icon appears on the album art top right corner when you select a whole album for offline.
It's so that if you decide you only like a couple songs off that album you can tap those songs and make them offline individually (giving the song the little icon) and then tap the album to remove all the rest you don't like with one button. (Same goes for playlists.) Offline has two layers so you can do that easier without having to delete one by one the songs you don't like from albums/playlists, keeping what you like and removing everything else.
Apple really needs to come out with a full manual, because there is so much misconception about the music app and Apple music flying around. (Between this and not knowing how to shuffle artists, I see so many complaints.)
Here ya go Apple Manuals