[edit] Purposefully Pausing music (not just removing airpods) will prevent the Audio/Music icon from displaying. Much easier than restarting the watch.
I have the same issue after updating to watchOS5. Finished a workout and pulled out my airpods. And hour later still flashing the red Audio/Music "icon". Happens most days and the only way I've found to remove it is to restart my AW3 LTE (saying that in case it only occurs with certain models). I'll try to remember to stop the music before removing my airpods.
Also, "killing" all apps doesn't help. (especially since "killing" apps, isn't really killing them)