If you use more than 1 set of normal headphones in different places it can be annoying. I leave mine plugged into my QC25s for now, but am thinking of moving it to my car's 3.5mm jack since I have BT headphones and don't really need to use the QC25.
Since they're only $9/each, I just bought a few so they can live permanently on my various wired headphones. That helps considerably with the annoyance factor.