Only when I need to swap a SIM or when I use the Olloclip lens attachment. Otherwise, I don't bother removing the case.
That said, I also don't keep cases that can't be removed fairly easily. Since the Olloclip models for the iPhone 4 & 5 series can't be used with a case on, the last thing on earth I'd want is to have to fuss with a difficult to remove case.
I'm figuring the Olloclip for the 6 will be the same way...