I had this problem with my iPhone 5 in its Survivor case, which has a bevel right around the edge of the screen. I finally found that if I put my finger just right on the very bottom of the screen and swipe quickly, the control center comes up quickly and reliably.
This problem is not unique. I have trouble doing anything that requires going to the very edge of the screen because of the bevel on the case (moving folders or icons from one screen to another, for example). But it has always been possible to do, just a matter of technique and some practice. The excellent protection provided by the case justifies a little inconvenience, at least for me.