I've tried doing it as fast as I possibly can. Works every time without fail. I do get a keyboard clicking sound sometimes, like it thinks I've hit one of they keys. But the control centre still opens. I really don't know what technique people are doing to open the control centre but It works for me all the time no matter how fast I swipe.Perhaps it's exactly the same as normal for you because you normally swipe up slowly, or normally swipe up well above the keyboard, regardless of whether the keyboard is present.
Do it fast and it doesn't really work (unless you reach up well above the keyboard).