I just took off my Speck Candygrip case, and i do have to apply some sort of pressure to bring the lip around. I really hate having to worry that I'll bend my phone since I change cases a lot. ugh. Sucks.
I truly don't think that's something you have to worry about.
I have the exact same case you do and i've taken it off about three times and nothing has happened.
I change my cases on an almost daily basis, and I don't think taking a case off, or putting one on requires nearly enough pressure that it takes to bend the device.