That they seem to fall apart at the corners. In my case all four corners. Luckily after six months Apple are sending me out a new one. I really like these cases and they do appear to protect the phone but I may have to reconsider when I replace my phone next.
I had this same experience four months into a silicone case. I agree on all the benefits of silicone, but the chipping was a dealbreaker; I Siam even moved to a Apple leather case.
Harder to get in and out of tight pockets. Sometimes they stick to certain types of pocket liner and that causes the pocket to turn inside out. In my experience they resist scuffing a bit more than leather cases so it's nice to use leather for everyday stuff (office, meetings, etc.) and silicone for when you're out and about (traveling, hiking, etc.).
I've heard MagSafe accessories stick to silicone cases better so that's another bonus.