I removed the bumper from my iPhone 4 the other day for the first time since fitting it well over a year ago (to take some photos to list it on eBay). I was horrified to find out that the metal side was now covered in loads of little scratches and dints, especially round the corners. I had heard people complaining before that the bumpers did this, but I have to admit I thought they were talking rubbish. Bottom line is that I'm always really, really careful with my gadgets; even though it is the first time I've never used a screen protector, after 16 months the screen and back of my iPhone 4 are in immaculate condition whereas the sides, which have always been fitted with an Apple bumper, are scratched and marked.
I don't think it's down to any roughness on the inside of the bumpers, I think little bits of dirt just work their way under and then get trapped and rub between the bumper and the metal. Maybe the new 4S bumpers won't suffer the same problem; I won't be getting one though. I'd recommend anyone fitting one checks their sides and cleans out the inside of their bumper every month or so, and also keeps an eye out for other people reporting problems with their bumpers causing scratches on their 4S's.