Had a space black S3 for a year, and there was no damage at all to the casing, but I did have an unfortunate incident with the space black milanese loop scratching up the display.
It wasn't the worst thing in the world, could only be seen in certain lighting. I ended up going with a silver stainless steel for S4, and while it does scratch like crazy on the casing, it's not really visible.
All that being said, I do miss the sexy space black look, how the display just melts away into the casing without any real visible seams. If I do go SBSS for S5, I'm definitely going to play it safe with the milanese loop.