What causes the bumper to scratch the band is particles getting in the bumper and rubbing it with the band surface.
Basically bumper will give you a little bit of protection, if you want that protection then go use it. Using it with no case/bumper will obviously give you absolutely no protection. If you want to use the bumper get a skin for the sides do that it won't scratch.