I wouldn't but it's entirely up to you, for me it was more the effort required to try get it swapped out.
Besides I look at my iP5 now, had it 6 months without a cover and there is almost no paint on the corners of the bezel anymore. But then you won't carry your iPad around in your pocket so I wouldn't worry about it.
Unless it affects the screen itself you'll be fine.