I had some marks where the smart pad connected but they rubbed right out. When I first saw it I thought they were scratches but I guess it was just dirt/dust build up.
That's how mine started to wear - I was really careful with it but it continued and wore down to bare metal right along the edge of the iPad where the face meets the back, right where the small portions of the smart cover hinge hit the case.
Apple replaced both the iPad and smart cover but I'm not taking any chances. I'm getting a back skin and have added a bit of moleskin to the inside face of the smart cover hinge (the small sections w/out the magnets).
I feel your pain. That's what mine looked like 4 days after purchase. I took it to the Genius Bar and even though the guy said it looks like something got in between the cover hinge and the device (causing the scratch) he replaced my iPad with a new one and refunded me the cost of the smart cover so I could buy another kind.
Always go to the Genius Bar. It's not always a sure thing but every now and then you'll get someone that will do a replace/exchange.