I decided to pop into the Genius Bar today and ask them about this. Before I did, I checked out all the display models. All of them were smooth apart from one small area on one handset.
Short story short, they replaced it with a white box device which is 100% smooth all the way around. I know some wouldn't be happy with a white box device, but I've never had an issue with a "refurb" in the past, so it's not an issue for me.
At this point into the release, I highly doubt you have a re-furbished X. Apple pulls stock to use specifically for their "white box" replacements.