It doesn't even have to be sand. A lot of incidental, tiny debris, like dust, can be remarkably abrasive. This kind of abrasion can happen just from not making sure the iPod/iPhone and the case are both very clean before you put them together.
Well i have that exact rubber case and the same dots everywhere on the back. you may notice that the beizer gets unpolished too.
i bought this case for $1 in hong kong.