Using anything for an extended period with your hands will deposit oils and dead skin cells on that object, it's unavoidable and part of human life, quite natural and not anything anyone should be overly concerned with. Even the magic mouse (which of course isn't really magic. 😉) will attract this residue.
However as others have attested to, due to its simple, largely unbroken shape it's easy to keep clean by wiping it off.
Also, excessive hand washing is really not a good thing. It can make your skin dry and parched, increasing the risk of rashes or allergies, and if you use anti-bacterial soap, liquids or other such products it will propagate bacterial resistance, making the little buggers tougher to kill and more dangerous to us. Very very bad!