Dust of course is too soft to do any harm. It's grit that does the damage. Some of us encounter grit more often than others. Grit is small bits of sand or other hard particles. I turn the pockets of pants inside out during the wash and I vacuum out the insides of my purses so I've not had problem with grit despite working in the garden.
You just need to be careful.