In the past, I have been using coloured cases but I switched to Spigen Ultra Hybrid with clear back and sides for iPhone 5s. Couldn't be happier because my preference is to see the design of the phone and there is usually less bulk associated with any clear cases.
Again for the dirt accumulation, it depends on what kind of environment the phone is being used.
Generally, if you have the habit of cleaning the screen and the exterior of the case daily or a few times a week, I don't see a substantial amount of dust being trapped between the phone and the case. Putting the phone in a pouch or sleeve would help the dust control as well which I have been doing. YMMV
But in addition to dirt accumulation, there may be some downsides of using clear case depending on the case design. Some involve
1. Scratches and wear on the case are more visible unless the back of the case has some scratch-resistant coating such as some cases from Spigen, Power Support, etc.
2. If the sides of the case is also clear, there is a chance of discolouration. (Yellowing is a non problem for high quality clear case as UV coat is added). Each person has different chemistry. You may see the side of the case becomes yellowish, greenish, etc.
