Well I dont clean mine and it's ok. The thing is, the plastic in the jack doesn't affect the sound. The jack connects to the metal connectors in the jack and quite frankly those are what you should worry about, fortunately apple makes good quality jacks and there's no issues with that so, in the words of a well repected author: Don't Panic.