I'll worry more about your headphones than the headphone jack itself. Headphones tend to be more brittle, and the tip of the plug is often susceptible/prone to damage more than the other areas. Therefore, I'm often more careful with that area when pulling the headphones out.