Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
The problem with poking things in there is that it can shove stuff in there tighter. The best thing I’ve ever found is poster putty. It goes into the holes without ever breaking off. When it comes out, all the pocket lint comes out with it.
 
The speaker holes in my XS are filthy, I must have really dirty pockets (likely related to my job).

I was thinking of getting something like these https://www.ebay.co.uk/itm/100pcs-Wood-Wooden-Handle-Stick-Cotton-Swabs-Buds-Cleaning-Tool-Medical-Makeup/272669745366?_trkparms=aid=555017&algo=PL.CASSINI&ao=1&asc=20151005190705&meid=a01ba1e447964cbfa59d396a1028baba&pid=100506&rk=1&rkt=1&&itm=272669745366&_trksid=p2045573.c100506.m3226 to clean them.

What’s in those holes, is it a form of speaker cloth? I’m concerned about poking around in there might damage something.

Don’t insert anything into the speaker grill holes. What I would do is get a soft bristle brush, and lightly use it in circles to remove any debris or use some canned air, but from a safe distance where it doesn’t damage any speaker internals.
 
The problem with poking things in there is that it can shove stuff in there tighter. The best thing I’ve ever found is poster putty. It goes into the holes without ever breaking off. When it comes out, all the pocket lint comes out with it.

Is poster putty Blu-tak?
 

After watching the above Video, that is the only way I clean the speaker holes of all my iPhones and those of my family since over two years now :)
Works like charm!
 
I use canned air sprayed at an angle so not to damage the speakers. For the top speaker earpiece I used a piece of scotch tape to lift dust away.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.