Try a sewing needle or safety pin to really get into the corners. That will probably do it.
But if those really don't work, try a tiny TINY drop of rubbing alcohol and then use the needle again to scrape in there. Hold the phone straight up the whole time so nothing drips into the speaker.
I'm serious about the smallness of this drop. Use a toothpick to apply one raindrop's worth of alcohol.