Can you provide us a picture of said port? Might give us some ideas...
In my experience, USB ports shouldn't be very hard to remove USB devices from. There's nothing actually holding onto the plug, just metal/plastic contact. One possible method of removal if the port/plug have expanded is to cool them or heat them to get it to relax a little. I'm not talking a blowtorch---maybe an icecube inside of a sandwich bag or something similar.
It really should be pretty simple to remove it---wiggle it a little!