The headphone port is limited only by the quality of the DAC and amplifier contained within the phone. The 3.5mm jack is effectively just a wire. Every ultra high end headphone and headphone amplifier uses this 'antiquated' jack because there is nothing wrong with it. Removing the jack just means that the DAC and amp just move to the headphone where they get smaller still and likely lower quality still except in some very expensive implementations like the Audeze lightning headphones. The conversion still happens though, still travels over a piece of wire where that analogue signal moves the driver that your ear hears. You'll notice that Apple didn't mention audio quality as one of their reasons for making this change.
I've always read that if you have the choice between the using the headphone port for audio vs using the USB, USB is always the way to go.
Also, I'm willing to bet one of the reasons they removed the headphone port was to close up another potential spot where water can get into the phone.