For digital music, any audiophile device would have to do at least two things:
1. Work with lossless formats
2. Send out a bit perfect digital signal (untouched by the cheap DACs and Amp in the portable).
The iPad can handle Apple lossless, so step 1 is out of the way.
It can also send out a digital signal via the camera connector. I have read on forums that it is bit perfect, but I have not seen any actual reviews from the establishment stating that it is. The camera connector has dual purposes, so it is not exactly expensive to get the digital output. However, you would need a USB DAC or something to convert the USB to a more popular digital output/input like optical or coax.
Also, Apple recently crippled the USB output to some extent, so DACs that aren't powered could have issues. However, a T3hub can be purchased if you need to trick the iPad into thinking it is connected to power (I have tested the T3hub with an older uDac and it works).
All that being said, I agree that the iPad is better as a remote if you are at home because you won't have to connect a bunch of cables to it. You can buy an Airport Express for less than $100 (which has been tested to be bit perfect from Stereophile) and use Airplay to send it a digital signal. Because it can also control your entire iTunes library, you will have the ability to have terabytes of music rather than what the iPad can store on its drive.
Finally, "audiophile quality" means different things to different people. While it won't satisfy everyone, the above should at least satisfy a lot of people that care about sound quality.