The "sell it and buy a PM" posters really are correct in this case; the XServe is designed as a server/cluster node, period. You're paying a large premium for the high-density form factor, hot-swappable drive bays, ECC RAM, and user-replaceable guts plus some management features (front-panel load lights, identifier lights, hardware lock, serial port, etc), plus the dual network interfacces.
In exchange for this, you sacrifice expandability, all graphics performance worth mentioning, nicities like wireless and sound, quiet (believe me, even with the fans on low the XServe is LOUD to keep those chips cool in such a small space), and the extra heat that can be generated in a big desktop closure (this is why the XServe maxes out at 2.3GHz while the PM goes to 2.7GHz).
The XServe is a very nice server, and I like the one my lab has, but I wouldn't even consider using it as a desktop--you're paying a lot for a bunch of features you don't need and not getting other features you need. Remote access for number crunching from a Mini would work, but from the sounds of it you need yourself a PowerMac, and for the price you could get for an unopened XServe you can easily buy a notably faster loaded-up PowerMac. Faster processors, much better graphics, and money to spare for lots of RAM, external drive enclosures, or whatever else you want.