Leaving licensing issues aside, I've set up smallish (5-20) workstation networks with both Mac and Windows servers, and to me I'd go OSXServer without a second thought.
Windows server was significantly more difficult to get set up when I wanted to do anything more than "default" stuff, and it was finicky for no discernible reason. OSXServer, in contrast, was easier to set up than I'd expected (though still not for the non-technical) and as been remarkably stable. The remote admin tools are much nicer as well.
It's really going to depend heavily on what you want to do with the server, though. In my day job environment (~15 Macs, 4 PCs, and a few laptops) the main purpose is file/print services, with a secondary bonus of OpenDirectory network logins (though we do not use network home folders). In terms of files and printing, this works wonderfully in a mixed-platform environment. Spotlight also works well on shared volumes--VERY fast once you enable network searching. Netboot images are also a possibility, though I haven't personally done more than mess around with them, and booting was rather slow.
If you're going to be running an Active Directory or similar Windoes-centric domain services, I'd assume Windows Server would do better, and might be the preferable choice. (Although Windows clients should be able to use accounts off the OSXServer, or so the relevant tab claims--I've never tested that.) I also like some of the automatic document versioning it can do, though I confess to never having gotten it working right. Windows used to have better granular ACL control, but OSX has caught up at this point.
I haven't been impressed by the speed of OSX network home folders, if you're planning on a completely centralized thin-client system, but to be fair I haven't used them recently and my experience with the Windows equivalent (as a user, not configuration) wasn't stellar, either.
As far as hardware, I'm certainly happy with my XServeG5--thing has been running solidly for 4.5 years in a dusty, very not-climate-controlled environment with the only failure being a stick of RAM after four years that gave plenty of warning with increasing ECC errors. The one Dell server of a similar vintage I set up was cheaper, but came with internal cables unplugged, a bad optical drive, the tape drive failed after a few months, and tech support was a pain to deal.