linux is the standard for servers, mac os server is not as capable as linux nor dose it have as much support
I would have to agree that you should get a linux server. That being said, I have to admit that I have never used a Mac server but I would think that having the GUI on it would be a waste of cpu cycles. The advantage of linux servers is easily found by google-ing it so I won't blather on about it. If you do need/want to have a gui to mangage your linux servers, I would recommend Webmin and not installing any window managers (though really...just learn the command line stuff, it is not as hard as you would think). Currently the company I work for runs debian linux on roughly 95% of our servers (we do have some windows servers, but they are limited in what they do (and being the lazy admins that we are, have found that in at least on case it's easier to do something on windows than on linux, not to mention that we can have the windows admins do the work and not us...now where is that remote control missle launcher again??))
Above all, I would definately not recommend using Windows servers. I don't really have too much problem with Windows per se, I just have not seen a windows server that was as solid and have the uptime that a linux/unix server does.
Oh, and I highly recommend debian linux or Ubuntu for your OS on the server, in that order. Ubuntu does have one advatage over Debian in that you can download a server edition that will let you set up a LAMP (Linux, Apache,Mysql, PHP) server in one go. With debian you have to install packages one at a time ( not that this is a time consuming effort).
Anyways, good luck
Cheers