I just found a way to get OS X Server to install and to boot. When turning on the virtual machine, hold down SHIFT. When the Apple logo appears, keep holding shift. Just after that, you are able to get it working. It's in safe-mode, but it works.
However when I don't use safe-mode when I want to boot OS X Server, it doesn't do anything. When running it in Verbose mode, I only see the message "RpcOut_Send returned a failure". At the end of line first line, ATSServer is mentoined, and at the second is loginwindow mentioned. Those lines keep appearing and I can't boot normally 😕
UPDATE: I've just updated Mac OS X Server to 10.5.6. Now I see the following message: AF ALERT: socketwall_cntl_updaterules ctl_enqueuedate rts err 55. I notice hige HDD activity, and then it just freezes.
UPDATE2: I solved the problem. First I've loaded the installation program of Mac OS X Server and used the Disk Utility to repair file permissions. Then updated Fusion to version 2.0.2. I then got a message that VMware tools was outdated. Installed VMware Tools, rebooted and then I could boot Mac OS X Server normally 🙂