After a quick google search, I get the impression that Virtualbox doesn't support hiDPI mode yet. (despite more than a year having passed since its introduction)
You could try SwitchResX to set 2880x1800 loDPI in OSX. Things will be very tiny but the guest should be able to use the full resolution and set its own hiDPI mode.