Video cards are an example of a device which assumes it is controlled by exactly one OS. If a guest were to be able to access a graphics card directly, it could draw anywhere on screen it wanted, affect host textures, etc. Even a well-intentioned guest would cause problems, because it wouldn't be aware of what the host is doing ("Hey, what's this texture? I don't recognize it, must not be important!" and then your windows/icon/desktop/menus/etc. disappear). It's not possible to dedicate an entire graphics card to the guest either, since the underlying buses are also not safe to pass through - see for example Re: Guest able to directly access PCI cards for a good explanation.
Because of this, we take the emulation approach. The guest sees a VMware video card, and we do the work of converting guest commands into something that's safe and usable for the host video card. There's no point in installing drivers for the host video card in the guest (with the exception of Boot Camp virtual machines, where you might want to native boot) since the guest never gets to speak directly to the host video card.
.
