Is my virtual "Mac Pro" better than your real MP?

[DearthnVader]

I've been using High Sierra in Qemu for a few months, and it runs really well, there are a few drawbacks.

1. I haven't figured out how to pass a PCI Firewire or XHCI USB card to my virtual MP, yet, in a way that they work correct.

2. I only have a 8 core 16 thread "Penryn" cpu, as I can't get my EFI firmware to boot when I select a CPU that would be "supported" by about this Mac.

3. Can't figure how to pass my Apple bluetooth.

Now the befits:

1. Boot screens for my RX580.
2. "native" support for M.2 NVME booting.
3. unmodified version of the macOS.
4. I can run Windows, Linux, and the macOS all at the same time, passing each on 8 cores and 16 threads, and the Linux host will load level CPU use with no real performance hit.
5. I can upgrade my system.
6. My system is open source, and I'm only really limited to how good of a coder I am, as to what I can make it do.

I passthrough a OEM Apple 802.11AC wifi card, an M.2 960 EVO, and an RX580, as well as any AHCI drives and USB( sans Apple bluetooth ).

I'd love to take the "Pepsi challenge" with folks with real Apple hardware, as far as what works and benchmarks, tho if you have a 12 core 24 thread machine you will likely beat me out, as well if you have a 1070/1080/Titan XP, I'm sure you can put up better numbers than me. However my system costs around $1200 to build, and I can upgrade my CPU, video card, and ram.

 

[eksu]

I've thought about multi-heading my 2009 4,1-5,1 Mac Pro. I work in MacOS but like to play Windows Games.

Here's what I was thinking:

• Host OS, Windows Server 2016 (with Desktop)
• Run MacOS in Hyper-V
  
• Passthrough Apple Raid Card
• Passthrough Wifi/Bluetooth Adapter (Handoff)
• Passthrough RX GPU

Ultimately I decided that I don't want MacOS to be the guest, I would much rather it be a host and my video games run in the guest. Maybe device passthrough with MacOS as the host is possible with some work on QEMU. We need a) fast cpu access and b) some way to disconnect a PCIE device from the host so we can attach it to the guest.

Recent eGPU support makes me think macOS now has a way to disconnect GPU/PCIE devices from the Host. I'm not sure if vtd.c get's loaded on anything pre ivy bridge, so I need to explore that a bit.

QEMU's hvf (hypervisor.framework acceleration) also isn't currently working on my machine, but I don't think it's stable yet.

 

[bookemdano]

I don't think Hyper-V supports true passthrough (VT-D) does it? I know it didn't when I was looking at Hypervisors a few years ago.

VMWare ESXi does (even on non-Apple hardware if you use the unlocker tool), and I had some degree of success passing through a Radeon card and a USB 3.0 card. In general it worked OK (this was ESXi 5.5) but it was still not as seamless as running natively. Was pretty fun to see it working though!

 

[eksu]

Hyper-V calls it 'Discrete Device Assignment', they support it on Server 2016.

I would go the ESXi route but would I need a third GPU at that point? If Windows & MacOS both supported dynamic CPU / Memory (adding AND removing) ESXi would work well enough for my workflow. The goal is to basically always have macOS running and sometimes spin up windows.

 

[h9826790]

Hyper-V calls it 'Discrete Device Assignment', they support it on Server 2016.

I would go the ESXi route but would I need a third GPU at that point? If Windows & MacOS both supported dynamic CPU / Memory (adding AND removing) ESXi would work well enough for my workflow. The goal is to basically always have macOS running and sometimes spin up windows.

If this work, it may be the perfect solution for me. I mainly gaming in Windows, so, really need the GPU power. Also, I use the hardware encoding in Windows now. This also require to access the real GPU but not virtual device.

 

[DearthnVader]

Sadly the macOS has fallen behind on VT-x and VT-d as far as not public API.

I'd rather run a macOS host.

Of course it wouldn't hurt if Apple gave us some real hardware to do this on, the cMP doesn't really have a proper power supply and is also long in the tooth, and the nMP lacks PCI-E. Even if we could passthrough Thunderbolt devices it only supports TB2.

 

[bookemdano]

Hyper-V calls it 'Discrete Device Assignment', they support it on Server 2016.

I would go the ESXi route but would I need a third GPU at that point?

Ah, nice to know MS added that functionality to Hyper-V. I'll have to check it out at some point.

Passing through a card in any hypervisor makes it completely unavailable for the host. You can easily-enough assign it to different VMs, but you can't do that while either of them are running, and they can't share it, naturally.

Also, with ESXi, a GFX card isn't strictly needed for the host, because there's not a whole lot you can do from the console. Don't get me wrong it's nice to be able to have the ability to display the host console directly, but you can get by just fine without it.

 

[eksu]

If I may ask, do you have full iCloud / iMessage support in your QEMU setup? Are you basically off in Hackintosh land at that point?

As long as the OS will run, Hyper-V on a 5,1 makes a lot of sense because Windows gets to be on bare metal and also you're still on Licensed Hardware.

Not sure what I will do when 5,1 support is dropped if the nMP isn't adequate / extendable / upgradeable.

 

[bookemdano]

Can't speak for QEMU but I think as of ESXi 6 full iMessage and iCloud support is working in MacOS guests.

I'm not sure it's even possible to run MacOS in Hyper-V (maybe with Hackintosh stuff?). I know it is natively and officially supported (and licensed) to do so in ESXI on a Mac Pro.

 

[toru173]

I've been planning to run KVM on my 1,1's for a while. How is the performance? Did you have much trouble getting it up and running? What guides (if any) did you follow?

 

[DearthnVader]

I've been planning to run KVM on my 1,1's for a while. How is the performance? Did you have much trouble getting it up and running? What guides (if any) did you follow?

It's kind of a pain to get everything working correct.

https://github.com/kholia/OSX-KVM/tree/master/HighSierra

That covers the basics, but qemu's command line is like an OS all to it self.

Here is my qemu launch command:

#!/bin/bash

# See https://www.mail-archive.com/qemu-devel@nongnu.org/msg471657.html thread.
#
# The "pc-q35-2.4" machine type was changed to "pc-q35-2.9" on 06-August-2017.
#
# The "media=cdrom" part is needed to make Clover recognize the bootable ISO
# image.

##################################################################################
# NOTE: Comment out the "MY_OPTIONS" line in case you are having booting problems!
##################################################################################

MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep"

/home/jam/QEMU/bin/qemu-system-x86_64 -enable-kvm -m 8192 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
          -machine pc-q35-2.9 \
          -smp 16,cores=8 \
      -usb -usbdevice host:046d:c534 \
          -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
          -device vfio-pci,host=27:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=./RX580.rom \
          -device vfio-pci,host=27:00.1,bus=root.1,addr=00.1 \
          -device vfio-pci,host=01:00.0,multifunction=on \
          -device ioh3420,id=root_port1,multifunction=on,chassis=1,addr=03.0,slot=3,bus=pcie.0 \
          -device vfio-pci,host=1f:00.0,bus=root.1,addr=01.0,multifunction=on  \
          -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
      -drive if=pflash,format=raw,readonly,file=OVMF_CODE-pure-efi.fd \
      -drive if=pflash,format=raw,file=OVMF_VARS-pure-efi-1024x768.fd \
      -smbios type=2 \
      -device ich9-intel-hda -device hda-duplex \
           -device ide-drive,bus=ide.0,drive=MacHDD \
      -drive id=MacHDD,if=none,file=./mac_hdd2.img \
          -netdev bridge,id=hn0 -device e1000-82545em,netdev=hn0 \
          -usb -usbdevice host:05ac:828d \
          -usb -usbdevice host:0a5c:4500 \
          -usb -usbdevice host:046d:0825 \
          -device nec-usb-xhci,id=xhci \
           -usb -usbdevice host:04e8:6860 \
          -vga none

[doublepost=1528271000][/doublepost]

If I may ask, do you have full iCloud / iMessage support in your QEMU setup? Are you basically off in Hackintosh land at that point?

As long as the OS will run, Hyper-V on a 5,1 makes a lot of sense because Windows gets to be on bare metal and also you're still on Licensed Hardware.

Not sure what I will do when 5,1 support is dropped if the nMP isn't adequate / extendable / upgradeable.

Yes, I have full support for iCould, Messages, FaceTime and the App Store.

 

[DearthnVader]

I thought I'd update this figured out how to pass one of the USB PCI devices to the guest OS and now my Bluetooth works 100%.

Also hot plugging USB devices works. It reports XHCI but when I connect a USB 3 HD it only works in EHCI mode, I'll have to get some more USB 3 devices and test.

#!/bin/bash

# See https://www.mail-archive.com/qemu-devel@nongnu.org/msg471657.html thread.
#
# The "pc-q35-2.4" machine type was changed to "pc-q35-2.9" on 06-August-2017.
#
# The "media=cdrom" part is needed to make Clover recognize the bootable ISO
# image.

##################################################################################
# NOTE: Comment out the "MY_OPTIONS" line in case you are having booting problems!
##################################################################################

MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep,+ssse3,+sse4.2"

/home/jam/QEMU/bin/qemu-system-x86_64 -enable-kvm -m 8192 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
          -machine pc-q35-2.9 \
          -smp 16,cores=8 \
          -device ide-drive,bus=ide.0,drive=MacHDD \
      -drive id=MacHDD,if=none,file=./mac_hdd2.img \
          -device ide-drive,bus=ide.2,drive=MacHDD2 \
      -drive id=MacHDD2,if=none,file=./Movaje.img \
          -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
          -device vfio-pci,host=23:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=./RX580.rom \
(RX580 )          -device vfio-pci,host=23:00.1,bus=root.1,addr=00.1 \
 ( HDMI AUDIO )          -device vfio-pci,host=01:00.0,multifunction=on \ ( 960 EVO NVME )
          -device ioh3420,id=root_port1,multifunction=on,chassis=1,addr=03.0,slot=3,bus=pcie.0 \
          -device vfio-pci,host=1f:00.0,bus=root.1,addr=01.0,multifunction=on \ ( Apple 802.11 AC Wifi and Bluetooth( Power only for the Bluetooth connection via internal USB2 header )
-device vfio-pci,host=03:00.0,bus=root.1,addr=02.0,multifunction=on \ ( USB 2 front usb3, two rear usb3, internal USB2 headers ) ( Leaving 2 rear USB 3.1 and 2 rear USB2 for the host system )
          -drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd \
          -drive if=pflash,format=raw,file=OVMF_VARS-1024x768.fd \
          -smbios type=2 \
      -device ich9-intel-hda -device hda-duplex \
          -netdev bridge,id=hn0 -device e1000-82545em,netdev=hn0 \
          -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
          -vga none

 

[startergo]

I thought I'd update this figured out how to pass one of the USB PCI devices to the guest OS and now my Bluetooth works 100%.

Also hot plugging USB devices works. It reports XHCI but when I connect a USB 3 HD it only works in EHCI mode, I'll have to get some more USB 3 devices and test.

#!/bin/bash

# See https://www.mail-archive.com/qemu-devel@nongnu.org/msg471657.html thread.
#
# The "pc-q35-2.4" machine type was changed to "pc-q35-2.9" on 06-August-2017.
#
# The "media=cdrom" part is needed to make Clover recognize the bootable ISO
# image.

##################################################################################
# NOTE: Comment out the "MY_OPTIONS" line in case you are having booting problems!
##################################################################################

MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep,+ssse3,+sse4.2"

/home/jam/QEMU/bin/qemu-system-x86_64 -enable-kvm -m 8192 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
          -machine pc-q35-2.9 \
          -smp 16,cores=8 \
          -device ide-drive,bus=ide.0,drive=MacHDD \
      -drive id=MacHDD,if=none,file=./mac_hdd2.img \
          -device ide-drive,bus=ide.2,drive=MacHDD2 \
      -drive id=MacHDD2,if=none,file=./Movaje.img \
          -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
          -device vfio-pci,host=23:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=./RX580.rom \
(RX580 )          -device vfio-pci,host=23:00.1,bus=root.1,addr=00.1 \
 ( HDMI AUDIO )          -device vfio-pci,host=01:00.0,multifunction=on \ ( 960 EVO NVME )
          -device ioh3420,id=root_port1,multifunction=on,chassis=1,addr=03.0,slot=3,bus=pcie.0 \
          -device vfio-pci,host=1f:00.0,bus=root.1,addr=01.0,multifunction=on \ ( Apple 802.11 AC Wifi and Bluetooth( Power only for the Bluetooth connection via internal USB2 header )
-device vfio-pci,host=03:00.0,bus=root.1,addr=02.0,multifunction=on \ ( USB 2 front usb3, two rear usb3, internal USB2 headers ) ( Leaving 2 rear USB 3.1 and 2 rear USB2 for the host system )
          -drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd \
          -drive if=pflash,format=raw,file=OVMF_VARS-1024x768.fd \
          -smbios type=2 \
      -device ich9-intel-hda -device hda-duplex \
          -netdev bridge,id=hn0 -device e1000-82545em,netdev=hn0 \
          -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
          -vga none

Are you running QEMU on the top of OS or bare-metal booting from QEMU cold boot? I tried pass-through ESXI cold-boot, but it did not accept changes even though it said it did

 

[DearthnVader]

Are you running QEMU on the top of OS or bare-metal booting from QEMU cold boot? I tried pass-through ESXI cold-boot, but it did not accept changes even though it said it did

The host OS is Debian 9.

 

[startergo]

The host OS is Debian 9.

So it looks like you use hypervisor 2:

"Note on hypervisors: Type 1 vs Type 2
April 15, 2017


Hypervisors
Type 1 hypervisor
Type 1 hypervisor means that vm is running natively (bare metal) on hardware.

Examples: VMWare vSphere / ESXi, KVM, Xen / Citrix XenServer

Type 2 hypervisor
Type 2 hypervisor means that vm is running on another peice of software. Basically on a host OS.

Example: VMWare workstation.

QEMU, KVM and QEMU with KVM
QEMU is a type 2 hypervisor but it can become Type 1 if used with KVM.

Here’s how:

Virtualization technologies like VT-x (intel) and AMD-V, map physical CPU directly to Virtual CPU.

KVM enables mapping of physical cpu to virtual cpu. This provides hardware acceleration for virtual machine.

QEMU can use KVM (Linux kernel module) as vert-type to take advantage of this hardware acceleration and thus can become type 1.

QEMU used without KVM, can use TCG (Tiny co de generator); if your cpu doesn’t support hardware virtualization. Then it’s a type 2 hypervisaor."

 

[DearthnVader]

I prefer, I think, type 2.

I like having OS X and Debian usable as two machines on one computer.

At one point I had Windows 10 running as well, but three kb/m, monitors was just overloading me, and I don't really have any use for windows.

 

[startergo]

I prefer, I think, type 2.

I like having OS X and Debian usable as two machines on one computer.

At one point I had Windows 10 running as well, but three kb/m, monitors was just overloading me, and I don't really have any use for windows.

I wanted to run alongside OSX and Windows or Linux on baremetal and pass a videocard, processor to each virtual machine. For example AMD to the OSX and NVIDIA to Windows. This way I don't need a boot screen, I am connected straight to the hardware using the full capabilities of the cMP

 

[mattspace]

I asked one of the MOSR MacRumors Forum admins about setting up a subforum specifically about these sort of visualisation questions, perhaps with posts preconfigured to declare / sort based on host / client systems as a way to organise it. Would anyone else be interested in that? There really isn't a good resource for "i've never done this and don't know where to start", the way the Mac Pro community here is for people looking at getting into / maintaining a cMP.
[doublepost=1555343443][/doublepost]

I wanted to run alongside OSX and Windows or Linux on baremetal and pass a videocard, processor to each virtual machine. For example AMD to the OSX and NVIDIA to Windows. This way I don't need a boot screen, I am connected straight to the hardware using the full capabilities of the cMP

I'm also interested in this sort of thing, and frankly, I just don't know if it can be done - but to have a workstation, with something running the hardware, that can pass through processor and Nvidia GPU to a Windows system for VR work / gaming, and processor and an AMD GPU (if necessary) to a macOS system for the rest of my work (perhaps even to multiple versions of macOS for old software / systems etc). I really don't want to run two separate computers, unless I build a PC that's effectively a smallish VR appliance.

Has anyone else looked at / run the Linux distribution elementaryOS? It really looks a lot calmer / less blingy than macOS now.

 

[Flint Ironstag]

I asked one of the MOSR Forum admins about setting up a subforum specifically about these sort of visualisation questions, perhaps with posts preconfigured to declare / sort based on host / client systems as a way to organise it. Would anyone else be interested in that? There really isn't a good resource for "i've never done this and don't know where to start", the way the Mac Pro community here is for people looking at getting into / maintaining a cMP.
[doublepost=1555343443][/doublepost]

I'm also interested in this sort of thing, and frankly, I just don't know if it can be done - but to have a workstation, with something running the hardware, that can pass through processor and Nvidia GPU to a Windows system for VR work / gaming, and processor and an AMD GPU (if necessary) to a macOS system for the rest of my work (perhaps even to multiple versions of macOS for old software / systems etc). I really don't want to run two separate computers, unless I build a PC that's effectively a smallish VR appliance.

Has anyone else looked at / run the Linux distribution elementaryOS? It really looks a lot calmer / less blingy than macOS now.

I'd be really interested in this as I've been struggling to get ESXi working.

[edit]
The more I think about it, here's what I'd be willing to do. I may need a week or 3 to get a spare Mac Mini for the house. If there's a member willing to take on the challenge of getting the combination of Z820, ESXi, and Mojave working, and document the process for the community, I'll ship them one of my Z820s to do the work on. I would need the machine back at the end of the build.

Any takers?

Specs:
Z820
dual Xeon E5 2687w 3.1GHz (3.8 boost) - 16 cores
32GB RAM
1TB SSD
240GB SSD
couple of 15k spinners
Radeon RX 580
Nvidia GT 1030
HP Thunderbolt 2 card

 

[mattspace]

If there's a member willing to take on the challenge of getting the combination of Z820, ESXi, and Mojave working, and document the process for the community, I'll ship them one of my Z820s to do the work on. I would need the machine back at the end of the build.

Fantastic idea - I hope someone can take you up on it. The more I think about it, provided it doesn't result in noticeable performance loss, that would be the most "pro" computer I can imagine - a workstation specifically designed for every task, even the most realtime demanding, to be virtualised.

 

[PeterJP]

I've been using High Sierra in Qemu for a few months, and it runs really well, there are a few drawbacks.

Did you already find Nick Sherlock's blog about this? Very interesting setup he's got: https://www.nicksherlock.com/2018/11/my-macos-vm-proxmox-setup/

Proxmox is a Debian style Linux with an ESXi style web interface for QEMU. I love it. I built a server and successfully installed macOS on it already. Looking to build a new one for myself as a permanent setup.

So it looks like you use hypervisor 2:

"Note on hypervisors: Type 1 vs Type 2

How interesting. I didn't know.

I asked one of the MOSR Forum admins about setting up a subforum specifically about these sort of visualisation questions, perhaps with posts preconfigured to declare / sort based on host / client systems as a way to organise it. Would anyone else be interested in that?

Yes, very much. I posted a question about a ProxMox setup on tonymacx86 and they removed it. I think I was lucky I didn’t get banned. They have a policy against virtualization: Apple hardware only (yeah right).

The question was about a laptop setup where I'd reroute the dGPU to the MacOS machine.

 

[mattspace]

Yes, very much. I posted a question about a ProxMox setup on tonymacx86 and they removed it. I think I was lucky I didn’t get banned. They have a policy against virtualization: Apple hardware only (yeah right).

That's interesting, because I don't think the prohibitions against running macOS on non-Apple hardware are likely to be legally enforceable in a lot of jurisdictions, so long as you bought a Mac, and still have it in your possession, or have sold it without the OS installed.

The source of that notion, is that in certain jurisdictions, there's already settled consumer law that when buying something that is a collection of separate parts, every singe part that comes with it is purchased, and therefore subject to the full range of consumer warranties, fitness for purpose, first sale doctrine etc, and we all know that a lot of EULA is basically unenforceable and unsubstantiated scare language. The operating system is no different to the (removable) hard drive in that respect.

In other words, there's no such thing as "buying Mac hardware and getting Photos / iTunes / macOS for free", each of those is considered a purchased part of an overall package / bundle. It does give an insight as to why macOS isn't available to purchase, however, and why Server was made an app - in theory you wouldn't be able to run more concurrent VMs of macOS, than you had Macs laying around unused.

 

[PeterJP]

That's interesting, because I don't think the prohibitions against running macOS on non-Apple hardware are likely to be legally enforceable in a lot of jurisdictions, so long as you bought a Mac, and still have it in your possession, or have sold it without the OS installed.

Interesting. Here in the EU, customer protection is quite strong so I think I’m legally ok

By the way, what's the MOSR forum?

 

[Flint Ironstag]

I'd be really interested in this as I've been struggling to get ESXi working.

[edit]
The more I think about it, here's what I'd be willing to do. I may need a week or 3 to get a spare Mac Mini for the house. If there's a member willing to take on the challenge of getting the combination of Z820, ESXi, and Mojave working, and document the process for the community, I'll ship them one of my Z820s to do the work on. I would need the machine back at the end of the build.

Any takers?

Specs:
Z820
dual Xeon E5 2687w 3.1GHz (3.8 boost) - 16 cores
32GB RAM
1TB SSD
240GB SSD
couple of 15k spinners
Radeon RX 580
Nvidia GT 1030
HP Thunderbolt 2 card

Just to be clear on my offer - if we agree on a period of say 4-6 weeks for the work to be done, feel free to also document QEMU, KVM, and any other acronym I may have overlooked.

Bueller?

 

[bookemdano]

Insanelymac has long had a virtualization forum. Not sure how active it is these days as I've been out of that realm for a couple of years now. It's primarily centered on VMWare solutions (ESXi and Workstation/Fusion) I think there is some info there about other hypervisors also. Worth checking out if you want to learn about the subject.