...I cannot even get the USB installer to boot.

(Updated)
I have Ubuntu 19.04 installed on an old hard drive in a Mac Pro 3,1 I use a test rig. I recently flashed some video cards using a live Linux DVD. See
How to Atiflash on a Mac Pro without Windows or boot screen. I managed to get a boot screen on an Radeon HD 7770. I left it in the Mac Pro 3,1 and moved the original Radeon HD 5870 Mac Edition to my second Mac Pro 1,1.
I also did some test on getting EFI shell to boot from an USB drive. This worked on the Mac Pro 3,1 with one of the 64-bit shell.efi files I found online. I even managed to add a .VolumeIcon.icns file to my USB drive that shows up on the boot screen.
For EFI Shell, maybe try the one from Clover build, as I believe it contains a later version of the shell which is compatible with MacPro3,1 EFI. Use the
ver
command to see the version of the EFI Shell. It will be 1 for early EFI Shell and 2 for a more modern version. For example, my Parallels VM has this version built-in:
There are differences in the commands supported by each version, including the help text. I think in the later version you can do
help * > help_a.txt
to get all the help into a text file. For the earlier version of the Shell which does not accept the * pattern in the help command, I create a help.nsh EFI script containing commands to get help for each command separately like this:
Code:
echo > help_a.txt
help alias >> help_a.txt
help attrib >> help_a.txt
...
Then you can compare the help of each version of the shell to see what most of the differences are.
To do all the experimenting, I had pulled out my Ubuntu 19.04 drive. Now when I reinserted it I cannot get the Ubuntu desktop to load. All I see is a purple background and a cursor. To fix the situation I decided to install Ubuntu 20.04 LTS on top of the 19.04.
I downloaded the
ubuntu-20.04.1-desktop-amd64.iso image and tried to flash it to a 16 GB SD card using balenaEtcher. I was immediately surprised by the result. I expected the drive to have FAT32 partition I could add a Ubuntu .VolumeIcon.icns file to. Instead it is totally weird! When looking at the drive with Disk Utility, it claims the drive has an Apple Partition Map and only a 4,1 MB FAT12 partition.
I do some searching and find that this is the expected result. The Ubuntu.iso image contains three different partition tables on top of each other. See
The app creates apple partitions on a flash to boot Ubuntu #2073. I was kind of not surprised when the SD card USB drive fails to show up on the Mac Pro boot screen.
Yes, the partitioning is weird. diskutil says it's APM, but dd of the first few blocks looks more like MBR/GPT hybrid.
Code:
sudo dd if=/dev/disk10 count=34 | xxd # block size 512
sudo fdisk /dev/disk10 # block size 512
sudo pdisk /dev/disk10 # block size is 2048 (0x800) so APM starting at block 1 means byte 0x800
sudo gpt -r show -l /dev/disk10 # block size 512
Actually, it looks like it is MBR/GPT/APM hybrid as you said (block #0 is MBR, block #1 has GPT with a backup GPT at the end of the disk pointed to be the primary GPT, block #4 has APM). The GPT is not viewable by the above commands without editing the disk. However, you can parse the
GPT manually to see that it does include info that corresponds to the partitions in the MBR and APM.
I suppose if you wanted to manually edit the EFI partition, you could dd it to a binary img file (without partition map, so it will just be a FAT image), mount it, make the changes, then dd it back to the USB disk.
Update: While writing this I decide to test the .iso image on a Kingston DataTraveler USB 3 stick. The boot screen now shows two external USB drives. The first one boots GRUB and Ubuntu.
Is there something wrong with SD cards or my Sony card reader? I have successfully ran OS installers from SD cards before. Maybe I need to buy some more USB sticks. But do I really need a 32 GB drive for a 2 GB .iso image?
I used balenaEtcher.app with a Kingston DataTraveller 100G3 (16 GB works). Maybe there's an issue with the type of USB interface. Maybe compare USB devices with USB Prober.app. I wonder if SD cards or SD card readers have some different behaviour that affects booting.
Update 2: Original issue solved. The problem is my badly flashed HD 7770 graphics card with a "jumping" boot screen. To see the boot screen on DisplayPort 2 I need to connect another monitor to DP 1, in this case an unpowered Cinema HD Display. To see the login screen after an OS boots, I need to turn both displays on or unplug the unpowered display. The Radeon HD 7770 reference design I have uses a Dashimaki framebuffer, while the HD 7950 the EFI rom came from uses the Hamachi framebuffer. I was able to
tweak the EFI, so that macOS recreates a RadeonFramefuffer after it boots. With Linux it does not work quite as well.
I haven't tried other GPUs in a long time. My receipt for the Mac Pro 2008 in my e-mail from July 25, 2008 says I bought it with a "Graphics Card 065-7537 NVIDIA GeForce 8800 GT 512MB".
I bought a "ATI RADEON HD 4870 512MB-ZML" Mar 5, 2009. I have both of these cards lying around still.
I also got a 4870 for a Dell desktop soon after.
The "EVGA GeForce GTX 680 Mac Edition Graphics Card" is from Jan 16, 2014.
I had Boot Camp booting issues with various SATA PCIe cards depending on the GPU installed, so now I don't have any other storage except USB 3.1 Gen 2 and built-in SATA (bay 1 to 4). USB 3.1 Gen 2 is faster than the fastest SATA so there's no point using anything else - except for booting - maybe should make a USB XHC driver for MacPro3,1 efi.