MP 7,1 MP 7,1 Thunderbolt pcie lanes, pool, allocation?

[JesterJJZ]

I remember reading about this and even seeing a diagram on how my 2019 MacPro splits up the Thunderbolt ports. I have a Vega Pro II Duo in one of my MPX slots.
Doing some rearranging and trying to see what my optimal connections should be.

Two 4k Displays, currently USB-C to Displayport from two Thunderbolt ports on my graphics card.
Optical Thunderbolt to my Qnap server in a third port on my Graphics card.
Blackmagic Ultrastudio on system Thunderbolt port on back of MP.
External Drive dock on system Thunderbolt port on back of MP.
Random USB and C drives to top Thunderbolt ports and USB on back.

I think ideally my main concern should be to split my displays and my server right?

Any input appreciated!

Thanks!

 

[arw]

Do you mean the diagram linked here?:

https://forums.macrumors.com/threads/pci-express-terminology.2438485/post-33450739

(Points to page 11 of https://www.apple.com/mac-pro/pdf/Mac_Pro_White_Paper_Aug_2021.pdf)

 

[joevt]

I think ideally my main concern should be to split my displays and my server right?

All Thunderbolt ports are connected to Pool B (in the Expansion Slot Utility).
https://www.apple.com/mac-pro/pdf/Mac_Pro_White_Paper_Aug_2021.pdf

But that's only for PCIe data. DisplayPort data is separate from PCIe data when the PCIe data and the DisplayPort data are coming from separate Thunderbolt buses. I think that's also true if they are coming from different ports of the same Thunderbolt bus. That means that two 4K displays connected to port 1 of bus 1 should have no effect on PCIe read/write speeds for port 2 of bus 1. But if there is an effect, then you will want both DisplayPorts coming from the same bus (say bus 1) and have PCIe data come from the other bus (e.g. bus 2) which is probably how you have it connected now - if the two DisplayPort outputs are coming from the two left or two right ports of the GPU.

USB ports not from a Thunderbolt port are not in any of the 4 pools. Same for Ethernet, WiFi, SATA, and NVMe. These are all connected to chipset DMI.

You'll want to keep PCIe data to under 2700 MB/s per Thunderbolt port and under Thunderbolt 2800 MB/s per bus.

I think DMI is limited to 4 lanes of 8 GT/s ≈ 3500 MB/s (no more than 3938 MB/s).

A pool is limited to 16 lanes of 8 GT/s ≈ 14000 MB/s (no more than 15753 MB/s).

You can use ATTO Disk Benchmark.app to test the bandwidth of multiple disks connected to different ports of the Mac to test the max bandwidth of a pool or DMI or bus or whatever without having to create a RAID 0.

 

[mattspace]

Random USB and C drives to top Thunderbolt ports and USB on back.

Specific to this - when you plug a USB device into a TB port, check your lane allocation for Pool A & B availability. My experience of plugging USB via cable adapters to the I/O card's TB ports (haven't tried the GPU) is that it causes an extra 4 PCI lanes to be assigned from the pool.

 

[JesterJJZ]

If all Thunderbolt is pool B, then it doesn't matter which ports I use?

 

[joevt]

If all Thunderbolt is pool B, then it doesn't matter which ports I use?

I/O in a computer is in a tree structure starting from the CPU. Each branch in the tree may be a bottleneck - meaning you can connect enough devices to max out the bandwidth of a branch such that some devices do not have max performance when some other devices are being accessed at the same time.

Just because all Thunderbolt ports are in Pool B doesn't mean you can connect everything to a single Thunderbolt port and expect max performance from all devices. Pool B supports 14000 MB/s but each Thunderbolt bus supports only 2800 MB/s and each port supports less than that.

CPU
    - 140000 MB/s - RAM
    - 3500 MB/s - DMI
        - 3400 MB/s - NVMe SSD
        -  960 MB/s - 10 GbE port 1
        -  960 MB/s - 10 GbE port 2
        -    ? MB/s - SATA controller
            - 540 MB/s - SATA port 1
            - 540 MB/s - SATA port 2
        -    ? MB/s - USB Controller
            -  460 MB/s - USB port 1
            -  460 MB/s - USB port 2
            -  460 MB/s - USB port internal
        -    ? MB/s - WiFi
        -    ? MB/s - Bluetooth
        -    ? MB/s - Audio
    - 14000 MB/s - MPX Slot 1
    - 14000 MB/s - MPX Slot 3
    - 14000 MB/s - Pool A
        - 14000 MB/s - Various PCI slots
    - 14000 MB/s - Pool B
        - 14000 MB/s - Various PCI slots
        -  2800 MB/s - Thunderbolt Bus 1
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 2
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 3
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 4
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 5
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 6
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2

Some of the numbers above are guesses or estimates. You can measure the bandwidth limits of MPX Slot 1, MPX Slot 2, Pool A, Pool B, the SATA controller, the SATA ports, the NVMe, the DMI, the Thunderbolt buses, the Thunderbolt ports, etc. if you have enough storage devices.

To measure the max bandwidth of a Thunderbolt port, connect the fastest device to a single port and measure its bandwidth. Then connect additional devices to the same port until the bandwidth no longer increases.

To measure the max bandwidth of a Thunderbolt bus, connect enough devices to max the bandwidth of the first port, spread the devices among both ports to see if the bandwidth increases - add devices until the bandwidth no longer increases.

The Expansion Slot Utility is an overly simplified representation of the above bandwidth limits. While a Thunderbolt controller does use 4 lanes of PCIe, the bandwidth it can use does not exceed 75% of that.

 

[JesterJJZ]

Isn't there a simpler breakdown of which busses go to which ports and where the bandwidth is split or shared?

Like top two thunderbolt ports share bus "x" etc...?

 

[joevt]

Top two Thunderbolt ports is a bus.
Rear two Thunderbolt ports of the I/O card is another bus.
A GPU in an MPX slot can have up to two Thunderbolt buses = two Thunderbolt ports each = four Thunderbolt ports total. Adjacent ports (port 1 and 2 and port 3 and 4) belong to the same bus.
You can check which Thunderbolt bus each Thunderbolt port belongs to by connecting a Thunderbolt device and checking the Thunderbolt tab of the System Information.app.

Like shown in my diagram, all Thunderbolt buses have an equal connection to Pool B.

 

[JesterJJZ]

Very helpful thanks!

So I guess the biggest takeaway is to avoid putting my biggest throughput devices on the same bus.

 

[joevt]

So I guess the biggest takeaway is to avoid putting my biggest throughput devices on the same bus.

If together they can max out any of their common upstream branches then its best to move a device to a different branch where that doesn't happen or it happens less.

Also consider if the devices will actually be used at the same time (writing to both or reading to both simultaneously). If that's a rare occurrence then it probably doesn't matter if together than can max out any of their common upstream branches.