MP 7,1 MP 7,1 Thunderbolt pcie lanes, pool, allocation?

[JesterJJZ]

I remember reading about this and even seeing a diagram on how my 2019 MacPro splits up the Thunderbolt ports. I have a Vega Pro II Duo in one of my MPX slots.
Doing some rearranging and trying to see what my optimal connections should be.

Two 4k Displays, currently USB-C to Displayport from two Thunderbolt ports on my graphics card.
Optical Thunderbolt to my Qnap server in a third port on my Graphics card.
Blackmagic Ultrastudio on system Thunderbolt port on back of MP.
External Drive dock on system Thunderbolt port on back of MP.
Random USB and C drives to top Thunderbolt ports and USB on back.

I think ideally my main concern should be to split my displays and my server right?

Any input appreciated!

Thanks!

 

[arw]

Do you mean the diagram linked here?:

https://forums.macrumors.com/threads/pci-express-terminology.2438485/post-33450739

(Points to page 11 of https://www.apple.com/mac-pro/pdf/Mac_Pro_White_Paper_Aug_2021.pdf)

 

[joevt]

I think ideally my main concern should be to split my displays and my server right?

All Thunderbolt ports are connected to Pool B (in the Expansion Slot Utility).
https://www.apple.com/mac-pro/pdf/Mac_Pro_White_Paper_Aug_2021.pdf

But that's only for PCIe data. DisplayPort data is separate from PCIe data when the PCIe data and the DisplayPort data are coming from separate Thunderbolt buses. I think that's also true if they are coming from different ports of the same Thunderbolt bus. That means that two 4K displays connected to port 1 of bus 1 should have no effect on PCIe read/write speeds for port 2 of bus 1. But if there is an effect, then you will want both DisplayPorts coming from the same bus (say bus 1) and have PCIe data come from the other bus (e.g. bus 2) which is probably how you have it connected now - if the two DisplayPort outputs are coming from the two left or two right ports of the GPU.

USB ports not from a Thunderbolt port are not in any of the 4 pools. Same for Ethernet, WiFi, SATA, and NVMe. These are all connected to chipset DMI.

You'll want to keep PCIe data to under 2700 MB/s per Thunderbolt port and under Thunderbolt 2800 MB/s per bus.

I think DMI is limited to 4 lanes of 8 GT/s ≈ 3500 MB/s (no more than 3938 MB/s).

A pool is limited to 16 lanes of 8 GT/s ≈ 14000 MB/s (no more than 15753 MB/s).

You can use ATTO Disk Benchmark.app to test the bandwidth of multiple disks connected to different ports of the Mac to test the max bandwidth of a pool or DMI or bus or whatever without having to create a RAID 0.

 

[mattspace]

Random USB and C drives to top Thunderbolt ports and USB on back.

Specific to this - when you plug a USB device into a TB port, check your lane allocation for Pool A & B availability. My experience of plugging USB via cable adapters to the I/O card's TB ports (haven't tried the GPU) is that it causes an extra 4 PCI lanes to be assigned from the pool.

 

[JesterJJZ]

If all Thunderbolt is pool B, then it doesn't matter which ports I use?

 

[joevt]

If all Thunderbolt is pool B, then it doesn't matter which ports I use?

I/O in a computer is in a tree structure starting from the CPU. Each branch in the tree may be a bottleneck - meaning you can connect enough devices to max out the bandwidth of a branch such that some devices do not have max performance when some other devices are being accessed at the same time.

Just because all Thunderbolt ports are in Pool B doesn't mean you can connect everything to a single Thunderbolt port and expect max performance from all devices. Pool B supports 14000 MB/s but each Thunderbolt bus supports only 2800 MB/s and each port supports less than that.

CPU
    - 140000 MB/s - RAM
    - 3500 MB/s - DMI
        - 3400 MB/s - NVMe SSD
        -  960 MB/s - 10 GbE port 1
        -  960 MB/s - 10 GbE port 2
        -    ? MB/s - SATA controller
            - 540 MB/s - SATA port 1
            - 540 MB/s - SATA port 2
        -    ? MB/s - USB Controller
            -  460 MB/s - USB port 1
            -  460 MB/s - USB port 2
            -  460 MB/s - USB port internal
        -    ? MB/s - WiFi
        -    ? MB/s - Bluetooth
        -    ? MB/s - Audio
    - 14000 MB/s - MPX Slot 1
    - 14000 MB/s - MPX Slot 3
    - 14000 MB/s - Pool A
        - 14000 MB/s - Various PCI slots
    - 14000 MB/s - Pool B
        - 14000 MB/s - Various PCI slots
        -  2800 MB/s - Thunderbolt Bus 1
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 2
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 3
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 4
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 5
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2
        -  2800 MB/s - Thunderbolt Bus 6
            - 2750 MB/s - Port 1
            - 2750 MB/s - Port 2

Some of the numbers above are guesses or estimates. You can measure the bandwidth limits of MPX Slot 1, MPX Slot 2, Pool A, Pool B, the SATA controller, the SATA ports, the NVMe, the DMI, the Thunderbolt buses, the Thunderbolt ports, etc. if you have enough storage devices.

To measure the max bandwidth of a Thunderbolt port, connect the fastest device to a single port and measure its bandwidth. Then connect additional devices to the same port until the bandwidth no longer increases.

To measure the max bandwidth of a Thunderbolt bus, connect enough devices to max the bandwidth of the first port, spread the devices among both ports to see if the bandwidth increases - add devices until the bandwidth no longer increases.

The Expansion Slot Utility is an overly simplified representation of the above bandwidth limits. While a Thunderbolt controller does use 4 lanes of PCIe, the bandwidth it can use does not exceed 75% of that.

 

[JesterJJZ]

Isn't there a simpler breakdown of which busses go to which ports and where the bandwidth is split or shared?

Like top two thunderbolt ports share bus "x" etc...?

 

[joevt]

Top two Thunderbolt ports is a bus.
Rear two Thunderbolt ports of the I/O card is another bus.
A GPU in an MPX slot can have up to two Thunderbolt buses = two Thunderbolt ports each = four Thunderbolt ports total. Adjacent ports (port 1 and 2 and port 3 and 4) belong to the same bus.
You can check which Thunderbolt bus each Thunderbolt port belongs to by connecting a Thunderbolt device and checking the Thunderbolt tab of the System Information.app.

Like shown in my diagram, all Thunderbolt buses have an equal connection to Pool B.

 

[JesterJJZ]

Very helpful thanks!

So I guess the biggest takeaway is to avoid putting my biggest throughput devices on the same bus.

 

[joevt]

So I guess the biggest takeaway is to avoid putting my biggest throughput devices on the same bus.

If together they can max out any of their common upstream branches then its best to move a device to a different branch where that doesn't happen or it happens less.

Also consider if the devices will actually be used at the same time (writing to both or reading to both simultaneously). If that's a rare occurrence then it probably doesn't matter if together than can max out any of their common upstream branches.

 

[StrongFront.tv]

Thank you for this thread. Question for y'all: Do you think it is ok to max it out at the threshold, or is it better to leave some room? My 2019 Mac Pro is full of PCIE cards as follows:

8-4X - MAC PRO CARD
7-8X - NVME RAID 16TB
6-8X - NVME RAID 16TB
5-16X - AFTERBURNER
3/4-16X - W5700X
1/2-16X - Radeon Pro Vega II

It runs great, Pool A is at 200% and Pool B at 100%. I may consider removing the W5700X to create some headroom. But love having more TB3 Ports. Reality is I don't need 12 ports but like separate buses when speed is required.

 

[JesterJJZ]

Thank you for this thread. Question for y'all: Do you think it is ok to max it out at the threshold, or is it better to leave some room? My 2019 Mac Pro is full of PCIE cards as follows:

8-4X - MAC PRO CARD
7-8X - NVME RAID 16TB
6-8X - NVME RAID 16TB
5-16X - AFTERBURNER
3/4-16X - W5700X
1/2-16X - Radeon Pro Vega II

It runs great, Pool A is at 200% and Pool B at 100%. I may consider removing the W5700X to create some headroom. But love having more TB3 Ports. Reality is I don't need 12 ports but like separate buses when speed is required.

I love having additional SSD storage in my tower. It's the thing that will suck most about moving to MacStudio.
MacStudio is a great machine don't get me wrong, but very much a TrashCan 2.0 spaghetti factory with external peripherals.

 

[handheldgames]

I love having additional SSD storage in my tower. It's the thing that will suck most about moving to MacStudio.
MacStudio is a great machine don't get me wrong, but very much a TrashCan 2.0 spaghetti factory with external peripherals.

Going to external works. It's a mess and legacy performance is a fraction of internal PCIe slots.

A spaghetti factory explains my current M1 Studio Ultra setup. To start off with I have 2 Highpoint 77101-A's from my MacPros shoved into x16 eGPU enclosures but... Unfortunately, TB3/4 is limited to around 3000MB/S R/W so I'm getting a fraction of the speed the cards are capable of. At least the SSD's are accessible and silent, since I was able to adapt AMD Wraith Coolers to replace the small high pitched fans on the highpoints. While internal SSD's on the M4/M5s are hitting 14k MB/S, so far externals top out at 6000MB/S.

I should also mention I also have 2 Thunderbay 4 enclosures to handle legacy HDD's and another eGPU enclosure to house an x8 BlackMagic Studio Quad 4k/60 HDMI capture card. In all, I have 3 x16 GPU enclosures, I picked up off of ebay for around $250 each. The 2 HDD enclosures on 2 shelves were around 300 each. The Studio is on it's side, in a tower configuration so everything can fit. Unfortunately the only TB5 eGPU enclosures is the Razer Core XV2, and it's internally limited to PCI Gen 4.

Far from optimal and != close to desktop performance no matter how one slices this apple.

 

[joevt]

It runs great, Pool A is at 200% and Pool B at 100%. I may consider removing the W5700X to create some headroom. But love having more TB3 Ports. Reality is I don't need 12 ports but like separate buses when speed is required.

The W5700 doesn't use any bandwidth unless a display is connected. A display doesn't use much bandwidth unless playing video or running games or doing 3D.

All the Thunderbolt controllers are in Pool B. That's not a problem unless you are maxing out 4 or 5 of the 6 Thunderbolt busses. A single Thunderbolt port can use ≈80% of a Thunderbolt bus so spreading the Thunderbolt PCIe devices between different Thunderbolt buses is the best way to go. Use the second Thunderbolt port of a bus for DisplayPort.