How to actually get 40 Gbps on an external NVMe via Thunderbolt 3 on M1?

[St0rMl0rD]

Hi,

So I've been planning an upgrade of my current slow external SSD to a Thunderbolt 3 NVMe. I purchased two things:

NVMe Gen4 drive, which has read speeds up to 4700 MB/s:

Sabrent M.2 NVME SSD Gen4 Solid State Drive

The Sabrent 1TB Rocket Q4 NVMe PCIe 4.0 M.2 2280 internal SSD offers all the benefits of flash disk technology with PCIe Gen4.0 x4 interface. Based on the TLC NAND flash memory, the performance speeds can reach up to 4700 MB / s (read) and 1800 MB / s (write) when using a PCIe Gen4 motherboard. W...

www.amazon.de

External Thunderbolt 3 enclosure, which has transfer speeds up to 40 Gb/s:

85.49US $ 43% OFF|ORICO 40Gbps USB4 M.2 SSD Case Compatible with Thunderbolt 3 4 with Cooling Fan M2 NVMe Case Enclosure for MacBook ProSamsung

Smarter Shopping, Better Living! Aliexpress.com

www.aliexpress.com

When I test this combo, Blackmagic Disk Speed Test gives me:
2500 MB/s read, 2400 MB/s write

Which is good for sure, but only about a half of what they advertise for the NVMe drive as well as that enclosure. My Mac is an M1 MacBook Air. So how do I achieve those highest speeds?

 

[joevt]

You don't.

To get 4700 MB/s to or from the NVMe device, you need a M.2 slot that supports PCIe 4.0 link rate (gen 4 in other words). To get that, you need a PC motherboard that support PCIe 4.0, or you need a PCIe 2.0 x16 or a PCIe 3.0 x8 slot with an installed PCIe card containing a PCIe 4.0 switch, such as the HighPoint SSD7505.
https://highpoint-tech.com/USA_new/series-ssd7505-overview.htm
The PCIe switch can convert between the slow & wide PCIe slot and the fast & narrow M.2 slot.

The downstream connection from the discrete Thunderbolt controller used in any Thunderbolt device such as the Orico Thunderbolt 3 NVMe Enclosure, to a PCIe device such as an NVMe device, is limited to PCIe 3.0 x4 (31.5 Gbps; ≈3500 MB/s after PCIe overhead). So you can't get full PCIe 4.0 speed. But you also can't get full PCIe 3.0 speed as described next:

The connection between the downstream Thunderbolt port of the host's Thunderbolt controller and the upstream Thunderbolt port of the Thunderbolt peripheral does support 40 Gbps (5000 MB/s), however that bandwidth is only fully usable when you tunnel a DisplayPort signal to a display connected to the Thunderbolt peripheral or a Thunderbolt peripheral further down the Thunderbolt chain. The remaining bandwidth that is not used for DisplayPort can be used for PCIe tunnelling, but for some unknown reason, PCIe data cannot exceed more than ≈22 Gbps (2750 MB/s, maybe 24 Gbps max 3000 MB/s). So that's why you are limited to 2500 MB/s. You can try AmorphousDiskMark.app to see if you can get closer to 2800 MB/s since that benchmark gives slightly higher numbers than Blackmagic Disk Speed Test.

For the upstream connection to a host computer's CPU, a discrete Thunderbolt controller (as used in Intel Macs that don't use Apple Silicon M1 or Intel Ice Lake or Intel Tiger Lake CPUs), which typically has two downstream Thunderbolt ports, is also limited to PCIe 3.0 x4. However, even if you connect two Thunderbolt NVMe devices, one per downstream Thunderbolt port of the discrete Thunderbolt controller, and try to do RAID 0 (or use the ATTO Disk Benchmark.app which can do multiple disks at once without having to create a RAID 0), you won't get more than ≈23 Gbps total.

An integrated Thunderbolt controller (such as in the Apple Silicon CPU, or Intel Ice Lake or Intel Tiger Lake CPUs) can exceed PCIe 3.0 x4 since they are part of the CPU (not using a real PCIe connection). This is true only when using multiple Thunderbolt ports simultaneously because each port is similarly limited to ≈22-24 Gbps like for discrete Thunderbolt controllers. Ice Lake and Tiger Lake can have four Thunderbolt ports, but the max bandwidth is ≈38 Gbps (tested for Ice Lake using AJA System Test Lite.app with a software RAID 0 - haven't seen Tiger Lake benchmarks).

 

[St0rMl0rD]

Thanks for the reply and for taking the time to write this extensive explanation. I'm sure others will also appreciate it.

 

[joevt]

A couple more things:
1) The 22 Gbps could be as high as 25 Gbps since I've seen benchmarks of slightly over 3000 MB/s for a single device.

2) We know the limit of a discrete Thunderbolt controller is between ≈22 Gbps (mentioned in previous post) and 25 Gbps (mentioned here in #1) even if two ports are used simultaneously (≈23 Gbps mentioned in the previous post for two NVMe) and we know that integrated Thunderbolt controllers don't have that limit for two ports (≈38 Gbps mentioned in the previous post for Ice Lake).
But what about two NVMe connected to the same port of an integrated Thunderbolt controller (either in a chain or with a Thunderbolt 4 hub)? Could that exceed 25 Gbps or 31.5 Gbps and get closer to 40 Gbps? I haven't seen that tested.

 

[sidorvm]

Wow!

You don't.

To get 4700 MB/s to or from the NVMe device, you need a M.2 slot that supports PCIe 4.0 link rate (gen 4 in other words). To get that, you need a PC motherboard that support PCIe 4.0, or you need a PCIe 2.0 x16 or a PCIe 3.0 x8 slot with an installed PCIe card containing a PCIe 4.0 switch, such as the HighPoint SSD7505.
https://highpoint-tech.com/USA_new/series-ssd7505-overview.htm
The PCIe switch can convert between the slow & wide PCIe slot and the fast & narrow M.2 slot.

The downstream connection from the discrete Thunderbolt controller used in any Thunderbolt device such as the Orico Thunderbolt 3 NVMe Enclosure, to a PCIe device such as an NVMe device, is limited to PCIe 3.0 x4 (31.5 Gbps; ≈3500 MB/s after PCIe overhead). So you can't get full PCIe 4.0 speed. But you also can't get full PCIe 3.0 speed as described next:

The connection between the downstream Thunderbolt port of the host's Thunderbolt controller and the upstream Thunderbolt port of the Thunderbolt peripheral does support 40 Gbps (5000 MB/s), however that bandwidth is only fully usable when you tunnel a DisplayPort signal to a display connected to the Thunderbolt peripheral or a Thunderbolt peripheral further down the Thunderbolt chain. The remaining bandwidth that is not used for DisplayPort can be used for PCIe tunnelling, but for some unknown reason, PCIe data cannot exceed more than ≈22 Gbps (2750 MB/s, maybe 24 Gbps max 3000 MB/s). So that's why you are limited to 2500 MB/s. You can try AmorphousDiskMark.app to see if you can get closer to 2800 MB/s since that benchmark gives slightly higher numbers than Blackmagic Disk Speed Test.

For the upstream connection to a host computer's CPU, a discrete Thunderbolt controller (as used in Intel Macs that don't use Apple Silicon M1 or Intel Ice Lake or Intel Tiger Lake CPUs), which typically has two downstream Thunderbolt ports, is also limited to PCIe 3.0 x4. However, even if you connect two Thunderbolt NVMe devices, one per downstream Thunderbolt port of the discrete Thunderbolt controller, and try to do RAID 0 (or use the ATTO Disk Benchmark.app which can do multiple disks at once without having to create a RAID 0), you won't get more than ≈23 Gbps total.

An integrated Thunderbolt controller (such as in the Apple Silicon CPU, or Intel Ice Lake or Intel Tiger Lake CPUs) can exceed PCIe 3.0 x4 since they are part of the CPU (not using a real PCIe connection). This is true only when using multiple Thunderbolt ports simultaneously because each port is similarly limited to ≈22-24 Gbps like for discrete Thunderbolt controllers. Ice Lake and Tiger Lake can have four Thunderbolt ports, but the max bandwidth is ≈38 Gbps (tested for Ice Lake using AJA System Test Lite.app with a software RAID 0 - haven't seen Tiger Lake benchmarks).

Wow! thanks for the details!

 

[Silly John Fatty]

You don't.

To get 4700 MB/s to or from the NVMe device, you need a M.2 slot that supports PCIe 4.0 link rate (gen 4 in other words). To get that, you need a PC motherboard that support PCIe 4.0, or you need a PCIe 2.0 x16 or a PCIe 3.0 x8 slot with an installed PCIe card containing a PCIe 4.0 switch, such as the HighPoint SSD7505.
https://highpoint-tech.com/USA_new/series-ssd7505-overview.htm
The PCIe switch can convert between the slow & wide PCIe slot and the fast & narrow M.2 slot.

The downstream connection from the discrete Thunderbolt controller used in any Thunderbolt device such as the Orico Thunderbolt 3 NVMe Enclosure, to a PCIe device such as an NVMe device, is limited to PCIe 3.0 x4 (31.5 Gbps; ≈3500 MB/s after PCIe overhead). So you can't get full PCIe 4.0 speed. But you also can't get full PCIe 3.0 speed as described next:

The connection between the downstream Thunderbolt port of the host's Thunderbolt controller and the upstream Thunderbolt port of the Thunderbolt peripheral does support 40 Gbps (5000 MB/s), however that bandwidth is only fully usable when you tunnel a DisplayPort signal to a display connected to the Thunderbolt peripheral or a Thunderbolt peripheral further down the Thunderbolt chain. The remaining bandwidth that is not used for DisplayPort can be used for PCIe tunnelling, but for some unknown reason, PCIe data cannot exceed more than ≈22 Gbps (2750 MB/s, maybe 24 Gbps max 3000 MB/s). So that's why you are limited to 2500 MB/s. You can try AmorphousDiskMark.app to see if you can get closer to 2800 MB/s since that benchmark gives slightly higher numbers than Blackmagic Disk Speed Test.

For the upstream connection to a host computer's CPU, a discrete Thunderbolt controller (as used in Intel Macs that don't use Apple Silicon M1 or Intel Ice Lake or Intel Tiger Lake CPUs), which typically has two downstream Thunderbolt ports, is also limited to PCIe 3.0 x4. However, even if you connect two Thunderbolt NVMe devices, one per downstream Thunderbolt port of the discrete Thunderbolt controller, and try to do RAID 0 (or use the ATTO Disk Benchmark.app which can do multiple disks at once without having to create a RAID 0), you won't get more than ≈23 Gbps total.

An integrated Thunderbolt controller (such as in the Apple Silicon CPU, or Intel Ice Lake or Intel Tiger Lake CPUs) can exceed PCIe 3.0 x4 since they are part of the CPU (not using a real PCIe connection). This is true only when using multiple Thunderbolt ports simultaneously because each port is similarly limited to ≈22-24 Gbps like for discrete Thunderbolt controllers. Ice Lake and Tiger Lake can have four Thunderbolt ports, but the max bandwidth is ≈38 Gbps (tested for Ice Lake using AJA System Test Lite.app with a software RAID 0 - haven't seen Tiger Lake benchmarks).

Thanks for taking the time to explain this! I most likely didn’t even understand everything.

Towards the end you’re saying 38 Gb/s are possible. Is that a theoretical thing or is it possible to get that on external storage connected to a new Mac Mini M2 Pro in real life?

 

[joevt]

Thanks for taking the time to explain this! I most likely didn’t even understand everything.

Towards the end you’re saying 38 Gb/s are possible. Is that a theoretical thing or is it possible to get that on external storage connected to a new Mac Mini M2 Pro in real life?

That was with using two Thunderbolt ports with raid 0.

 

[Silly John Fatty]

That was with using two Thunderbolt ports with raid 0.

Finally, that's what I've been looking for the entire time! Does this stress the Mac's hardware in any way?

 

[joevt]

Finally, that's what I've been looking for the entire time! Does this stress the Mac's hardware in any way?

No. The ports were made to connect stuff without blowing up your computer.