Thunderbolt 3 + NVME: m.2 enclosures & pci-e expansion boxes

[aurieg92]

Dear Forum,

I saw this nice thread about NVME enclosures: https://forums.macrumors.com/threads/thunderbolt-3-m-2-nvme-ssd-enclosures.2027925/ It's a bit long and old but has some nice links in it.

A common issue with NVME M.2 enclosures (if I'm reading the thread right) is that the chipset can sometimes bottleneck the disk, or you can run into some compatibility problems.

I'm looking to run NVME off a new 2020 iMac 27 inch and here's my idea: 1 thunderbolt 3 connection is 4 PCI-e lanes, and that's enough for 2 decent NVME M.2 drives (e.g. Sabrent Rocket or 970 EVO Plus). I do understand one M.2 socket is effectively 4 PCI-e lanes, but today's drives don't fully saturate. Also, down the road, NVME density will probably go up, or I might want to switch to some generic PCI-e 3.0 storage (e.g. an Optane card).

Does anybody have experience putting NVME drives in PCI-e expansion boxes like the Sonnet Echo Express line, the OWC Mercury Helios, StarTech & others e-GPU boxes? What about when used in conjunction with M.3 PCI-e cards like the Asus Hyper M.2, the Sonnet Fusion M.2, or even some of cheap $20 single-drive M.2 cards out there (Sabrent has one even)? I understand that any of the x8 or x16 cards won't actually give full speed since Thunerbolt 3 is only supplying 4 lanes, but maybe it'll still work?? An be faster than some of the NVME single-drive enclosures?

Personally I was also thinking that getting a PCI-e breakout box would be a better investment in the long run because it can take more than just NVME stuff. I personally wouldn't ever get a GPU, but hey what if DDR4 prices plummet in the next 2 years and those DDR-PCI-e disk things look good?

Perf goal here, for me at least, is more skewed towards latency & random R/W (databases) versus sequential R/W (video).

 

[joevt]

Thunderbolt 3 is 4 PCIe lanes, but actual PCIe traffic is limited to approximately 22 Gbps (2750 MB/s) but some benchmark have seen up to 2800 or 2900 MB/s (Ice Lake, or Tiger Lake, or Thunderbolt 4?).

Some NVMe drives have a strange slow down in a Thunderbolt 3 enclosure (write is 800 MB/s instead of the usual higher number?).

I have four NVMe drives connected to an Amfeltec Gen 3 four M.2 carrier card inside a Sonnet Echo Express III-D (Thunderbolt 3 Edition). That's one Thunderbolt connection and two PCIe switches between my Mac mini 2018 and the NVMe.

AJA System Test Lite, 5120x2700 5K RED, 4 GB, 16bit RGB (repeat test until numbers stop increasing)

Read/Write (HFS+ partitions)
Samsung 950 Pro 512MB: 2415/993
Samsung 950 Pro 512MB: 2418/991
Samsung 960 Pro 1TB: 2555/803
Samsung 960 Pro 1TB: 2554/809

I recently got a XPG SX8200 Pro 2TB and installed it in a Trebleet M.2 Thunderbolt enclosure.
1962/1400

I have various single slot enclosures (OWC Mercury Helios 3, Sonnet Echo Express SE I (Thunderbolt 3 Edition), Sonnet Echo Express SEL (Thunderbolt 3 Edition) but I think they have similar performance. Here is the XPG in the OWC using a StarTech PCIe to M.2 adapter:
1919/1364

Here is 960 Pro in the Trebleet:
2572/976
so it seems the two PCIe switches in the Sonnet III-D + Amfeltec configuration does drop performance slightly.

950 Pro in the Trebleet:
2414/1170

I'll be getting a OWC Mercury Helios S3 which has a Titan Ridge controller instead of Alpine Ridge.
Also a Sabrent Rocket which is PCIe Gen 4 (but of course it will be running at Gen 3 speed until I get the Highpoint Gen 4 card).

 

[aurieg92]

Wow thanks for these benchmarks!!! Huh those Treebleet results look better than I've seen for some of the other single M.2 enclosures... clearly the chipset makes a big difference. This is tremendously helpful, thank you!

Is there a Mac that supports PCI-e 4 now (for the Highpoint card)? Or was this for some other platform?

 

[joevt]

Wow thanks for these benchmarks!!! Huh those Treebleet results look better than I've seen for some of the other single M.2 enclosures... clearly the chipset makes a big difference. This is tremendously helpful, thank you!

Is there a Mac that supports PCI-e 4 now (for the Highpoint card)? Or was this for some other platform?

The Trebleet is a bus powered single Thunderbolt port Alpine Ridge.
The Helios is an externally powered dual Thunderbolt port Alpine Ridge.
The XPG in the Trebleet was only slightly faster - maybe because of shorter Thunderbolt cable, or fewer adapters, or fewer ports...

The Highpoint card has a gen 4 switch. The upstream will be PCIe 1.0 in a MacPro1,1 or MacPro2,1 and PCIe 2.0 in a MacPro3,1, MacPro4,1, MacPro5,1 and PCIe 3.0 in a MacPro7,1. The downstream M.2 slots will be PCIe 4.0 for gen 4 NVMe drives. PCIe 4.0 x4 is similar to PCIe 2.0 x16 so the HighPoint gen 4 card wouldn't be worthwhile in a PCIe 1.0 slot.

Here are some more numbers for the Helios. Very similar to the Trebleet.
960 Pro in Helios 3:
2545/944

950 Pro Helios 3:
2413/1139

 

[aurieg92]

Ok I looked around a bit more and I think here's what I learned:

In order to use an M.2 PCIe card like the Asus Hyper M.2 (PCIe 3 version https://www.amazon.com/ASUS-M-2-X16-V2-Threadripper/dp/B07NQBQB6Z ), you need a host that supports (at least) PCIe bifurcation. I'm not sure which (if any) eGPU enclosures do this, or if the Sonnet Echo line does (since they have some units that have multiple slots ..?).

That explains why the Sonnet Fusion RAID card ( https://www.amazon.com/SoNNeT-Fusio...raid+card&qid=1604009937&s=electronics&sr=1-2 ) or the HighPoint have particular value and cost more. The Asus card is just a pass-through (which is what I wanted actually, but I don't have a normal ATX mobo here, just a potential breakout box).

The OWC Express 4M2, while cheaper than an expensive RAID card + housing (though perhaps more expensive than housing + Asus Hyper M.2) appears to limit each M.2 drive to 700MB/s no matter what you put in there. So that's great if it were a couple years ago, but with today's prices and drives it appears to have only niche value. It definitely made sense when M.2 drives were only 1TB or smaller.

It's looking like for one Thunderbolt 3 port (which is what I have), the best bang for the buck might be just going with one 2TB drive. For more money, perhaps a 4TB M.2 is actually a better value than a complicated breakout box plus multiple drives. Over Thunderbolt 3, the speed of a single 4TB M.2 is probably comparable to any sort of fancy RAIDed option.

Looking forward, I wonder what the best future-proof approach is. Today there's an 8TB Sabrent Rocket on the market that's solidly more than some sort of RAIDed external solution. But clearly prices should come down. And storage density can still go up?

On the other hand, some sort of eGPU box with a PCIe slot has in theory the most agility.

Again goal here, at least for me, is more slanted towards IOPs and random reads, with sequential R/W and space (at least 2TB+) being a secondary considerations. For pure sequential + space, the OWC Express 4M2 might be the best value, although it seems like for some reason single M.2 enclosures can still achieve better total sequential thruput... because of the RAID software maybe?

In terms of single M.2 enclosure options, it looks like there are no Titan Ridge ones? Just Alpine Ridge, and for some reason the Trebleet ( https://www.amazon.com/gp/product/B07N67P39W/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1 ) appears to benchmark fastest.

 

[joevt]

In order to use an M.2 PCIe card like the Asus Hyper M.2 (PCIe 3 version https://www.amazon.com/ASUS-M-2-X16-V2-Threadripper/dp/B07NQBQB6Z ), you need a host that supports (at least) PCIe bifurcation. I'm not sure which (if any) eGPU enclosures do this, or if the Sonnet Echo line does (since they have some units that have multiple slots ..?).

No eGPU supports bifurcation. Bifurcation support is a property of the motherboard.
Bifurcation divides an x16 slot into x4x4x4x4 but an eGPU slot is only x4 electrically so there's nothing to bifurcate.
Bifurcation does not depend on support for multiple slots like in the Sonnet Echo Express III - it depends circuitry connected to a single slot.

The only thing connected to the slot of an eGPU is the Thunderbolt controller. The Thunderbolt controller has 4 modes:
x1x1x1x1
x1x1x2
x2x2
x4
I suppose the first 3 modes could be used in a bifurcation situation but the M.2 bifurcation cards depend on divisions of x4x4x4x4. Some bifurcation boards do x8x8 for dual slot support but that's not going to work with the Thunderbolt controller that only has 4 lanes to use.

The Sonnet Echo Express SE III, SE IIIe, III-D, III-R uses a PCIe switch and provide x4 and x8 slots so they can't do x4x4x4x4 or x8x8 bifurcation.

In either case, I don't think there's software for the Thunderbolt controller or the PCIe switch that can change the lane configuration.

That explains why the Sonnet Fusion RAID card ( https://www.amazon.com/SoNNeT-Fusio...raid+card&qid=1604009937&s=electronics&sr=1-2 ) or the HighPoint have particular value and cost more.

The explanation for why these cards are expensive is because they have a PCIe switch. It has nothing to do with the host (your not buying a host when you buy one of these cards).

The Asus card is just a pass-through (which is what I wanted actually, but I don't have a normal ATX mobo here, just a potential breakout box).

Yes, it's just a pass-through so it is less expensive (no PCIe switch cost).

The OWC Express 4M2, while cheaper than an expensive RAID card + housing (though perhaps more expensive than housing + Asus Hyper M.2) appears to limit each M.2 drive to 700MB/s no matter what you put in there. So that's great if it were a couple years ago, but with today's prices and drives it appears to have only niche value. It definitely made sense when M.2 drives were only 1TB or smaller.

You are comparing the price of something that works (OWC Express 4M2 $279.75) with something that cannot work (an Thunderbolt 3 enclosure $220 + Asus Hyper M.2 $52) which makes no sense, plus the prices are very similar.

It's looking like for one Thunderbolt 3 port (which is what I have), the best bang for the buck might be just going with one 2TB drive. For more money, perhaps a 4TB M.2 is actually a better value than a complicated breakout box plus multiple drives. Over Thunderbolt 3, the speed of a single 4TB M.2 is probably comparable to any sort of fancy RAIDed option.

The value of OWC Express 4M2 is that it will let you connect four M.2 drives, RAID them together with Software RAID, and get better performance from the Thunderbolt connection than you can usually get with a single M.2 drive. If you want bang for the buck, avoid Thunderbolt 3 and just get a USB-C to NVMe enclosure - the enclosure is a quarter of the price and has almost half the performance (actually, it might have better write performance for some NVMe drives).

On the other hand, some sort of eGPU box with a PCIe slot has in theory the most agility.

eGPU box gets you nothing in terms of M.2 storage. No M.2 device will require 300W of power or litres of space. The PCIe slot is x4 just like any other Thunderbolt enclosure.

Again goal here, at least for me, is more slanted towards IOPs and random reads, with sequential R/W and space (at least 2TB+) being a secondary considerations. For pure sequential + space, the OWC Express 4M2 might be the best value, although it seems like for some reason single M.2 enclosures can still achieve better total sequential thruput... because of the RAID software maybe?

I'm not sure if Software RAID is good for IOPs or random reads. I think it should improve sequential writes, but I have not seen benchmarks for the OWC Express 4M2.

In terms of single M.2 enclosure options, it looks like there are no Titan Ridge ones? Just Alpine Ridge,

Not that I've seen. Alpine Ridge is probably less expensive. Titan Ridge gives DisplayPort 1.4 support, but you don't need that for a drive enclosure. The Mercury Helios S3 has Titan Ridge. I'll try that later.

and for some reason the Trebleet ( https://www.amazon.com/gp/product/B07N67P39W/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1 ) appears to benchmark fastest.

Only slightly faster - maybe just because it was tested with a shorter cable and without PCIe to M.2 adapters and only has one Thunderbolt port and no other ports.

 

[aurieg92]

Thanks @joevt for your very technical response, this really helps clear up my misconceptions! I think my major takeaway is that my use case that slants towards IOPS is pretty niche, so trying to squeeze it through a Thunderbolt 3 connection is just (today) a dumb idea.

One of my key use cases is docker, and on linux docker will use the same virtual memory system as the host... but on Mac I learned (even with hyperkit) the VM virtual memory is distinct. So on linux, you can take an NVME drive and basically use it as slow ram (if your NVME has good IOPS, and Octane ROCKS here), but Mac+docker this is not going to be easy at all. So my quest to have "N terabytes of RAM" in dockerized Mac isn't going to work. (It should work in theory! Because Thunderbolt 3 is just the PCIe bus, and latency appears in benchmarks to be comparable to any slotted NVME device ...)... Lesson learned: 3990X + optane PCIe 4 is the cheapest route to terabytes of "ram".

eGPU box gets you nothing in terms of M.2 storage. No M.2 device will require 300W of power or litres of space. The PCIe slot is x4 just like any other Thunderbolt enclosure.

yes totally true. the main thing i like about eGPU enclosures I've seen is that they have max agility through (1) lots of physical space and (2) a PCIe slot. I'm pretty optimistic about the hardware leaps we might see in the next 5 years, so who knows.

... there's also sentimental value for me.. once upon a time I hand-built a hard drive enclosure for a mac mini (with a Western Digital Raptor X (with exposed window through the front of the enclosure!)), so I have a soft spot for breakout boxes

https://imgur.com/a/ftay6R2

(for the curious, that's a SATA cable drilled out of the mini and into the breakout box [eSata card forwarded to drive], which had a Raptor X via SATA and additionally two SATA drives by firewire 800 card. And a pair of 92mm fans and even an ATX power supply for the cards and drives. Chassis was some very random thing of metal found in east California).

I'm not sure if Software RAID is good for IOPs or random reads. I think it should improve sequential writes, but I have seen benchmarks for the OWC Express 4M2.

for spinning disks, I'm pretty sure RAID actually can help in terms of aggregate (and amortized ..) IOPS because of command queuing. not sure about NVME, but maybe you only get a boost with fancy hardware raid like the SONNET or HighPoint cards.

Not that I've seen. Alpine Ridge cheap is probably cheaper. Titan Ridge gives DisplayPort 1.4 support, but you don't need that for a drive enclosure. The Mercury Helios S3 has Titan Ridge. I'll try that later.

right right right my bad, Titan Ridge value is mainly just DisplayPort 1.4. i guess i figured Titan Ridge might also be a bit more compatible (e.g. some Alpine Ridge enclosures have Evo 970 Plus problems, but that could be just Mac stuff and not Alpine Ridge). Thanks for calling out the OWC Mercury Helios S3-- it's really hard to track down chipset details.

[With respect to Trebleet] Only slightly faster - maybe just because it was tested with a shorter cable and without PCIe to M.2 adapters and only has one Thunderbolt port and no other ports.

oh thanks for calling this out! I totally missed the cable issue.




In summary: I'm a 2020 iMac (Core i9) buyer who did not opt for the Apple 2TB+ drive because the >$600 upgrade price looked crazy, yet I still wanted a lot of flash. What I learned is that you can add 2TB of decent NVME, but the enclosure story is complicated. If you want IOPS, a good enclosure will cost over $100 and you're best just getting that plus one drive. If you want _space_, then OWC 4M2 is probably the cheapest play, but to beat Apple today you likely have to be wanting 4TB or more of additional storage.

If you want more RAM for your 2020 iMac, the story is more straightforward: buy anything BUT apple RAM. (Unless you're an AppleCare fan, perhaps ...)...

 

[Kcetech1]

Is there a Mac that supports PCI-e 4 now (for the Highpoint card)? Or was this for some other platform?

No there isnt. 11th gen Intel chips might support it finally, and it may take longer to come into the new xeon platform.

But at this time only AMD Threadripper, Epyc and I believe newer Ryzen chips support PCIE v4. I have it on both a threadripper and Epyc setup and read of around 4760 and write of 4325 is about my normal in a benchmark. Real world .... its pretty noticeable over stock Mac Pro 8TB SSD cards.

Others are bumping near the 5000. ( and they do RAID 0 nicely on some motherboards as well.)


Id love to play with one of those 5000MB/s Sabrent units.

 

[aurieg92]

just got a sabrent rocket 4 and Q4. there was actually a deal recently on amazon where rocket Q4 was cheaper than evo 970 plus (!!). i don’t have any PCIe 4 boxes to bench tho was just trying to be future proof, and also going for the ”cheap” IOPs

 

[joevt]

right right right my bad, Titan Ridge value is mainly just DisplayPort 1.4. i guess i figured Titan Ridge might also be a bit more compatible (e.g. some Alpine Ridge enclosures have Evo 970 Plus problems, but that could be just Mac stuff and not Alpine Ridge). Thanks for calling out the OWC Mercury Helios S3-- it's really hard to track down chipset details.

I've received the OWC Helios S3 but my Sabrent Rocket is lost somewhere. I'll test it on the weekend.

oh thanks for calling this out! I totally missed the cable issue.

I don't know if it's a cable issue. That seems unlikely. I will redo the tests by swapping the cables used by the old Helios and the Trebleet. If it's not the cable then it's one of the other things I mentioned, or the different chip, or the different firmware, or heck, maybe a different port of my MacMini 2018.

 

[aurieg92]

iMac (Retina 5K, 27-inch, 2020), OSX 10.15.7


Alright so I've done a bunch more research, and some testing, and here's some findings that I hope might be useful to others looking around here. Given that NVME will probably be the leading form factor going forward, and given where thunderbolt 4 is going ( https://9to5mac.com/2020/07/08/thunderbolt-4/ ) (especially versus PCIe 4), I think there's a compelling case to start preparing for all-flash non-SATA arrays.

Also, the Samsung 970 Evo Plus just went on holiday discount this week, so that changes the playing field quite a bit. It's now at $125-$150 per TB: https://www.bhphotovideo.com/c/product/1454172-REG/samsung_mz_v7s2t0b_am_2tb_970_evo_plus.html

I bet the 870 Evo Plus will remain the PCIe 3.0 leader indefinitely because of how much DRAM they embed. (And also they're Samsung, they know something about DRAM). Who knows what happens with PCIe 4.0, but (1) the EVO PLus is probably the 3.0 king and (2) EVO Plus prices won't fall too much more (see below where SATA prices are).

(Note: no referral links below. I don't have any connections).

tl;dr:
* Don't buy M.2 SATA; stick to only 2.5 inch SATA
* The OWC 4M2 is the best value for a Thunderbolt / NVME solution today. For best NVME RAID, you need a real computer (real PCIe x16 slot); Software RAID over Thunderbolt works ONLY for sequential I/O.


M.2 SATA vs NVME vs 2.5 inch SSDs: perf and $ per TB

First of all, NVME prices have fallen, and there are now NVME drives that carry a very slight premium over SATA (both M.2 and 2.5 inch). Going onto https://pcpartpicker.com/ and sorting drives by $ per GB, here are some relevant drives:

M.2 SATA. These max out at 2TB (WD Blue) and are probably dead going forward:
* ~$75/TB TEAMGROUP drives like the MS30 ( https://www.newegg.com/team-group-ms30-1tb/p/N82E16820331233?Item=N82E16820331233 ). Some of these have TLC flash and decent warranties. Some also benchmark decently for sustained read / write.
* ~$100/TB Western Digital Blue (e.g. https://www.amazon.com/dp/B073SBW3VD?tag=pcpapi-20&linkCode=ogi&th=1 ). These benchmark a bit better than the TEAMGROUP stuff and have been around for years.
* ~$100/TB Crucial MX500 (e.g. https://www.amazon.com/dp/B0784SY515?tag=pcpapi-20&linkCode=ogi&th=1&psc=1 ). Comparable to WD Blue but benchmarks a little better.

2.5 inch SATA. These are mostly dead but Samsung might still keep going higher:
* $110/ TB Samsung 870 QVO (e.g. https://www.amazon.com/dp/B089C3TZL9/ref=twister_B08C4K4SMW?_encoding=UTF8&psc=1 ). Benchmarks here are interesting.

Here are some useful benchmarks / reviews:
* The WD Blue is clearly old stuff. The Crucial MX500 does a bit better, but everything in this generation is "about the same," especially given today's better-performing drives: https://www.tomshardware.com/reviews/western-digital-blue-ssd-review,4767-2.html
* There is a bit of a spread across stuff newer than the WD Blue: https://www.anandtech.com/show/10741/the-western-digital-blue-1tb-ssd-review/3
* Great review of the brand new QV0. You do pay a performance penalty for the higher density, though relevant to NVME (as we'll see later) the perf hit may or may not be relevant to you. https://www.anandtech.com/show/15887/the-samsung-870-qvo-1tb-4tb-ssd-review-qlc-refreshed/3

For very useful context, the 970 EVO Plus (which is NVME) is a lot faster than any of these. Destroyer review shows https://www.anandtech.com/show/13761/the-samsung-970-evo-plus-ssd-review/3 :
* The 970 Evo Plus is about 3x faster overall than anything M.2 SATA
* Mean Latencies in the Destroyer test for M.2 SATA are about 1 millisecond. Mean Latency for the 970 Evo Plus is 140 MICROseconds -- the 970 Evo Plus is nearly 10x better!

Ok but what about entry-level NVME like the Crucial P1 or P2?
* It's about $100/TB, so about the same price as M.2 SATA https://www.amazon.com/Crucial-500GB-NAND-NVMe-2400MB/dp/B086BGWNY8
* You get comparable average performance to Crucial M.2 SATA, but you get the better PCIe sequential read speeds that you can't get thru SATA: https://www.tomshardware.com/review...329.433614884.1605644125-567693706.1605315663

Key Takeaways:
* NVMe is now about as cheap as M.2 SATA. But NVMe gives you better perf: at the very least you'll get better sequential thruput via PCIe just having more bandwidth than SATA III.
* M.2 SATA is probably not getting more dense. The 2.5 inch form factor may continue to get more density. There is simply more SATA hardware out there versus M.2 B / B+M key hardware.


Multiple M.2 SATA Drives in External USB / Thunderbolt

Couple interesting tools here:
* QNAP 2x M.2 SATA to 2.5 inch SATA RAID housing: https://www.amazon.com/QNAP-Dual-SATA-Adapter-Converter/dp/B07RLKVN9N NOTE: THESE WILL ONLY TAKE B or B+M key drives!!! So NO NVME!! (I had an oopsie here)
* QNAP 2x 2.5 inch RAID to 1x 3.5 inch: https://www.amazon.com/QNAP-Dual-SATA-Adapter-Converter/dp/B07RLM8W68

Using those above tools, you could get four M.2 SATA drives in a single 2.5 inch slot. But will that really get you max density? NO! As we saw above, M.2 NVME is almost strictly better than M.2 SATA. So today, we have 2TB M.2 SATA drives available (WD Blue), but all the density improvements are going to M.2 NVME. I personally think it's unlikely we'll see 8TB for M.2 SATA (or at least not much competition there).

Moreover, the Samsung high density stuff (e.g. 870 EVO) is not much more than the cost of two M.2 SATA drives plus adapter.

So for the greatest flash density, get a normal 3.5 inch or 2.5 inch enclosure, and potentially use the QNAP 2x 2.5 inch RAID adapters if you have a 3.5 inch enclosure and you want agility. With the QNAP device, you can now get 16TB of flash in a single 3.5 inch slot.

In my own case, I went with the OWC 4-bay USB enclosure: https://www.amazon.com/OWC-Mercury-...keywords=elite+pro+quad&qid=1605721288&sr=8-2

I liked this one because:
* I have some old 3.5 spinning disks I need to use and I didn't need hardware RAID.
* Yottamaster and some others are a little cheaper but the other ones appear to not offer full USB 10g speed.
* 2 slot enclosures are roughly $100, so this 4-bay doesn't have a huge premium.
* I wanted both USB A and C for linux interop.



Multiple M.2 NVME Drives in External Thunderbolt 3

Returning to the original focus of this thread: can external Thunderbolt enclosures give us high-IOPs storage? Most SATA SSDs max out at about 100k IOPS or so. NVME PCIe 3.0 drives in contrast can do 500k, and 4.0 drives can do 750k-1m IOPS.

(What about high-thruput storage? Well 2+ NVME drives can definitely saturate a Thunderbolt 3 line. To get there with SATA, you'd likely need both at least 6 SATA-III drives as well as hardware RAID to make it worth it. Today, NVME is about the same price-per-TB though).


I ended up trying a QNAP 4xNVME card plus a HighPoint Thunderbolt 3 enclosure:
* https://www.qnap.com/en-us/product/qm2-m.2ssd
* https://www.bhphotovideo.com/c/prod...t_rs6661a_rocketstor_6661a_thunderbolt_3.html

Why these parts?
* The HighPoint appears to be the cheapest Thunderbolt 3 enclosure that will take a full-length PCI-e card. I also trusted HighPoint more than some of the other off-name brands.
* Sonnet and OWC have good eGPU enclosures that would also work, but they're more expensive. (Support for nVIDIA on Mac also looks lacking, and the apps I need don't use OpenCL/Radeon, so I gave up on the "maybe the enclosure might also be useful for a GPU one day" idea).
* The QNAP card is a lot lot cheaper than NVME RAID cards, and there's a 4x version that would match the HighPoint perfectly.

I looked on Ali Express and it looks like:
* Almost all Thunderbolt 3 hardware is at least $100. So the HighPoint and QNAP parts aren't ripoffs.
* There are indeed dual NVME Thunderbolt enclosures out there! But they're $250+ and who knows about the manufacturers: https://www.aliexpress.com/item/100...earchweb0_0,searchweb201602_,searchweb201603_


My test setup:
* HighPoint + QNAP + dual Samsung Evo 970 Plus
* Also shown: Sabrent Rocket 4 in a single-nvme TB 3 enclosure.
* New iMac 27 inch with Core i9

My Findings:
* Sequential I/O looks good! Two NVME can definitely saturate the Thunderbolt 3 line.
* RAID 0 random read sucks!!!!
* Random reads concurrently from non-RAIDed drives sucks!!!

Here are some benches:

https://imgur.com/a/i4zIzwQ

Key findings:
* Apple Software RAID 0 DOES yield Thunderbolt-saturating sequential I/O.
* Concurrent I/O over Thunderbolt SUCKs! Both in RAID 0 and when using the drives independently (but concurrently). You lose 10x of your NVME drive perf!! Bad investment!

Other Key findings:
* Apple Random 4k Queue depth 1 is crazy high. They must either pack the hardware with DRAM, or there's caching that the benchmark can't turn off, or both, or ???
* Single EVO 970 Plus is competitive with Apple NVME. Surprisingly, the 970 Evo has much better Random 4k Queue depth 64 performance. Wonder if the Apple wins at QD1 because the 970 EVO is thru a PCIe switch.
* Single EVO 970 Plus over Thunderbolt is faster than single Sabrent Rocket 4. So the premium for PCE 4 in this setting doesn't appear worth it.

Buyer beware:
* The fan on the QNAP card is HELLA LOUD. After 2 minutes of benchmarks, it went full speed.
* The Thunderbolt 3 cable for the Highpoint is extremely short.

To conclude, I decided to go with the OWC 4M2. It's $100 cheaper than the HighPoint+QNAP combo and provides most of the same agility.

External enclosure + NVME? Only if NVME hardware RAID cards become cheap.

 

[aurieg92]

iMac (Retina 5K, 27-inch, 2020), OSX 10.15.7

Got the OWC 4m2 and while it's quieter than the QNAP under full load, the performance is actually a bit behind the QNAP+Highpoint setup.

Benchmarks:

https://imgur.com/a/xItGL4o

tl;dr:
* The OWC 4M2 sequential R/W is well behind the QNAP using Apple Raid 0 for two drives; the OWC can't saturate the Thunderbolt 3 line with just two drives.
* The OWC 4M2 sees Random 4k QD64 performance *half* versus the QNAP for two drives NOT in RAID with concurrent load.
* The OWC 4M2 does recover that Random 4k QD64 performance a bit when using Apple RAID 0
* OWC 4M2 random read 4k QD1 perf is slightly less.

So if you want max performance for the $$ with software RAID, the QNAP+HighPoint solution is perhaps the best option. While the QNAP fan was too loud for me, it can be unplugged, and the QNAP heatsink is both giant and includes thermal pads. Also the HighPoint has built-in fans... I didn't try doing a thermal test for this approach.

What's interesting here is the OWC 4M2 is essentially the same overall architecture: it's clearly a Thunderbolt 3 card plugged into a PCI-e bridge plugged into a 4xM2 PCI-e card. All these parts are clearly bespoke for OWC and since it's all only PCI-e 4-lane, the form factor can be really small. But the OWC very clearly limits each M.2 drive to 1 PCIe lane, where as the QNAP appears to be a full 4-lane switch.

I did notice the OWC fan started to give some really bad noises, but I shook the unit a little and it went away.

I don't think I'm going to have the time & money to try out a hardware RAID solution (like the $550 HighPoint unit https://www.amazon.com/Highpoint-RocketStor-6661A-NVMe-Thunderbolt-Adapter/dp/B07CTH5PND ) but I imagine that to get the full IOPS of all four 970 EVO Pluses you'd need hardware RAID.

For software RAID 0, it seems there still needs to be some work done for NVME.

For best IOPS plus TB/$, the 4TB Sabrent Rocket Q w/ enclosure is right now not a lot more than a pair of 970 EVO pluses + OWC M2. The 8TB Rocket Q carries a bit more of a premium. Both these drives see a hit in IOPs though for the size.

 

[joevt]

Got the OWC 4m2 and while it's quieter than the QNAP under full load, the performance is actually a bit behind the QNAP+Highpoint setup.

Benchmarks:

https://imgur.com/a/xItGL4o

tl;dr:
* The OWC 4M2 sequential R/W is well behind the QNAP using Apple Raid 0 for two drives; the OWC can't saturate the Thunderbolt 3 line with just two drives.

Each drive can only do 800 MB/s because of PCIe 3.0 x1 in the OWC 4M2. You need to raid all 4 drives to saturate Thunderbolt 3.

* The OWC 4M2 sees Random 4k QD64 performance *half* versus the QNAP for two drives NOT in RAID with concurrent load.
* The OWC 4M2 does recover that Random 4k QD64 performance a bit when using Apple RAID 0
* OWC 4M2 random read 4k QD1 perf is slightly less.

So if you want max performance for the $$ with software RAID, the QNAP+HighPoint solution is perhaps the best option. While the QNAP fan was too loud for me, it can be unplugged, and the QNAP heatsink is both giant and includes thermal pads. Also the HighPoint has built-in fans... I didn't try doing a thermal test for this approach.

But you didn't try four drives in RAID 0 which is what you need to get max performance out of the 4M2.

What's interesting here is the OWC 4M2 is essentially the same overall architecture: it's clearly a Thunderbolt 3 card plugged into a PCI-e bridge plugged into a 4xM2 PCI-e card. All these parts are clearly bespoke for OWC and since it's all only PCI-e 4-lane, the form factor can be really small. But the OWC very clearly limits each M.2 drive to 1 PCIe lane, where as the QNAP appears to be a full 4-lane switch.

The PCIe bridge in the 4M2 is the Thunderbolt controller itself (one lane per NVMe). An external PCIe bridge (four lanes per NVMe) would add an extra level to the PCIe bus and extra cost, but would give better single NVMe performance.

 

[aurieg92]

@joevt Have you seen any benchmarks with random read results for the OWC 4M2 with four drives? I’m concerned even with four that the random read perf would be worse than that of a single drive. I have only seen Black Magic & sequential benchmarks so far ...

 

[joevt]

@joevt Have you seen any benchmarks with random read results for the OWC 4M2 with four drives? I’m concerned even with four that the random read perf would be worse than that of a single drive. I have only seen Black Magic & sequential benchmarks so far ...

I usually don't do random read benchmarks since they are usually miserable.

 

[lowendlinux]

Were these benches done in Big Sur or Catalina..

I'm not a big storage benchmark guy but reading this thread made me re-bench my Envoy Pro EX and my Server with spinning rust my Envoy now benches faster than it did in the reference thread and while my Server is faster it's within the margin of error.

 

[gwerhart0800]

Got the OWC 4m2 and while it's quieter than the QNAP under full load, the performance is actually a bit behind the QNAP+Highpoint setup.

Benchmarks:

https://imgur.com/a/xItGL4o

This is what I get from an 4m2 with 4 Inland 2T "premium" NVME SSDs installed running the full version of SoftRAID. Micro Center quotes these Inland drives (house brand) as north of 3000MB/s. The higher scores are raid 0 across all drives and the lower ones are raid 5.

 

[lowendlinux]

Here's my envoy

 

[robsojon]

What a fantastic thread, thanks for the hardwork! I've been debating between ordering a Mac mini m1 with 512gb internal plus an external nvme solution for fast access to my Lightroom library, versus ordering one with 2tb internal (enough for my needs). I think from reading this thread, to get similar performance in an external solution, I'm looking at the same or higher cost than the apple internal memory. Admittedly it would likely outlive the Mac.

This thread certainly gave me a lot to think about, think ultimately it has helped me decide to order the 2tb m1 Mac mini.

 

[aurieg92]

Were these benches done in Big Sur or Catalina..

iMac (Retina 5K, 27-inch, 2020), OSX 10.15.7

I updated the posts to include this info. Thanks for the call-out!

 

[aurieg92]

This is what I get from an 4m2 with 4 Inland 2T "premium" NVME SSDs installed running the full version of SoftRAID. Micro Center quotes these Inland drives (house brand) as north of 3000MB/s. The higher scores are raid 0 across all drives and the lower ones are raid 5.

@gwerhart0800 oh wow thank you for posting this! This is very interesting because your posted random read (QD64) is ~8x faster, but random writes is worse than 1/10th the speed versus my 2x 970 Evo Plus numbers. That is really hard to digest because the 970 Evo Plus should be great here, even with only 2x instead of 4x. I will need to try SoftRAID instead of Apple's software raid to see if things improve.

I get that sequential read should be capped in the 4m2 because of thunderbolt, but I'm really trying to see if the random read can get better. My app is very database-like (not really video-like) and I can't understand why 4m2 kills random perf so bad even with just a single drive.

 

[aurieg92]

What a fantastic thread, thanks for the hardwork! I've been debating between ordering a Mac mini m1 with 512gb internal plus an external nvme solution for fast access to my Lightroom library, versus ordering one with 2tb internal (enough for my needs). I think from reading this thread, to get similar performance in an external solution, I'm looking at the same or higher cost than the apple internal memory. Admittedly it would likely outlive the Mac.

This thread certainly gave me a lot to think about, think ultimately it has helped me decide to order the 2tb m1 Mac mini.

I was on the fence for going 2TB in the iMac for the +$600 fee. I went with 1TB because I was certain I could beat Apple. I think you made the right call, and if I could return the iMac today I think I'd go ahead and do 2TB Apple NVME. While the upcharge is overpriced, their SSD actually has strangely super-high random read perf (key for a start-up disk), and as I've learned here the Thunderbolt solutions are not no-brainers.

The other added benefit of going 2TB in an M1 mac is that they limit you to 16GB of RAM, so if you have a bigger boot disk then you will have more swap available. Right now I have Chrome using 50GB of RAM (lots of tabs) so that extra NVME helps. On my linux box, using NVME for 100GB+ of "RAM" (swap) has been a lifesaver.

 

[aurieg92]

Alright, well the OWC 4M.2 was making bad poppy noises (like exploding transformer), so I decided to try the Orico dual m.2 USB-C enclosure:

ORICO Dual-Bay NVME Enclosure External Hard Drive Enclosure Aluminum USB3.1 Gen2 10Gbps M.2 Type-C SSD Docking Station for PCIe M-Key B&M Key 2230/2242/2260/2280 SSD Up to 4TB-M2P2J-Update

【Good Heat Dissipation】By adapting aluminum alloy striped rib shell to expand the heat dissipation area and increase the heat dissipation air duct can effectively reduce the damage casued by heat generated by the high-speed work of the SSD. (Note: SSD will generate a lot of heat during high-speed...

www.amazon.com

Here are the benchmarks with dual 970 2TB EVO Pluses:

https://imgur.com/a/3iS1XbC

Sadly it's essentially all-around slower than the OWC 4M.2 on random read. I knew sequential would get capped, but it looks as if the USB-C bus just runs at a slower clock rate than Thunderbolt 3? Because the random 4k QD1 is really really reduced.

Note that this Orico unit is only USB 3.1 gen 2, not USB 3.2, so perhaps that's why it can do about max sequential thruput but random read is bad. Doh!

 

[joevt]

Note that this Orico unit is only USB 3.1 gen 2, not USB 3.2, so perhaps that's why it can do about max sequential thruput but random read is bad. Doh!

No difference between USB 3.2 and USB 3.1, unless you are talking about gen 2x2 speed - I haven't seen anyone try a 20 Gbps USB device in macOS.

 

[rynow204]

Hey Just wondering if anyone has recently put together a Trebleet 2 bay tb3 enclosure ?
I'm considering them as they seem to be the only dual bay nvme enclosure out there.