MP 7,1 Samsung NVME Extremely Slow Random Read/Write

[betthefarm]

I'm using a Sonnet 4x4 PCIe card, and I've got two brand new drives currently hooked up to it.

The 9100 Pro seems to be operating fine, but the Samsung 990 Pro is posting extremely slow Random Write times. Random Reads are good.

This seems abnormal and not an issue related to running on PCIe 3.0. I've also posted results for the 9100 for comparison.

These are running on the same PCIe card. Ive changed the PCI slots to a different x16 slot and same results. I've run Samsung Magician and the Firmware is up to date.

I'm not sure what could be causing this. Bad drive?

 

[reader50]

Leave the Samsung 990 idle in the background for a few hours while you do other things on the computer.

I'm thinking the overprovisioning (effectively a write cache) is loaded up. The card will do background maintenance while idle, which should restore your write speed. A manual TRIM pass might help. Had you just finished an OS install on the 990?

 

[betthefarm]

No, just installed it brand new and newly formatted. MacOS is on the main Apple SSD. I'll try that and report back.

 

[betthefarm]

I let the Samsung idle overnight, so far no luck. Still getting about 8MB/sec Random 4k Writes at QD1.

Wondering if there is another app to test it with. Could it be a bad result that's unique to Amorphous Disk Mark? Still seems strange though given that the other Samsung 9100 drive is fine.

 

[bzgnyc2]

I let the Samsung idle overnight, so far no luck. Still getting about 8MB/sec Random 4k Writes at QD1.

Wondering if there is another app to test it with. Could it be a bad result that's unique to Amorphous Disk Mark? Still seems strange though given that the other Samsung 9100 drive is fine.

This is a strange result and Amorphous has gotten better numbers from that drive on other systems:

https://forums.macrumors.com/threads/thunderbolt-3-m-2-nvme-ssd-enclosures.2027925/post-32668247

https://forums.macrumors.com/threads/owc-express-1m2-owners-what-ssd-did-you-buy.2444541/?post=33617639#post-33617639

Given that your connection to that NVMe is relatively direct and the configurations used in the above had their drives in an external enclosure (first over TB3 to an Intel machine and second over USB4v1 to a M4 Pro), I would have thought if anything you would have gotten a touch more performance from that drive in your system.

Is it possible that drive doesn't like the Sonnet for some reason? Do you have any way of testing that drive on a single drive extension card or a TB3 enclosure? Have you checked the card's settings in Expansion Slot Utility?

 

[mattspace]

I'm using a Sonnet 4x4 PCIe card, and I've got two brand new drives currently hooked up to it.

The 9100 Pro seems to be operating fine, but the Samsung 990 Pro is posting extremely slow Random Write times. Random Reads are good.

This seems abnormal and not an issue related to running on PCIe 3.0. I've also posted results for the 9100 for comparison.

These are running on the same PCIe card. Ive changed the PCI slots to a different x16 slot and same results. I've run Samsung Magician and the Firmware is up to date.

I'm not sure what could be causing this. Bad drive?



View attachment 2499995View attachment 2499996

What numbers do you get from Blackmagic?

Did you check the drives have the latest firmware (by installing them in a PC) first?

 

[reader50]

The results suggest your stick is at fault. Can you post the raw SMART values for your 990 stick? (you may have been sold a heavily-used one) Maybe pictures of the front & back of the 990 also?

I've read of people buying imitation Samsung sticks from Chinese marketplaces. The fakes often omit DRAM cache and other performance-boosting features. While the box they come in shows a real Samsung stick, the actual stick has a different component layout.

 

[betthefarm]

Is it possible that drive doesn't like the Sonnet for some reason? Do you have any way of testing that drive on a single drive extension card or a TB3 enclosure? Have you checked the card's settings in Expansion Slot Utility?

Maybe. But it's weird that the 9100 works well, and the website advertises results with a Samsung 970 EVO as the benchmark (albeit from a few years ago). Is NVME a universal standardized language? I know the card doesn't need a driver.

I tried removing all PCIe drives in case it's a Pool allocation problem. Tried PCI Expansion Utility to put the card on its own Pool. Also tried a different x16 Slot in case it was a Motherboard issue. No luck.

I also checked to see if it was a Sonoma issue, but I was able to reproduce the problem on Catalina.

What numbers do you get from Blackmagic?

Did you check the drives have the latest firmware (by installing them in a PC) first?

Blackmagic only tests sequential Read/Write. Those numbers are fine. It's the Random writes that are off, which is a much more real-world test.

When I drag a 1TB folder with thousands of small files, after about 10min it says it will take 8 hours to complete. The same folder copies in less than an hour on the Samsung 9100. That seems to backup that there is a real world problem with Random Writes.

The Drive has the latest firmware according to Samsung's website.

The results suggest your stick is at fault. Can you post the raw SMART values for your 990 stick? (you may have been sold a heavily-used one) Maybe pictures of the front & back of the 990 also?

I ordered it directly from Samsung's store on Amazon. I've ordered a second 990 Pro drive just to check. That should hopefully tell me if this one is a lemon.

I downloaded SSD Reporter and the 990 gets 100% health, all readings/indicators at 100. Disk Utility is saying SMART status is "Verified".

When I get the new drive, I can post the results to see if therey're any different. Any ideas of what to check in the meantime?

 

[mattspace]

Blackmagic only tests sequential Read/Write. Those numbers are fine. It's the Random writes that are off, which is a much more real-world test.

My results on a Highpoint non-hardware-raid card, with 2 out of 4 NVME slots filled, on an x8 pci slot.

FWIW.

 

[betthefarm]

My results on a Highpoint non-hardware-raid card, with 2 out of 4 NVME slots filled, on an x8 pci slot.

FWIW.

That's super helpful, thank you. Most likely a bad drive or compatibility issue with the Sonnet.

I've got a 7101a, but work wise I can't break the Raid right now to test out the Samsung stick.

Which Highpoint card are you using?

 

[mattspace]

Which Highpoint card are you using?

Rocket 1104f

 

[reader50]

I ordered it directly from Samsung's store on Amazon. I've ordered a second 990 Pro drive just to check. That should hopefully tell me if this one is a lemon.

I downloaded SSD Reporter and the 990 gets 100% health, all readings/indicators at 100. Disk Utility is saying SMART status is "Verified".

When I get the new drive, I can post the results to see if they're any different. Any ideas of what to check in the meantime?

Ordering via Amazon is good - I've heard too many warnings about buying on Temu.

I'd still eyeball the stick, especially if you got an open-box deal. I've heard of a scam, where people order the same item from both Amazon and a Chinese marketplace. They get the immediate shipping from Amazon Prime. A week or two later, the Chinese part arrives, which they return to Amazon for a full refund. ie - they end up with the cheap China price, but the fast Amazon shipping. And some later customer gets an open-box deal on a possible imitation part.

I've used Samsung SSDs for years now, they're usually my top choice. But I really don't like those low write numbers in your first post. The 990 is supposed to be an above-average SSD in my opinion. Your numbers don't seem to track with benchmarks posted by other owners on newegg, for example.

 

[betthefarm]

I've heard of a scam, where people order the same item from both Amazon and a Chinese marketplace.

The seller is listed as Amazon.com. The drive is currently on sale. Here's the link in case I was had:

https://www.amazon.com/SAMSUNG-Computing-Workstations-MZ-V9P4T0B-AM/dp/B0CHGT1KFJ/ref=sr_1_4?dib=eyJ2IjoiMSJ9.h1VWh7AbYWtFKgCktjwnFzksEXnxQcme6EaQ680mYRL_eN6q9nfvifS28DJSk6ESS5VVq1-jMDXptFoggwk5LE18PcUD70Tg_H4KxlsssDUmPZ4eDoVL-rCwgd1ZMwx62OKWLPGnjF5El2YPbn1mq5fmZjVsSmDMHBmvaHkx5hm_4lJcYny_mlq_nsZHE0CFkVx2oq5KF28ipKk3uUSTakpB8QJstNs4cK0Gl7kXB_SR7t4rX0i66EaIarEmyjzIEL5pXIa5Nhbt6q1GvvPENgfuuhbzedPmdGR2RVkEgLs.4jJas9tr6riFdgRiV_0MJnCt1E2QNIpDtBqZCX4y-6o&dib_tag=se&qid=1744166084&refinements=p_n_feature_three_browse-bin%3A6797522011&rnid=6797515011&s=pc&sr=1-4&th=1

But I really don't like those low write numbers in your first post.

Possibly? Although if it was counterfeit I'm surprised that the 4k QD1 Random Read was so strong. Faster than any other drive I have. That's an impressive result for any drive to achieve.

I did order a second 990 Pro drive to rule out a lemon. The numbers were the same. So either I was scammed twice or there is something else going on with my system.

The next big thing to rule out would be to try a different NVME adapter. I'm thinking compatibility issue.

I should be able to try a different PCIe adapter tomorrow.

 

[mattspace]

The seller is listed as Amazon.com. The drive is currently on sale. Here's the link in case I was had:

Did you intentionally use a Macrumors affiliate link, or is that something the forum is doing now when you post an amazon link?

 

[bzgnyc2]

Possibly? Although if it was counterfeit I'm surprised that the 4k QD1 Random Read was so strong. Faster than any other drive I have. That's an impressive result for any drive to achieve.

When it comes to NVMe SSD, fast reads are easier than fast writes.

I did order a second 990 Pro drive to rule out a lemon. The numbers were the same.

Confirming -- you ordered that second 990 Pro from a different seller?

So either I was scammed twice or there is something else going on with my system.

The next big thing to rule out would be to try a different NVME adapter. I'm thinking compatibility issue.

I should be able to try a different PCIe adapter tomorrow.

It is possible -- I have seen strange slow downs in other situations depending on the combination of host controller/firmware/OS, enclosure controller/firmware, SSD/firmware. Sequential writes at half speed but only in an external enclosure connected to a Mac and then all fixed with a firmware update. Or one enclosure/SSD combination great with a Mac M1 USB4 but then half speed with an Intel Mac/TB3. But same enclosure with a different SSD works fine with the same Intel Mac/TB3. Modern NVMe SSD work great when they do but there doesn't seem much recourse other than randomly swapping out things when they don't.

I would think your configuration is more straightforward than those but maybe that drive doesn't quite love being in the Sonnet 4x4. Is your alternate PCIe adapter a single NVMe slot design?

 

[betthefarm]

Rocket 1104f

Ok, so I ordered this card and installed the drive. I'm still getting the same exact low random write result. I can rule out that it's a controller issue. Highpoint is behaving the same way.

When it comes to NVMe SSD, fast reads are easier than fast writes.



Confirming -- you ordered that second 990 Pro from a different seller?



It is possible -- I have seen strange slow downs in other situations depending on the combination of host controller/firmware/OS, enclosure controller/firmware, SSD/firmware. Sequential writes at half speed but only in an external enclosure connected to a Mac and then all fixed with a firmware update. Or one enclosure/SSD combination great with a Mac M1 USB4 but then half speed with an Intel Mac/TB3. But same enclosure with a different SSD works fine with the same Intel Mac/TB3. Modern NVMe SSD work great when they do but there doesn't seem much recourse other than randomly swapping out things when they don't.

I would think your configuration is more straightforward than those but maybe that drive doesn't quite love being in the Sonnet 4x4. Is your alternate PCIe adapter a single NVMe slot design?

The Second 990 Pro was from my local Best Buy, and both are having the same low reads. So far I have tried them in the following cards: Sonnet 4x4 and the Highpoint 1104. The Highpoint I recently ordered as @mattspace has said he has the same card on my same generation Mac Pro 2019 and gets expected results.

Firmware on both drives is 4B2QJXD7. Wondering what else to try.

 

[mattspace]

Ok, so I ordered this card and installed the drive. I'm still getting the same exact low random write result. I can rule out that it's a controller issue. Highpoint is behaving the same way.

Assuming you have the 1104, no the 1104f (long discontinued), how're the temperatures?

 

[mattspace]

Wondering what else to try.

Boot from a clean OS, running on one of the Samsungs?

 

[betthefarm]

Assuming you have the 1104, no the 1104f (long discontinued), how're the temperatures?

The 1104, yes not the 1104f. According to SMART data drives are running at ~100F, SMART raw data says 313 which is measured in Kelvin.

Boot from a clean OS, running on one of the Samsungs?

I'm running a clean install on the Samsung 9100. I can try it on the 990 Pro and see what happens.

I would think your configuration is more straightforward than those but maybe that drive doesn't quite love being in the Sonnet 4x4. Is your alternate PCIe adapter a single NVMe slot design?

I have a friend with a OWC Adapter that is single NVME. I'm going to borrow and report back. Because both the Highpoint and the Sonnet are 4x4, I could see if it just doesn't like negotiating with other drives for the controller.

 

[mattspace]

The 1104, yes not the 1104f. According to SMART data drives are running at ~100F, SMART raw data says 313 which is measured in Kelvin.

DriveDX shows my 990s at 100F, but I suspect the fan in the 1104f's cooler is primarily directed at the I/O chip on the card. At 100F the 990's only around halfway to their maximum rated temperature.

 

[betthefarm]

Boot from a clean OS, running on one of the Samsungs?

This solved it! When running MacOS off of the 990 Pro, I got the following result:

Which means this may have just been a Amorphous Disk Mark issue all along. (!) Something about its Random Write programming doesn't gel with an external 990 Pro.

While talking to tech support Highpoint recommended I test with ATTO Disk Benchmark. I hadn't done this yet because they dropped Catalina support. Highpoint just sent me a link that allowed me to install it. All tests including Random Writes perform normally on the 990 Pro.

If anyone else finds this be skeptical of Amorphous DiskMark measurements, always check on a separate program.

Thank you @mattspace @bzgnyc2 @reader50 for your help.

 

[reader50]

That is so much better, inline with results others have posted for 990 Pros. I was wondering if Samsung had something to confess about current 990 batches.

Glad to see it was probably just a formatting issue. Or maybe the clean install did a TRIM on all the free space.

I'm guessing the fix worked for both of your 990 Pros?

 

[mattspace]

This solved it! When running MacOS off of the 990 Pro, I got the following result:

View attachment 2502455


Which means this may have just been a Amorphous Disk Mark issue all along. (!) Something about its Random Write programming doesn't gel with an external 990 Pro.

Interesting - I can say that I saw similar (good) performance on both my 990 Pros which are in the Highpoint card, and I only boot from one of them. It could be something abut the particular OS / App install on the 9100. I'd be tempted to bot back to the 9100, and use App Cleaner to remove Amorphous, then reinstall it and see if the results change.

 

[betthefarm]

I'm guessing the fix worked for both of your 990 Pros?

I haven't checked since I'd have to copy over MacOS to test it out. I'm satisfied with ATTO results. I suspect an issue with Amorphous' testing script and Samsung's turbo write, as the performance would decline with each interval measurement.

So first interval would read 37MBs, then 24MBs, and by the 3rd measurement/interval it would settle down to 9-10MBs.

When running the OS off of the Samsung 990, the first interval would be 37MBs, but then jump up to 150+MBs on the following intervals. Testing on the external 990 it would always give low random write results.

With ATTO, both drives test normally, which is honestly enough for me. Kind of wished I had tried that sooner.

Interesting - I can say that I saw similar (good) performance on both my 990 Pros which are in the Highpoint card, and I only boot from one of them. It could be something abut the particular OS / App install on the 9100. I'd be tempted to bot back to the 9100, and use App Cleaner to remove Amorphous, then reinstall it and see if the results change.

So that's the weird part. I got the same low test results from both the 9100 on Catalina and my internal Apple SSD running Sonoma. Same low random writes.

I would have stayed on Sonoma, but the 9100 is super snappy, and Catalina happens to be extremely stable for my work software.

ATTO performs as expected, so the culprit here seems to be Amorphous. In the end just a goose chase but glad this is up for others to stumble upon.

 

[bzgnyc2]

This solved it! When running MacOS off of the 990 Pro, I got the following result:

View attachment 2502455


Which means this may have just been a Amorphous Disk Mark issue all along. (!) Something about its Random Write programming doesn't gel with an external 990 Pro.

While talking to tech support Highpoint recommended I test with ATTO Disk Benchmark. I hadn't done this yet because they dropped Catalina support. Highpoint just sent me a link that allowed me to install it. All tests including Random Writes perform normally on the 990 Pro.

If anyone else finds this be skeptical of Amorphous DiskMark measurements, always check on a separate program.

Thank you @mattspace @bzgnyc2 @reader50 for your help.

Thanks betthefarm for sharing your results as you worked through a solution. I am trying to understand the solution. You suggest that the problem was measurement errors from Amorphous DiskMark but you did eventually get normally results from it on that drive. Then it seems the drive worked better for random writes when you booted off it?

I haven't used ATTO Disk Benchmark for the same reasons you mention even though it is more comprehensive. Amorphous DiskMark has been a goto first pass tool for testing I/O subsystems on the Mac -- especially for typical users -- but of course there's no point if it is unreliable.

Also to confirm all your testing with these drives has been under Catalina?