nMP - USB3 vs Thunderbolt drive speed - your experience?

USB3 V TB external speeds

Hi,

I have a nMacPro and I have all of my LR5 and PS CS6 files on an external CalDigit T3 Thunderbolt drive running spinning HDs. The T3 will also accept SSDs which I will move to when the prices drop.

TB literally moves data faster than a fast external drive can read or write. TB2 is even faster. For LR5 even TB is overkill. If you are editing a lot of video files or doing 4K then you may begin to push the limits of TB with an external as fast as a CalDigit. CalDigit is about to release their T4 which uses TB2 which is even faster ( ludicrous speed ) For LR5 and PS, if you get an external with the massive throughput of a CalDigit T3 TB will be superb. USb 3 is also good.... but about 1/2 as fast in throughput.
Go with TB.

hualon said:

I have read in a few places that USB3 external drives drastically underperform Thunderbolt drives. From your experience is that accurate? Will I have problems working on USB3 mass storage or should I be shopping for a Thunderbolt enclosure?

My nMP arrives in a couple weeks so I haven't tested yet. My plan is to keep my Lightroom library file on the internal storage and the RAW files (2.5TB of them) on a Seagate USB3 external drive that I have. Is this going to be a good idea?

Click to expand...

After I managed to pay for my nMP I just added RAM from a third party and could not justify any more important expenses.
Therefore I simply bought 3 external USB3 enclosures to put the 3 HD previously placed in my former cMP and use them in this way.
I also added a 4 port USB3 hub (with power supply) to get more ports than the 4 USB3 ports of the nMP.
Even those external enclosures were bought only because my different old USB2 enclosures did not recognize HD larger than 1.5 TB (and those HD were 4 TB each), something I did not know in the past. I thought that any USB external enclosure takes any size of HD inside but this isn't so.
Unfortunately this is not always mentioned among the specs of USB3 enclosures although very important!
I cannot therefore speak about any experience with TB enclosures I do not have but am perfectly satisfied with the USB3 connections.

Thanks Weasel... very interesting.

The other drives I have are CalDigit VR2 USB3 externals which I also highly recommend.

hualon said:

I have read in a few places that USB3 external drives drastically underperform Thunderbolt drives. From your experience is that accurate?

Click to expand...

Depends upon storage technology used and whether USB 3.0 is kneecapped by legacy mode.

Single HDD both should do OK.

A USB external drive that supports UAS ( http://www.tomshardware.com/reviews/usb-3-uas-turbo,3215-4.html driver support is more widespread 2 years after this article. ) won't be handicapped with faster HDDs over time (e.g., a hybrid or modest SSD ).

1st generation USB 3.0 (likely lacking UAS support ) versus TB v2 ... might see some difference but that is is as much time difference than some huge gap with single HDD inputs.

My nMP arrives in a couple weeks so I haven't tested yet. My plan is to keep my Lightroom library file on the internal storage and the RAW files (2.5TB of them) on a Seagate USB3 external drive that I have. Is this going to be a good idea?

Click to expand...

Are the sidecar XMP files also out with the RAW files. ? Or "Lightroom library file" implictily including these also?

Generally, yes it should work OK. Lightroom is going to grab catalog/library data more often in total 'bulk' than it will need to grind through all of your RAW files.

If workflow have smaller, focused catalogs then keeping the "current project " (or two) on the internal drive and then migrating them later to the external USB 3.0 drive can also work. But if keeping broad, spanning catalogs the SSD vs. HDD split should work.

I just wanted to open up a thread about this.
It completely depends on your drives..

I've got a Pegasus R6 which runs at 680 mb/s - impossible via usb 3.

But today I got a new Crucial mx100 SSD in an external usb 3 enclosure. It runs at 430 mb/s on my macbook air, very nice.

But on the nMP it only gets up to 350 mb/s and gets capped there.
Is this due to the non-standard usb 3 implementation (dont know how else to call it, but I read that they somehow put a 3rd party chipset on the board since the cpu chipsets didn't support usb 3 yet).
This is quite a bummer, since I would've gone straight away for a thunderbolt enclosure...

chfilm said:

I just wanted to open up a thread about this.
It completely depends on your drives..

I've got a Pegasus R6 which runs at 680 mb/s - impossible via usb 3.

But today I got a new Crucial mx100 SSD in an external usb 3 enclosure. It runs at 430 mb/s on my macbook air, very nice.

But on the nMP it only gets up to 350 mb/s and gets capped there.
Is this due to the non-standard usb 3 implementation (dont know how else to call it, but I read that they somehow put a 3rd party chipset on the board since the cpu chipsets didn't support usb 3 yet).
This is quite a bummer, since I would've gone straight away for a thunderbolt enclosure...

Click to expand...

I think other people on this forum have also reported not getting maximum speed from a USB3 device on the Mac Pro even if the device can deliver the speed on other computers.

They do use a third party chip by necessity, but that's not the problem. In fact, ironically, the manufacturer touts that chip as providing maximum USB3 speed on all four ports simultaneously (which not all USB3 controllers can do).

In order to get that speed, you need to provide enough PCI lines to the chip. Well, here's the layout according to the Anandtech review: the chip set has 40 PCIe 3 lines, but 32 of those are gobbled up by the graphics boards and the remaining 8 are routed to a multiplex chip to service the Thunderbolt controllers.

That leaves 8 PCIe2 lines that the chip set also provides. Four of those go to the internal SSD drive, and out of the other four, one goes to each of the two Ethernet ports, one goes to the wireless, and one goes to the USB3 controller.

So that's the problem, only one PCIe2 line to the USB ports. It's especially annoying because they blew three lines on the networking stuff when one line was probably sufficient for all three; and the second Ethernet port probably goes unused by most people to boot.

Well, for regular old single hard disks it's probably not an issue -- they don't come close to saturating a USB3 line anyway, and Thuderbolt would be overkill for those. However, if you are attaching anything high speed like an SSD or a RAID array, I personally would avoid using USB3 on this machine.

yea, lesson learned ;(

I'll keep saving for the lacie little big disk thunderbolt 2 monster

demars said:

....
In order to get that speed, you need to provide enough PCI lines to the chip. Well, here's the layout according to the Anandtech review: ...
That leaves 8 PCIe2 lines that the chip set also provides. Four of those go to the internal SSD drive, and out of the other four, one goes to each of the two Ethernet ports, one goes to the wireless, and one goes to the USB3 controller.

So that's the problem, only one PCIe2 line to the USB ports. It's especially annoying because they blew three lines on the networking stuff when one line was probably sufficient for all three; and the second Ethernet port probably goes unused by most people to boot.

Click to expand...

The 1x PCIe2 lane isn't the problem because most other discrete USB3.0 suffer from the same limitation. The problem is clear on the diagram from the Anandtech review.


( http://www.anandtech.com/show/7603/mac-pro-review-late-2013/8 )

Given this is correct layout, the 2Gb/s from C600 to CPU package and 2Gb/s from the SSD presents the primary chokepoint issue. The vast majority of the other discrete USB 3.0 controllers would be 1x PCIe v2 also. Same exact cap. But try to run the USB 3.0 at top seed at the same time as top speed PCIe SSD in the same direction and should see the issue.

Merging USB 3.0 into the core chipset doesn't particularly solve the problem much when crank up the internal PCIe SSD usage. Can have higher amounts of oversubscribed bandwidth, but it is still oversubscribed.

hualon said:

...
Is it fair to say though that I will be hating life if I'm attempt to edit 4k in FCPX running off of a spinning-HDD over USB3?

Click to expand...

That has alot less to do with USB 3 (versus say Thunderbolt) and far more to do with 4K data file bulk on a single spinning HDD. Tweaking clips from a single 4K camera with limited storage probably would not be too bad.

I suppose I should just mentally reserve a couple hundred GB on my internal storage and work off of that in FCPX and then move it off to spinning media when I'm done.

Click to expand...

I wouldn't drive the free capacity of the internal SSD down close to zero on a regular basis. If don't give SSD drive controllers room to work then can see more hiccups long term. Until the Mac Pro 3rd party SSD market arrives and demonstrates some longer term viability I wouldn't abuse the internal SSDs. Not saying treat with kid gloves but not drive up 98% capacity usage and bang hard on it either.

If need more "fast" storage there are external routes for that too.

Some are RAW with sidecar but mostly what's actually on there are DNG files. I also generate smart previews in LR5 for whatever that's worth.

Click to expand...

The smart previews make the catalog folder bigger. I was trying to gage how much data was doing to be on SSD. Sidecar or DNG with metadata out of the disk is fine but it is going to be slower to access.

I went with the big-catalog method so I only have three catalogs: Real Estate, Timelapse, and Other. The latter is the largest and most frequently used.

Click to expand...

Over time it would be good to plot how fast those catalog folders are growing so can plan out when would need to transition to a bigger SSD. With a limited set of catalogs over time that will happen.

I suppose I could do the same thing in reverse when I want to work on older files if I feel like I need the speed boost in editing. I could just move the older files onto internal storage, process, and then put back later.

Click to expand...

If working on older files in bulk. If it is just to grab a couple of pictures from spinning storage and duplicate/branch it for some edit/insertion into something else then the temporary move-edit-and-back may not be worth it.

However, in may workflows pictures are heavily worked on for a time and then are hardly ever tweaked/modifed. Static, not often accessed bulk data is quite well handled by HDD storage and USB 3.0 wouldn't make a huge difference.

deconstruct60 said:

Given this is correct layout, the 2Gb/s from C600 to CPU package and 2Gb/s from the SSD presents the primary chokepoint issue. The vast majority of the other discrete USB 3.0 controllers would be 1x PCIe v2 also. Same exact cap. But try to run the USB 3.0 at top seed at the same time as top speed PCIe SSD in the same direction and should see the issue.

Merging USB 3.0 into the core chipset doesn't particularly solve the problem much when crank up the internal PCIe SSD usage. Can have higher amounts of oversubscribed bandwidth, but it is still oversubscribed.

Click to expand...

I agree that the situation is not ideal, but I'm not sure I agree that it is the main problem. The 2GB/s is a maximum bandwidth figure; it's not as if there is 2GB/s continually running to the SSD. During times that the SSD is idle the USB 3 ports are still being limited by having one PCI v2 line. I agree the bandwidth will periodically be hammered by SSD activity but on the average they would sill perform better with more than one line going to that controller.

This is kind of water under the bridge anyway. This is the design of the current generation Mac Pro, for better or worse, and the next generation will almost certainly use the new chip set with integrated USB 3. So in this case, no use complaining that Apple should change the design, we know the design will change.

USB3 vs TB

hualon said:

I have read in a few places that USB3 external drives drastically underperform Thunderbolt drives. From your experience is that accurate? Will I have problems working on USB3 mass storage or should I be shopping for a Thunderbolt enclosure?

My nMP arrives in a couple weeks so I haven't tested yet. My plan is to keep my Lightroom library file on the internal storage and the RAW files (2.5TB of them) on a Seagate USB3 external drive that I have. Is this going to be a good idea?

Click to expand...

TB is pretty much is a winner as far as fast, efficiency data storage.
But TB could not beat USB3.0 's price
If you want FAST, use Thunderbolt, but a single drive there is not much different from USB3.0 vs Thunderbolt

I would look at the sBOX-TJ
http://www.amazon.com/Thunderbolt-JBoD-SATAIII-Enclosure-Windows/dp/B00HSA1AOA/

You can use multiple SSD/SATA 3.5 drives, create RAID, if need

demars said:

...During times that the SSD is idle the USB 3 ports are still being limited by having one PCI v2 line. I agree the bandwidth will periodically be hammered by SSD activity but on the average they would sill perform better with more than one line going to that controller.

Click to expand...

The presumption is that the USB 3.0 controller gets a "wider" bandwidth slice just because it is integrated into the PCH. PCH chipsets have cost and internal bandwidth constraints on them also. The path to a cheaper PCH chispet is just to link the USB 3.0 subsystem on the PCIe controller. No major bandwidth increase at all; just cheaper.

, and the next generation will almost certainly use the new chip set with integrated USB 3. So in this case, no use complaining that Apple should change the design, we know the design will change.

Click to expand...

I would expect an incremental jump more so from using the Intel controller ( since Apple has spent last 3 years of USB 3.0 existence in denial that any others were useful.) The incrementally more mature drivers being the difference. Not as much from PCH USB 3.0 being 'crazy faster' than discrete USB 3.0 offerings.

Throw a couple of USB 2.0 devices onto a USB 3.0 controller and throughput often sags a bit. Throw on top USB 3.0 overhead that's why most discrete USB controllers get by just fine on a 1x PCIe v2 link.

Copying 50GBs worth of video to a Samsung 840Pro via USB 3 yields roughly 350MB/s. That same file and drive over thunderbolt yields only 420MB/s. It may also be limited by which chip set is being used in the external enclosure it seems.

The same setup internally yields 520MB/s (done on my laptop)

Weaselboy said:

Are you using an enclosure that supports UASP on a Mac that also supports it?

Click to expand...

I don't know of any mac that supports UASP mode... My enclosure does support it and my laptop does as well. Disabling it yields around a 20-30MB/s decrease in performance as well as a decrease in 4K (sectors) performance. I am just giving this from a connection type stand point. the USB 3 on my Macbook pro doesn't get anywhere near this performance funny enough.

I figured a Samsung 840Pro would be a good way to test this.. I'll see what I find when doing an Anvil Storage utilities benchmark with Thunderbolt and USB 3

hualon said:

I have read in a few places that USB3 external drives drastically underperform Thunderbolt drives. From your experience is that accurate? Will I have problems working on USB3 mass storage or should I be shopping for a Thunderbolt enclosure?

My nMP arrives in a couple weeks so I haven't tested yet. My plan is to keep my Lightroom library file on the internal storage and the RAW files (2.5TB of them) on a Seagate USB3 external drive that I have. Is this going to be a good idea?

Click to expand...

USB3 is slower than Thunderbolt 1 in theoretical and actual speeds. On the new Mac Pro… performance is going to be worse because the Xeon E5 v2 processors don't actually have native USB3 chipset support, so all those USB ports are on the same PCIe lane.

EDIT: Deconstruct's post does suggest an alternate explanation. Either way hopefully Haswell-E will clean up the architecture...

Weaselboy said:

UASP is supported on Macs starting with the 2012 Ivy Bridge models, that's why I was wondering what you tested on. Sounds like you are on a PC though.

There is a Black Magic speedtest here by SaSaSushi with a Samsung EVO showing a little better (391/414) speeds than you are getting. Interestingly in that post the TB enclosure got slight slower speeds. There is also a report in that same thread from a user getting 430/430 with the same USB setup.

Click to expand...

Guess Mac does support it cool. It could also be the chipset that my thunderbolt/USB 3 enclosure is using. Maybe just not as fast..