Thunderbolt 2.0: capable of PCIE 3.0 16x cards in external enclosures?

darkgoob · Jun 10, 2013

Is Thunderbolt 2.0 capable of PCIE 3.0 16x external enclosures? I mean can you have an external bay with one PCIE 3.0 16x graphics card in it? What bout two?

I reckon that the GB/Sec. of Thunderbolt 2.0 certainly indicates it's capable of this, and it's touted as PCIE 3.0 tech. We've already seen there are currently PCIE 2.0 16x capable enclosures on Thunderbolt 1.0, although the performance is maybe 10% less than a real slot (I think?) due to extra lag from the Thunderbolt controllers and signal propagating down a 3-foot cable etc. I kind of doubt Apple would introduce a Mac Pro that CAN'T have PCIE 2.0 16x slots added. Am I wrong?

Not that you'll likely need extra graphics card power beyond what appears to be DUAL AMD FirePro W9000's, which currently run about $3000 APIECE... but hey. Just curious!

lewdvig · Jun 10, 2013

There will be all kinds of crazy carriages spawned from this.

IE: The current MP is meagre for storage if you do 4k video. with 6x Thunderbolt 2 you can have lots of fast storage. Or maybe you want an array of 100,000 CUDA cores for rendering.

People complaining about the new mac have no imagination. I doubt they represent the true 'pro' this machine is targeted at.

calaverasgrande · Jun 10, 2013

more importantly does it come in black and will it reflect the new design language of the Mac Pro?

blackhand1001 · Jun 10, 2013

darkgoob said:
Is Thunderbolt 2.0 capable of PCIE 3.0 16x external enclosures? I mean can you have an external bay with one PCIE 3.0 16x graphics card in it? What bout two?

I reckon that the GB/Sec. of Thunderbolt 2.0 certainly indicates it's capable of this, and it's touted as PCIE 3.0 tech. We've already seen there are currently PCIE 2.0 16x capable enclosures on Thunderbolt 1.0, although the performance is maybe 10% less than a real slot (I think?) due to extra lag from the Thunderbolt controllers and signal propagating down a 3-foot cable etc. I kind of doubt Apple would introduce a Mac Pro that CAN'T have PCIE 2.0 16x slots added. Am I wrong?

Not that you'll likely need extra graphics card power beyond what appears to be DUAL AMD FirePro W9000's, which currently run about $3000 APIECE... but hey. Just curious!

Thunderbolt 2 has no where near the bandwidth of a true pci 3.0 16x slot.
Thunderbolt 2 maxes out 20 gigabits per second. PCI express 3.0 has 16 gigabytes per second aka 128 gigabits per second.

darkgoob · Jun 10, 2013

blackhand1001 said:
Thunderbolt 2 has no where near the bandwidth of a true pci 3.0 16x slot.
Thunderbolt 2 maxes out 20 gigabits per second. PCI express 3.0 has 16 gigabytes per second aka 128 gigabits per second.

Ouch. So, I guess that means you can't use something like this:
http://www.netstor.com.tw/_03/03_02.php?MTEx# ...?

Or is there an actual PCIE 3.0 slot hiding in that new Mac Pro somewhere? I guess it just does not seem like Thunderbolt alone will cut it. But then, perhaps this Mac Pro design may see some revisions before it's officially sold, like maybe a way to give full PCIE 3.0 16x speeds to external devices?

I don't need it, but I am sure many will.

Tanax · Jun 11, 2013

blackhand1001 said:
Thunderbolt 2 has no where near the bandwidth of a true pci 3.0 16x slot.
Thunderbolt 2 maxes out 20 gigabits per second. PCI express 3.0 has 16 gigabytes per second aka 128 gigabits per second.

Yes, those numbers are indeed very far apart - however does a GPU like Titan even use 128 Gbps?

Inconsequential · Jun 11, 2013

Yes.

Proved many times that new gen cards are starting to hit very high transfer rates and are hitting PCI-E 2 limits let along TB2 limits!!!

d-m-a-x · Jun 11, 2013

so where does the red rocket pci 2.0 fit in? it needs 16x bandwith, and do they make a chassis that it will fit in?

What is red supposed to do? sell their card in a tb enclosure effectively getting less bandwith?

theSeb · Jun 11, 2013

d-m-a-x said:
so where does the red rocket pci 2.0 fit in? it needs 16x banwith, and do they make a chassis that it will fit in?

Unfortunately nowhere at this point in time. Or a non-Apple workstation.

Edit: looking at the sonnet website it seems that I was wrong, but I have no idea now well the red rocket runs in the sonnet chassis. The marketing speak does specifically mention a red rocket though.

bsbeamer · Jun 11, 2013

d-m-a-x said:
so where does the red rocket pci 2.0 fit in? it needs 16x bandwith, and do they make a chassis that it will fit in?

What is red supposed to do? sell their card in a tb enclosure effectively getting less bandwith?

Check these out:
http://www.mlogic.com/products/mlink-r
http://www.maxxdigital.com/mobile-thunderbolt-redrocket.html
http://www.sonnettech.com/support/charts/thunderbolt/index.html

I have not used them, but they've been available for some time. Options do exist for Thunderbolt v1, would expect to see more with v2.

cube · Jun 11, 2013

darkgoob said:
Ouch. So, I guess that means you can't use something like this:
http://www.netstor.com.tw/_03/03_02.php?MTEx# ...?

Or is there an actual PCIE 3.0 slot hiding in that new Mac Pro somewhere? I guess it just does not seem like Thunderbolt alone will cut it. But then, perhaps this Mac Pro design may see some revisions before it's officially sold, like maybe a way to give full PCIE 3.0 16x speeds to external devices?

I don't need it, but I am sure many will.

So, the Tube is obsolete already.

MacVidCards · Jun 11, 2013

cube said:
So, the Tube is obsolete already.

Maybe, and maybe not.

One of the paid Apple shills (KADS) is here saying he has run a Titan using TB2 and it runs just peachy keen. When asked for something resembling details he got a little cagey. He DID include a lovely photo of a GTX560 with someone's old heathkit crystal radio project attached to it.

See his fine thread, named "missed connections " or something like that.

Tesselator · Jun 11, 2013

Yeah, I thought that was pretty funny.

But to be clear there will be PCIe expansions for the MP6,1. They even mentioned it in the announcement (err, sneak peak). But it will be limited to TB2 speeds. Now TB2 speeds are fast enough for most things (I guess all games for sure, and certainly any 3D apps like Lightwave, C4D, Maya, xsi, 3DS Max, Houdini, and so on... but not fast enough for all things. If what you're doing with your MP6,1 is of that later category you're going to be disappointed in the performance of the TB2 attached PCIe device in question.

BTW, that TB2 limit is about that same as 3 fast SSD drives in RAID0.

Any way you look at it it's not obsolete tho. You can either fit it into your pipeline (hardware work environment) successfully or you can't. It's either something useful to you or it's not robust enough. For me as a retired person messing around with music, photography and keeping up on my CG chops, it looks very interesting. If the list price is right, I'm getting one.

CaptainChunk · Jun 11, 2013

d-m-a-x said:
so where does the red rocket pci 2.0 fit in? it needs 16x bandwith, and do they make a chassis that it will fit in?

What is red supposed to do? sell their card in a tb enclosure effectively getting less bandwith?

The "classic" RED Rocket calls for 8 lanes of PCIe 2.0 bandwidth. But I've seen plenty of cases where it functions just fine in a TB 1.0 enclosure, which would provide "only" 2GB/s bandwidth. And this is still real-time 4K rendering...

The upcoming RED Rocket X card mentions requiring a x16 mechanical slot (the "classic" card also requires this), but the specs make no mention of how many lanes are required. I would imagine that this will also perform well in a TB enclosure, but only time will tell...

slughead · Jun 17, 2013

darkgoob said:
Is Thunderbolt 2.0 capable of PCIE 3.0 16x external enclosures? I mean can you have an external bay with one PCIE 3.0 16x graphics card in it? What bout two?

No, Thunderbolt 2 (~2GBps) is about 1/8 the speed per cable as PCIe 3.0 16x (~16GBps). On the 7970 benchmarks by anandtech, bottlenecking was showing with anything below 8GBps (PCIe 16x 2.0 or PCIe 8x 3.0), and severe bottlenecking at below 4GBps... TB2 (which hasn't even been tested yet) is 2GBps.

Therefore, TB2 will be worthless for modern high-end graphics cards, even if it's 1 card per TB2 port.

If you were looking to buy a Mac Pro now with low-end FirePro and then, say, 4 years from now buy the "TITAN 2 XXXXXTREEEEME" edition in SLI in a TB2 PCIe Chassis, you will be sorely disappointed.

----------

MacVidCards said:
Maybe, and maybe not.

One of the paid Apple shills (KADS) is here saying he has run a Titan using TB2 and it runs just peachy keen. When asked for something resembling details he got a little cagey. He DID include a lovely photo of a GTX560 with someone's old heathkit crystal radio project attached to it.

See his fine thread, named "missed connections " or something like that.

That's so weird that Anandtech would have such severe bottlenecking with a 7970 below 4GBps yet the much more powerful "titan" is running on "some internet guy's" chassis running at half the speed. Surely we should believe this man immediately and not ask for proof.

deconstruct60 · Jun 17, 2013

blackhand1001 said:
Thunderbolt 2 maxes out 20 gigabits per second. PCI express 3.0 has 16 gigabytes per second aka 128 gigabits per second.

There is also a difference between TB v2's top bandwidth and the subset of PCI-e data top bandwidth. There is little to no indications that TB v2 is moving the x4 PCI-e v2.0 cap on transport. TB v2 should deliver a more reliable and maximized PCI-e v2.0 transport.

The primary reason for the bandwidth re-assignment ( there was no aggregate bandwidth increase with TB v2.0 ) was to enable 4K video traffic. TB v2.0 is not focused on throwing all of the 20Gbps at PCI-e at all. The 10Gbps normally hardware reservation for PCI-e data traffic was removed so that video can soak up more of the aggregate TB v2 bandwidth. If there happens to be no "road hogging" video traffic the PCI-e data traffic will run with better Quality of Service (QoS) but not necessarily faster. Just like how the remote USB 3.0 sockets doesn't run any faster when remotely connected by Thunderbolt to the host box. Thunderbolt is significantly faster than USB 3.0 so that the remote socket works OK and delivers acceptable QoS in a remote box.

theSeb · Jun 17, 2013

slughead said:
No, Thunderbolt 2 (~2GBps) is about 1/8 the speed per cable as PCIe 3.0 16x (~16GBps). On the 7970 benchmarks by anandtech, bottlenecking was showing with anything below 8GBps (PCIe 16x 2.0 or PCIe 8x 3.0), and severe bottlenecking at below 4GBps... TB2 (which hasn't even been tested yet) is 2GBps.

Therefore, TB2 will be worthless for modern high-end graphics cards, even if it's 1 card per TB2 port.

If you were looking to buy a Mac Pro now with low-end FirePro and then, say, 4 years from now buy the "TITAN 2 XXXXXTREEEEME" edition in SLI in a TB2 PCIe Chassis, you will be sorely disappointed.

----------

That's so weird that Anandtech would have such severe bottlenecking with a 7970 below 4GBps yet the much more powerful "titan" is running on "some internet guy's" chassis running at half the speed. Surely we should believe this man immediately and not ask for proof.

Isn't Thunderbolt One's 2 GBps bandwidth including up and down? So real bandwidth available in one direction is actually 1 Gigabyte/s? Have I gone mad?

Anyway, same results found on tomshardware, even with midrange GPUs. Internet guys are not always reliable.

deconstruct60 · Jun 17, 2013

slughead said:
No, Thunderbolt 2 (~2GBps) is about 1/8 the speed per cable as PCIe 3.0 16x (~16GBps). On the 7970 benchmarks by anandtech, bottlenecking was showing with anything below 8GBps (PCIe 16x 2.0 or PCIe 8x 3.0), and severe bottlenecking at below 4GBps... TB2 (which hasn't even been tested yet) is 2GBps.

Eh?

http://www.anandtech.com/show/5458/the-radeon-hd-7970-reprise-pcie-bandwidth-overclocking-and-msaa

Given most monitors at 60Hz, not much of a loss. The x2 v3.0 on chart is equivalent to x4 v2.0 ( what TB v2 is likely provisioning). There is another chart with some other games like DIRT where there are bigger gaps but both of those approaches are biased ( cherry picking only the least or most impacted games ).

As long as use games software optimized for x16 lane throttled mainstream PC systems this isn't going to be much of a problem for a reasonable number of apps that target that context.

For a heavy data traffic OpenCL application it would be much more substantive bottlenecking. For higher end 3D apps with non pre-compressed data. Again much bigger impact. It is a much larger impact if the application is trying to leverage PCI-e v3.0 to send a higher amount of data to the card.

Therefore, TB2 will be worthless for modern high-end graphics cards, even if it's 1 card per TB2 port.

Per port isn't as material as per TB controller. Also whether the TB controllers are themselves are on switches ( The 2013 Mac Pro is using some switches internally someone where. Either GPU and PCI-e SSD are sharing or at least two of the TB controllers are sharing. )

Boomhowler · Jun 17, 2013

I find that anandtech-article really interesting. It actually says that with higher resolutions the differences will be smaller since the video card will spend more energy spitting out pixels than transferring data between cpu and gpu (i.e. the settings will probably have to be lower in the game if you want to keep framerates ok), so the bottleneck will NOT be in the connectivity. At least with the current, and maybe next, generation.

I would gladly use a GTX 670 or similar with a macbook air, it would be really cool. It would even work decently with a 2560*1440 monitor.

For some games (Dirt apparently) the bottleneck will be very visible though so the "mileage may vary", but for many applications it will be a very good addition.

theSeb · Jun 17, 2013

Here is the tomshardware article on the subject

http://www.tomshardware.com/reviews/pci-express-graphics-thunderbolt,3263.html

deconstruct60 · Jun 17, 2013

Boomhowler said:
I find that anandtech-article really interesting. It actually says that with higher resolutions the differences will be smaller since the video card will spend more energy spitting out pixels than transferring data between cpu and gpu (i.e. the settings will probably have to be lower in the game if you want to keep frame rates ok),

It is that plus the factor that the amount of data doesn't really go up all that much as increase the screen size. The textures used in the smaller view typically are largely the same set of textures used the larger view. The game also likely caches textures whatever are likely to transition to immediately also. The needs for specific data are much more highly predictable in games, because the games controls the parameters of "the world" playing inside of.

So the x4 PCI-e v2.0's 2GB/s it only takes about a 2 seconds to fill the VRAM full of new data. If game has stylized transitions through different sections that is plenty of time to reload the cache while the high rendering action is stopped.

As the VRAM storage on full sized GPU cards has gotten bigger games just pre-deploy more data onto into the VRAM.

Games are a pretty poor benchmark for bandwidth sensitivity since almost all of them are designed with a bucketload of work-arounds for limited bandwidth. They much more effectively measure if the work-arounds are still effective far more so as to whether the card is using more data. ( In fact it is usually the cast that the more "sensitive" games here will also show wider variability in performance between Nvidia and AMD. That's largely indicative that they are heavily dependent upon some proprietary mechanism to enable their preferred work-arounds. )

so the bottleneck will NOT be in the connectivity. At least with the current, and maybe next, generation.

It really depends upon what the objective is. If it is to run at "ultimate" performance levels there is impact. If just want to run faster than the embedded GPU several years into the future and focused primarily on gaming... then not so much as long as the compensation mechanisms are weaved into the external card.

deconstruct60 · Jun 17, 2013

theSeb said:
Here is the tomshardware article on the subject

http://www.tomshardware.com/reviews/pci-express-graphics-thunderbolt,3263.html

Similar results to the same work-arounds.

Where the long term vision is probably a bit off though is casting this opposite of HD3000 graphics. The new IrisPro HD is just first step in minimizing the need for external GPUs. As that goes mainstream there isn't as big of a potential market as the article's results suggest.

Sure later discrete PCI-e card graphics will start leverage eDRAM also so the lead gap will remain but at some point the frame rates get stupid crazy high that don't make any difference. There are still human eyeballs to view this stuff and there are still monitors with limited refresh cycles that they are presenting to the human eyeballs.

The "compute gap" will get bigger as folks wring more actual (as opposed to just theoretical) performance out of GPGPUs. But Thunderbolt may eventually get reasonably affordable fiber to help solve its bandwidth problem.

Cubemmal · Jun 17, 2013

Also in principle you might be able to gang up TB (two cables) for double the bandwidth. Out of my expertise though, I don't know if you'd need them all to be on the same clock or any of the details.

Regardless Apple has no problem releasing incremental hardware and making changes. TB is ultimately shooting for a much higher bandwidth, I recall 100Mbs but can't find a reference. Anyhow I wouldn't be too surprised if they don't support it a few years down the road.

Inconsequential · Jun 17, 2013

deconstruct60 said:
Eh?
Image
http://www.anandtech.com/show/5458/the-radeon-hd-7970-reprise-pcie-bandwidth-overclocking-and-msaa

Given most monitors at 60Hz, not much of a loss. The x2 v3.0 on chart is equivalent to x4 v2.0 ( what TB v2 is likely provisioning). There is another chart with some other games like DIRT where there are bigger gaps but both of those approaches are biased ( cherry picking only the least or most impacted games ).

Yes, this chart:

Also here: http://www.tomshardware.com/reviews/pci-express-graphics-thunderbolt,3263-7.html

I'm not convinced that a 7970 or 8970 or GTX680 would not be impacted by the loss of bandwidth, increased latency etc.

We need benchmarks, but I can't see myself using such a solution, especially as the 780MX in the next gen iMac will almost certainly proove to have better performance than an external GPU via TB2.

deconstruct60 · Jun 17, 2013

Concorde Rules said:
Yes, this chart:

Also here: http://www.tomshardware.com/reviews/pci-express-graphics-thunderbolt,3263-7.html

Which as I said when toss the outlier of DIRT doesn't really show a huge drop off.

Between x16 and x2 the changes are

Battlefield 3 -5%
Civ -1%
Crysis -3%
Cyrsis min -2%
Metro -3%

Batman -3%
Portal -8%
Shogun -19%

average so far is -5.5% : median -3%

DIRT3 -27%
DIRT3 Min -39%

average is -11% : median -5%

What have is a highly skewed data set at the end. But the average and median performance is hardly "serve bottlenecking". The majority even with the reductions are above 60Hz.

I'm not convinced that a 7970 or 8970 or GTX680 would not be impacted by the loss of bandwidth, increased latency etc.

It isn't whether the peak hardware is, it is where the software is running on the hardware.

We need benchmarks, but I can't see myself using such a solution, especially as the 780MX in the next gen iMac will almost certainly proove to have better performance than an external GPU via TB2.

The conclusion will vary by which software packages are used to compose the benchmark. However, it isn't 780MX ( or likely 880MX in 2013 iMac ) versus what is available in TB v2 external now. It is going to be far more so what happens in 2015 with the fixed 780MX and whatever is available then. Similar to how tom's hardware article is matched up against Intel HD.

Is someone going to "fix" their 2013 Mac Pro in 2015-16 with dual external GPUs in two enclosures on two independent TB controllers? For OpenCL work, probably not as well as a then current Mac Pro. For the legacy game they like to occasionally play... that is not so clear (only would need on eGPU and if stuffed to the brim with eDRAM cache and 8-9GB of VRAM it might work. )

Thunderbolt 2.0: capable of PCIE 3.0 16x cards in external enclosures?

macrumors 6502

macrumors 65816

macrumors 65816

macrumors 68030

macrumors 6502

macrumors 65816

macrumors 68000

macrumors 6502a

macrumors 604

macrumors 601

Suspended

Suspended

macrumors 601

macrumors 68020

macrumors 68040

macrumors G5

macrumors 604

macrumors G5

macrumors 6502

macrumors 604

macrumors G5

macrumors G5

macrumors 6502a

macrumors 68000

macrumors G5

Our Staff