Something odd about Thunderbolt specs?

[fpnc]

The Intel white paper on Thunderbolt says:

A Thunderbolt connector is capable of providing two full-duplex channels. Each channel provides bi-directional 10 Gbps of band-width.

Furthermore, the website Anandtech notes:

Thunderbolt is dual-channel, with each channel supporting 10 Gbps of bidirectional bandwidth. That’s a potential 20 Gbps of upstream and 20 Gbps of downstream bandwidth.

In addition, Apple says the following on their Thunderbolt website (my emphasis on the two channels):

Thunderbolt I/O technology gives you two channels on the same connector...

Thus, if you believe all of the above there should be a total of 20Gbps of output (or input) on the MacBook's single Thunderbolt port.

However, all of the marketing materials for the new MacBook Pros simply state "data transfer rates up to 10 Gbps."

Thus, my question would be, since it appears that one channel can transfer up to 10Gbps what happened to the second channel's 10Gbps?

The simply explanation would be that a single device can be served by only one channel (i.e. the channels can't be bonded, which makes some sense), and thus when Apple says "data transfer rates up to 10Gbps" they are talking about i/o to a single device. But if that is the case, why don't they mention that in a daisy-chain a second device can also receive up to 10Gbps by using the second channel?

Intel's white paper seems to hint at that possibility:

Because Thunderbolt technology delivers two full-bandwidth channels, the user can realize high bandwidth on not only the first device attached, but on downstream devices as well.

Intel also says:

Unlike bus-based I/O architectures, each Thunderbolt port on a computer is capable of providing the full bandwidth of the link in both directions with no sharing of band-width between ports or between upstream and downstream directions.

So, what is it? Does the MacBook's Thunderbolt port provide a total maximum of 20Gbps input and 20Gbps out, or does it only provide a total of 10Gbps in each direction?

I think I may have an answer (but it could be a bit controversial), so I wanted to pose this question to see if anyone had a good explanation as to why the numbers don't seem to add up.

 

[billgates99]

it's up to 10Gb/s total for each lane in either direction and simultaneously.

USB 3.0 is the same in that respect (max 5.0Gb/s bi-directional)

 

[Evil Spoonman]

Thunderbolt contains two 10Gbps full duplex lines for a total of 20Gbps in both directions at the same time (requires a PCIe 4x lane). I'm not sure yet how devices are given priority on the chain, nor do I know how the lines are split (if they are at all). The possibilities are.

- The full 20Gbps is split between all seven devices, controllers arbitrate QoS.
- The line is split, 10Gbps for DisplayPort and 10Gbps for data.
- The line is split, device 1 takes one line, device 2 takes another line.

Really the second two don't seem logical, I'd bank on the first one.

Source (PDF)

 

[Pressure]

Where is the confusion?

Two lanes:

<---------- <- 10Gbit/s -> ---------->
<---------- <- 10Gbit/s -> ---------->

10Gbit/s in each direction. No single device can access both lanes for a 20Gbit/s bandwidth, so it is limited to 10Gbit/s bidirectional transfers.
They cannot advertise 20Gbit/sec because that is being divided over the two lanes and/or more devices.

So in short, the Thunderbolt Port does provide 20Gbit/s bandwidth bidirectional but so single device can access both lanes.

 

[Erasmus]

It was my understanding that mini displayport can transfer well over 10 Gbit/s (Wikipedia says DisplayPort can do over 17 Gbit/s, I assume the Mini DisplayPort can do the same).

Perhaps the Thunderbolt connection simply adds an extra 3Gbit/s through extra wires, and restructures the data transfer when it's used instead of mDP?

One would assume the two lanes are effectively used to help with daisy chaining somehow, and would not be available to a single device at once. ie, I expect you can upload and download data at 10Gbit/s, but you can't upload OR download at 20Gbit/s. Hopefully you can upload and download from two devices at once at the full 10Gbit/s.

I expect a single channel would result in painful latencies when trying to use multiple devices at once, which wouldn't happen with two channels.

 

[Evil Spoonman]

Where is the confusion?

Two lanes:

<---------- <- 10Gbit/s -> ---------->
<---------- <- 10Gbit/s -> ---------->

10Gbit/s in each direction. No single device can access both lanes for a 20Gbit/s bandwidth, so it is limited to 10Gbit/s bidirectional transfers.
They cannot advertise 20Gbit/sec because that is being divided over the two lanes and/or more devices.

So in short, the Thunderbolt Port does provide 20Gbit/s bandwidth bidirectional but so single device can access both lanes.

That makes a lot more sense. So each lane is arbitrated individually, and each host connects to both lanes. No device can use two lanes at once. So effectively any single device is limited to 10Gbps full duplex. I wonder what happens if one device suddenly wants to use a whole lane, do the other hosts switch lanes dynamically? Is there a QoS implementation (your 1920x1080 DisplayPort link won't get stuttery when your line is maxed)? Conveniently I believe 2560x1600@60Hz is about 10Gbps, so one lane would be consumed by the display.

 

[fpnc]

Where is the confusion?

Two lanes:

<---------- <- 10Gbit/s -> ---------->
<---------- <- 10Gbit/s -> ---------->

10Gbit/s in each direction. No single device can access both lanes for a 20Gbit/s bandwidth, so it is limited to 10Gbit/s bidirectional transfers.
They cannot advertise 20Gbit/sec because that is being divided over the two lanes and/or more devices...

Well, yes, that is what I suggested as the simple explanation.

However, if that is true then I'm pretty sure they would advertise 20Gbps of total bandwidth per port against the 5Gbps for USB3. You have one port or connection and it can transfer a total of 20Gbps in both directions. However, no one from Apple has actually said that since it would indicate that you could be outputting 20Gbps of non-DisplayPort data at one time through that single port.

I think the truth is that one channel is reserved for DisplayPort and that may be because of a possible limitation in the current Thunderbolt controller. What I mean, is that they may have been unsure as to whether they could actually get the multiplexing of PCIe and DisplayPort working over the same channel and therefore they decided to reserve one channel for DisplayPort. In fact, I'm pretty sure I remember reading just such a claim in a technical overview of the new MacBooks on another website (not that there was a "limitation," just that one of the channels was reserved for DisplayPort).

This may not have been that big of a compromise or problem, since it seems that the PCIe portion of Thunderbolt is connected by 4 lanes of PCIe v2.0 and thus is limited to a total of 16Gbps anyway (bi-directional). Thus, they could never get a full 20Gbps of non-DisplayPort data over this one connection.

So, this is what I'm proposing:

MacBook's PCIe <---------- <- 10Gbit/s -> ----------> PCIe-based external device
MacBook's DisplayPort (full-time dedicated) <---------- <- 10Gbit/s -> ----------> DisplayPort device

Effectively nearly the same thing, but if true somewhat unfortunate if you really wanted a full 20Gbps (or 16Gbps) of pure PCIe bandwidth.

 

[Evil Spoonman]

Well, yes, that is what I suggested as the simple explanation.

However, if that is true then I'm pretty sure they would advertise 20Gbps of total bandwidth per port against the 5Gbps for USB3. You have one port or connection and it can transfer a total of 20Gbps in both directions. However, no one from Apple has actually said that since it would indicate that you could be outputting 20Gbps of non-DisplayPort data at one time through that single port.

I think the truth is that one channel is reserved for DisplayPort and that may be because of a possible limitation in the current Thunderbolt controller. What I mean, is that they may have been unsure as to whether they could actually get the multiplexing of PCIe and DisplayPort working over the same channel and therefore they decided to reserve one channel for DisplayPort. In fact, I'm pretty sure I remember reading just such a claim in a technical overview of the new MacBooks on another website (not that there was a "limitation," just that one of the channels was reserved for DisplayPort).

This may not have been that big of a compromise or problem, since it seems that the PCIe portion of Thunderbolt is connected by 4 lanes of PCIe v2.0 and thus is limited to a total of 16Gbps anyway (bi-directional). Thus, they could never get a full 20Gbps of non-DisplayPort data over this one connection.

So, this is what I'm proposing:

MacBook's PCIe <---------- <- 10Gbit/s -> ----------> PCIe-based external device
MacBook's DisplayPort (full-time dedicated) <---------- <- 10Gbit/s -> ----------> DisplayPort device

Effectively nearly the same thing, but if true somewhat unfortunate if you really wanted a full 20Gbps (or 16Gbps) of pure PCIe bandwidth.

That looks plausible. This might explain some of the confusion I've been having with understanding Thunderbolt's implementation. Some of the information seems to indicate that the Thunderbolt controller requires access to the GPU to function. This doesn't really make much sense, and seems incredibly limiting. Unless they are talking solely about its usage in laptop integration. There you will want to pipe both Thunderbolt and DisplayPort over the same line, so the Thunderbolt controller must have access to the GPU.

In a desktop you could use the Thunderbolt port as a data-only port with a PCIe x8 card and get the full 20Gbps.

 

[fpnc]

Oh, in my diagram I should have written:

MacBook's PCIe <---------- <- 10Gbit/s -> ----------> PCIe-based external device(s)

Since that one channel could serve several devices.

I guess almost the same could be true for the DisplayPort channel -- perhaps up to two devices.

 

[Evil Spoonman]

Oh, in my diagram I should have written:

MacBook's PCIe <---------- <- 10Gbit/s -> ----------> PCIe-based external device(s)

Since that one channel could serve several devices.

I guess almost the same could be true for the DisplayPort channel -- perhaps up to two devices.

Bumping this thread to say that you are indeed correct, fpnc. It is one channel DisplayPort and one channel Thunderbolt data.

http://www.tidbits.com/article/11993

 

[dime21]

the link speed is 10 Gb/s. It is full duplex. The numeric speed is the link speed in a single direction. Look up the term "full duplex" if you don't understand.

Just like Gigabit Ethernet is 1 Gb/s in each direction, at the same time, and therefore 2 Gb/s in total possible bandwidth.

 

[fpnc]

the link speed is 10 Gb/s. It is full duplex. The numeric speed is the link speed in a single direction. Look up the term "full duplex" if you don't understand.

Just like Gigabit Ethernet is 1 Gb/s in each direction, at the same time, and therefore 2 Gb/s in total possible bandwidth.

Well, obviously that's true for a single channel, but that's not really what we are talking about here.

 

[fpnc]

Bumping this thread to say that you are indeed correct, fpnc. It is one channel DisplayPort and one channel Thunderbolt data.

http://www.tidbits.com/article/11993

That does seem to confirm my suspicion, but if this is really true it means that you'll never get more than 10Gbps combined bandwidth in one direction for all of your PCIe devices. Of course, this is completely consistent with what the demos showed and what Apple says about "data transfer rates up to 10 Gbps."

So, for PCIe you have bi-directional 10Gbps over ONE channel (10Gbps output and 10Gbps in, agrees with spec and descriptions from Apple) and the other channel is dedicated or reserved for just DisplayPort (this part is not really that clear based upon what we've been shown by Apple and Intel).

Now the unfortunate part of this is that the second channel will go unused and remain completely idle unless you are also attached to a display. Frankly, if this is true (and I'm still not 100% convinced that it is) this seems to be a rather disappointing finding. The difference may be only around 5Gbps in combined PCIe bandwidth but that's still something of a loss (I say 5Gbps because with four PCIe lanes you'd be limited to 16Gbps anyway, so 16Gbps - 10Gbps means around 5Gbps underutilized).

 

[Evil Spoonman]

That does seem to confirm my suspicion, but if this is really true it means that you'll never get more than 10Gbps combined bandwidth in one direction for all of your PCIe devices. Of course, this is completely consistent with what the demos showed and what Apple says about "data transfer rates up to 10 Gbps."

So, for PCIe you have bi-directional 10Gbps over ONE channel (10Gbps output and 10Gbps in, agrees with spec and descriptions from Apple) and the other channel is dedicated or reserved for just DisplayPort (this part is not really that clear based upon what we've been shown by Apple and Intel).

Now the unfortunate part of this is that the second channel will go unused and remain completely idle unless you are also attached to a display. Frankly, if this is true (and I'm still not 100% convinced that it is) this seems to be a rather disappointing finding. The difference may be only something like 5Gbps in combined PCIe bandwidth but that's still something of a loss (I say 5Gbps because with four PCIe lanes you'd be limited to 16Gbps anyway, so 16Gbps - 10Gbps means around 5Gbps underutilized).

Yes, I see what you are saying. I am not clear on what happens to the reserved lane when there is no display attached either. It would make sense for the controllers to just use the lane (albeit at a reduced total of 16Gbps), and only reserve it for display data when there was indeed one attached.

 

[cpnotebook80]

I did see a video on youtube where a guy was doing on review on 2 mbps 13" incl the new model from last week. Using a normal usb stick to transfer like 10gb, it took less time on the usb port of new mbp by half or more compared to the old usb port of last years mbp. NOt sure how thunderbolt affects that or other ports in general.
it was very interesting indeed.

 

[fpnc]

Yes, I see what you are saying. I am not clear on what happens to the reserved lane when there is no display attached either. It would make sense for the controllers to just use the lane (albeit at a reduced total of 16Gbps), and only reserve it for display data when there was indeed one attached.

The problem with that mode of operation is what happens when you hot plug a display when the second channel is already being used for PCIe transfers. To actually use the display you'd either have to stop the PCIe transfer or switch the PCIe data over to the other channel. The latter would be the obvious solution, but maybe they couldn't get that to work.

In any case, we're still going some amount of hand-waving here. I don't think anyone has really given a satisfactory explanation and I have yet to read a completely unambiguous and unimpeachable source that explains exactly how the two Thunderbolt channels are actually being used.

 

[Evil Spoonman]

The problem with that mode of operation is what happens when you hot plug a display when the second channel is already being used for PCIe transfers. To actually use the display you'd either have to stop the PCIe transfer or switch the PCIe data over to the other channel. The latter would be the obvious solution, but maybe they couldn't get that to work.

In any case, we're still going some amount of hand-waving here. I don't think anyone has really given a satisfactory explanation and I have yet to read a completely unambiguous and unimpeachable source that explains exactly how the two Thunderbolt channels are actually being used.

Agree, this is all highly speculative. I'll keep my eyes open as Thunderbolt devices are released and see if I can't get my paws on some hardware, a 2011 MBP, and do some tests. If I come to any verifiable answers I will post back to this thread with them.

 

[mfordfm1]

any new info?

In searching for info on Thunderbolt, I came across this thread. However, it appears to be dated. Is there any new info out there on the specs, etc?

 

[oripash]

One device using both channels...

So..
A mid-2012 macbook air has a dual-channel thunderbolt controller.
Sonnet has a dual-channel enclosure:
http://store1.sonnettech.com/product_info.php?&products_id=403

They claim that while the mechanical plug in their enclosure is x16, the electrical connection is PCIe 2.0 x4, which is consistent with *two* thunderbolt channels.

Looking inside their product, there is
[a] A daughterboard with a dual-channel thunderbolt controller that has a male PCIe x4 plug. This daughterboard is oriented vertically.
Another daughterboard oriented horizontally with one PCIe x4 female connector on the bottom (where daughterboard [a] feeds in) and two PCIe x16 female connectors on top (where your PCI devices plug in).

I daresay that if they use PCIe x4 as a transport (what physically runs over that x4 mechanical plug) between thunderbolt controller and the end-user devices (and advertise their product as "four electrical PCIe lanes) it's one big 20Gbit pipe, not two discrete 10Gbit pipes only a single one of which can be used by an end-user device. At least insofar as the Sonnet product is concerned.

My 2 cents...

 

[drambuie]

That makes a lot more sense. So each lane is arbitrated individually, and each host connects to both lanes. No device can use two lanes at once. So effectively any single device is limited to 10Gbps full duplex. I wonder what happens if one device suddenly wants to use a whole lane, do the other hosts switch lanes dynamically? Is there a QoS implementation (your 1920x1080 DisplayPort link won't get stuttery when your line is maxed)? Conveniently I believe 2560x1600@60Hz is about 10Gbps, so one lane would be consumed by the display.

Actually, 2560 x 1600 x 24 bits at 60Hz equals 5.9Gb/s, so it's well within the single lane bandwidth limit. Even full 32 bit color would only need 7.86Gb/s. The limit would be reached with 40 bit color at 9.8Gb/s.