iMac Pro's Nbase-T Ethernet with support for 1Gb, 2.5Gb, 5Gb, and 10Gb port's performance

[bxs]

Subject: iMac Pro's NBase-T Ethernet with support for 1Gb, 2.5Gb, 5Gb, and 10Gb port's performance

Hi there....

I'm about to deploy 10GbE networking in the office and will be using an iMac Pro (iMP) having the 10Gb Ethernet port.

I've been told and understand that some people using this 10Gb ethernet port aren't seeing the expected 10G speeds and that they are not any better than 5G.

The issue appears to be that Apple has made their 10Gb ethernet port be 10GNbase-T ethernet and not the 10GBase-T ethernet.

In order to get the full or better than 10GNbase-T speeds one needs to use 10GBase-T and one way to get this is to connect a Sonnet Echo Express box to one of the iMP's TB3 ports and install a 10GBase-T PCIe card in it.

So what I'm thinking of doing is this....

1) Use a Sonnet Echo Express III (Thunderbolt 3 Edition) that has room for three PCIe Cards. Only need one PCIe slot really, but having an extra two covers any future need for additional PCIe cards we may need. This enclosure appeared to be priced at $499, PART NO. ECHO-EXP-SE3-T3 at http://www.sonnettech.com/product/echoexpressse3.html

2) Install a Dual port PCIe P2E10G-2-T card with the Intel X540 Chip in the Sonnet enclosure. Prices seem to range from $400 to $500 for this card.

3) Use shielded Cat7 cables for networking to a 10GbE switch. Cat7 cables from OWC are at a reasonable cost.

Question(s)
1. For people using the stock/native 10GNbase-T port in their iMP, what speeds are you seeing ?

2. For people that have connected a Sonnet enclosure + PCIe Intel card (as described above) to their iMP's TB3 port, what speeds are you seeing ?

3. Will I need special software driver code for this Sonnet+PCIe card or will the stock High Sierra 10.13.4 have the necessary driver code already ?

4. If I need special driver code where would I obtain that from ?

Thanks for any feedback on this 10GNbase-T vs. 10Gbase-T aspect.

 

[kschendel]

I don't have any answers, but I am a bit curious as to what you're doing that the factor of 2 transfer speed difference between 5G and 10G is worth a thousand bucks plus some unknown number of hours of time? I don't deny that such situations exist, I'm just being nosy...

 

[bxs]

Not wanting to be too snippy in my response.... but speed trumps every thing in my world.

The Intel 10GBase-T PCIe 2-port card can be had for around $400 and the Sonnet enclosure for holding a single PCIe card is $199. So total would be $599 as a minimum cost presumably.

I'm also told that placing the PCIe card in a 4x lane slot would yield some 65-70% of the 10G speed whereas placing the card in a 16x lane slot should yield a good 1200 MB/s. If this is correct then the difference between a 4x and a 16x slot is very significant.

My approach will be as follows.... (software for the Small-Tree Intel PCIe dual 10G card can be obtained from Small-Tree free of charge for High Sierra)

1) Run varies speed tests using the stock 10G port on the iMacPro.

2) Run same tests as in 1) using the iMacPro's TB3 port connected to the Sonnet enclosure with the PCIe dual 10G port installed.

3) Compare the 1) & 2) test results and if there's little to no difference then return the Sonnet and PCIe card for full refund.

Of course my hope is that the results from 3) support what I have mentioned in my OP.

 

[jaytv111]

I've never heard of 10 GN-Base-T, as far as I know there's only 10 GBase-T, and NBase-T just means 2.5 or 5 Gbit/s speeds being an option for cheaper cabling setups. https://en.wikipedia.org/wiki/10_Gigabit_Ethernet

So as far as I know it supports 10G and if you have the appropriate cables and switches then it should have the right speed (you also have to check that the network is reporting it at 10 Gbits in System Preferences).

Sometimes you don't observe the full speeds because the application isn't taking full advantage of the extra speed. Why that would be is anyone's guess; could be thread priority, lack of multithreading, a parameter being set to limit the network transfer speed, etc. I had hooked up a Mac to a PC with TB3 (the network bridge feature of TB3) and it reported 20 Gbit/s in the link speed but then when I transferred files over SMB it basically got 1G speeds in my observations, not sure why since it was using SSDs with at least 500 MB/s sequential read and write speed.

Also, re: the Thunderbolt enclosure, just an FYI that Thunderbolt 3 can only offer an x4 link, it cannot offer x16, so if you're using Thunderbolt 3 for anything it cannot physically connect at x16 speeds. The PCIe link speed can only go up to x16 if it's internal (ie in a desktop tower straight on the motherboard).

 

[bxs]

Yes, terminology is at odds here.... sorry for that.

You wrote: So as far as I know it supports 10G

Are you saying the iMacPro's ethernet port supports both 10 GBase-T, and NBase-T ? The Apple specs state its NBase-T.

I intend to test the 10G network speeds using iperf, dd, AJA and Blackmagic.

We are using Thunderbolt Bridge for three of our MP6,1 machines today and get very reliable transfer speeds of some 800 MB/s. This is with grabbing data from one of the MP6,1's having a Pegasus2 R8 RAID-5 attached.

THANKS for the insight to the Sonnet enclosure providing x4 link only. However, here's what I see stated on the Sonnet web site...

The Echo Express SE III’s PCIe 3.0 x8 slots support three half-length+, full-height, single-width PCIe cards, or one single-width card plus one double-width card.

Even the Sonnet Echo Express SEL support x8 as well.

 

[jaytv111]

N-Base-T refers to 5 and 2.5 Gbit/s, 10 GBase-T refers to 10 Gbit/s. The iMac Pro supports each of these (as well as earlier 1 G ethernet). Quote from the site:
10Gb Ethernet

• Nbase-T Ethernet with support for 1Gb, 2.5Gb, 5Gb, and 10Gb Ethernet using RJ‑45 connector
  

So summary: NBase-T = 1, 2.5, and 5. The 10 Gb Ethernet part of that sentence refers to 10 GBase-T.

And Sonnet is referring to the internal connector of the enclosure. PCIe cards can be in different form factors, there's x1, x4, and x16. If it supports x16 then that means you can fit a card with x16, but that doesn't mean the interface to the Mac is running at x16. In fact, TB3 can only support x4 because of bandwidth limitations. It has up to 40 Gbit/s of line speed, PCIe 3.0 is running at 1 GByte/s per lane, and that's 8 Gbit/s per lane, times 4 lanes = 32 Gbit/s, 8 left over for protocol overhead (or perhaps a video port). There isn't any more speed available to offer x8 or x16. Therefore, any card running through TB3 will be running at most with an x4 link. And PCIe cards can indeed run at lower link speeds when lanes aren't available (there's some motherboards for instance that have like 3 or more x16 slots, but only offer you x8 speeds if you max out 2 of the x16s).

 

[ColdCase]

So the convention shortcut is that N is just the * number holding spot in the term *Base-T. NBase-T is a shortcut term and is not a different network type. It is not sufficient to describe the interface. A number, or a series of numbers needs to follow. When and interface only supports one speed, the N shortcut is not usually used.

 

[jaytv111]

Yes, NBase-T seems to be an informal way of saying 2.5 and 5 Gbits, as those seem to be paired together and in order to differentiate from 10 Gbits which needs more expensive cabling and the chips tend to cost more for 10 G.

 

[bxs]

N-Base-T refers to 5 and 2.5 Gbit/s, 10 GBase-T refers to 10 Gbit/s. The iMac Pro supports each of these (as well as earlier 1 G ethernet). Quote from the site:
10Gb Ethernet

• Nbase-T Ethernet with support for 1Gb, 2.5Gb, 5Gb, and 10Gb Ethernet using RJ‑45 connector

So summary: NBase-T = 1, 2.5, and 5. The 10 Gb Ethernet part of that sentence refers to 10 GBase-T.

And Sonnet is referring to the internal connector of the enclosure. PCIe cards can be in different form factors, there's x1, x4, and x16. If it supports x16 then that means you can fit a card with x16, but that doesn't mean the interface to the Mac is running at x16. In fact, TB3 can only support x4 because of bandwidth limitations. It has up to 40 Gbit/s of line speed, PCIe 3.0 is running at 1 GByte/s per lane, and that's 8 Gbit/s per lane, times 4 lanes = 32 Gbit/s, 8 left over for protocol overhead (or perhaps a video port). There isn't any more speed available to offer x8 or x16. Therefore, any card running through TB3 will be running at most with an x4 link. And PCIe cards can indeed run at lower link speeds when lanes aren't available (there's some motherboards for instance that have like 3 or more x16 slots, but only offer you x8 speeds if you max out 2 of the x16s).

I'm not disputing what you've posted at all. My concern is that I've been told on good authority that many people are very dissatisfied with the iMacPro's 10G port's capability and that their observed data rates are far below what is expected. From this I was assuming it was because maybe Apple pooped out and only offers the NBase-T ethernet and not the GBase-T.

From the iMac Pro's System Information -> Ethernet Cards we see Link Width: x4 for en0.

Again, thanks for the clarification of the Sonnet enclosure's stated PCIe card lane info. No matter, the x4 lanes offers in theory 4 GB/s (4000 MB/s) and the TB3 offers the 5 GB/s (5000 MB/s) and the GBase-T presumable offers 1.2 GB/s (1200 MB/s). Our file server can throw out a good 1000 MB/s so I'm hoping it being attached to a iMacPro via TB3 and then over the 10G network there's an expectation to see around 1000 MB/s being received at the other end or in flight at least. From what you've said the iMacPro's ethernet port does support GBase-T so we will see what our test reveals.

I'd be very interested in knowing what others who have employed the iMacPro's ethernet port for a 10G network are seeing for speeds along with their overall setup (network architecture, online disk types and so on).

 

[bxs]

I'm not disputing what you've posted at all. My concern is that I've been told on good authority that many people are very dissatisfied with the iMacPro's 10G port's capability and that their observed data rates are far below what is expected. From this I was assuming it was because maybe Apple pooped out and only offers the NBase-T ethernet and not the GBase-T.

From the iMac Pro's System Information -> Ethernet Cards we see Link Width: x4 for en0.

Again, thanks for the clarification of the Sonnet enclosure's stated PCIe card lane info. No matter, the x4 lanes offers in theory 4 GB/s (4000 MB/s) and the TB3 offers the 5 GB/s (5000 MB/s) and the GBase-T presumable offers 1.2 GB/s (1200 MB/s). Our file server can throw out a good 1000 MB/s so I'm hoping it being attached to a iMacPro via TB3 and then over the 10G network there's an expectation to see around 1000 MB/s being received at the other end or in flight at least. From what you've said the iMacPro's ethernet port does support GBase-T so we will see what our test reveals.

I'd be very interested in knowing what others who have employed the iMacPro's ethernet port for a 10G network are seeing for speeds along with their overall setup (network architecture, online disk types and so on).

Here's a block diagram for the iMacPro, and the 10G ethernet port gets PCIe 2.0 x4.

 

[jaytv111]

It could be a lousy controller, or lousy drivers. I don't know. I just would say that the application layer tends to show up as a bottleneck more often than not.

 

[rkuo]

I'm not disputing what you've posted at all. My concern is that I've been told on good authority that many people are very dissatisfied with the iMacPro's 10G port's capability and that their observed data rates are far below what is expected. From this I was assuming it was because maybe Apple pooped out and only offers the NBase-T ethernet and not the GBase-T.

Got a source for this? I haven't heard anything of the sort myself, but I'd be genuinely curious if this was true.

 

[bxs]

Got a source for this? I haven't heard anything of the sort myself, but I'd be genuinely curious if this was true.

It has been word&mouth from Small-Tree hearing back from their customers.

 

[bxs]

It has been word&mouth from Small-Tree hearing back from their customers.

OK.... here's a bit more info on the issue related to what folks are saying about the iMacPro's 10G ethernet port....

The iMacPro does have a 10G NBase-T type connection.... which of course supports 10Gbase-T but it does NOT use Intel's chipset and that is the reported issue........ all NBase-T ports are suspect bad.... so one has to go or should go with Intel.... This would be the reason for using a TB3 cable to a Sonnet enclosure having an Intel-base PCIe 10G card installed in it to avoid the performance/operational issues that others are reporting.

For example.... Use Small-Tree's Single Port Intel X540 based PCI Express 10Gb Ethernet Network Adapter - 10GBASE-T card P2E10G-1-T 10GbE 10GBase-T or the dual port card P2E10G-2-T 10GbE 10GBase-T, https://www.small-tree.com/categories/10gb-ethernet-cards/ and have say Small-Tree's software driver installed on the iMacPro - https://www.small-tree.com/support/download_category?cat_id=6.

IMO it's simply a case of not ignoring what others are saying about the iMacPro's native 10G ethernet port. Vendor's such as Small-Tree have hundreds of customers and if some are reporting issues with using the iMacPro's native 10G port it's difficult to ignore out of hand.