High CPU usage, lower than expected throughput with 10 gigabit network

[tmiw]

I recently bought an OWC 10 gigabit Thunderbolt adapter and have been running some tests locally with my 2017 13" MBP on 10.14.5 (still need to update to .6, I know). While I was able to get ~9 gigabits a second with iperf3 while transferring from my NAS to my MBP, I don't get nearly as good performance the other direction:

NAS->MBP:

Accepted connection from 10.0.1.164, port 53787
[ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53788
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.06 GBytes 9.07 Gbits/sec 63 1.32 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.58 MBytes
[ 5] 2.00-3.00 sec 1.06 GBytes 9.12 Gbits/sec 0 1.80 MBytes
[ 5] 3.00-4.00 sec 1.07 GBytes 9.17 Gbits/sec 0 2.00 MBytes
[ 5] 4.00-5.00 sec 1.06 GBytes 9.10 Gbits/sec 0 2.00 MBytes
[ 5] 5.00-6.00 sec 1.06 GBytes 9.13 Gbits/sec 0 2.00 MBytes
[ 5] 6.00-7.00 sec 1.06 GBytes 9.10 Gbits/sec 20 1.31 MBytes
[ 5] 7.00-8.00 sec 1.06 GBytes 9.07 Gbits/sec 0 1.57 MBytes
[ 5] 8.00-9.00 sec 1.07 GBytes 9.18 Gbits/sec 0 1.80 MBytes
[ 5] 9.00-10.00 sec 1.04 GBytes 8.98 Gbits/sec 22 1.00 MBytes
[ 5] 10.00-10.00 sec 1.25 MBytes 4.81 Gbits/sec 0 1.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.6 GBytes 9.12 Gbits/sec 105 sender

MBP->NAS (default window size):

Accepted connection from 10.0.1.164, port 53789
[ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53790
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 340 MBytes 2.86 Gbits/sec
[ 5] 1.00-2.00 sec 261 MBytes 2.19 Gbits/sec
[ 5] 2.00-3.00 sec 250 MBytes 2.10 Gbits/sec
[ 5] 3.00-4.00 sec 245 MBytes 2.06 Gbits/sec
[ 5] 4.00-5.00 sec 244 MBytes 2.05 Gbits/sec
[ 5] 5.00-6.00 sec 249 MBytes 2.09 Gbits/sec
[ 5] 6.00-7.00 sec 246 MBytes 2.07 Gbits/sec
[ 5] 7.00-8.00 sec 243 MBytes 2.04 Gbits/sec
[ 5] 8.00-9.00 sec 243 MBytes 2.04 Gbits/sec
[ 5] 9.00-10.00 sec 243 MBytes 2.04 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec receiver

MBP->NAS (512K window size):

Accepted connection from 10.0.1.164, port 53791
[ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53792
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 726 MBytes 6.09 Gbits/sec
[ 5] 1.00-2.00 sec 749 MBytes 6.29 Gbits/sec
[ 5] 2.00-3.00 sec 765 MBytes 6.42 Gbits/sec
[ 5] 3.00-4.00 sec 772 MBytes 6.48 Gbits/sec
[ 5] 4.00-5.00 sec 771 MBytes 6.46 Gbits/sec
[ 5] 5.00-6.00 sec 769 MBytes 6.45 Gbits/sec
[ 5] 6.00-7.00 sec 770 MBytes 6.46 Gbits/sec
[ 5] 7.00-8.00 sec 773 MBytes 6.48 Gbits/sec
[ 5] 8.00-9.00 sec 774 MBytes 6.49 Gbits/sec
[ 5] 9.00-10.00 sec 754 MBytes 6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.45 GBytes 6.40 Gbits/sec receiver

Something I noticed while doing the last transfer is that kernel_task seems to take a very large amount of CPU while it's happening, as shown below:

Has anyone else seen similar behavior, either with the OWC adapter or another one? Any idea how to resolve? I'm open to returning my adapter and buying a different one if it's something inherent to OWC's implementation.

Adapter info:

System info:

 

[HDFan]

~9 gigabits a second with iperf3

What were the iPerf command lines that you were using? What NAS did you test with?

 

[tmiw]

What were the iPerf command lines that you were using? What NAS did you test with?

FreeNAS side: iperf3 -s
OSX (2gb/sec test): iperf3 -c freenas
OSX (6gb/sec test): iperf3 -c freenas -w 512k
OSX (9gb/sec test): iperf3 -c freenas -R

 

[HDFan]

I tested on my iMac with iperf, AJA System Test and Blackmagic on a QNAP TS-453bt3. My
QNAP seems to be quite a bit slower than the NAS you are using.

My experience has been that reading from my NAS is generally rather fast, writing tends to be slower. I do see a spike in kernel cpu while transfers are in process.

iperf:

./iperf3 -c host -w 512k [iMac to NAS]
[ 4] 0.00-1.00 sec 139 MBytes 1.16 Gbits/sec

./iperf3 -c host -R [NAS to iMac]

[ 4] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec

AJA system test which is the reverse of what I usually see:



Blackmagic




I have no idea why different tools give such different results. Even using the same tool I see a ton of variation.

 

[tmiw]

I tested on my iMac with iperf, AJA System Test and Blackmagic on a QNAP TS-453bt3. My
QNAP seems to be quite a bit slower than the NAS you are using.

My experience has been that reading from my NAS is generally rather fast, writing tends to be slower. I do see a spike in kernel cpu while transfers are in process.

iperf:

./iperf3 -c host -w 512k [iMac to NAS]
[ 4] 0.00-1.00 sec 139 MBytes 1.16 Gbits/sec

./iperf3 -c host -R [NAS to iMac]

[ 4] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec

AJA system test which is the reverse of what I usually see:

View attachment 852733

Blackmagic

View attachment 852734


I have no idea why different tools give such different results. Even using the same tool I see a ton of variation.

The Blackmagic results you got are actually almost exactly what I get on mine, so it sounds like I might be more drive limited than network limited at this point. Still, it would be nice to be able to get full throughput in both directions in case that were to ever change.

 

[HDFan]

Still, it would be nice to be able to get full throughput in both directions in case that were to ever change.

I spent a lot of time working with QNAP support trying to determine what my "best" numbers were. Trying to eliminate the network as a bottle neck we ran the tests via thunderbolt. Theoretical results are (scroll down a bit) 356/514 MB/s.

https://www.qnap.com/en-us/product/ts-453bt3

My Blackmagic numbers on one test for thunderbolt are 369/629 MB/s, a bit better. But it is extremely variable, uploads are usually in the 100 - 200 MBs range.

My thunderbolt 2 hardware RAID comes in at 828/1045 MB/s W/R. I would guess that you probably have full throughout in both directions. All of my tests were on RAID 5 system. Maybe another RAID level would give you better results.

 

[tmiw]

I spent a lot of time working with QNAP support trying to determine what my "best" numbers were. Trying to eliminate the network as a bottle neck we ran the tests via thunderbolt. Theoretical results are (scroll down a bit) 356/514 MB/s.

https://www.qnap.com/en-us/product/ts-453bt3

My Blackmagic numbers on one test for thunderbolt are 369/629 MB/s, a bit better. But it is extremely variable, uploads are usually in the 100 - 200 MBs range.

My thunderbolt 2 hardware RAID comes in at 828/1045 MB/s W/R. I would guess that you probably have full throughout in both directions. All of my tests were on RAID 5 system. Maybe another RAID level would give you better results.

I'm running effectively a RAID10 setup at the moment with 8 drives. bonnie++ gives me around 646MB sequential read and ~392MB write (command line: bonnie++ -u root -r 1024 -s 65535 -d /benchmark -f -b -n 1). Unfortunately I'm not sure if it'd even be possible to do a direct TB connection to my NAS--at least, not without additional components and setup.

However, I might try directly connecting the Ethernet cable to the NAS instead of going through a switch. That way, setting the MTU to 9000 hopefully won't screw up anything else on the network and should allow me to see if my MBP is CPU limited with regard to network speeds.