High CPU usage, lower than expected throughput with 10 gigabit network

Discussion in 'macOS Mojave (10.14)' started by tmiw, Aug 11, 2019.

  1. tmiw macrumors 68020

    Joined:
    Jun 26, 2007
    Location:
    San Diego, CA
    #1
    I recently bought an OWC 10 gigabit Thunderbolt adapter and have been running some tests locally with my 2017 13" MBP on 10.14.5 (still need to update to .6, I know). While I was able to get ~9 gigabits a second with iperf3 while transferring from my NAS to my MBP, I don't get nearly as good performance the other direction:

    NAS->MBP:

    Accepted connection from 10.0.1.164, port 53787
    [ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53788
    [ ID] Interval Transfer Bitrate Retr Cwnd
    [ 5] 0.00-1.00 sec 1.06 GBytes 9.07 Gbits/sec 63 1.32 MBytes
    [ 5] 1.00-2.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.58 MBytes
    [ 5] 2.00-3.00 sec 1.06 GBytes 9.12 Gbits/sec 0 1.80 MBytes
    [ 5] 3.00-4.00 sec 1.07 GBytes 9.17 Gbits/sec 0 2.00 MBytes
    [ 5] 4.00-5.00 sec 1.06 GBytes 9.10 Gbits/sec 0 2.00 MBytes
    [ 5] 5.00-6.00 sec 1.06 GBytes 9.13 Gbits/sec 0 2.00 MBytes
    [ 5] 6.00-7.00 sec 1.06 GBytes 9.10 Gbits/sec 20 1.31 MBytes
    [ 5] 7.00-8.00 sec 1.06 GBytes 9.07 Gbits/sec 0 1.57 MBytes
    [ 5] 8.00-9.00 sec 1.07 GBytes 9.18 Gbits/sec 0 1.80 MBytes
    [ 5] 9.00-10.00 sec 1.04 GBytes 8.98 Gbits/sec 22 1.00 MBytes
    [ 5] 10.00-10.00 sec 1.25 MBytes 4.81 Gbits/sec 0 1.00 MBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval Transfer Bitrate Retr
    [ 5] 0.00-10.00 sec 10.6 GBytes 9.12 Gbits/sec 105 sender

    MBP->NAS (default window size):

    Accepted connection from 10.0.1.164, port 53789
    [ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53790
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-1.00 sec 340 MBytes 2.86 Gbits/sec
    [ 5] 1.00-2.00 sec 261 MBytes 2.19 Gbits/sec
    [ 5] 2.00-3.00 sec 250 MBytes 2.10 Gbits/sec
    [ 5] 3.00-4.00 sec 245 MBytes 2.06 Gbits/sec
    [ 5] 4.00-5.00 sec 244 MBytes 2.05 Gbits/sec
    [ 5] 5.00-6.00 sec 249 MBytes 2.09 Gbits/sec
    [ 5] 6.00-7.00 sec 246 MBytes 2.07 Gbits/sec
    [ 5] 7.00-8.00 sec 243 MBytes 2.04 Gbits/sec
    [ 5] 8.00-9.00 sec 243 MBytes 2.04 Gbits/sec
    [ 5] 9.00-10.00 sec 243 MBytes 2.04 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec receiver

    MBP->NAS (512K window size):

    Accepted connection from 10.0.1.164, port 53791
    [ 5] local 10.0.1.177 port 5201 connected to 10.0.1.164 port 53792
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-1.00 sec 726 MBytes 6.09 Gbits/sec
    [ 5] 1.00-2.00 sec 749 MBytes 6.29 Gbits/sec
    [ 5] 2.00-3.00 sec 765 MBytes 6.42 Gbits/sec
    [ 5] 3.00-4.00 sec 772 MBytes 6.48 Gbits/sec
    [ 5] 4.00-5.00 sec 771 MBytes 6.46 Gbits/sec
    [ 5] 5.00-6.00 sec 769 MBytes 6.45 Gbits/sec
    [ 5] 6.00-7.00 sec 770 MBytes 6.46 Gbits/sec
    [ 5] 7.00-8.00 sec 773 MBytes 6.48 Gbits/sec
    [ 5] 8.00-9.00 sec 774 MBytes 6.49 Gbits/sec
    [ 5] 9.00-10.00 sec 754 MBytes 6.33 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-10.00 sec 7.45 GBytes 6.40 Gbits/sec receiver

    Something I noticed while doing the last transfer is that kernel_task seems to take a very large amount of CPU while it's happening, as shown below:

    Screen Shot 2019-08-11 at 8.28.08 PM.png

    Has anyone else seen similar behavior, either with the OWC adapter or another one? Any idea how to resolve? I'm open to returning my adapter and buying a different one if it's something inherent to OWC's implementation.

    Adapter info:

    Screen Shot 2019-08-11 at 9.14.40 PM.png

    System info:

    Screen Shot 2019-08-11 at 9.14.48 PM.png
     
  2. HDFan macrumors 65816

    Joined:
    Jun 30, 2007
    #2
    What were the iPerf command lines that you were using? What NAS did you test with?
     
  3. tmiw thread starter macrumors 68020

    Joined:
    Jun 26, 2007
    Location:
    San Diego, CA
    #3
    FreeNAS side: iperf3 -s
    OSX (2gb/sec test): iperf3 -c freenas
    OSX (6gb/sec test): iperf3 -c freenas -w 512k
    OSX (9gb/sec test): iperf3 -c freenas -R
     
  4. HDFan macrumors 65816

    Joined:
    Jun 30, 2007
    #4
    I tested on my iMac with iperf, AJA System Test and Blackmagic on a QNAP TS-453bt3. My
    QNAP seems to be quite a bit slower than the NAS you are using.

    My experience has been that reading from my NAS is generally rather fast, writing tends to be slower. I do see a spike in kernel cpu while transfers are in process.

    iperf:

    ./iperf3 -c host -w 512k [iMac to NAS]
    [ 4] 0.00-1.00 sec 139 MBytes 1.16 Gbits/sec

    ./iperf3 -c host -R [NAS to iMac]

    [ 4] 0.00-1.00 sec 580 MBytes 4.87 Gbits/sec

    AJA system test which is the reverse of what I usually see:

    AJA System Test QNAP 453bt3 network.png

    Blackmagic

    Blackmagic QNAP 453bt3 SSD ethernet.png


    I have no idea why different tools give such different results. Even using the same tool I see a ton of variation.
     
  5. tmiw thread starter macrumors 68020

    Joined:
    Jun 26, 2007
    Location:
    San Diego, CA
    #5
    The Blackmagic results you got are actually almost exactly what I get on mine, so it sounds like I might be more drive limited than network limited at this point. Still, it would be nice to be able to get full throughput in both directions in case that were to ever change.
     
  6. HDFan macrumors 65816

    Joined:
    Jun 30, 2007
    #6
    I spent a lot of time working with QNAP support trying to determine what my "best" numbers were. Trying to eliminate the network as a bottle neck we ran the tests via thunderbolt. Theoretical results are (scroll down a bit) 356/514 MB/s.

    https://www.qnap.com/en-us/product/ts-453bt3

    My Blackmagic numbers on one test for thunderbolt are 369/629 MB/s, a bit better. But it is extremely variable, uploads are usually in the 100 - 200 MBs range.

    My thunderbolt 2 hardware RAID comes in at 828/1045 MB/s W/R. I would guess that you probably have full throughout in both directions. All of my tests were on RAID 5 system. Maybe another RAID level would give you better results.
     
  7. tmiw thread starter macrumors 68020

    Joined:
    Jun 26, 2007
    Location:
    San Diego, CA
    #7
    I'm running effectively a RAID10 setup at the moment with 8 drives. bonnie++ gives me around 646MB sequential read and ~392MB write (command line: bonnie++ -u root -r 1024 -s 65535 -d /benchmark -f -b -n 1). Unfortunately I'm not sure if it'd even be possible to do a direct TB connection to my NAS--at least, not without additional components and setup.

    However, I might try directly connecting the Ethernet cable to the NAS instead of going through a switch. That way, setting the MTU to 9000 hopefully won't screw up anything else on the network and should allow me to see if my MBP is CPU limited with regard to network speeds.
     

Share This Page

6 August 11, 2019