MP 1,1-5,1 Using a NETSTOR NA211A (G3?) with a cMP 5,1 (PCIe switch experts needed)

[zedex]

A while ago I purchased a NETSTOR Turbobox (Model NA211A) on eBay that included a Target card but no Host card. I was lucky to find an official Netstor host card in the aftermarket and my PCIe slot expansion dreams came true.

After researching the switch capabilities/specifications (PCIe 2.0 x8) for the Turbobox, I reasoned that by inserting two (2) passive PCIe 3.0 x4 HBAs - each containing one (1) m.2 NVME PCIe SSD (matching 512GB Samsung SM951 NVME-only SSDs), in to each of the available PCIe 2.0 x8 slots on the Turbobox target board, then I would have assembled the equivalent of a Highpoint SSD7101 or a Syba PEX40129 HBA.

To be clear, the HIGHPOINT uses a PLX switch, the Syba uses a ASM switch and the Turbobox uses an IDT switch (just like the cMP 5,1).

The issue is that when I SoftRAID my matching SM951s to get full PCIe 2.0 x8 data transfer speeds (nominally 3000MB/s), I still get the typical 1500MB/s (PCIe 2.0 x4) speeds offered by each of these disks individually.

Yesterday I ran the 'pcitree' bash script provided to forum members by @joevt (results below) and I need some assistance interpreting the bottleneck limiting the link speed between the cMP and the Turbobox to PCIe 2.0 x4.

ANY feedback - particularly on how to read the second and third columns on link width - is appreciated..

 

[joevt]

Text would be more useful than a picture.

The second column is link rate/width capability (the max allowed)
The third column is link rate/width status (the current setting)

link rate:
g1 = gen 1 = PCIe 1.0 = 2.5 GT/s
g2 = gen 2 = PCIe 2.0 = 5 GT/s
g3 = gen 3 = PCIe 3.0 = 8 GT/s
g4 = gen 4 = PCIe 4.0 = 16 GT/s

link width (PCIe lanes):
x1
x2
x4 = four PCI lanes
x8
x16
x32 (I've never seen x32)

00:03.0 is Slot 1? It is occupied by a 96 lane gen 2 switch (16 lanes upstream, 6 downstream slots = x16,x16,x8,x8,x16,x16) which has your x16 GPU (AMD 5770), x4 dual SATA controllers (two of them), second x16 GPU (AMD 580 or similar), and x8 HighPoint RAID controller.

00:07.0 is Slot 2? The host and target card appear to be connected using only x4.
In the pcitree output, it looks like the upstream at 13:00.0 can support x8 so maybe you can upgrade the host and target card to x8 or x16? I don't know if that's possible though.
An x16 card uses two x8 cables which may comprise two x4 cables connected to each x8 connector. If the target is x8 then only one of the x8 cables is required. The NA255A comes with PCIe 3.0 x16 host and target cards that use two x8 cables.
The tech specs at https://www.netstor.com.tw/product_info.aspx?SerID=6&Title=TurboBox&ArID=22&PID=PID_170324380784203 only mentions x4 upstream support. They also mention PCIe 3.0 but yours is only PCIe 2.0 so maybe you have a different product? Maybe the NA211A-GPU? The picture at http://dynapowerusa.com/product/pcie-expansion-chassis-3-slot-gpu-na211a-gpu/ seems to show an x4 cable. The tech specs for that say x4 upstream, x8, x8, x4 downstream but yours is x8 upstream, x4, x4, x8 downstream.

The SSD7101 and PEX40129 have x16 upstream so there's no way to make the equivalent of those with this x8 or x4 upstream product.