MP All Models Mellanox ConnectX-3 40 GbE using ATTO FastFrame macOS driver

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
Hi,

Did anybody else notice that the ATTO FastFrame NQ41 and NQ42 are rebranded Mellanox ConnectX-3 CX353A and CX354A cards? ATTO provides the bin firmware files on their website, all you likely need to do to get a generic Mellanox CX3 card to work is cross flash its firmware with ATTO's bin file. This is easy with mst / flint using Mellanox's Firmware Tools package.

1. Download Mellanox Firmware Tools: https://www.mellanox.com/products/adapter-software/firmware-tools
2. Download ATTO NQ41/NQ42 Firmware: https://www.atto.com/downloads/174/driver/29300356
3. Run the following:

mst start;

# NQ41 - Single Port Card
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq41_2017_09_05.bin burn;

# NQ42 - Dual Port Card
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq42_2017_09_05.bin burn;

4. Put device in a Mac and install ATTO's FastFrame macOS drivers.

I haven't tested this yet but it should work just fine, I cross flash Dell branded Mellanox cards all the time back to stock Mellanox firmware. I don't have a QSFP cable that is long enough to reach my Mac, but as soon as I do I'll flash one of mine and report back. CX3 cards can be picked up on eBay for about $40, the part numbers you should be looking for are CX353A-FCBT or CX354A-FCBT, these support up to 56 GbE. The CX353A-QCBT or CX354A-QCBT may also work, these support up to 40 GbE. At this time, the ATTO driver appears to only work with ConnectX-3 Pro CX313A and CX314A cards; the supported device id type (i.e. chipset model) is 4103. MCX313A-BCBT and MCX314A-BCBT are presumed to not be compatible because their device id is reportedly 4099.
 
Last edited:

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
It appears the ATTO cards are device id 4103, which are ConnectX-3 Pro devices.

Code:
root@r920-cmwhv52:~# flint -i flsbundle_nq41_2017_09_05.bin q
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050
MACs:                                       0002c9000001     0002c9000002
VSD:                   n/a
PSID:                  ATT1060111023

root@r920-cmwhv52:~# flint -d /dev/mst/mt4099_pci_cr0 q
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 506b4b03004cfac0 506b4b03004cfac1 506b4b03004cfac2 506b4b03004cfac3
MACs:                                       506b4b4cfac1     506b4b4cfac2
VSD:
PSID:                  DEL1100001019

root@r920-cmwhv52:~# flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i flsbundle_nq41_2017_09_05.bin burn;

    Current FW version on flash:  2.42.5000
    New FW version:               2.42.5000

    Note: The new FW version is the same as the current FW version on flash.

 Do you want to continue ? (y/n) [n] : y


    You are about to replace current PSID on flash - "DEL1100001019" with a different PSID - "ATT1060111023".
    Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
-E- Burning FS2 image failed: Device/Image mismatch: FW image file cannot be programmed to device ConnectX-3 A1, it is intended for: ConnectX-3Pro only
 

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
Code:
Stumbleine:Downloads root# pkgutil --expand ATTOFastFrame2_1110f1.pkg ./ATTOFastFrame2

Stumbleine:Downloads root# cd ATTOFastFrame2

Stumbleine:ATTOFastFrame2.pkg root# ls -l
total 1488
-rw-r--r--  1 root  wheel   36340 Sep 12  2018 Bom
-rw-r--r--@ 1 root  wheel     652 Sep 12  2018 PackageInfo
-rw-r--r--  1 root  wheel  715930 Sep 12  2018 Payload
drwxr-xr-x  4 root  wheel     128 Mar  7 09:52 Scripts

Stumbleine:ATTOFastFrame2.pkg root# file *
Bom:         Mac OS X bill of materials (BOM) file
PackageInfo: ASCII text
Payload:     gzip compressed data, from Unix, original size 2054144
Scripts:     directory

Stumbleine:ATTOFastFrame2.pkg root# mv Payload Payload.gz

Stumbleine:ATTOFastFrame2.pkg root# gunzip Payload.gz

Stumbleine:ATTOFastFrame2.pkg root# ls -l
total 4096
-rw-r--r--  1 root  wheel    36340 Sep 12  2018 Bom
-rw-r--r--@ 1 root  wheel      652 Sep 12  2018 PackageInfo
-rw-r--r--  1 root  wheel  2054144 Sep 12  2018 Payload
drwxr-xr-x  4 root  wheel      128 Mar  7 09:52 Scripts

Stumbleine:ATTOFastFrame2.pkg root# file *
Bom:         Mac OS X bill of materials (BOM) file
PackageInfo: ASCII text
Payload:     ASCII cpio archive (pre-SVR4 or odc)
Scripts:     directory

Stumbleine:ATTOFastFrame2.pkg root# mv Payload Payload.cpio

Stumbleine:ATTOFastFrame2.pkg root# cpio -idv < Payload.cpio
.
./usr
./usr/local
./usr/local/libexec
./usr/local/libexec/ATTOFastFrame2_tmp
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/_CodeSignature
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/_CodeSignature/CodeResources
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/MacOS
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/MacOS/ATTOFastFrame2
./usr/local/libexec/ATTOFastFrame2_tmp/ATTOFastFrame2.kext/Contents/Info.plist
./Applications
./Applications/ATTO
./Applications/ATTO/FastFrame2
./Applications/ATTO/FastFrame2/Uninstall.command
./Applications/ATTO/FastFrame2/atflash2
./Applications/ATTO/FastFrame2/ReadMe.htm
./Applications/ATTO/FastFrame2/atnicnvr
./Applications/ATTO/FastFrame2/atnetstat
./Applications/ATTO/FastFrame2/atinfo
4012 blocks
 

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
Code:
root@r920-cmwhv52:~# flint -i flsbundle_nq41_2017_09_05.bin dc
;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3pro_MCX313A_40g_2.prs
;;
;; PRS  FILE FOR KESTREL BENTAL
;; $Id$



[PS_INFO]
Name = 4779-313A-X00_Ax
Description = ConnectX-3 Pro EN network interface card; 40/56GbE; single-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name    = cx3pro_MCX313A_40g_2.prs

[ADAPTER]
PSID = ATT1060111023
pcie_gen2_speed_supported = true
pcie_gen3_speed_supported = true
adapter_dev_id = 0x1007
silicon_rev = 0x00

gpio_mode1 = 0x08000001
gpio_mode0 = 0x04c04032
gpio_default_val = 0x0f306023
gpio_pull_up = 0xff2baf2f
gpio_pull_enable = 0xfbbbbfff

receiver_detect_en = true
vdd_change_to_1_offset = 5

nv_cfg_en = true

nv_config_sectors = 2

[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0090
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x0f
dpdp_en = false

[IB]
mlpn_en_port0 = true
phy_type_port1 = XFI
ext_phy_board_port1 = FALCON
gen_guids_from_mac = true
do_sense = false
ref_clk_to_use = 0
module_power_level_supported_port0 = 5

num_of_ports = One_Port
new_gpio_scheme_en = true
read_cable_params_port1_en = true

cx3_spec1_3_ib_support_port0 = true
cx3_spec1_2_ib_support_port0 = true
spec1_3_fdr14_ib_support_port0 = true
spec1_3_fdr10_ib_support_port0 = true
mellanox_ddr_ib_support        = true
mellanox_qdr_ib_support        = true

port1_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port1_802_3ap_56kr4_ability = true

center_mix90phase = true

;;Logic lane to Serdes mapping
tx_logic_0_serdes = 0
tx_logic_1_serdes = 1
tx_logic_2_serdes = 2
tx_logic_3_serdes = 3
rx_logic_0_serdes = 3
rx_logic_1_serdes = 2
rx_logic_2_serdes = 1
rx_logic_3_serdes = 0

eth_tx_lane_polarity_port1 = 0xf
eth_rx_lane_polarity_port1 = 0x0
tx_lane_polarity_port1 = 0xf

; start of '#include "include_QSFP_serdes_prams_bental.h"'

;;Serdes parameters
port0_nego_fdr_mask_en = 0xfffc
port1_nego_fdr_mask_en = 0xfffc
port0_nego_fdr10_mask_en = 0xfffc
port1_nego_fdr10_mask_en = 0xfffc

nego_rx4_slicer_ind_en = 255
nego_rx4_slicer1_enable = 8
nego_rx4_slicer2_enable = 8
nego_rx4_ffe_tap0 = 94
nego_rx4_ffe_tap1 = 134
nego_rx4_ffe_tap2 = 245
nego_rx4_ffe_tap3 = 135
nego_rx4_ffe_tap4 = 171

nego_rx9_ffe_tap0=84
nego_rx9_ffe_tap1=164
nego_rx9_ffe_tap2=251
nego_rx9_ffe_tap3=132
nego_rx9_ffe_tap4=140

nego_rx15_ffe_tap3 = 140
nego_rx15_ffe_tap1 = 140

nego_rx10_ffe_tap3 = 140
nego_rx10_ffe_tap1 = 140

nego_rx8_ffe_tap3 = 140
nego_rx8_ffe_tap1 = 140

force_rx0_slicer_ind_en = 0x0
force_rx0_slicer1_enable = 0x0
force_rx0_slicer2_enable = 0x0
force_rx0_ffe_tap0 = 0xff
force_rx0_ffe_tap1 = 0x80
force_rx0_ffe_tap2 = 0x80
force_rx0_ffe_tap3 = 0x80
force_rx0_ffe_tap4 = 0x80

force_tx0_ob_preemp_pre = 0x40
force_tx0_ob_preemp_post = 0x0
force_tx0_ob_preemp_main = 0x7f
force_tx0_preemp = 0x0
force_tx0_pre_polarity = 0x1
force_tx0_post_polarity = 0x1
force_tx0_main_polarity = 0x0

force_rx2_slicer_ind_en = 0xeb
force_rx2_slicer1_enable = 0x0
force_rx2_slicer2_enable = 0x0
force_rx2_ffe_tap0 = 0x64
force_rx2_ffe_tap1 = 0x80
force_rx2_ffe_tap2 = 0xde
force_rx2_ffe_tap3 = 0x80
force_rx2_ffe_tap4 = 0x46

force_tx2_ob_preemp_pre = 0x30
force_tx2_ob_preemp_post = 0x0
force_tx2_ob_preemp_main = 0x7f
force_tx2_preemp = 0x0
force_tx2_pre_polarity = 0x1
force_tx2_post_polarity = 0x1
force_tx2_main_polarity = 0x0

force_rx3_slicer_ind_en = 0xff
force_rx3_slicer1_enable = 0x8
force_rx3_slicer2_enable = 0x8
force_rx3_ffe_tap0 = 0x6c
force_rx3_ffe_tap1 = 0x80
force_rx3_ffe_tap2 = 0xff
force_rx3_ffe_tap3 = 0x80
force_rx3_ffe_tap4 = 0x80

force_tx3_ob_preemp_pre = 0xc
force_tx3_ob_preemp_post = 0x7f
force_tx3_ob_preemp_main = 0x45
force_tx3_preemp = 0x0
force_tx3_pre_polarity = 0x1
force_tx3_post_polarity = 0x0
force_tx3_main_polarity = 0x1
force_tx3_ob_bias = 0xa

auto_ddr_tx_options = 2
auto_ddr_rx_options = 1

auto_qdr_tx_options = 6
auto_qdr_rx_options = 7

preset_tx_fdr_set12_ob_preemp_pre = 17
preset_tx_fdr_set12_ob_preemp_post = 0
preset_tx_fdr_set12_ob_preemp_main=25
preset_tx_fdr_set12_preemp = 0
preset_tx_fdr_set12_pre_polarity = 1
preset_tx_fdr_set12_post_polarity = 1
preset_tx_fdr_set12_main_polarity = 0
preset_tx_fdr_set12_ob_bias = 5

preset_tx_fdr_set13_ob_preemp_main =40
preset_tx_fdr_set13_ob_preemp_pre = 28
preset_tx_fdr_set13_ob_preemp_post = 0
preset_tx_fdr_set13_preemp = 0
preset_tx_fdr_set13_pre_polarity = 1
preset_tx_fdr_set13_post_polarity = 1
preset_tx_fdr_set13_main_polarity = 0
preset_tx_fdr_set13_ob_bias = 5

preset_tx_fdr_set14_ob_preemp_main = 35
preset_tx_fdr_set14_ob_preemp_pre = 25
preset_tx_fdr_set14_ob_preemp_post = 0
preset_tx_fdr_set14_preemp = 0
preset_tx_fdr_set14_pre_polarity = 1
preset_tx_fdr_set14_post_polarity = 1
preset_tx_fdr_set14_main_polarity = 0
preset_tx_fdr_set14_ob_bias = 5

preset_tx_fdr_set15_ob_preemp_main = 30
preset_tx_fdr_set15_ob_preemp_pre = 20
preset_tx_fdr_set15_ob_preemp_post = 0
preset_tx_fdr_set15_preemp = 0
preset_tx_fdr_set15_pre_polarity = 1
preset_tx_fdr_set15_post_polarity = 1
preset_tx_fdr_set15_main_polarity = 0
preset_tx_fdr_set15_ob_bias = 5

preset_tx_mask = 0xfffe

aba_mask0_start = 0
aba_mask0_end   = 3
aba_mask0 = 0x1000
aba_mask1_start = 4
aba_mask1_end   = 5
aba_mask1 = 0x8000
aba_mask2_start = 6
aba_mask2_end   = 10
aba_mask2 = 0x4000
aba_mask3_start = 11
aba_mask3_end   = 16
aba_mask3 = 0x2000

; ABA 40GE
aba_tx2_ob_preemp_pre = 20
aba_tx2_ob_preemp_main = 42
aba_tx2_ob_preemp_post = 8
aba_tx2_ob_bias = 8
aba_tx2_pre_polarity = 1
aba_tx2_post_polarity = 1
aba_tx2_main_polarity = 0

;;3m
aba_tx3_ob_preemp_pre = 22
aba_tx3_ob_preemp_main = 42
aba_tx3_ob_preemp_post = 5
aba_tx3_ob_bias = 8
aba_tx3_pre_polarity = 1
aba_tx3_post_polarity = 1
aba_tx3_main_polarity = 0

aba_tx4_ob_preemp_pre = 26
aba_tx4_ob_preemp_main = 42
aba_tx4_ob_preemp_post = 3
aba_tx4_ob_bias = 8
aba_tx4_pre_polarity = 1
aba_tx4_post_polarity = 1
aba_tx4_main_polarity = 0

aba_tx5_ob_preemp_pre = 60
aba_tx5_ob_preemp_main = 90
aba_tx5_ob_preemp_post = 8
aba_tx5_ob_bias = 8
aba_tx5_pre_polarity = 1
aba_tx5_post_polarity = 1
aba_tx5_main_polarity = 0

aba_tx6_ob_preemp_pre = 80
aba_tx6_ob_preemp_main = 110
aba_tx6_ob_preemp_post = 10
aba_tx6_ob_bias = 8
aba_tx6_pre_polarity = 1
aba_tx6_post_polarity = 1
aba_tx6_main_polarity = 0

aba_tx7_ob_preemp_pre = 75
aba_tx7_ob_preemp_main = 110
aba_tx7_ob_preemp_post = 15
aba_tx7_ob_bias = 8
aba_tx7_pre_polarity = 1
aba_tx7_post_polarity = 1
aba_tx7_main_polarity = 0

aba_fdr_tx16_ob_preemp_pre = 17
aba_fdr_tx16_ob_preemp_post = 0
aba_fdr_tx16_ob_preemp_main=25
aba_fdr_tx16_preemp = 0
aba_fdr_tx16_pre_polarity = 1
aba_fdr_tx16_post_polarity = 1
aba_fdr_tx16_main_polarity = 0
aba_fdr_tx16_ob_bias = 5

aba_fdr_tx17_ob_preemp_main =46
aba_fdr_tx17_ob_preemp_pre = 32
aba_fdr_tx17_ob_preemp_post = 0
aba_fdr_tx17_preemp = 0
aba_fdr_tx17_pre_polarity = 1
aba_fdr_tx17_post_polarity = 1
aba_fdr_tx17_main_polarity = 0
aba_fdr_tx17_ob_bias = 3

aba_fdr_tx18_ob_preemp_main = 50
aba_fdr_tx18_ob_preemp_pre = 32
aba_fdr_tx18_ob_preemp_post = 0
aba_fdr_tx18_preemp = 0
aba_fdr_tx18_pre_polarity = 1
aba_fdr_tx18_post_polarity = 1
aba_fdr_tx18_main_polarity = 0
aba_fdr_tx18_ob_bias = 3

aba_fdr_tx19_ob_preemp_main = 60
aba_fdr_tx19_ob_preemp_pre = 30
aba_fdr_tx19_ob_preemp_post = 0
aba_fdr_tx19_preemp = 0
aba_fdr_tx19_pre_polarity = 1
aba_fdr_tx19_post_polarity = 1
aba_fdr_tx19_main_polarity = 0
aba_fdr_tx19_ob_bias = 3

aba_index0_start = 0
aba_index0_end   = 3
aba_index0 = 0
aba_index1_start = 4
aba_index1_end   = 5
aba_index1 = 3
aba_index2_start = 6
aba_index2_end   = 9
aba_index2 = 2
aba_index3_start = 10
aba_index3_end   = 16
aba_index3 = 1

aba_rx2_slicer_ind_en = 0xeb
aba_rx2_slicer1_enable = 0x0
aba_rx2_slicer2_enable = 0x0
aba_rx2_ffe_tap0 = 0x80
aba_rx2_ffe_tap1 = 0x68
aba_rx2_ffe_tap2 = 0xd7
aba_rx2_ffe_tap3 = 0x80
aba_rx2_ffe_tap4 = 0x5a

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;; SFP+ section. all QSFP can be converted to SFP+ using QSA adapter.;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; ETH connected to third party device
aba_non_mlpn_tx8_ob_preemp_pre = 5
aba_non_mlpn_tx8_ob_preemp_post = 0
aba_non_mlpn_tx8_ob_preemp_main = 65
aba_non_mlpn_tx8_ob_bias = 8
aba_non_mlpn_tx8_pre_polarity = 1
aba_non_mlpn_tx8_post_polarity = 1
aba_non_mlpn_tx8_main_polarity = 0
aba_non_mlpn_tx8_preemp = 0


nego_eth_rx12_slicer_ind_en = 0xff
nego_eth_rx12_slicer1_enable= 0x8
nego_eth_rx12_slicer2_enable= 0x8
nego_eth_rx12_ffe_tap0=241
nego_eth_rx12_ffe_tap1=128
nego_eth_rx12_ffe_tap2=61
nego_eth_rx12_ffe_tap3=99
nego_eth_rx12_ffe_tap4=128

; end of '#include "include_QSFP_serdes_prams_bental.h"'

[PLL]
lbist_en  = 0
lbist_shift_freq  = 3
flash_div = 0x3
lbist_array_bypass = 1
lbist_pat_cnt_lsb = 0x2
core_f = 60
core_r = 14
core_od = 2
en_427_mhz = true

[FW]
flash_has_suspend_resume = 0
log_flashdev_size = 21
log_flash_sector_size = 2
Code:
root@r920-cmwhv52:~# flint -i flsbundle_nq42_2017_09_05.bin dc
;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3pro_MCX314A_40g.prs
;
;; PRS  FILE FOR FALCON BENTAL 40G
;; $Id$



[PS_INFO]
Name = 4779-314A-X00_Ax
Description = ConnectX-3 Pro EN network interface card; 40/56GbE; dual-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
PRS_name    = cx3pro_MCX314A_40g.prs

[ADAPTER]
PSID = ATT1090111023
pcie_gen2_speed_supported = true
pcie_gen3_speed_supported = true
adapter_dev_id = 0x1007
silicon_rev = 0x00

gpio_mode1 = 0x08000001
gpio_mode0 = 0x04d042fe
gpio_default_val = 0x0f287f9f
gpio_pull_up = 0xff2baf9f
gpio_pull_enable = 0xfbabbfef

receiver_detect_en = true
vdd_change_to_1_offset = 7

nv_cfg_en = true

[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0091
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x00
dpdp_en = false
cmpl_timeout_default_value_en = 1

[IB]
mlpn_en_port0 = true
mlpn_en_port1 = true
phy_type_port1 = XFI
phy_type_port2 = XFI
module_power_level_supported_port0 = 5
module_power_level_supported_port1 = 5

ext_phy_board_port1 = FALCON
ext_phy_board_port2 = FALCON
gen_guids_from_mac = true
do_sense = false
ref_clk_to_use = 0
new_gpio_scheme_en = true
read_cable_params_port1_en = true
read_cable_params_port2_en = true

cx3_spec1_3_ib_support_port0 = true
cx3_spec1_3_ib_support_port1 = true
cx3_spec1_2_ib_support_port0 = true
cx3_spec1_2_ib_support_port1 = true
spec1_3_fdr14_ib_support_port0 = true
spec1_3_fdr14_ib_support_port1 = true
spec1_3_fdr10_ib_support_port0 = true
spec1_3_fdr10_ib_support_port1 = true
mellanox_ddr_ib_support        = true
mellanox_qdr_ib_support        = true

port1_802_3ap_cr4_enable = true
port2_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port2_802_3ap_cr4_ability = true
port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_ability = true

center_mix90phase = true

;;Logic lane to Serdes mapping
tx_logic_0_serdes = 0
tx_logic_1_serdes = 1
tx_logic_2_serdes = 2
tx_logic_3_serdes = 3
rx_logic_0_serdes = 3
rx_logic_1_serdes = 2
rx_logic_2_serdes = 1
rx_logic_3_serdes = 0

tx_logic_4_serdes = 4
tx_logic_5_serdes = 5
tx_logic_6_serdes = 6
tx_logic_7_serdes = 7
rx_logic_4_serdes = 7
rx_logic_5_serdes = 6
rx_logic_6_serdes = 5
rx_logic_7_serdes = 4

eth_tx_lane_polarity_port1 = 0xf
eth_tx_lane_polarity_port2 = 0xf
eth_rx_lane_polarity_port1 = 0x0
eth_rx_lane_polarity_port2 = 0xf
tx_lane_polarity_port1 = 0xf
tx_lane_polarity_port2 = 0xf

; start of '#include "include_QSFP_serdes_prams_bental.h"'

;;Serdes parameters
port0_nego_fdr_mask_en = 0xfffc
port1_nego_fdr_mask_en = 0xfffc
port0_nego_fdr10_mask_en = 0xfffc
port1_nego_fdr10_mask_en = 0xfffc

nego_rx4_slicer_ind_en = 255
nego_rx4_slicer1_enable = 8
nego_rx4_slicer2_enable = 8
nego_rx4_ffe_tap0 = 94
nego_rx4_ffe_tap1 = 134
nego_rx4_ffe_tap2 = 245
nego_rx4_ffe_tap3 = 135
nego_rx4_ffe_tap4 = 171

nego_rx9_ffe_tap0=84
nego_rx9_ffe_tap1=164
nego_rx9_ffe_tap2=251
nego_rx9_ffe_tap3=132
nego_rx9_ffe_tap4=140

nego_rx15_ffe_tap3 = 140
nego_rx15_ffe_tap1 = 140

nego_rx10_ffe_tap3 = 140
nego_rx10_ffe_tap1 = 140

nego_rx8_ffe_tap3 = 140
nego_rx8_ffe_tap1 = 140

force_rx0_slicer_ind_en = 0x0
force_rx0_slicer1_enable = 0x0
force_rx0_slicer2_enable = 0x0
force_rx0_ffe_tap0 = 0xff
force_rx0_ffe_tap1 = 0x80
force_rx0_ffe_tap2 = 0x80
force_rx0_ffe_tap3 = 0x80
force_rx0_ffe_tap4 = 0x80

force_tx0_ob_preemp_pre = 0x40
force_tx0_ob_preemp_post = 0x0
force_tx0_ob_preemp_main = 0x7f
force_tx0_preemp = 0x0
force_tx0_pre_polarity = 0x1
force_tx0_post_polarity = 0x1
force_tx0_main_polarity = 0x0

force_rx2_slicer_ind_en = 0xeb
force_rx2_slicer1_enable = 0x0
force_rx2_slicer2_enable = 0x0
force_rx2_ffe_tap0 = 0x64
force_rx2_ffe_tap1 = 0x80
force_rx2_ffe_tap2 = 0xde
force_rx2_ffe_tap3 = 0x80
force_rx2_ffe_tap4 = 0x46

force_tx2_ob_preemp_pre = 0x30
force_tx2_ob_preemp_post = 0x0
force_tx2_ob_preemp_main = 0x7f
force_tx2_preemp = 0x0
force_tx2_pre_polarity = 0x1
force_tx2_post_polarity = 0x1
force_tx2_main_polarity = 0x0

force_rx3_slicer_ind_en = 0xff
force_rx3_slicer1_enable = 0x8
force_rx3_slicer2_enable = 0x8
force_rx3_ffe_tap0 = 0x6c
force_rx3_ffe_tap1 = 0x80
force_rx3_ffe_tap2 = 0xff
force_rx3_ffe_tap3 = 0x80
force_rx3_ffe_tap4 = 0x80

force_tx3_ob_preemp_pre = 0xc
force_tx3_ob_preemp_post = 0x7f
force_tx3_ob_preemp_main = 0x45
force_tx3_preemp = 0x0
force_tx3_pre_polarity = 0x1
force_tx3_post_polarity = 0x0
force_tx3_main_polarity = 0x1
force_tx3_ob_bias = 0xa

auto_ddr_tx_options = 2
auto_ddr_rx_options = 1

auto_qdr_tx_options = 6
auto_qdr_rx_options = 7

preset_tx_fdr_set12_ob_preemp_pre = 17
preset_tx_fdr_set12_ob_preemp_post = 0
preset_tx_fdr_set12_ob_preemp_main=25
preset_tx_fdr_set12_preemp = 0
preset_tx_fdr_set12_pre_polarity = 1
preset_tx_fdr_set12_post_polarity = 1
preset_tx_fdr_set12_main_polarity = 0
preset_tx_fdr_set12_ob_bias = 5

preset_tx_fdr_set13_ob_preemp_main =40
preset_tx_fdr_set13_ob_preemp_pre = 28
preset_tx_fdr_set13_ob_preemp_post = 0
preset_tx_fdr_set13_preemp = 0
preset_tx_fdr_set13_pre_polarity = 1
preset_tx_fdr_set13_post_polarity = 1
preset_tx_fdr_set13_main_polarity = 0
preset_tx_fdr_set13_ob_bias = 5

preset_tx_fdr_set14_ob_preemp_main = 35
preset_tx_fdr_set14_ob_preemp_pre = 25
preset_tx_fdr_set14_ob_preemp_post = 0
preset_tx_fdr_set14_preemp = 0
preset_tx_fdr_set14_pre_polarity = 1
preset_tx_fdr_set14_post_polarity = 1
preset_tx_fdr_set14_main_polarity = 0
preset_tx_fdr_set14_ob_bias = 5

preset_tx_fdr_set15_ob_preemp_main = 30
preset_tx_fdr_set15_ob_preemp_pre = 20
preset_tx_fdr_set15_ob_preemp_post = 0
preset_tx_fdr_set15_preemp = 0
preset_tx_fdr_set15_pre_polarity = 1
preset_tx_fdr_set15_post_polarity = 1
preset_tx_fdr_set15_main_polarity = 0
preset_tx_fdr_set15_ob_bias = 5

preset_tx_mask = 0xfffe

aba_mask0_start = 0
aba_mask0_end   = 3
aba_mask0 = 0x1000
aba_mask1_start = 4
aba_mask1_end   = 5
aba_mask1 = 0x8000
aba_mask2_start = 6
aba_mask2_end   = 10
aba_mask2 = 0x4000
aba_mask3_start = 11
aba_mask3_end   = 16
aba_mask3 = 0x2000

; ABA 40GE
aba_tx2_ob_preemp_pre = 20
aba_tx2_ob_preemp_main = 42
aba_tx2_ob_preemp_post = 8
aba_tx2_ob_bias = 8
aba_tx2_pre_polarity = 1
aba_tx2_post_polarity = 1
aba_tx2_main_polarity = 0

;;3m
aba_tx3_ob_preemp_pre = 22
aba_tx3_ob_preemp_main = 42
aba_tx3_ob_preemp_post = 5
aba_tx3_ob_bias = 8
aba_tx3_pre_polarity = 1
aba_tx3_post_polarity = 1
aba_tx3_main_polarity = 0

aba_tx4_ob_preemp_pre = 26
aba_tx4_ob_preemp_main = 42
aba_tx4_ob_preemp_post = 3
aba_tx4_ob_bias = 8
aba_tx4_pre_polarity = 1
aba_tx4_post_polarity = 1
aba_tx4_main_polarity = 0

aba_tx5_ob_preemp_pre = 60
aba_tx5_ob_preemp_main = 90
aba_tx5_ob_preemp_post = 8
aba_tx5_ob_bias = 8
aba_tx5_pre_polarity = 1
aba_tx5_post_polarity = 1
aba_tx5_main_polarity = 0

aba_tx6_ob_preemp_pre = 80
aba_tx6_ob_preemp_main = 110
aba_tx6_ob_preemp_post = 10
aba_tx6_ob_bias = 8
aba_tx6_pre_polarity = 1
aba_tx6_post_polarity = 1
aba_tx6_main_polarity = 0

aba_tx7_ob_preemp_pre = 75
aba_tx7_ob_preemp_main = 110
aba_tx7_ob_preemp_post = 15
aba_tx7_ob_bias = 8
aba_tx7_pre_polarity = 1
aba_tx7_post_polarity = 1
aba_tx7_main_polarity = 0

aba_fdr_tx16_ob_preemp_pre = 17
aba_fdr_tx16_ob_preemp_post = 0
aba_fdr_tx16_ob_preemp_main=25
aba_fdr_tx16_preemp = 0
aba_fdr_tx16_pre_polarity = 1
aba_fdr_tx16_post_polarity = 1
aba_fdr_tx16_main_polarity = 0
aba_fdr_tx16_ob_bias = 5

aba_fdr_tx17_ob_preemp_main =46
aba_fdr_tx17_ob_preemp_pre = 32
aba_fdr_tx17_ob_preemp_post = 0
aba_fdr_tx17_preemp = 0
aba_fdr_tx17_pre_polarity = 1
aba_fdr_tx17_post_polarity = 1
aba_fdr_tx17_main_polarity = 0
aba_fdr_tx17_ob_bias = 3

aba_fdr_tx18_ob_preemp_main = 50
aba_fdr_tx18_ob_preemp_pre = 32
aba_fdr_tx18_ob_preemp_post = 0
aba_fdr_tx18_preemp = 0
aba_fdr_tx18_pre_polarity = 1
aba_fdr_tx18_post_polarity = 1
aba_fdr_tx18_main_polarity = 0
aba_fdr_tx18_ob_bias = 3

aba_fdr_tx19_ob_preemp_main = 60
aba_fdr_tx19_ob_preemp_pre = 30
aba_fdr_tx19_ob_preemp_post = 0
aba_fdr_tx19_preemp = 0
aba_fdr_tx19_pre_polarity = 1
aba_fdr_tx19_post_polarity = 1
aba_fdr_tx19_main_polarity = 0
aba_fdr_tx19_ob_bias = 3

aba_index0_start = 0
aba_index0_end   = 3
aba_index0 = 0
aba_index1_start = 4
aba_index1_end   = 5
aba_index1 = 3
aba_index2_start = 6
aba_index2_end   = 9
aba_index2 = 2
aba_index3_start = 10
aba_index3_end   = 16
aba_index3 = 1

aba_rx2_slicer_ind_en = 0xeb
aba_rx2_slicer1_enable = 0x0
aba_rx2_slicer2_enable = 0x0
aba_rx2_ffe_tap0 = 0x80
aba_rx2_ffe_tap1 = 0x68
aba_rx2_ffe_tap2 = 0xd7
aba_rx2_ffe_tap3 = 0x80
aba_rx2_ffe_tap4 = 0x5a

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;; SFP+ section. all QSFP can be converted to SFP+ using QSA adapter.;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; ETH connected to third party device
aba_non_mlpn_tx8_ob_preemp_pre = 5
aba_non_mlpn_tx8_ob_preemp_post = 0
aba_non_mlpn_tx8_ob_preemp_main = 65
aba_non_mlpn_tx8_ob_bias = 8
aba_non_mlpn_tx8_pre_polarity = 1
aba_non_mlpn_tx8_post_polarity = 1
aba_non_mlpn_tx8_main_polarity = 0
aba_non_mlpn_tx8_preemp = 0


nego_eth_rx12_slicer_ind_en = 0xff
nego_eth_rx12_slicer1_enable= 0x8
nego_eth_rx12_slicer2_enable= 0x8
nego_eth_rx12_ffe_tap0=241
nego_eth_rx12_ffe_tap1=128
nego_eth_rx12_ffe_tap2=61
nego_eth_rx12_ffe_tap3=99
nego_eth_rx12_ffe_tap4=128

; end of '#include "include_QSFP_serdes_prams_bental.h"'

[PLL]
lbist_en  = 0
lbist_shift_freq  = 3
flash_div = 0x3
lbist_array_bypass = 1
lbist_pat_cnt_lsb = 0x2
core_f = 60
core_r = 14
core_od = 2
en_427_mhz = true

[FW]
flash_has_suspend_resume = 0
log_flashdev_size = 21
log_flash_sector_size = 6
 

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
I sorta got it working with the regular ConnectX-3 MCX353A-FCBT, all I did was generate a new firmware image using the [HCA] section from the INI file above and also changed the PSID. My device is a single port, so everything below relates to the single port ATTO FastFrame NQ41, steps for the dual port could be slightly different.

These are the commands / steps I did:

# Backup current firmware on the Mellanox card and the INI file:
flint -d /dev/mst/mt4099_pci_cr0 ri DEL1100001019.bin;
flint -d /dev/mst/mt4099_pci_cr0 dc > DEL1100001019.ini;

# Create a new INI file using the INI file from the previous step as the foundation:
cp -a DEL1100001019.ini ATT1060111023.ini;

# Swap out the [HCA] section and change PSID under the [ADAPTER] section with the following:
vim ATT1060111023.ini

[ADAPTER]
PSID = ATT1090111023
[HCA]
hca_header_subsystem_vendor_id = 0x117c
hca_header_device_id = 0x1007
hca_header_subsystem_id = 0x0091
hca_header_class_code = 0x020000
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x00
dpdp_en = false
cmpl_timeout_default_value_en = 1
# Download Mellanox's firmware development kit, you need the fw-ConnectX3-rel.mlx file.
# I can't find "fw-ConnectX3-rel.mlx" version 2.42.5000, if you have a copy send it my way please.
wget http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz;
tar -axf ConnectX3-rel-2_40_5030.tgz;

# Download Mellanox's Firmware Tools package and install, and also install the extra mft-oem package, mft-oem contains the tools necessary to create new firmware images.
wget http://content.mellanox.com/MFT/mft-4.14.0-105-x86_64-deb.tgz;
tar -axf mft-4.14.0-105-x86_64-deb.tgz;
cd mft-4.14.0-105-x86_64-deb;
./install.sh;
dpkg -I ./DEBS/mft-oem_4.14.0-105_amd64.deb;

# Create a new firmware image using the fw-ConnectX3-rel.mlx and ATT1060111023.ini files, output file is fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin:
mlxburn -wrimage fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin -fw fw-ConnectX3-rel.mlx -conf ATT1060111023.ini;

# Start mst services if not already started:
mst start;

# Flash new firmware image onto the Mellanox card:
flint --allow_psid_change -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_40_5030-ATT1060111023.bin burn;

# Reboot for the new firmware image to take effect:
reboot;

After rebooting the Linux host, the Subsystem now identifies as ATTO, vfio-pci is attached because I'm running macOS Mojave inside of a KVM VM on a Ubuntu 20.04 system, so I'm doing PCIe device passthrough of the Mellanox card into the VM... this could potentially be the cause for why it's not working right... I still need to get a 10 meter QSFP cable to reach all the way to my Mac Pro 2012.

Code:
root@r920-cmwhv52:~/Mellanox# lspci -vs 41:00
41:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
    Subsystem: ATTO Technology, Inc. FastFrame NQ41
    Flags: bus master, fast devsel, latency 0, IRQ 139, NUMA node 2
    Memory at d3d00000 (64-bit, non-prefetchable) [disabled] [size=1M]
    Memory at d3000000 (64-bit, prefetchable) [disabled] [size=8M]
    Expansion ROM at d3e00000 [disabled] [size=1M]
    Capabilities: [40] Power Management version 3
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
    Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [148] Device Serial Number 50-6b-4b-03-00-4c-fa-c0
    Capabilities: [154] Advanced Error Reporting
    Capabilities: [18c] Secondary PCI Express
    Kernel driver in use: vfio-pci
From within the macOS Mojave VM I get the following:

Code:
Stumbleine:FastFrame2 root# ./atinfo -c 1 -i all

######################################################################
Channel 1: ATTO FastFrame NQ41
######################################################################

Node Address:
Permanent MAC Address:   00:00:00:00:00:00
Current MAC Address:     00:00:00:00:00:00
PCI Vendor ID:           15B3
PCI Device ID:           1007
PCI Subsystem Vendor ID: 117C
PCI Subsystem Device ID: 0090
PCI Location:            1.0.0
PCI Class Code:          020000
PCI Revision ID:         0
Current PCI Link Width:  x8
Maximum PCI Link Width:  x8
Current PCI Link Speed:  8 GT/s (Gen3)
Maximum PCI Link Speed:  8 GT/s (Gen3)
Interrupt Mode:          Legacy
Driver Name:             ATTOFastFrame2.kext
Driver Version:          1.11.0f1
Firmware Version:        2.40.5030
Controller State:        Degraded
 
Last edited:

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
I found a Mellanox QSFP to SFP+ adapter so I was able to try this in my Mac Pro 2012, unfortunately it did not work... but it feels very close to working... I'm willing to bet if I had a genuine ConnectX-3 Pro (I only have regular CX3 right now, not the Pro version) it would work. In the screenshot below it appears the card lost it's MAC addresses. I checked with flint and it's still there, but on closer investigation it appears the ATTO driver can't access the NVRAM region of the regular non-pro CX3 card. I'm going to stop here until I can get my hands on a real CX3 Pro.

atto-screenshot-1.png

atto-screenshot-2.png


atto-screenshot-3.png


atto-screenshot-4.png
 
  • Like
Reactions: hwojtek

simonjii

macrumors newbie
Apr 20, 2020
3
0
Hi. This is extremely interesting. How sure are you that it'll work with the ConnectX-3 Pro? I'm about to get one on Ebay to test that with our 2013 MacPro (in external TB PCI box) and our server.
Do you think those specs should be good? Model No: CX314A, P/N: MCX314A-BCCT, Pro version, CONNECTX-3Pro EN 40 Gigabit Ethernet Adapter PCI-E 2-Port MELLANOX.
 

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
Hi. This is extremely interesting. How sure are you that it'll work with the ConnectX-3 Pro? I'm about to get one on Ebay to test that with our 2013 MacPro (in external TB PCI box) and our server.
Do you think those specs should be good? Model No: CX314A, P/N: MCX314A-BCCT, Pro version, CONNECTX-3Pro EN 40 Gigabit Ethernet Adapter PCI-E 2-Port MELLANOX.
Yes I would be willing to bet that it'll work, and the MCX314A-BCCT card has the correct device id of 4103. The most important thing to ensure is that the device id matches the one in the ATTO card, and this one does. Best of luck, please post your results... unfortunately I haven't had the spare cash to buy one of these cards to test yet due to the whole covid-19 pandemic. Your worst case is the driver simply won't work, and if that happens then you can flash it back to the stock firmware and just return it to the seller on eBay.

Mellanox MCX314A-BCCT.png
 

evilzardoz

macrumors newbie
Oct 19, 2008
28
3
Hey,

This is really cool! Shame all my Mellanox Connect-X 3 cards are either the CX354A or CX353A. Would be curious if there was a way to get those to work!
 

nbritton

macrumors regular
Original poster
May 22, 2008
151
106
Hey,

This is really cool! Shame all my Mellanox Connect-X 3 cards are either the CX354A or CX353A. Would be curious if there was a way to get those to work!
There is probably a way to hack the ATTO driver to use stock Mellanox cards but I don’t know how to do that off the top of my head and I don’t have any slack time to research it right now.
 

simonjii

macrumors newbie
Apr 20, 2020
3
0
Hi. I just ordered the MCX314A-BCCT and should have it by 1st of June at the latest. I'll report here or ask questions.
 

simonjii

macrumors newbie
Apr 20, 2020
3
0
Hi nbritton

I got my Connect X3-pro and flashed the firmware in Linux. Put it in my Mac Pro.
But unfortunately it looks like I'm at the exact same point like you were:
1590743177465.png

1590743236822.png


I did manage to flash the firmware again on the mac but it doesn't change anything. Looks like the same issue with NVRAM. Do you have any ideas on how to proceed?

Edit: I got a bit further. I could restore the NVRAM to default settings with atinicnvr and now the NVRAM not found message disappeared. Still the same in the ATTO config tool. Maybe we would need a proper ATTO Nvram file to copy onto the card?
 
Last edited:
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.