Introduction
Although old, Mellanox ConnectX-3 has been a good card with decent performance and a good price on second-hand markets. This makes it the straightforward choice for a high-performance home or lab network. (If you intend to run InfiniBand on ESXi 7.0 or higher, go for CX4 or higher.) Sometimes you would find a card with a strange firmware or one not compatible with your existent gear. Luckily you can change the port configuration on all CX3 cards with a simple firmware flash. Here’s how.
Disclaimer:
- Flashing firmware may brick your card and I’m not responsible for it
- Do not power off or reboot your computer during firmware flashing
Preparation
I’ll assume you are using Windows Server 2019 with all the latest patches; all programs are available under Linux too so you can apply all the instructions for a Linux server with only minor deviations.
All the download links are for ConnectX-3 non-pro version; If you have a ConnectX-2 or ConnectX-3 Pro, you need to download a different firmware.
Downloads
Download and install Mellanox firmware tools. Install ALL the components (some of them are not selected by default).
Download the ConnectX-3 VPI firmware package. EN firmware can only run Ethernet; VPI firmware run both Ethernet and IB. Thus, VPI firmware is always recommended.
Check Your Card
First of all, we need the card’s machine-dependent device ID. This device ID is used for all the commands below.
PS C:\Users\Administrator> mst status
MST devices:
------------
mt4099_pci_cr0
mt4099_pciconf0
In this case, the card’s ID is mt4099_pci_cr0 . Then we make sure the card communicates:
PS C:\Users\Administrator> flint -d mt4099_pci_cr0 query
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: [redacted]
MACs: [redacted]
VSD:
PSID: MT_1090120019
Now we are good to go.
Firmware Backup
ALWAYS make a backup so you can go back when something went wrong.
flint -d <id> query full > flint_query.txt
flint -d <id> hw query > flint_hwinfo.txt
flint -d <id> ri orig_firmware.mlx
flint -d <id> dc orig_firmware.ini
flint -d <id> rrom orig_rom.mlx
mlxburn -d <id> -vpd > orig_vpd.txt
Manually open all the txt & ini files to make sure they are not empty and no error messages exist.
Firmware Flashing
Unzip the firmware downloaded from Mellanox, change the file extension from .bin to .mlx , then verify the firmware is OK:
PS C:\Users\Administrator\Desktop> flint -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx query full
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
MIC Version: 2.0.0
Config Sectors: 1
PRS Name: cx3-1_MCX354A_fdr_09v.prs
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050
MACs: 0002c9000001 0002c9000002
VSD: n/a
PSID: MT_1090120019
PS C:\Users\Administrator\Desktop> flint -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx verify
FS2 failsafe image. Start address: 0x0. Chunk size 0x80000:
NOTE: The addresses below are contiguous logical addresses. Physical addresses on
flash may be different, based on the image start address and chunk size
/0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
/0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
/0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
/0x00003924-0x0001d14b (0x019828)/ (ROM) - OK
/0x0001d14c-0x0001d18f (0x000044)/ (GUID) - OK
/0x0001d190-0x0001d31b (0x00018c)/ (Image Info) - OK
/0x0001d31c-0x0002a6b3 (0x00d398)/ (DDR) - OK
/0x0002a6b4-0x0002b707 (0x001054)/ (DDR) - OK
/0x0002b708-0x0002bad7 (0x0003d0)/ (DDR) - OK
/0x0002bad8-0x00069aaf (0x03dfd8)/ (DDR) - OK
/0x00069ab0-0x0006e933 (0x004e84)/ (DDR) - OK
/0x0006e934-0x00072e07 (0x0044d4)/ (DDR) - OK
/0x00072e08-0x000738ff (0x000af8)/ (DDR) - OK
/0x00073900-0x000a29f3 (0x02f0f4)/ (DDR) - OK
/0x000a29f4-0x000a659f (0x003bac)/ (DDR) - OK
/0x000a65a0-0x000bb493 (0x014ef4)/ (DDR) - OK
/0x000bb494-0x000bb59b (0x000108)/ (DDR) - OK
/0x000bb59c-0x000c061f (0x005084)/ (DDR) - OK
/0x000c0620-0x000c1e1b (0x0017fc)/ (Configuration) - OK
/0x000c1e1c-0x000c1e8f (0x000074)/ (Jump addresses) - OK
/0x000c1e90-0x000c280f (0x000980)/ (FW Configuration) - OK
/0x00000000-0x000c280f (0x0c2810)/ (Full Image) - OK
-I- FW image verification succeeded. Image is bootable.
Make sure the PSID from the first command is the target PSID (model number) you want; and make sure the second command outputs “Image is bootable”. Then we proceed to actually burn the card. (For demonstration purposes, I burned firmware with a different PSID.)
PS C:\Users\Administrator\Desktop> flint -d mt4099_pci_cr0 -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-QCB_Ax-FlexBoot-3.4.752.mlx -allow_psid_change burn
Current FW version on flash: 2.42.5000
New FW version: 2.42.5000
Note: The new FW version is the same as the current FW version on flash.
Do you want to continue ? (y/n) [n] : y
You are about to replace current PSID on flash - "MT_1090120019" with a different PSID - "MT_1090110018".
Note: It is highly recommended not to change the PSID.
Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature - OK
Reboot to use the new firmware.
FAQ/Miscellaneous
Firmware Compatibility?
In theory, all firmware for the same chip are compatible with all the cards with that chip. In practice, only flash a firmware that contains the same physical configuration as your card. For example, if your card have one physical port, do not flash any 2-port firmware. If your card have SFP+ ports, do not flash any QSFP firmware.
How to Switch Port Protocol (Ethernet/IB)
In devmgmt.msc , expand System Devices, double click on Mellanox ConnectX-3 VPI (MT04099) Network Adapter, go to Port Protocol tab, uncheck HW Defaults, select your desired port type. It should look like this:
Unable to Switch Port Protocol
If the port protocol is locked to Ethernet or IB and the options are all grayed out, it is likely that some boot-time configuration is set incorrectly. First read them out:
PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 query
Device #1:
----------
Device type: ConnectX3
Device: mt4099_pci_cr0
Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 IB(1)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)
For link type fields, 1 means IB only, 2 means Ethernet only, 3 means online configurable. In this case, we configure them all to 3:
PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 set LINK_TYPE_P1=3 LINK_TYPE_P2=3
Device #1:
----------
Device type: ConnectX3
Device: mt4099_pci_cr0
Configurations: Next Boot New
LINK_TYPE_P1 IB(1) VPI(3)
LINK_TYPE_P2 IB(1) VPI(3)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Reboot the machine and profit.
SR-IOV？
mlxconfig -d <id> set SRIOV_EN=1 NUM_OF_VFS=8
Then reboot. For detailed instructions, see the official documentation.
DO NOT configure too many VFs (e.g. NUM_OF_VFS=64 ) as the card might become IB only on that configuration even if it is in VPI mode (mode 3).
“Compile” the Firmware?
It’s known that old CX3 firmware need to be “compiled” (basically you bake a configuration into the firmware) then be flashed. New stock firmware doesn’t need this process anymore. For the completeness of this article, I’ll put the commands below.
# Compile
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.mlx
# Verify
flint -i mlnx_firmware.mlx query full
flint -i mlnx_firmware.mlx verify
# Flash
flint -d <id> -i mlnx_firmware.mlx -allow_psid_change burn
The ini files are available for download here.
OEM Card Firmware?
If you really want OEM (HP, IBM, etc.) ConnectX-3 firmware for some reason, download from here.
Bricked Card?
If you unfortunately have a bricked card, power off, try to find the DFU pins (2-pin jumper on the card), short them, then power on the computer. The card will boot in DFU mode and you might be able to flash a new firmware onto it. Then power off the computer, disconnect DFU jumper, power on again.