Dev.to AI 🤖 Ai 👁 0 📖 11 min read

Mellanox ConnectX-4/5/6/7 NIC Migration: Compatibility, Drivers and Fabric Impact

Migrating a fleet from ConnectX-4 to ConnectX-5, 6 or 7 is a slot-budget and driver-stack exercise before it is a bandwidth decision. Each generation roughly doubles the per-port rate, and each one also moves the PCIe re

Migrating a fleet from ConnectX-4 to ConnectX-5, 6 or 7 is a slot-budget and driver-stack exercise before it is a bandwidth decision. Each generation roughly doubles the per-port rate, and each one also moves the PCIe requirement up a tier: ConnectX-4 wants PCIe 3.0 x16 to feed one 100G port, ConnectX-6 Dx wants PCIe 4.0 x16 to feed 200G, and ConnectX-7 needs PCIe 5.0 x16 before a 400G port can run anywhere near line rate. If your server generation cannot give the card that slot, the upgrade will negotiate at the new speed and then bottleneck in the root complex. Match the fabric first, the slot second, and the card model third, and most of the migration risk disappears before you touch a driver.

What Actually Changes Between Generations

The four families share a programming model and a largely common toolset, which is why migrations tend to look deceptively easy. The differences that matter are physical and structural: a new cage form factor, a higher PCIe tier, a different default firmware branch, and in one specific case a change in what the silicon can do at all. Mellanox adapter naming carries the information if you read it correctly. The MCX4xxxx part numbers are ConnectX-4, MCX5xxxx are ConnectX-5, MCX6xxxx are ConnectX-6, and the trailing letters identify the interface options and the cooling and bracket variants.

The one structural break worth memorising is that ConnectX-6 Dx is an Ethernet-only part. It does not run InfiniBand. If a cluster still uses an InfiniBand fabric and someone proposes a Dx card because the price per port looks attractive, that proposal is wrong regardless of the rest of the specification. The VPI variants of ConnectX-4, ConnectX-5 and ConnectX-6 are the ones that can present either protocol.

Port Speed and PCIe Lane Budget

This is where most upgrade plans fail quietly. A single 100G port moves 12.5 GB/s of payload, a 200G port 25 GB/s, and a 400G port 50 GB/s. PCIe 3.0 x16 carries about 15.75 GB/s of raw bandwidth, which after encoding and transaction overhead leaves roughly 12 to 13 GB/s of usable throughput, so a Gen3 x16 slot is just barely enough for one saturated 100G port and not enough for two. That is why dual-port 100G adapters in Gen3 servers so often measure half of what the label promises.

PCIe 4.0 x16 doubles the raw figure to about 32 GB/s, which comfortably feeds 200G and covers dual-port 100G without contention. PCIe 5.0 x16 doubles again to roughly 64 GB/s, and 400G needs that headroom because a Gen4 x16 slot tops out below the 50 GB/s a single NDR400 port can consume. The practical rule: do not buy a port rate your slot cannot feed. A ConnectX-7 in a Gen4 slot is not a 400G adapter; it is an expensive 200G adapter.

The Comparison Table

Figures below follow NVIDIA's published adapter specifications. Where a family has multiple SKUs, the highest-rate variant is listed.

Family Max port rate Port form factor Host interface InfiniBand Typical fit
ConnectX-4 100 Gb/s QSFP28 PCIe 3.0 x16 EDR 100G (VPI SKUs) 100G leaf access, existing Gen3 servers
ConnectX-4 Lx 25 Gb/s per port SFP28 PCIe 3.0 x8 No 25G server access, storage nodes
ConnectX-5 100 Gb/s QSFP28 PCIe 3.0 x16 EDR / HDR100 (VPI SKUs) 100G with RoCEv2, general compute
ConnectX-5 Ex 100 Gb/s QSFP28 PCIe 4.0 x16 EDR / HDR100 Dual 100G at line rate on Gen4 hosts
ConnectX-6 Dx 200 Gb/s QSFP56 PCIe 4.0 x16 No (Ethernet only) 200G Ethernet, inline IPsec and MACsec
ConnectX-6 VPI 200 Gb/s QSFP56 PCIe 4.0 x16 HDR200 200G IB clusters, mixed-protocol estates
ConnectX-7 400 Gb/s QSFP112 PCIe 5.0 x16 NDR400 / NDR200 GPU fabric endpoints, 400G spine access

QSFP28, QSFP56 and QSFP112: What Fits Where

The cages are physically the same width and the newer ones accept the older modules, so a QSFP28 optic will seat in a QSFP56 or QSFP112 cage, and the port will simply train at the rate the module supports. The reverse is not true: a QSFP56 or QSFP112 module in a QSFP28 port either does not train or falls back to a lower rate, and it is a common source of "the link is up but slow" tickets. When a mixed fleet shares a cable plant, the module at each end sets the real ceiling for that link, not the port capability printed on the switch.

Backward compatibility is a convenience, not a strategy. Running a 400G ConnectX-7 as a 100G adapter through an old QSFP28 module wastes the exact capability you paid for, and the card may not expose the lower-rate options you assumed. For a staged migration, it is usually cheaper to buy the right generation of adapter for the speed the host can actually use, and reserve the 400G parts for nodes whose slot, switch port and workload all justify them. A 32-port 100G QSFP28 platform such as the SN2700 has no use for a 400G endpoint unless the plan is to replace the switch next.

Firmware, Drivers and the OFED Transition

Each generation sits on its own firmware branch, and firmware is not interchangeable across families, so a fleet with three generations will have three upgrade tracks to maintain. The tools are the same: mlxfwmanager and mstflint for firmware, mlxconfig for burning adapter configuration, mlxlink for reading link state and eye information, plus ibstat or mlx5_core behaviour under ethtool -m for module diagnostics. Learn those five and you can diagnose almost any generation without a vendor console.

The bigger change is the software stack. MLNX_OFED, the long-standing driver and library bundle, has been superseded by DOCA-OFED, and the older adapter families are supported on a narrower set of releases than the current ones. That matters in practice: a ConnectX-4 that ran happily for years on an older OFED may not have a matching build for a brand-new kernel, and it may fall back to the in-box rdma-core driver with fewer tuning knobs. In-box drivers cover basic RoCEv2 on ConnectX-4 and later, so a card does not become useless, but features like advanced congestion control, hardware timestamping options and some offload paths may be absent.

Treat the driver question as a gate, not an afterthought. Before ordering a batch of adapters, confirm three things for the exact OS and kernel you run: that a driver build exists, that the firmware branch is still receiving fixes, and that your boot path supports the card. UEFI and PXE support across generations is broadly consistent, but Secure Boot environments will reject an unsigned UEFI driver, and that failure appears at boot rather than at install time.

InfiniBand and Ethernet SKUs: The VPI Trap

A VPI adapter can present either protocol, which makes it flexible and makes second-hand purchasing confusing. Two cards with nearly identical part numbers can differ only in whether the InfiniBand stack is enabled, and the difference is not always visible in an auction listing. If the fabric is InfiniBand, verify the SKU against NVIDIA's product brief rather than assuming from the family name, and confirm the expected rate: ConnectX-4 tops out at EDR 100G, ConnectX-5 covers EDR and HDR100, ConnectX-6 VPI covers HDR200, and ConnectX-7 covers NDR200 and NDR400.

Rate naming is where buyers get caught out. HDR100 delivers 100 Gb/s over InfiniBand using two lanes of a QSFP56 port, so an adapter like the ConnectX-6 MCX653106A is a genuine 100G InfiniBand part despite sitting in the 200G generation. Reading the rate off the family name alone would lead you to expect double what the card delivers.

Switch Fabric Matching

An adapter upgrade only pays off when the port on the other end can use it. A 200G ConnectX-6 talking to a 100G leaf trains at 100G, which is fine if that was the plan and wasteful if it was not. Map the intended rate to the switch family before you commit: 100G QSFP28 leaves pair naturally with ConnectX-4, ConnectX-5 and the 100G SKUs of later generations; 200G QSFP56 leaves pair with ConnectX-6, including high-density platforms such as the SN4600 VS2RC; and 400G needs both a QSFP112-capable switch port and a Gen5 host slot.

There is also a cabling consequence that surprises people mid-project. Going from 100G to 200G usually means new modules at both ends, and going to 400G means new modules and often a different cable grade. If the plan is to reuse existing optics, the rate ceiling stays where it was, and the adapter upgrade delivers latency and feature improvements rather than bandwidth. That can still be worth doing, but it should be a deliberate choice rather than a discovery six weeks into a rollout.

NUMA, Slot Placement and Thermal Reality

Two identical adapters in the same server can differ by a large margin in measured latency purely because of which PCIe root complex they sit behind. On a dual-socket host, place the NIC in a slot attached to the same socket as the memory and the GPUs or accelerators that will drive it. Cross-socket traffic adds a hop that shows up in RDMA round-trip numbers and in GPU-direct paths, and no amount of driver tuning recovers it.

Cooling is the second physical constraint. These adapters are passively cooled and rely on server airflow, and the higher-rate parts dissipate more heat in the same footprint. A card designed for a front-to-back airflow chassis will run hot in a chassis with reversed or obstructed airflow, and thermal throttling looks exactly like a driver problem in the logs. Check the airflow direction of the bracket and heatsink variant before assuming a card from one platform will behave in another.

A Migration Sequence That Survives Contact With Production

Run the transition in four passes, and stop at any point where a pass fails. First, inventory the host fleet by PCIe tier and free slot, because that single table predicts which generations are even eligible. Second, pick one representative node per tier and validate end to end: firmware flash, driver install, module pairing, a bandwidth test with ib_write_bw or the equivalent perftest tool, and an application-level check. Third, standardise on one module type per speed band so the spares pool stays small. Fourth, roll out by rack or by role, not by convenience.

Keep a rollback path for each node: the previous adapter, the firmware version it was running, and the driver build that matched it. Firmware downgrades are possible but not always painless, and having the old card on the shelf is the cheapest insurance available. For fleets that mix generations for a year or more, document which node runs which firmware branch in the same place you document IP addressing, because the person diagnosing an intermittent link failure will not be the person who installed the card.

Choosing the Generation That Fits, Not the Fastest One

The 25G access tier is now the cheapest entry point, and a ConnectX-4 Lx-class adapter such as the MCX4121A still does that job with a Gen3 x8 slot, which means it fits servers that cannot host a 100G card at all. The 100G tier is where most refreshes land, and a dual-port QSFP28 adapter like the ConnectX-5 MCX516A covers it on either Gen3 or Gen4 hosts depending on the variant. The 200G tier belongs to ConnectX-6, and the 400G tier belongs to ConnectX-7 on Gen5 hosts with a matching fabric.

Buying one generation above your fabric is the most common waste in this market. The card negotiates down, the slot may not feed it, and the extra spend buys nothing you can measure. Buying one generation below your fabric is the second most common mistake: the switch ports sit idle at half their rated rate and the refresh comes round again a year later. The correct answer is always the generation that saturates its own port on the slot you actually have.

Frequently Asked Questions

Can I put a ConnectX-6 in a server that originally shipped with a ConnectX-4?

Usually yes, with two conditions. The slot must be PCIe 4.0 x16 if you want 200G at anything close to line rate, and the server must have BIOS and UEFI support for the card. Older platforms often enumerate a newer adapter correctly but fail to boot from it or expose reduced features. Test one node before buying a batch, and check the airflow variant matches the chassis direction.

Is the ConnectX-6 Dx usable on an InfiniBand fabric?

No. ConnectX-6 Dx is an Ethernet-only adapter and carries no InfiniBand capability. If the fabric is InfiniBand, you need a VPI SKU: ConnectX-6 VPI for HDR200, ConnectX-5 for EDR or HDR100, or ConnectX-7 for NDR. This distinction is easy to miss because the family name and the port form factor are identical.

Will a ConnectX-7 run at 400G in a PCIe 4.0 x16 slot?

It will run, but not at 400G. A PCIe 4.0 x16 link carries roughly 32 GB/s of raw bandwidth, while a saturated 400G port needs about 50 GB/s. The card negotiates down to the rate the host can sustain, so you get 200G-class throughput and pay for 400G silicon. Full 400G needs a Gen5 x16 slot.

Do I need DOCA-OFED, or is the in-box driver enough?

The in-box rdma-core driver supports RoCEv2 on ConnectX-4 and later and is adequate for straightforward IP-over-fabric and storage traffic. DOCA-OFED adds tuning knobs, newer offload paths and tooling that the in-box stack does not expose. For latency-sensitive or GPU-direct workloads, install the vendor stack and pin the version; for general-purpose servers, the in-box driver is often sufficient and far easier to maintain across kernel upgrades.

Can I mix ConnectX-4, 5, 6 and 7 adapters in the same RDMA fabric?

Yes, as long as each link trains at a rate both ends support and the congestion control settings are consistent. Practical problems come from mismatched firmware behaviour and from hosts with different PCIe tiers delivering very different throughput to the same application. Standardise firmware branches within a role group, keep RoCEv2 PFC and ECN settings identical cluster-wide, and expect the slowest host in a job to set the pace.

Where to Start

The cheapest way to plan this migration is to write down two columns per server: the PCIe tier of each free slot, and the switch port rate available to it. Everything else follows from that table. If you want a second opinion on a specific pairing before you commit budget, the adapter and switch listings on zhongxindata.hk carry the port types, form factors and host interfaces for each model, and you can send your slot inventory and target rate on WhatsApp to +86 13068616914 and we will flag the combinations that will negotiate down before you buy them.

Published by Zhongxin Data | November 2026

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.