Open Fabrics Enterprise Distribution (OFED) ConnectX driver (mlx4) in OFED 1.4 Release Notes December 2008 =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Supported Firmware Versions 3. VPI (Virtual Process Interconnect) 4. Infiniband new features and bug fixes 5. Known Issues =============================================================================== 1. Overview =============================================================================== mlx4 is the low level driver implementation for the ConnectX adapters designed by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter, as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports Infiniband and Ethernet NIC configurations. To accommodate the supported configurations, the driver is split into three modules: - mlx4_core Handles low-level functions like device initialization and firmware commands processing. Also controls resource allocation so that the InfiniBand and Ethernet functions can share the device without interfering with each other. - mlx4_ib Handles InfiniBand-specific functions and plugs into the InfiniBand midlayer - mlx4_en A new 10G driver named mlx4_en was added to drivers/net/mlx4. It handles Ethernet specific functions and plugs into the netdev mid-layer. =============================================================================== 2. Supported Firmware Versions =============================================================================== - This release was tested with FW 2.6.000. - The minimal version to use is 2.3.000. - To use both IB and Ethernet use FW version 2.6.0 =============================================================================== 3. VPI (Virtual Protocol Interconnect) =============================================================================== VPI enables ConnectX to be configured as an Ethernet NIC and/or an Infiniband adapter. o Overview: The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and Infiniband drivers. It supplies the user with the ability to run Infiniband and Ethernet protocols on the same HCA (separately or at the same time). For more details on the Ethernet driver see MLNX_EN_README.txt. o Firmware: The VPI driver works with FW 25408 version 2.6.000 or higher. One needs to use INI files that allow different protocols over same HCA. o Installing the mlnx_en driver: To install the OFED 1.4 with the Ethernet driver make sure that "mlx4_en=y" in file ofed.conf o Loading drivers: To load the Ethernet driver one should set "MLX4_EN_LOAD=yes" in file /etc/infiniband/openib.conf. If "MLX4_EN_LOAD" is not marked as "yes", the Ethernet driver can be loaded by running "/sbin/modprobe mlx4_en". o Port type management: By default both ConnectX ports are initialized as Infiniband ports. If you wish to change the port type use the connectx_port_config script after the driver is loaded. Running "/sbin/connectx_port_config -s" will show current port configuration for all ConnectX devices. Port configuration is saved in file: /etc/infiniband/connectx.conf. This saved configuration is restored at driver restart only if done via "/etc/init.d/openibd restart". Possible port types are: "eth" - Always Ethernet. "ib" - Always Infiniband. "auto" - Link sensing mode - detect port type based on the attached network type. If no link is detected, the driver retries link sensing every few seconds. Port link type can be configured for each device in the system at run time using the "/sbin/connectx_port_config" script. This utility will prompt for the PCI device to be modified (if there is only one it will be selected automatically). At the next stage the user will be prompted for the desired mode for each port. The desired port configuration will then be set for the selected device. Note: This utility also has a non interactive mode: "/sbin/connectx_port_config [[-d|--device ] -c|--conf ]". - The following configurations are supported by VPI: Port1 = eth Port2 = eth Port1 = ib Port2 = ib Port1 = auto Port2 = auto Port1 = ib Port2 = eth Port1 = ib Port2 = auto Port1 = auto Port2 = eth Note: the following options are not supported: Port1 = eth Port2 = ib Port1 = eth Port2 = auto Port1 = auto Port2 = ib =============================================================================== 4. Infiniband new features and bug fixes =============================================================================== Features that are enabled with FW 2.5.0 only: - Send with invalidate and Local invalidate send queue work requests. - Resize CQ support. Features that are enabled with FW 2.6.0 only: - Fast register MR send queue work requests. - Local DMA L_Key. - Raw Ethertype QP support (one QP per port) -- receive only. Non FW dependent features: - Allow 4K messages for UD QPs. - Allocate/free fast register MR page lists. - More efficient MTT allocator. - RESET->ERR QP state transition no longer supported (IB Spec 1.2.1). - Pass congestion management class MADs to the HCA. - Enable firmware diagnostic counters available via sysfs. - Enable LSO support for IPOIB. - IB_EVENT_LID_CHANGE is generated more appropriately. - Fixed race condition between create QP and destroy QP (bugzilla 1389) =============================================================================== 5. Known Issues =============================================================================== - mlx4_en driver is not supported on PPC64 and IA64 - The mlx4_en module uses a Linux implementation for Large Receive Offload (LRO) in kernel 2.6.24 and later. These kernels require installing the "inet_lro" module. - The SQD feature is not supported: - To load the driver on machines with 64KB default page size UAR bar must be enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium with 64KB page size enabled. Perform the following three steps: 1. Add the following line in the firmware configuration (INI) file under the [HCA] section: log2_uar_bar_megabytes = 5 2. Burn a modified firmware image with the changed INI file. 3. Reboot the system. =============================================================================== 6. mlx4 Available Parameters =============================================================================== In order to set mlx4 parameters, add the following line(s) to /etc/modpobe.conf: options mlx4_core parameter= and/or options mlx4_ib parameter= and/or options mlx4_en parameter= mlx4_core parameters: msi_x: attempt to use MSI-X if nonzero (default 1) enable_qos: Enable Quality of Service support in the HCA if > 0, (default 0) block_loopback Block multicast loopback packets if > 0 (default: 1) internal_err_reset: Reset device on internal errors if non-zero (default 1) debug_level: Enable debug tracing if > 0 (default 0) log_num_qp: log maximum number of QPs per HCA (default is 17; max is 20) log_num_srq: log maximum number of SRQs per HCA (default is 16; max is 20) log_rdmarc_per_qp: log number of RDMARC buffers per QP (default is 4; max is 7) log_num_cq: log maximum number of CQs per HCA (default is 16 max is 19) log_num_mcg: log maximum number of multicast groups per HCA (default is 13; max is 21) log_num_mpt: log maximum number of memory protection table entries per HCA (default is 17; max is 20) log_num_mtt: log maximum number of memory translation table segments per HCA (default is 20; max is 20) log_num_mac: log maximum number of MACs per ETH port (1-7) (int) log_num_vlan: log maximum number of VLANs per ETH port (0-7) (int) use_prio: Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool) mlx4_ib parameters: debug_level: Enable debug tracing if > 0 (default 0) mlx4_en parameters: rss_xor: Use XOR hash function for RSS 0 (default is xor) rss_mask: RSS hash type bitmask (default is 0xf) num_lro: Number of LRO sessions per ring or disabled (0) (default is 32) pptx: Pause policy on TX: 0 never generate pause frames 1 generate pause frames according to RX buffer threshold (default is 1) pprx: Pause policy on RX: 0 ignore received pause frames 1 respect received pause frames (default is 1) pfctx: Priority based Flow Control policy on TX[7:0]. Per priority bit mask (default is 0) pfcrx: Priority based Flow Control policy on RX[7:0]. Per priority bit mask (default is 0) rx_moder_cnt: Max coalesced descriptors for Rx interrupt moderation rx_moder_time: Timeout following last packet for Rx interrupt moderation auto_moder: Enable dynamic interrupt moderation (default is 1) rx_ring_num1: Number or Rx rings for port 1 (0 = #cores) (default is 0) rx_ring_num2: Number or Rx rings for port 2 (0 = #cores) (default is 0) tx_ring_size1: Tx ring size for port 1 (default is 1024) tx_ring_size2: Tx ring size for port 2 (default is 1024) rx_ring_size1: Rx ring size for port 1 (default is 1024) rx_ring_size2: Rx ring size for port 2 (default is 1024) inline_thold: treshold for using inline data (default is 128)