Open Fabrics Enterprise Distribution (OFED) Version 3.18-3 Release Notes Feb 2017 =============================================================================== Table of Contents =============================================================================== 1. Overview, which includes: - OFED Distribution Rev 3.18-3 Contents - Supported Platforms and Operating Systems - Supported HCA and RNIC Adapter Cards and Firmware Versions - Tested Switch Platforms - Third party Test Packages - OFED sources 2. Change log 3. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of OpenFabrics Enterprise Distribution (OFED) release 3.18-3. The OFED software package is composed of several software modules, and is intended for use on a computer cluster constructed as an InfiniBand Fabric, an iWARP Network or a RoCE Fabric. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. 1.1 OFED 3.18-3 Contents ----------------------- The OFED package contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, mlx5, qib, ehca) - iWARP RNIC driver (cxgb3, cxgb4, nes) - RoCE drivers (mlx4, ocrdma) - IB core - Upper Layer Protocols: IPoIB, SRP Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA*. - OpenFabrics utilities: - OpenSM (OSM): InfiniBand Subnet Manager - Diagnostic tools - Performance tests - Extra packages: - infinipath-psm: Performance-Scaled Messaging API, an accelerated interface to Intel(R) HCAs - Packages for Intel(R) Xeon Phi(TM) coprocessor systems (libibscif, ibpd) - IBSCIF Driver (ibscif) - libfbric - library that exports interfaces for fabric services to applications - CCL-Direct host-side drivers for Intel(R) Xeon Phi(TM) coprocessor systems: - HCA proxy (ibp_server) - Connection Manager proxy (ibp_cm_server) - Subnet Administrator proxy (ibp_sa_server) - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) - Documentation 1.2 Supported Platforms and Operating Systems --------------------------------------------- o CPU architectures: - x86_64 - x86 - ppc64 o Linux Operating Systems: - RedHat EL6.5 2.6.32-431.el6 - RedHat EL6.6 2.6.32-504.el6 - RedHat EL6.7 2.6.32-573.el6 - RedHat EL6.8 2.6.32-642.el6 - RedHat EL7.0 3.10.0-123.el7 - RedHat EL7.1 3.10.0-229.el7 - RedHat EL7.2 3.10.0-327.el7 - SLES11 SP3 3.0.76-0.9.1 - SLES11 SP4 3.0.101-63 - SLES12 3.12.28-4 - SLES12.1 3.12.49-11.1 - kernel.org 3.18 * * Minimal QA for these versions. 1.3 HCAs and RNICs Supported ---------------------------- This release supports IB HCAs by IBM, Intel and Mellanox Technologies, iWARP RNICs by Chelsio Communications and Intel and RoCE adapters by Emulex, IBM and Mellanox. InfiniBand Adapters o IBM HCAs: - GX Dual-port SDR 4x IB HCA - GX Dual-port SDR 12x IB HCA - GX Dual-port DDR 4x IB HCA - GX Dual-port DDR 12x IB HCA o Intel (formerly QLogic) HCAs: - Intel(R) True Scale DDR PCIe x8 and x16 HCAs - Intel(R) True Scale QDR PCIe x8 Gen2 HCAs o Mellanox Technologies HCAs (SDR, DDR and QDR Modes are Supported): - ConnectX(R) and ConnectX EN (Rev 2.9.1000) - ConnectX-2 (Rev 2.9.1200 and above) o Mellanox Technologies HCAs (FDR and FDR10 Modes are Supported): - ConnectX-3 (Rev 2.33.5100 and above) - ConnectX-3 Pro (Rev 2.33.5100 and above) o Mellanox Technologies HCAs (FDR and FDR10 Modes are Supported): - Connect-IB (Rev 10.10.5054 and above) For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download iWARP Adapters o Chelsio RNICs: - S310/S320 10GbE Storage Accelerators - R310/R320 10GbE iWARP Adapters - T4: T420-CR, T440-CR, T422-CR, T404-BT, T440-LP-CR, T420-LL-CR, T420-CX - T5: T502-BT, T580-CR, T580-LP-CR, T520-LL-CR, T520-CR, T522-CR, T540-CR o Intel RNICs: - NE020 10Gb iWARP Adapter RoCE Adapters o Emulex - using ocrdma - Emulex OCe14102 2-port 10 GbE RoCE - Emulex OCe14401 1-port 40 GbE RoCE o IBM - IBM Flex System EN4132 2-port 10 GbE RoCE - IBM EL27 PCIe LP 2-Port 10GbE RoCE SFP+ adapter - IBM EC28 PCIe 2-Port 10GbE RoCE SFP+ adapter o Mellanox - ConnectX-2 EN (Rev 2.9.1200 and above) - ConnectX-3 EN (Rev 2.31.5050 and above) 1.4 Switches Supported ---------------------- This release was tested with switches and gateways provided by the following companies: InfiniBand Switches o Flextronics - F-X430044 o Intel (formerly QLogic) - 12200 o Mellanox - MLNX-OS MSX6036/SX6025 w/w MLNX-OS version 3.3.4304 - Grid Director 4036 w/w Grid Director version 3.9.2-992 - FabricIT EFM IS5035 w/w FabricIT EFM version 1.1.3000 - FabricIT BXM MBX5020 w/w FabricIT BXM version 2.1.2000 iWARP Switches o Fujitsu - XG2000C 10Gb Ethernet Switch RoCE Switches o Arista o BLADE Network Technologies (BNT) o Mellanox - SX1036 - SX1024 - SX1016 1.5 Third Party Packages ------------------------ The following third party packages have been tested with OFED 3.18-3: - Open MPI - 1.8 - Intel MPI 4.1.3 - MVAPICH2 1.6 OFED Sources ---------------- All sources are located under git://git.openfabrics.org/ Linux: ------ URL: git://git.openfabrics.org/compat-rdma/linux-3.18.git Branch: master - Linux kernel sub-tree that includes files relevant for the OFED project only. Based on v3.18. Used to shorten git clone time. Note: the regular Linux git tree can be used as well. compat: ------- URL: git://git.openfabrics.org/compat-rdma/compat.git Branch: ofed - Based on compat project (https://github.com/mcgrof/compat). The compat module provides functionality introduced in newer kernels to older kernels through a set of header files and exported symbols. See https://github.com/mcgrof/compat/wiki for details. - Used to replace kernel_addons in the previous OFED kernel tree. compat-rdma: ------------ URL: git://git.openfabrics.org/compat-rdma/compat-rdma.git Branch: master User level Sources are downloaded from http://www.openfabrics.org/downloads/ as written in the BUILD_ID The kernel sources are based on Linux 3.18 mainline kernel. Its patches are included in the OFED sources directory. For details see HOWTO.build_ofed. The list of maintainers is available under: http://www.openfabrics.org/downloads/MAINTAINERS =============================================================================== 2. Change log =============================================================================== OFED-3.18-3-RC1 Main Changes from OFED 3.18-2 ------------------------------------------------------------------------------- 1. Added support for RHEL 6.8 2. install.pl: Fixed bug 2598 OFED-3.18-2 Main Changes from OFED 3.18-2-rc3 ------------------------------------------------------------------------------- - Updated docs OFED-3.18-2-rc3 Main Changes from OFED 3.18-2-rc2 ------------------------------------------------------------------------------- 1. Updated packages: - ibsim-0.7.tar.gz - mstflint-4.4.0 - opensm-3.3.20 - perftest-3.0-0.21.g21dc344 - rds-tools-2.0.7 2. compat-rdma changes - xeon-phi: add dependency for scif.symvers for ibscif OFED-3.18-2-rc2 Main Changes from OFED 3.18-2-rc1 ------------------------------------------------------------------------------- 1. Updated packages: - ibpd-1.0.1 - libiwpm-1.0.5 - mstflint-4.3.0 2. compat-rdma changes - IB/security: Restrict use of the write() interface - xeon-phi: adjust mlx5_ib bug fix releated to large memory region registration OFED-3.18-2-rc1 Main Changes from OFED 3.18-1 ------------------------------------------------------------------------------- 1. Updated packages: - dapl-2.1.9 - fabtests-1.3.0 - ibacm-1.2.1 - infinipath-psm-3.3-19_g67c0807_open - libcxgb4-1.3.6 - libfabric-1.3.0 - libocrdma-1.0.7 - librdmacm-1.1.0 - mstflint-4.3.0-1.49.g9b9af70 - perftest-3.0-0.16.gb2f2e82 2. compat-rdma changes - xeon-phi: allow full ibscif ud mtu - xeon-phi: allow rdma_bind to mic0 device - NFSRDMA: RHEL7.2 backport - cxgb4: update firmware to 1.4.14.0 - cxgb4: RHEL7.2 backports - cxgb4: fix up bonding.h include paths - IB/ipoib: Do not update neigh validity if not resolved yet - mlx4_en: Fixed compilation on RHEL6.6 - makefile: Added distclean target - xeon-phi: mlx5_ib bug fix releated to large memory regions registration. - RDMA/ocrdma: ocrdma sync up patch series - be2net: backport fixes for RHEL 7.2 - mlx4: Added support for RHEL7.2 OFED-3.18-1 Main Changes from OFED 3.18-1-rc2 ------------------------------------------------------------------------------- 1. Updated packages: - libocrdma-1.0.6 - perftest-3.0-0.12 OFED-3.18-1-rc2 Main Changes from OFED 3.18-1-rc1 ------------------------------------------------------------------------------- 1. Updated packages: - dapl-2.1.7 - infiniband-diags-1.6.6 - infinipath-psm-3.3-7_g05f6f14_open - libiwpm-1.0.3 - libfabric-1.1.1 - fabtests-1.1.1 2. compat-rdma changes - xeon-phi: ibscif bug 2575: Allow xeon-phi support for Knights Landing - xeon-phi: ibscif bug 2570: correction to communitate between host and card OFED-3.18-1-rc1 Main Changes from OFED 3.18 ------------------------------------------------------------------------------- 1. Updated packages: - infinipath-psm-3.3-6_gca75825_open - dapl-2.1.6 - ibacm-1.1.0 - libfabric-1.1.0rc4 - libiwpm-1.0.3rc1 - mstflint-4.1.0-1.45.g1d75d01 - perftest-3.0-0.11.g9bd522a - fabtests-1.1.0rc2 2. compat-rdma - Added RHEL7.1 support on ppc64le OFED-3.18 Main Changes from OFED 3.18-rc3 ------------------------------------------------------------------------------- 1. Updated packages: - libiwpm-1.0.2 - mstflint-4.0.1-1.43.g97d7275 - perftest-3.0-0.9.g214990b 2. compat-rdma changes - Adding backport to neigh resolve fix for RHEL 6.5 and RHEL 6.6 - ibscif: Fixed bring up of scif device on kernels >= 3.11 - openibd: Unload in-box drivers upon boot - RDMA/nes: Fix for incorrect recording of the MAC address - RDMA/nes: Fix for resolving the neigh - RDMA/iwpm: Updated fixes for the client registration functionality OFED-3.18-rc3 Main Changes from OFED 3.18-rc2 ------------------------------------------------------------------------------- 1. Updated packages: - dapl-2.1.5 - fabtests-1.0.0 - libfabric-1.0.0 - librdmacm-1.0.21 - mstflint-4.0.1-1.42.g0d3b372 2. compat-rdma changes - RDMA/ocrdma: insert Dual/BSD license. - IB/ipoib: Change To max_cm_mtu when changing mode to connected - RDMA/ocrdma : ocrdma upstream sync up patch. - NFS/RDMA: fix for RHEL7.1 - be2net: Fixing the build failure for SLES11 SP4 - Fix XEON PHI compilation - compat-rdma/linux-next-pending: Adding changes to enable the use of the tos field for iwarp providers - gen-compat-config.sh: Added XPS related staff - openibd: Check if iwpmd is running before unloading modules - compat-rdma.spec: Updated content of compat-rdma-devel RPM - compat-rdma: add backport to fix ofed-bz2539 - compat-rdma: add backport to fix ofed-bz2539 - Fixed openibd stop on OSes using systemd - compat-rdma.spec: Fixed openibd header - mlx4/5: Added bug fixes - IB/core: Added fixes from upstream - ibdev2netdev: Added dev_port support - mlx4/backport: Fixed dev_id - iw_cxgb4: use wildcard mapping for getting remote addr info - cxgb4: update firmware to 1.3.32 - compat-rdma.spec: Added systemd support for all relevant OSes OFED-3.18-rc2 Main Changes from OFED 3.18-rc1 ------------------------------------------------------------------------------- 1. Updated packages: - dapl-2.1.4 - libfabric-1.0.0rc4 - librdmacm-1.0.20 2. Added RHEL7.1 support 3. compat-rdma changes - IB/uverbs: Prevent integer overflow in ib_umem_get address arithmetic - NFSoRDMA: backport for RHEL 6.5 and 6.6 - Fixed mlx4 backport - Added RHEL7.1 backport patches - RDMA/ocrdma: rebasing the upstream sync up patch - be2net-ocrdma: Fixing the RH 6.5/6 backport patch - be2net-ocrdma: move backport patches to the correct folder - Updated XEON-Phi patches - Updated compat-rdma.spec for XEON-Phi - NFS/RDMA: SLES11SP3 backport OFED-3.18-rc1 Main Changes from OFED 3.12-1 GA ------------------------------------------------------------------------------- 1. Updated packages: - compat-rdma-3.18 - dapl-2.1.3 - infiniband-diags-1.6.5 - libfabric-1.0.0rc2 - libibmad-1.3.12 - libibumad-1.3.10.2 - libiwpm-1.0.1 - libmlx5-1.0.2 - libocrdma-1.0.5 - mstflint-3.8.0-1.27.gf3d39b6 - opensm-3.3.19 - perftest-2.4-0.8.gd3c2b22 - srptools-1.0.3 =============================================================================== 3. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of the OFED 3.18-3 release. 01. When upgrading from an earlier OFED version, the installation script does not stop the earlier OFED version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to OFED 3.18-3 or reboot the server after OFED installation. 02. Memory registration by the user is limited according to administrator setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for system configuration. 03. Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. 04. The qib driver is supported only on 64-bit platforms. 05. IPoIB: brctl utilities do not work on IPoIB interfaces. The reason for that is that these utilities support devices of type Ethernet only. 06. In case uninstall is failing, check the error log and remove the remaining RPMs manually using 'rpm -e '. 07. On SLES11.x, set allow_unsupported_modules parameter to 1 in file: /etc/modprobe.d/unsupported-modules. Without this the modules will not load. 08. RDS is not supported. 09. Bug 2515: when an Intel HCA is attached directly to a Mellanox ConnectX3 and the OpenSM is started on the Intel HCA, the link will not go to active. The workaround is to start the OpenSM on the Mellanox HCA. 10. Bug 2553: Unable to run UD traffic using qperf on a RoCE R-NIC 11. Bug 2561: Devices do not receive the echo reply of the first ping sequence when the payload size is 65493 or higher. This is still an issues in the 4.3 kernel and should be resolved in a future OFED 4.x release 12. Bug 2574: Installing OFED (with --with_xeon_phi and qib=y in ofed.conf file) makes the access to the scif char device fail. psmd uses scif functions to allow psm to connect to the Phi devices. It also makes sure it can connect to them when the service is started. Scif functions cannot be called unless the mpss service is running. Since openibd starts the psmd service, it should be loaded after mpss 13. The ipath and eHCA drivers will be deprecated in the next OFED build that uses a kernel >= 4.3 14. Bug 2583: "iwpm_mapping_error_cb: Received msg seq = 381240 err code = 12 client = 3" seen intermittently 15. Bug 2587: autoconf is needed by ibacm-1.1.0-1.x86_64 16. Bug 2596: Host crash related to mlx5 driver, ConnectIB HCAs, ib_srp, and performing IO 17. Bug 2600: SRPT compilation fails on kernels from kernel.org starting: 3.18.33 18. Bug 2602: RHEL 6.6 crashed during Switch-to-controller cable pull less than failover Note: See the release notes of each component for additional issues.