Open Fabrics Enterprise Distribution (OFED) Version 1.4 README December 2008 This is the OpenFabrics Enterprise Distribution (OFED) version 1.4 software package supporting InfiniBand and iWARP fabrics. It is composed of several software modules intended for use on a computer cluster constructed as an InfiniBand subnet or an iWARP network. *** Note: If you plan to upgrade OFED on your cluster, please upgrade all its nodes to this new version. This document includes the following sections: 1. HW and SW Requirements 2. OFED Package Contents 3. Installing OFED Software 4. Starting and Verifying the IB Fabric 5. MPI (Message Passing Interface) 6. Related Documentation OpenFabrics Home Page: http://www.openfabrics.org The OFED rev 1.4 software download available in http://www.openfabrics.org/builds/ofed-1.4/release/ Please email bugs and error reports to your InfiniBand vendor, or use bugzilla https://bugs.openfabrics.org/ 1. HW and SW Requirements: ========================== 1) Server platform with InfiniBand HCA or iWARP RNIC (see OFED Distribution Release Notes for details) 2) Linux operating system (see OFED Distribution Release Notes for details) 3) Administrator privileges on your machine(s) 4) Disk Space: - For Build & Installation: 300MB - For Installation only: 200MB 5) For the OFED Distribution to compile on your machine, some software packages of your operating system (OS) distribution are required. These are listed here. OS Distribution Required Packages --------------- ---------------------------------- General: o Common to all gcc, glib, glib-devel, glibc, glibc-devel, glibc-devel-32bit (to build 32-bit libraries on x86_86 and ppc64), zlib-devel o RedHat, Fedora kernel-devel, rpm-build o SLES 9.0 kernel-source, udev, rpm o SLES 10.0 kernel-source, rpm Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit glibc-devel should be installed. Specific Component Requirements: o Mvapich a Fortran Compiler (such as gcc-g77) o Mvapich2 libstdc++-devel, sysfsutils (SuSE), libsysfs-devel (RedHat5.0, Fedora C6) o Open MPI libstdc++-devel o ibutils tcl-8.4, tcl-devel-8.4, tk, libstdc++-devel o tvflash pciutils-devel o mstflint libstdc++-devel (32-bit on ppc64), gcc-c++ Note: The installer will warn you if you attempt to compile any of the above packages and do not have the prerequisites installed. *** Important Note for open-iscsi users: Installing iSER as part of OFED installation will also install open-iscsi. Before installing OFED, please uninstall any open-iscsi version that may be installed on your machine. Installing OFED with iSER support while another open-iscsi version is already installed will cause the installation process to fail. 2. OFED Package Contents ======================== The OFED Distribution package generates RPMs for installing the following: o OpenFabrics core and ULPs - HCA drivers (mthca, mlx4, ipath, ehca) - iWARP driver (cxgb3, nes) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER Initiator and target, RDS, uDAPL, qlgc_vnic and NFS-RDMA. o OpenFabrics utilities - OpenSM: InfiniBand Subnet Manager - Diagnostic tools - Performance tests o MPI - OSU MVAPICH stack supporting the InfiniBand and iWARP interface - Open MPI stack supporting the InfiniBand and iWARP interface - OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface - MPI benchmark tests (OSU BW/LAT, Intel MPI Benchmark, Presta) o Extra packages - open-iscsi: open-iscsi initiator with iSER support - ib-bonding: Bonding driver for IPoIB interface o Sources of all software modules (under conditions mentioned in the modules' LICENSE files) o Documentation 3. Installing OFED Software ============================ The default installation directory is: /usr Install Quick Guide: 1) Download and extract: tar xzvf OFED-1.4.tgz file. 2) Change into directory: cd OFED-1.4 3) Run as root: ./install.pl 4) Follow the directions to install required components. For details, please see OFED_Installation_Guide.txt under OFED-1.4/docs. Notes: 1. The install script removes previously installed IB packages and re-installs from scratch. You will be prompted to acknowledge the deletion of the old packages. However, configuration files (.conf) will be preserved and saved with a ".rpmsave" extension. 2. After the installer completes, information about the OFED installation such as the prefix, the kernel version, and installation parameters can be found by running /etc/infiniband/info. 3. Information on the driver version and source git trees can be found using the ofed_info utility 4. Starting and Verifying the IB Fabric ======================================= 1) If you rebooted your machine after the installation process completed, IB interfaces should be up. If you did not reboot your machine, please enter the following command: /etc/init.d/openibd start 2) Check that the IB driver is running on all nodes: ibv_devinfo should print "hca_id: " on the first line. 3) Make sure that a Subnet Manager is running by invoking the sminfo utility. If an SM is not running, sminfo prints: sminfo: iberror: query failed If an SM is running, sminfo prints the LID and other SM node information. Example: sminfo: sm lid 0x1 sm guid 0x2c9010b7c2ae1, activity count 20 priority 1 To check if OpenSM is running on the management node, enter: /etc/init.d/opensmd status To start OpenSM, enter: /etc/init.d/opensmd start Note: OpenSM parameters can be set via the file /etc/opensm/opensm.conf 4) Verify the status of ports by using ibv_devinfo: all connected ports should report a "PORT_ACTIVE" state. 5) Check the network connectivity status: run ibchecknet to see if the subnet is "clean" and ready for ULP/application use. The following tools display more information in addition to IB info: ibnetdiscover, ibhosts, and ibswitches. 6) Alternatively, instead of running steps 3 to 5 you can use the ibdiagnet utility to perform a set of tests on your network. Upon finding an error, ibdiagnet will print a message starting with a "-E-". For a more complete report of the network features you should run ibdiagnet -r. If you have a topology file describing your network you can feed this file to ibdiagnet (using the option: -t ) and all reports will use the names they appear in the file (instead of LIDs, GUIDs and directed routes). 7) To run an application over SDP set the following variables: env LD_PRELOAD='stack_prefix'/lib/libsdp.so LIBSDP_CONFIG_FILE='stack_prefix'/etc/libsdp.conf (or LD_PRELOAD='stack_prefix'/lib64/libsdp.so on 64 bit machines) The default 'stack_prefix' is /usr 5. MPI (Message Passing Interface) ================================== In Step 2 of the main menu of install.pl, options 2, 3 and 4 can install one or more MPI stacks. Multiple MPI stacks can be installed simultaneously -- they will not conflict with each other. Three MPI stacks are included in this release of OFED: - MVAPICH 1.1.0 - Open MPI 1.2.8 - MVAPICH2 1.2p1 OFED also includes 4 basic tests that can be run against each MPI stack: bandwidth (bw), latency (lt), Intel MPI Benchmark and Presta. The tests are located under: /mpi///tests/. Please see MPI_README.txt for more details on each MPI package and how to run the tests. 6. Related Documentation ======================== 1) Release Notes for OFED Distribution components are to be found under OFED-1.4/docs and, after the package installation, under /usr/share/doc/ofed-docs-1.4 for RedHat /usr/share/doc/packages/ofed-docs-1.4 for SuSE. 2) For a detailed installation guide, see OFED_Installation_Guide.txt. 3) For more information, please visit the OFED web-page http://www.openfabrics.org For more information contact your vendor.