Open Fabrics Enterprise Distribution (OFED) SRP in OFED 1.4 Release Notes December 2008 ============================================================================== Table of contents ============================================================================== 1. Overview 2. Changes and Bug Fixes since OFED 1.3.1 3. Software Dependencies 4. Major Features 5. Loading SRP Initiator 6. Manually Establishing an SRP Connection 7. SRP Tools - ibsrpdm and srp_daemon 8. Automatic Discovery and Connecting to Targets 9. Multiple Connections from Initiator IB Port to the Target 10. High Availability 11. Shutting Down SRP 12. Known Issues 13. Vendor Specific Notes ============================================================================== 1. Overview ============================================================================== The SRP standard describes the message format and protocol definitions required for transferring commands and data between a SCSI initiator port and a SCSI target port using RDMA communication service. ============================================================================== 2. Changes and Bug Fixes since OFED 1.3.1 ============================================================================== * Check for scsi_id in scmnd to prevent scan/rescan keep adding new scsi devices ie. echo "- - -" > /sys/class/scsi_host/hostXX/scan * Bug fixing ============================================================================== 4. Software Dependencies ============================================================================== The SRP Initiator depends on the installation of the OFED Distribution stack with OpenSM running. ============================================================================== 5. Major Features ============================================================================== This SRP Initiator is based on source taken from openib.org gen2 implementing the SCSI RDMA Protocol-2 (SRP-2), Doc. no. T10/1524-D. See: www.t10.org/ftp/t10/drafts/srp2/srp2r00a.pdf The SRP Initiator supports: - Basic SCSI Primary Commands -3 (SPC-3) (www.t10.org/ftp/t10/drafts/spc3/spc3r21b.pdf) - Basic SCSI Block Commands -2 (SBC-2) (www.t10.org/ftp/t10/drafts/sbc2/sbc2r16.pdf) - Basic functionality, task management and limited error handling ============================================================================== 6. Loading SRP Initiator ============================================================================== To load the SRP module, either execute the "modprobe ib_srp" command after the OFED driver is up, or change the value of SRP_LOAD in /etc/infiniband/openib.conf to "yes" (causing the srp module to be loaded at driver boot). NOTE: When loading the ib_srp module, it is possible to set the module parameter srp_sg_tablesize. This is the maximum number of gather/scatter entries per I/O (default: 12). a. modprobe ib_srp srp_sg_tablesize=32 or b. edit /etc/modprobe.conf and add the following line: options ib_srp srp_sg_tablesize=32 ============================================================================== 7. Manually Establishing an SRP Connection ============================================================================== The following steps describe how to manually load an SRP connection between the Initiator and an SRP Target. Section 8 explains how to do this automatically. - Make sure that the ib_srp module is loaded, the SRP Initiator is reachable by the SRP Target, and that an SM is running. - To establish a connection with an SRP Target and create SRP (SCSI) device(s) for that target under /dev, use the following command: echo -n id_ext=[GUID value],ioc_guid=[GUID value],dgid=[port GID value],\ pkey=ffff,service_id=[service[0] value] > \ /sys/class/infiniband_srp/srp-mthca[hca number]-[port number]/add_target Notes: a. Execution of the above "echo" command may take some time b. The SM must be running while the command executes c. It is possible to include additional parameters in the echo command: > max_cmd_per_lun - Default: 63 > max_sect (short for max_sectors) - sets the request size of a command > io_class - Default: 0x100 as in rev 16A of the specification Note: In rev 10 the default was 0xff00 > initiator_ext - Please refer to Section 9 (Multiple Connections...) d. See SRP Tools below for instructions on how the parameters in the echo command above may be obtained. - To list the new SCSI devices that have been added by the echo command, you may use either of the following two methods: a. Execute "fdisk -l". This command lists all devices; the new devices are included in this listing. b. Execute "dmesg" or look at /var/log/messages to find messages with the names of the new devices. ============================================================================== 8. SRP Tools - ibsrpdm and srp_daemon ============================================================================== To assist in performing the steps in Section 6, the OFED 1.3.1 distribution provides two utilities which: - Detect targets on the fabric reachable by the Initiator (for step 1) - Output target attributes in a format suitable for use in the above "echo" command (step 2) These utilities are: ibsrpdm and srp_daemon. The utilities can be found under /usr/local/ofed/sbin/ (or /sbin/), and are part of the srptools RPM that may be installed using the OFED custom installation. Detailed information regarding the various options for these utilities are provided by their man pages. Below, several usage scenarios for these utilities are presented. ibsrpdm usage ------------- 1. Detecting reachable targets a. To detect all targets reachable by the SRP initiator via the default umad device (/dev/umad0), execute the following command: > ibsrpdm This command will output information on each SRP target detected, in human-readable form. Sample output: IO Unit Info: port LID: 0103 port GID: fe800000000000000002c90200402bd5 change ID: 0002 max controllers: 0x10 controller[ 1] GUID: 0002c90200402bd4 vendor ID: 0002c9 device ID: 005a44 IO class : 0100 ID: LSI Storage Systems SRP Driver 200400a0b81146a1 service entries: 1 service[ 0]: 200400a0b81146a1 / SRP.T10:200400A0B81146A1 b. To detect all the SRP Targets reachable by the SRP Initiator via another umad device, use the following command: > ibsrpdm -d 2. Assistance in creating an SRP connection a. To generate output suitable for utilization in the "echo" command of section 5, add the "-c" option to ibsrpdm: >ibsrpdm -c Sample output: id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4, dgid=fe800000000000000002c90200402bd5,pkey=ffff,service_id=200400a0b81146a1 b. To establish a connection with an SRP Target (Section 6) using the output from the "libsrpdm -c" example above, execute the following command: echo -n id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4, dgid=fe800000000000000002c90200402bd5,pkey=ffff,service_id=200400a0b81146a1 > /sys/class/infiniband_srp/srp-mthca0-1/add_target The SRP connection should now be up; the newly created SCSI devices should appear in the listing obtained from the "fdisk -l" command. srp_daemon ---------- The srp_daemon utility is based on ibsrpdm and extends its functionality. In addition to the ibsrpdm functionality described above, srp_daemon can also - Establish an SRP connection by itself (without the need to issue the "echo" command described in Section 6) - Continue running in background, detecting new targets and establishing SRP connections with them (daemon mode) - Discover reachable SRP targets given an infiniband HCA name and port, rather than just by /dev/umad where is a digit - Enable High Availability operation (together with Device-Mapper Multipath) - Have a configuration file that determines the targets to connect to a. srp_daemon commands equivalent to ibsrpdm: "srp_daemon -a -o" is equivalent to "ibsrpdm" "srp_daemon -c -a -o" is equivalent to "ibsrpdm -c" Note: These srp_daemon commands can behave differently than the equivalent ibsrpdm command when /etc/srp_daemon.conf is not empty. b. srp_daemon extensions to ibsrpdm - To discover SRP Targets reachable from HCA device , port , (and generate output suitable for 'echo') you may execute srp_daemon -c -a -o -i -p - To both discover the SRP Targets and establish connections with them, just add the -e option to the above command. - Executing srp_daemon over a port without the -a option will only display the reachable targets via the port and to which the initiator is not connected. If executing with the -e option it is better to omit -a. - It is recommended to use the -n option. This option adds the initiator_ext to the connecting string. (See Section 9 for more details). - srp_daemon has a configuration file that can be set, where the default is /etc/srp_daemon.conf. Use the -f to supply a different configuration file that configures the targets srp_daemon is allowed to connect to. The configuration file can also be used to set values for additional parameters (e.g., max_cmd_per_lun, max_sect). - A continuous background (daemon) operation, providing an automatic ongoing detection and connection capability. See Section 8. ============================================================================== 9. Automatic Discovery and Connecting to Targets ============================================================================== - Make sure that the ib_srp module is loaded, the SRP Initiator can reach an SRP Target, and that an SM is running. - To connect to all the existing Targets in the fabric, execute srp_daemon -e -o. This utility will scan the fabric once, connect to every Target it detects, and then exit. NOTE: srp_daemon will follow the configuration it finds in /etc/srp_daemon.conf. Thus, it will ignore a target that is disallowed in the configuration file. - To connect to all the existing Targets in the fabric and to connect to new targets that will join the fabric, execute srp_daemon -e. This utility continues to execute until it is either killed by the user or encounters connection errors (such as no SM in the fabric). - To execute SRP daemon as a daemon you may execute run_srp_daemon (found under /usr/local/ofed/sbin/ or /sbin/), providing it with the same options used for running srp_daemon. Note: Make sure only one instance of run_srp_daemon runs per port. - To execute SRP daemon as a daemon on all the ports, execute srp_daemon.sh (found under /usr/local/ofed/sbin/ or /sbin/). srp_daemon.sh sends its log to /var/log/srp_daemon.log. - It is possible to configure this script to execute automatically when the InfiniBand driver starts by changing the value of SRP_DAEMON_ENABLE in /etc/infiniband/openib.conf to "yes". Another option to to configure this script to execute automatically when the InfiniBand driver starts is by changing the value of SRPHA_ENABLE in /etc/infiniband/openib.conf to "yes". However, this option also enables SRP High Availability that has some more features. (Please read the High Availability section). ============================================================================== 10. Multiple Connections from Initiator IB Port to the Target ============================================================================== Some system configurations may need multiple SRP connections from the SRP Initiator to the same SRP Target: to the same Target IB port, or to different IB ports on the same Target HCA. In case of a single Target IB port, i.e., SRP connections use the same path, the configuration is enabled using a different initiator_ext value for each SRP connection. The initiator_ext value is a 16-hexadecimal-digit value specified in the connection command. Also in case of two physical connections (i.e., network paths) from a single initiator IB port to two different IB ports on the same Target HCA, there is need for a different initiator_ext value on each path. The conventions is to use the Target port GUID as the initiator_ext value for the relevant path. If you use srp_daemon with -n flag, it automatically assigns initiator_ext values according to this convention. For example: id_ext=200500A0B81146A1,ioc_guid=0002c90200402bec,dgid=fe800000000000000002c90200402bed,\ pkey=ffff,service_id=200500a0b81146a1,initiator_ext=ed2b400002c90200 Notes: a. It is recommended to use the -n flag for all srp_daemon invocations. b. ibsrpdm does not have a corresponding option. c. srp_daemon.sh always uses the -n option (whether invoked manually by the user, or automatically at startup by setting SRPHA_ENABLE or SRP_DAEMON_ENABLE to yes). ============================================================================== 11. High Availability (HA) ============================================================================== High Availability Overview -------------------------- High Availability works using the Device-Mapper (DM) multipath and the SRP daemon. Each initiator is connected to the same target from several ports/HCAs. The DM multipath is responsible for joining together different paths to the same target and for fail-over between paths when one of them goes offline. Multipath will be execute on newly joined SCSI devices. Each initiator should execute several instances of the SRP daemon, one for each port. At startup, each SRP daemon detects the SRP targets in the fabric and sends requests to the ib_srp module to connect to each of them. These SRP daemons also detect targets that subsequently join the fabric, and send the ib_srp module requests to connect to them as well. High Availability Operation --------------------------- When a path (from port1) to a target fails, the ib_srp module starts an error recovery process. If this process gets to the reset_host stage and there is no path to the target from this port, ib_srp will remove this scsi_host. After the scsi_host is removed, multipath switches to another path to this target (from another port/HCA). When the failed path recovers, it will be detected by the SRP daemon. The SRP daemon will then request ib_srp to connect to this target. Once the connection is up, there will be a new scsi_host for this target. Multipath will be executed on the devices of this host, returning to the original state (prior to the failed path). High Availability Prerequisites ------------------------------- Installation for RHEL4 and RHEL5: (Execute once) - Verify that the standard device-mapper-multipath rpm is installed. If not, install it from the RHEL distribution. Installation for SLES10: (Execute once) - Verify that multipath is installed. If not, install it from the installation (You can use yast). - Update udev: (Execute once - for manual activation of High Availability only) - Add a file to /etc/udev/rules.d/ (you can call it 91-srp.rules) This file should have one line: ACTION=="add", KERNEL=="sd*[!0-9]", RUN+="/sbin/multipath %M:%m" Note: When SRPHA_ENABLE is set to "yes" (see Automatic Activation of High Availability below), this file is created upon each boot of the driver and is deleted when the driver is unloaded. Manual Activation of High Availability -------------------------------------- Initialization: (Execute after each boot of the driver) 1) Execute modprobe dm-multipath 2) Execute modprobe ib-srp 3) Make sure you have created file /etc/udev/rules.d/91-srp.rules as described above 4) Execute for each port and each HCA: srp_daemon -c -e -R 300 -i -p (You can use another value for -R. See under the Known Issues section the workaround for the rare race condition.) This step can be performed by executing srp_daemon.sh, which sends its log to /var/log/srp_daemon.log. Now it is possible to access the SRP LUNs on /dev/mapper/. NOTE: It is possible for regular (non-SRP) LUNs to also be present; the SRP LUNs may be identified by their names. You can configure the /etc/multipath.conf file to change multipath behavior. Automatic Activation of High Availability ----------------------------------------- - Set the value of SRPHA_ENABLE in /etc/infiniband/openib.conf to "yes". - From the next loading of the driver it will be possible to access the SRP LUNs on /dev/mapper/ NOTE: It is possible that regular (not SRP) LUNs may also be present; the SRP LUNs may be identified by their name. - It is possible to see the output of the SRP daemon in /var/log/srp_daemon.log ============================================================================== 12. Shutting Down SRP ============================================================================== SRP can be shutdown by using "rmmod ib_srp", or by stopping the OFED driver ("/etc/init.d/openibd stop"), or as a by-product of a complete system shutdown. Prior to shutting down SRP, remove all references to it. The actions you need to take depend on the way SRP was loaded. There are three cases. a. Without High Availability ------------------------------------ When working without High Availability, you should unmount the SRP partitions that were mounted prior to shutting down SRP. b. After Manual Activation of High Availability ----------------------------------------------- If you manually activated SRP High Availability, perform the following steps: 1) Unmount all SRP partitions that were mounted 2) Kill the SRP daemon instances 3) Make sure there are no multipath instances running. If there are multiple instances, wait for them to end or kill them. 4) Execute multipath -F c. After Automatic Activation of High Availability -------------------------------------------------- If SRP High Availability was automatically activated, SRP shutdown must be part of the driver shutdown ("/etc/init.d/openibd stop") which performs steps 2-4 of case b above. However, you still have to unmount all SRP partitions that were mounted before driver shutdown. HAL Issue --------- The HAL (Hardware Abstraction Layer) system includes a daemon that examines all devices in the system. In this process, it frequently holds a reference to the ib_srp module. If you attempt to shutdown SRP while this daemon is holding a reference to ib_srp, the shutdown will fail. Therefore, you should make sure this will not occur. One solution may be to stop "haldaemon" (/etc/init.d/haldaemon stop) prior to SRP shutdown. ============================================================================== 13. Known Issues ============================================================================== - There is a very rare race condition which can cause the SRP daemon to miss a target that joins the fabric. The race can occur if a target joins and leaves the fabric several times in a short time (e.g., if the cable is not connected well). In such a case, the SM may ignore this quick change of state and may not send an InformInfo to the srp_daemon. Workaround: Execute the srp_daemon command with the -R option. This option causes the SRP daemon to perform a full rescan of the fabric every seconds. - The srp_daemon does not support different pkeys other than the default pkey=ffff - It is recommended to use an SM that supports the enhanced capability mask matching feature (errata MGTWG8372). With SMs which support this feature, the SRP daemon generates significantly less communication traffic. - When booting OFED with SRP High Availability enabled, executing multipath for all LUNs on all connections may take some time (several minutes). However, it is possible to start working while this process is in progress. - Stopping the driver while SRP High Availability is enabled kills all multipath processes. Consider appropriate actions in case multipath is used for other purposes. - AS High Availability is based on Device Mapper multipath, it embodies multipath limitations and also its configuration and tuning options. See http://christophe.varoqui.free.fr/wiki/wakka.php?wiki=Home for information on multipath. To modify and tune multipath configuration, edit the file /etc/multipath.conf according to instructions and tips listed in /usr/share/doc/packages/multipath-tools/multipath.conf.* - In case your topology has two physical connections (i.e., network paths) from a single initiator IB port to two different IB ports on the same Target HCA, and you wish to have an SRP connection on the one path coexist with an SRP connection on the second path, you must set a different initiator_ext value on each path. See Section 9, "Multiple Connections from Initiator IB Port to the Target" for details. - The srp_daemon tool reads by default the configuration file /etc/srp_daemon.conf. In case this configuration file disallows connecting to a certain target, srp_daemon will ignore the target. If you find out that srp_daemon ignores a target, please check the /etc/srp_daemon.conf file. ============================================================================== 14. Vendor Specific Notes ============================================================================== Hosts connected to Qlogic SRP Targets must perform one of the following steps after upgrading to OFED 1.3.1 to continue accessing their storage successfully: 1. When issuing the "echo" command to add a new SRP Target, the host must append the string ",initiator_ext=0000000000000001" to the original echo string. Example: 'ibsrpdm -c' output is as follows: id_ext=0000000000000001,ioc_guid=00066a0138000165,dgid=fe8000000000000 000066a0260000165,pkey=ffff,service_id=0000494353535250,io_class=ff00 id_ext=0000000000000001,ioc_guid=00066a0238000165,dgid=fe8000000000000 000066a0260000165,pkey=ffff,service_id=0000494353535250,io_class=ff00 To connect to the first target, the echo command must be: echo -n \ id_ext=0000000000000001,ioc_guid=00066a0138000165,\ dgid=fe8000000000000000066a0260000165,pkey=ffff,\ service_id=0000494353535250,io_class=ff00,\ initiator_ext=0000000000000001 > \ /sys/class/inifiniband_srp/srp-mthca0-1/add_target 2. Change the SRP map on the Qlogic SRP Target to set the expected initiator extension to 0. For details on how to change the SRP map on a Qlogic SRP Target, please refer to product documentation.