IB Bonding =============================================================================== 1. Introduction 2. How to work with ib-bond 3. How to work with interface configuration scripts 3.1 Configuration with initscripts support 3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7) 3.1.2 Writing network scripts under Redhhat-EL5 3.2 Configuration with sysconfig support 3.2.1 Writing network scripts under SLES-10 3.3 Configuring Ethernet slaves 1. Introduction ------------------------------------------------------------------------------- ib-bonding is a High Availability solution for IPoIB interfaces. It is based on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB. ib-bonding package contains a bonding driver and a utility called ib-bond to manage and control the driver operation. 2. How to work with ib-bond ------------------------------------------------------------------------------- * Creating a bonding network interface --bond-name: sets the name of the bonding network interface. Default is bond0 --bond-ip : sets the IP address of bond0. If MASK is not given it is set to 255.255.255.0 (24 bits). Note that MASK should be the number of 1 bits in the netmask. --slaves: a comma separated list of slave ib devices. If not given ib0 and ib1 will be used as slaves. Child interfaces are allowed. --miimon: the MII monitoring interval in mSec. Default is 100 * Deleting a bonding network interface --stop: unenslave slaves and delete a specific bonding network interface (use with --bond-name) --stop-all: unenslave slaves and delete all bonding network interfaces * Querying a bonding network interface --status: show the status of a specific bonding network interface (use with --bond-name) --status-all: show the status of all bonding network interfaces Examples: * To bring up bond0 with ib0 and ib2 as slaves (assumes 2 HCAs) ib-bond --bond-ip 192.186.10.100 --slaves ib0,ib2 * To bring up bond1 with ib0.f1f1 1and ib1.f1f1 as slaves with non default netmask ib-bond --bond-name bond1 --bond-ip 192.186.10.100/25 --slaves ib0.f1f1,ib1.f1f1 * To query the status of bond1 ib-bond --bond-name bond1 --status * To query the status of all bonding interfaces ib-bond --status-all * To stop bond1 ib-bond --bond-name bond1 --stop * To stop all bonding interfaces ib-bond --stop-all 3. How to work with interface configuration scripts ------------------------------------------------------------------------------- Using ib-bond to configure interfaces doesn't save the configuration anywhere, so whenever the master or one of the slaves is destroyed the configuration should be restored by running ib-bond again (e.g. after system reboot). It is possible to avoid that if you create an interface configuration script for the ibX and bondX interfaces. To do that, you should use the standard syntax to create the bonding configuration (depending on your OS). 3.1 Configuration with initscripts support ------------------------------------------ Note: This feature is available only for Redhat-AS4 (Update 4, Update 5, Update 6 or Update 7) and for Redhat-EL5 and above. 3.1.1 Writing network scripts under Redhat-AS4 (Update 4, 5, 6 or 7) ----------------------------------------------------------------- * In the master (bond) interface script add the line: TYPE=Bonding Exmaple: for bond0 (master) the file is named /etc/sysconfig/network-scripts/ifcfg-bond0 with the following text in the file: DEVICE=bond0 IPADDR=192.168.1.1 NETMASK=255.255.255.0 NETWORK=192.168.1.0 BROADCAST=192.168.1.255 ONBOOT=yes BOOTPROTO=none USERCTL=no TYPE=Bonding * In the slave (ib) interface script put the following lines: SLAVE=yes MASTER= TYPE=InfiniBand Example: the script for ib0 (slave) would be named /etc/sysconfig/network-scripts/ifcfg-ib0 with the following text in the file: DEVICE=ib0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none TYPE=InfiniBand After the configuration is saved, restart the network service by running: /etc/init.d/network restart 3.1.2 Writing network scripts under Redhhat-EL5 ----------------------------------------------- Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4) with the following changes: * In the bondX (master) script - the line TYPE=Bonding is not needed. * in the ibX (slave) script - the line TYPE=InfiniBand necessary when using bonding over devices configured with partitions ( p_key) Example: ifcfg-ibX.8003 and ifcfg-ibY.8003 must include TYPE=InfiniBand line in their configuration files, when using as slaves for bondX device * in /etc/modprobe.conf add the following lines alias bond0 bonding options bond0 miimon=100 mode=1 max_bonds=1 If you want more than one bonding interface, name them bond1, bond2... and just add the necessary lines in /etc/modprobe.conf and change max_bonds=1 to max_bonds=N where N=number_of_bonding_interfaces Note: restarting OFED doesn't keep the bonding configuration via initscripts. You have to restart the network service in order to recreate the bonding interface. 3.2 Configuration with sysconfig support ---------------------------------------- Note: This feature is available only for SLES-10 and above. 3.2.1 Writing network scripts under SLES-10 ----------------------------------------------- * In the master (bond) interface script add the lins: BONDING_MASTER=yes BONDING_MODULE_OPTS="mode=active-backup miimon=" BONDING_SLAVE0=slave0 BONDING_SLAVE1=slave1 Exmaple: for bond0 (master) the file is named /etc/sysconfig/network/ifcfg-bond0 with the following text in the file: BOOTPROTO="static" BROADCAST="10.0.2.255" IPADDR="10.0.2.10" NETMASK="255.255.0.0" NETWORK="10.0.2.0" REMOTE_IPADDR="" STARTMODE="onboot" BONDING_MASTER="yes" BONDING_MODULE_OPTS="mode=active-backup miimon=100" BONDING_SLAVE0=ib0 BONDING_SLAVE1=ib1 * The slave (ib) interace script should look like this: BOOTPROTO='none' STARTMODE='off' PRE_DOWN_SCRIPT=/etc/sysconfig/network/unenslave.sh After the configuration is saved, restart the network service by running: /etc/init.d/network restart 3.3 Configuring Ethernet slaves ------------------------------- It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the same bonding master. It is possible however that a bonding master of Ethernet slaves and a bonding master of IPoIB slaves will co-exist in one machne. To configure Ethernet slaves under a bonding master use the same instructions as for IPoIB slaves (according to the OS) with one exception. When working under Redhat-AS4 do the following when configuring a bonding master with Ethernet slaves - In the master configuration file add the line SLAVEDEV=1 - In the slave configuration file leave the line TYPE=InfiniBand When using both type of bonding under SLES-10, it is neccessary to update the MANDATORY_DEVICES environment variable in /etc/sysconfig/network/config with the names of the InfiniBand devices ( ib0, ib1, etc. ). Otherwise, bonding devices will be created before InfiniBand devices at boot time.