fm10k Linux* Base Driver for the Intel(R) Ethernet Switch FM10000 Host Interface =============================================================================== =============================================================================== October 1, 2015 =============================================================================== Contents -------- - Overview - Building and Installation - Command Line Parameters - Intel(R) fm10k Ethernet Flow Director - Additional Features & Configurations - Known Issues ================================================================================ Important Notes --------------- Configuring SR-IOV for improved network security ------------------------------------------------ In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV, the virtual function (VF) may be subject to malicious behavior. Software- generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb (priority based flow-control), and others of this type, are not expected and can throttle traffic between the host and the virtual switch, reducing performance. To resolve this issue, configure all SR-IOV enabled ports for VLAN tagging. This configuration allows unexpected, and potentially malicious, frames to be dropped. Overview -------- This document describes the fm10k Linux* Base Driver for the Intel(R) Ethernet Switch FM10000 Host Interface. The Linux* base driver supports the following kernel versions: 2.6.32 and newer Driver information can be obtained using ethtool, lspci, and iproute2 ip command. Instructions on updating ethtool can be found in the section Additional Configurations later in this document. ================================================================================ Building and Installation ------------------------- To build a binary RPM* package of this driver, run 'rpmbuild -tb fm10k-<x.x.x>.tar.gz', where <x.x.x> is the version number for the driver tar file. NOTES: - For the build to work properly, the currently running kernel MUST match the version and configuration of the installed kernel sources. If you have just recompiled the kernel reboot the system before building. - RPM functionality has only been tested in Red Hat distributions. 1. Move the base driver tar file to the directory of your choice. For example, use '/home/username/fm10k' or '/usr/local/src/fm10k'. 2. Untar/unzip the archive, where <x.x.x> is the version number for the driver tar file: tar zxf fm10k-<x.x.x>.tar.gz 3. Change to the driver src directory, where <x.x.x> is the version number for the driver tar: cd fm10k-<x.x.x>/src/ 4. Compile the driver module: # make install The binary will be installed as: /lib/modules/<KERNEL VERSION>/kernel/drivers/net/fm10k/fm10k.[k]o The install location listed above is the default location. This may differ for various Linux distributions. 5. Load the module using the modprobe command: modprobe <fm10k> [parameter=port1_value,port2_value] Make sure that any older fm10k drivers are removed from the kernel before loading the new module: rmmod fm10k; modprobe fm10k 6. Assign an IP address to the interface by entering the following, where ethX is the interface name that was shown in dmesg after modprobe: ip address add <IP_address>/<netmask bits> dev ethX 7. Verify that the interface works. Enter the following, where IP_address is the IP address for another machine on the same subnet as the interface that is being tested: ping <IP_address> NOTE: For certain distributions like (but not limited to) RedHat Enterprise Linux 7 and Ubuntu, once the driver is installed the initrd/initramfs file may need to be updated to prevent the OS loading old versions of the fm10k driver. The dracut utility may be used on RedHat distributions: # dracut --force For Ubuntu: # update-initramfs -u ================================================================================ Command Line Parameters ----------------------- If the driver is built as a module, the following optional parameters are used by entering them on the command line with the modprobe command using this syntax: modprobe fm10k [<option>=<VAL1>] For example: modprobe fm10k max_vfs=7 The default value for each parameter is generally the recommended setting, unless otherwise noted. RSS --- Valid Range: 0-128 0 = Assign up to the lesser value of the number of CPUs or the number of queues X = Assign X queues, where X is less than or equal to the maximum number of queues (128 queues). VMDQ ---- Valid Range: 0-4 on 82575-based adapters; 0-8 for 82576/82580-based adapters Supports enabling VMDq pools as this is needed to support SR-IOV. 0 = Disabled 1 = Sets the netdev as pool 0 2+ = Add additional queues but they currently are not used This parameter is forced to 1 or more if the max_vfs module parameter is used. In addition, the number of queues available for RSS is limited if this is set to 1 or greater. NOTE: When either SR-IOV mode or VMDq mode is enabled, hardware VLAN filtering and VLAN tag stripping/insertion will remain enabled. max_vfs ------- Valid Range: 0-64 NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x and above, use sysfs to enable VFs. For example: #echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs //enable VFs #echo 0 > /sys/class/net/$dev/device/sriov_numvfs //disable VFs The parameters for the driver are referenced by position. Thus, if you have a dual port adapter, or more than one adapter in your system, and want N virtual functions per port, you must specify a number for each port with each parameter separated by a comma. For example: modprobe fm10k max_vfs=4,1 NOTE: Caution must be used in loading the driver with these parameters. Depending on your system configuration, number of slots, etc., it is impossible to predict in all cases where the positions would be on the command line. This parameter adds support for SR-IOV. It causes the driver to spawn up to max_vfs worth of virtual functions. NOTE: When SR-IOV mode is enabled, hardware VLAN filtering and VLAN tag stripping/insertion will remain enabled. Please remove the old VLAN filter before the new VLAN filter is added. For example, ip link set eth0 vf 0 vlan 100 // set vlan 100 for VF 0 ip link set eth0 vf 0 vlan 0 // Delete vlan 100 ip link set eth0 vf 0 vlan 200 // set a new vlan 200 for VF 0 Configuring SR-IOV for improved network security ------------------------------------------------ In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV, the virtual function (VF) may be subject to malicious behavior. Software- generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb (priority based flow-control), and others of this type, are not expected and can throttle traffic between the host and the virtual switch, reducing performance. To resolve this issue, configure all SR-IOV enabled ports for VLAN tagging. This configuration allows unexpected, and potentially malicious, frames to be dropped. Configuring VLAN tagging on SR-IOV enabled adapter ports -------------------------------------------------------- To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the following command. The VLAN configuration should be done before the VF driver is loaded or the VM is booted. $ ip link set dev <PF netdev id> vf <id> vlan <vlan id> For example, the following instructions will configure PF eth0 and the first VF on VLAN 10. $ ip link set dev eth0 vf 0 vlan 10 . ================================================================================ Additional Features and Configurations ------------------------------------------- Configuring the Driver on Different Distributions ------------------------------------------------- Configuring a network driver to load properly when the system is started is distribution dependent. Typically, the configuration process involves adding an alias line to /etc/modules.conf or /etc/modprobe.conf as well as editing other system startup scripts and/or configuration files. Many popular Linux distributions ship with tools to make these changes for you. To learn the proper way to configure a network device for your system, refer to your distribution documentation. If during this process you are asked for the driver or module name, the name for the Base Driver is fm10k. Viewing Link Messages --------------------- Link messages will not be displayed to the console if the distribution is restricting system messages. In order to see network driver link messages on your console, set dmesg to eight by entering the following: dmesg -n 8 NOTE: This setting is not saved across reboots. Jumbo Frames ------------ Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU) to a value larger than the default value of 1500. Use the ifconfig command to increase the MTU size. For example, enter the following where <x> is the interface number: ifconfig eth<x> mtu 9000 up This setting is not saved across reboots. The setting change can be made permanent by adding 'MTU=9000' to the file: /etc/sysconfig/network-scripts/ifcfg-eth<x> for RHEL or to the file /etc/sysconfig/network/<config_file> for SLES. NOTES: - The maximum MTU setting for Jumbo Frames is 15342. This value coincides with the maximum Jumbo Frames size of 15364 bytes. - This driver will attempt to use multiple page sized buffers to receive each jumbo packet. This should help to avoid buffer starvation issues when allocating receive packets. ethtool ------- The driver utilizes the ethtool interface for driver configuration and diagnostics, as well as displaying statistical information. The latest ethtool version is required for this functionality. Download it at http://ftp.kernel.org/pub/software/network/ethtool/ Supported ethtool Commands and Options -------------------------------------- -n --show-nfc Retrieves the receive network flow classification configurations. rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 Retrieves the hash options for the specified network traffic type. -N --config-nfc Configures the receive network flow classification. rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 m|v|t|s|d|f|n|r... Configures the hash options for the specified network traffic type. udp4 UDP over IPv4 udp6 UDP over IPv6 f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet. n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet. NAPI ---- NAPI (Rx polling mode) is supported in the fm10k driver. For more information on NAPI, see ftp://robur.slu.se/pub/Linux/net-development/NAPI/usenix-paper.tgz. Flow Control ------------ The Intel(R) Ethernet Switch FM10000 Host Interface does not support Flow Control. It will not send pause frames. This may result in dropped frames. Support for UDP RSS ------------------- This feature adds an ON/OFF switch for hashing over certain flow types. Only UDP can be turned on. The default setting is enabled . VXLAN Overlay HW Offloading --------------------------- VXLAN Overlay HW Offloading is enabled by default. The fm10k Linux driver features VXLAN Overlay HW Offloading support. To view and configure VXLAN on a VXLAN-overlay offload enabled device, use the following commands: # ethtool -k ethX (This command displays the offloads and their current state.) # ethtool -K ethX tx-udp_tnl-segmentation [off|on] (This enables/disables VXLAN support in the driver.) For more information on configuring your network for VXLAN overlay support, refer to the Intel Technical Brief, "Creating Overlay Networks Using Intel Ethernet Converged Network Adapters" (Intel Networking Division, August 2013): http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/ overlay-networks-using-converged-network-adapters-brief.pdf Data Center Bridging (DCB) -------------------------- TBD ================================================================================ Known Issues/Troubleshooting ---------------------------- Driver does not compile on SLES 11 (no SP) ------------------------------------------ The fm10k driver does not compile on SUSE Enterprise Linux Server 11 (no Service Pack). Installing Service Pack 1 (sp1) or newer will resolve this issue. ================================================================================ Support ------- For general information, go to the Intel support website at: www.intel.com/support/ or the Intel Wired Networking project hosted by Sourceforge at: http://sourceforge.net/projects/fm10k If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to fm10k-devel@lists.sf.net. ================================================================================ License ------- This program is free software; you can redistribute it and/or modify it under the terms and conditions of the GNU General Public License, version 2, as published by the Free Software Foundation. This program is distributed in the hope it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. The full GNU General Public License is included in this distribution in the file called "COPYING". Intel(R) Ethernet Switch FM10000 Host Interface driver Copyright(c) 2015 Intel Corporation. ================================================================================ Trademarks ---------- Intel, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. * Other names and brands may be claimed as the property of others.