[PATCH] S2io: Updated documentation

S2IO Technologies XFrame 10 Gig adapter.

Release notes for Neterion's (Formerly S2io) Xframe I/II PCI-X 10GbE driver.

-------------------------------------------

Contents

I. Module loadable parameters.

=======

When loaded as a module, the driver provides a host of Module loadable

- 1.  Introduction

parameters, so the device can be tuned as per the users needs.

- 2.  Identifying the adapter/interface

A list of the Module params is given below.

- 3.  Features supported

(i)	ring_num: This can be used to program the number of

- 4.  Command line parameters

		 receive rings used in the driver.

- 5.  Performance suggestions

(ii)	ring_len: This defines the number of descriptors each ring

- 6.  Available Downloads 

		 can have. There can be a maximum of 8 rings.

(iii)	frame_len: This is an array of size 8. Using this we can 

		 set the maximum size of the received frame that can

1.	Introduction:

		 be steered into the corrsponding receive ring.	

This Linux driver supports Neterion's Xframe I PCI-X 1.0 and

(iv)	fifo_num: This defines the number of Tx FIFOs thats used in

Xframe II PCI-X 2.0 adapters. It supports several features 

		 the driver. 

such as jumbo frames, MSI/MSI-X, checksum offloads, TSO, UFO and so on.

(v)	fifo_len: Each element defines the number of 

See below for complete list of features.

 		 Tx descriptors that can be associated with each 

All features are supported for both IPv4 and IPv6.

		 corresponding FIFO. There are a maximum of 8 FIFOs.

(vi)	tx_prio: This is a bool, if module is loaded with a non-zero

2.	Identifying the adapter/interface:

		value for tx_prio multi FIFO scheme is activated.

a. Insert the adapter(s) in your system.

(vii)	rx_prio: This is a bool, if module is loaded with a non-zero

b. Build and load driver 

		value for tx_prio multi RING scheme is activated.

# insmod s2io.ko

(viii)	latency_timer: The value given against this param will be

c. View log messages

		 loaded	into the latency timer register in PCI Config

# dmesg | tail -40

		 space, else the register is left with its reset value.

You will see messages similar to:

eth3: Neterion Xframe I 10GbE adapter (rev 3), Version 2.0.9.1, Intr type INTA

II. Performance tuning.

eth4: Neterion Xframe II 10GbE adapter (rev 2), Version 2.0.9.1, Intr type INTA

 By changing a few sysctl parameters.

eth4: Device is on 64 bit 133MHz PCIX(M1) bus

	Copy the following lines into a file and run the following command,

	"sysctl -p <file_name>"

The above messages identify the adapter type(Xframe I/II), adapter revision,

### IPV4 specific settings

driver version, interface name(eth3, eth4), Interrupt type(INTA, MSI, MSI-X).

net.ipv4.tcp_timestamps = 0 # turns TCP timestamp support off, default 1, reduces CPU use

In case of Xframe II, the PCI/PCI-X bus width and frequency are displayed

net.ipv4.tcp_sack = 0 # turn SACK support off, default on

as well.

# on systems with a VERY fast bus -> memory interface this is the big gainer

net.ipv4.tcp_rmem = 10000000 10000000 10000000 # sets min/default/max TCP read buffer, default 4096 87380 174760

To associate an interface with a physical adapter use "ethtool -p <ethX>".

net.ipv4.tcp_wmem = 10000000 10000000 10000000 # sets min/pressure/max TCP write buffer, default 4096 16384 131072

The corresponding adapter's LED will blink multiple times.

net.ipv4.tcp_mem = 10000000 10000000 10000000 # sets min/pressure/max TCP buffer space, default 31744 32256 32768

3.	Features supported:

### CORE settings (mostly for socket and UDP effect)

a. Jumbo frames. Xframe I/II supports MTU upto 9600 bytes,

net.core.rmem_max = 524287 # maximum receive socket buffer size, default 131071

modifiable using ifconfig command.

net.core.wmem_max = 524287 # maximum send socket buffer size, default 131071

net.core.rmem_default = 524287 # default receive socket buffer size, default 65535

b. Offloads. Supports checksum offload(TCP/UDP/IP) on transmit

net.core.wmem_default = 524287 # default send socket buffer size, default 65535

and receive, TSO.

net.core.optmem_max = 524287 # maximum amount of option memory buffers, default 10240

net.core.netdev_max_backlog = 300000 # number of unprocessed input packets before kernel starts dropping them, default 300

c. Multi-buffer receive mode. Scattering of packet across multiple

---End of performance tuning file---

buffers. Currently driver supports 2-buffer mode which yields

significant performance improvement on certain platforms(SGI Altix,

IBM xSeries).

d. MSI/MSI-X. Can be enabled on platforms which support this feature

(IA64, Xeon) resulting in noticeable performance improvement(upto 7%

on certain platforms).

e. NAPI. Compile-time option(CONFIG_S2IO_NAPI) for better Rx interrupt 

moderation.

f. Statistics. Comprehensive MAC-level and software statistics displayed

using "ethtool -S" option.

g. Multi-FIFO/Ring. Supports up to 8 transmit queues and receive rings, 

with multiple steering options.

4.  Command line parameters

a. tx_fifo_num

Number of transmit queues

Valid range: 1-8

Default: 1

b. rx_ring_num

Number of receive rings

Valid range: 1-8

Default: 1

c. tx_fifo_len

Size of each transmit queue

Valid range: Total length of all queues should not exceed 8192

Default: 4096

d. rx_ring_sz 

Size of each receive ring(in 4K blocks)

Valid range: Limited by memory on system

Default: 30 

e. intr_type

Specifies interrupt type. Possible values 1(INTA), 2(MSI), 3(MSI-X)

Valid range: 1-3

Default: 1 

5.  Performance suggestions

General:

a. Set MTU to maximum(9000 for switch setup, 9600 in back-to-back configuration)

b. Set TCP windows size to optimal value. 

For instance, for MTU=1500 a value of 210K has been observed to result in 

good performance.

# sysctl -w net.ipv4.tcp_rmem="210000 210000 210000"

# sysctl -w net.ipv4.tcp_wmem="210000 210000 210000"

For MTU=9000, TCP window size of 10 MB is recommended.

# sysctl -w net.ipv4.tcp_rmem="10000000 10000000 10000000"

# sysctl -w net.ipv4.tcp_wmem="10000000 10000000 10000000"

Transmit performance:

a. By default, the driver respects BIOS settings for PCI bus parameters. 

However, you may want to experiment with PCI bus parameters 

max-split-transactions(MOST) and MMRBC (use setpci command). 

A MOST value of 2 has been found optimal for Opterons and 3 for Itanium.  

It could be different for your hardware.  

Set MMRBC to 4K**.

For example you can set 

For opteron

#setpci -d 17d5:* 62=1d 

For Itanium

#setpci -d 17d5:* 62=3d 

For detailed description of the PCI registers, please see Xframe User Guide.

b. Ensure Transmit Checksum offload is enabled. Use ethtool to set/verify this 

parameter.

c. Turn on TSO(using "ethtool -K")

# ethtool -K <ethX> tso on

Receive performance:

a. By default, the driver respects BIOS settings for PCI bus parameters. 

However, you may want to set PCI latency timer to 248.

#setpci -d 17d5:* LATENCY_TIMER=f8

For detailed description of the PCI registers, please see Xframe User Guide.

b. Use 2-buffer mode. This results in large performance boost on

on certain platforms(eg. SGI Altix, IBM xSeries).

c. Ensure Receive Checksum offload is enabled. Use "ethtool -K ethX" command to 

set/verify this option.

d. Enable NAPI feature(in kernel configuration Device Drivers ---> Network 

device support --->  Ethernet (10000 Mbit) ---> S2IO 10Gbe Xframe NIC) to 

bring down CPU utilization.

** For AMD opteron platforms with 8131 chipset, MMRBC=1 and MOST=1 are 

recommended as safe parameters.

For more information, please review the AMD8131 errata at

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/26310.pdf

6.  Available Downloads

Neterion "s2io" driver in Red Hat and Suse 2.6-based distributions is kept up 

to date, also the latest "s2io" code (including support for 2.4 kernels) is 

available via "Support" link on the Neterion site:  http://www.neterion.com.

For Xframe User Guide (Programming manual), visit ftp site ns1.s2io.com,

user: linuxdocs password: HALdocs

7. Support 

For further support please contact either your 10GbE Xframe NIC vendor (IBM, 

HP, SGI etc.) or click on the "Support" link on the Neterion site:  

http://www.neterion.com.