                        ===========
                        R E A D M E
                        ===========

          Broadcom Advanced Server Program (BASP) Driver
                 for Linux 2.4.x and 2.6.x kernels

            Copyright (c) 2000-2004 Broadcom Corporation
                     All rights reserved.

Table of Contents
-----------------

    1.  Overview
    2.  Limitation
    3.  Installation
    4.  Install - TAR archive
    5.  Install - RPM package
    6.  Files
    7.  Configuration - Red Hat & United Linux based Distribution
    8.  Configuration - Suse Distribution
    9.  Configuration and Startup for Other Linux Distribution
    10. Startup scripts - Red Hat distributions
    11. Configuration scripts - Red Hat Distributions
    12. Broadcom NICE patches
    13. Uninstall - RPM package
    14. Offload Features
    15. Removal of physical interface in Generic Trunking 
        and 802.3ad mode
    16. LiveLink
    17. Known Problems


1. Overview
-----------

BASP is a kernel module designed for 2.4.x and 2.6.x kernels and provides 
load-balancing, fault-tolerance, and VLAN features. These features 
are provided by creating teams that consist of multiple 
NIC interfaces. A team can consist of 1 to 8 NIC interfaces and each 
interface can be designated primary, or hot-standby (SLB team only). 
All primary NIC interfaces in a team will participate in 
Load-balancing operations by sending and receiving a portion of the 
total traffic. Hot-standby interfaces will take over in the event 
that all primary interfaces have lost their links. VLANs can be 
added to a team to allow multiple VLANs with different VLAN IDs. 
A virtual device is created for each VLAN added.

BASP supports Smart Load-balance (SLB<tm>), SLB (Auto-Fallback Disable),
Generic trunking and IEEE 802.3ad Link Aggregation. In both SLB modes
and 802.3ad mode, all the NIC drivers must support Broadcom
NIC Extension (NICE). In this release, several NIC drivers patched
with NICE are included. 
    * Both SLB modes work with all Ethernet switches without configuring 
      the switch ports to any special trunking mode. Only IP traffic 
      will be load-balanced in both inbound and outbound directions. 
    * Generic trunking mode does not require NICE and can work with 
      any NIC, however, it requires the Ethernet switch to support 
      the technology and be properly configured. This mode is 
      protocol-independent and all traffic should be load-balanced 
      and fault-tolerant.
    * 802.3ad mode requires NICE drivers and Ethernet switches 
      supporting IEEE 802.3ad Link Aggregation. This mode is 
      protocol-independent and all traffic should be load-balanced 
      and fault-tolerant. All the physical interfaces in the 
      802.3ad teams are defaulted to be LACP active. A 802.3ad team 
      requires all the member NICs supporting NICE. All the member
      NICs, once in the 802.3ad team, will be set with the same 
      MAC address. 




2. Limitation
-------------

The current version of BASP has been tested on the latest Red Hat
and United Linux distributions, as well as other similar Linux 
distributions for i386, ia64, and x86_64 CPU architectures using 
2.4.x and 2.6.x kernels. The module has been tested up to kernel
version 2.4.24 and 2.6.3.

Minor modification to the Makefile may be required if problems are 
experienced when compiling BASP on other i386 Linux distributions.

VLANs are only supported by Broadcom NetXtreme Gigabit Ethernet. As 
opposed to VLANs support in other platforms, e.g. Windows and Netware, 
VLANs are not supported by Alteon Acenic driver (acenic.c).

For full functionality, Broadcom driver for NetXtreme Gigabit Ethernet
bcm5700 version 6.2.2 or newer should be used.


3. Installation
---------------

For installing RPM on Linux i386 distribution, follow instructions 
in "Install - RPM package" section.

For installing TAR archive on i386, IA64 and AMD 64 distributions, 
follow instructions in "Install - TAR archive" section.




4. Install - TAR archive
------------------------

BASP for Linux is shipped in mixed forms, where the platform and 
kernel specific files are in source code, and the core file is in 
object form. Three packages are shipped in this release: two tar 
archives and two RPM package.

"basplnx-{version}.i386.tgz" is the tar archive for i386 platform,  
"basplnx-{version}.ia64.tgz" is the tar archive for IA-64 platform,
"basplnx-{version}.x86_64.tgz" is the tar archive for AMD 64 bit platform.

To uncompress and expand the tar archive, run

    % tar xvfz basplnx-{version}.{arch}.tgz

The installation process involves the following steps:

(1) To build kernel module, "basp.o" or "basp.ko" on 2.6 kernels,

    % make

    Note that the Make process will automatically build the correct 
    module for different kernel options, e.g. symbol versioning and 
    SMP support. There is NO need to define -DMODVERSIONS in the 
    Makefile.

(2) To create device file and to copy files,

    % make install

(3) To update the module reference,

    % depmod -a

(4) To load the driver,

    % insmod basp

(5) Follow the "Configuration and Startup for Other Linux Distribution" 
    section to set up the teams.



5. Install - RPM package
------------------------
(1) To install the RPM source package, run

    % rpm -i basplnx-{version}.src.{arch}.rpm

(2) Change directory to the RPM path and build the binary driver for 
    the kernel

    % cd /usr/src/redhat
    % rpm -bb SPECS/basplnx.spec

    Note that the RPM path is different for different Linux 
    distributions. For those distributions with RPM v4.1 or later,
    use "rpmbuild" comannd instead of "rpm".

(3) Install the newly built package

    % rpm -i RPMS/i386/basplnx-{version}.{arch}.rpm

    The driver and other required files will be automatically 
    installed.

(4) To load the driver,

    % insmod basp

(5) Follow the Configuration section to set up the teams.



6. Files
--------

    Makefile                    makefile
    baspcfg                     precompiled configuration utility
    bcmtype.h                   commonly use type header file
    blf.c                       BASP module entry points
    blf.h                       ioctl interface
    blfcore.h                   core interface
    blfcore.o                   pre-compiled core object
    blfopt.h                    automatic generated header file by Make
    blfver.h                    version header file
    nicext.h                    NICE header file
    pal.c                       platform abstraction implementation
    pal.h                       header for paltform abstraction
    release.txt                 this file
    nice-2.2.16                 NICE enabled driver for 2.2 kernel
    nice-2.4.16                 NICE enabled driver for 2.4 kernel
    scripts                     contains sample scripts
    scripts/basp                init script, goes to /etc/init.d
    scripts/baspteam            start/stop script, goes to /etc/basp
    scripts/baspif              start/stop network i/f, goes to /etc/basp
    scripts/team-sample         sample script of SLB team with 3 NICs
    scripts/team-gec            sample script of GEC team with 3 NICs
    scripts/team-vlan           sample script of SLB team with 2 VLANs
    basp.4                      man page
    baspcfg.8                   man page for baspcfg utility



7. Configuration - Red Hat & United Linux based Distribution
------------------------------------------------------------

The BASP distribution includes a utility program and several scripts 
for team configuration. Most of the steps are only required to be 
performed after the first time installation. Step 2 "Modify the 
configuration script" should be performed whenever there is any 
change to the team configuration. 

Since Linux distributions do not automatically load drivers for 
network devices unless the device is configured with an IP address, 
users must manually configure a network-script file for all physical 
adapters that will be team members. Network script files are located 
under /etc/sysconfig/network-scripts (Red Hat), or 
/etc/sysconfig/network (United Linux 1.x). The file name must be 
prefixed with "ifcfg-" then the physical adapter alias. For 
interface eth0, you would create a file with the name ifcfg-eth0, 
then add the below content. 

Example: 

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes 

For users of other Linux distributions, follow instructions in the 
"BASP Configuration and Startup for Other Linux Distributions" section.

The configuration process involves the following steps:

Copy a configuration script from the "/etc/basp/samples" directory 
to the "/etc/basp" directory. Note that the configuration script 
name must be prefixed with "team-".

Modify the configuration script to:

(a) change the team type
(b) add/delete the physical network interfaces
(c) add/delete the virtual network interfaces
(d) assign IP address to each virtual network interface. 

The syntax of the configuration script can be found in the BASP 
Configuration Scripts for Red Hat Distributions section or in the 
/etc/basp/sample/team-sample script file itself. Note that when 
configuring Teaming, at least one Primary Adapter is required.

Manually start the team for the first time: 
% /etc/init.d/basp start

NOTE - This step is only required for the first time installation. 
The team configuration will be automatically started on subsequent 
reboots. 

Note that if not all the virtual network interfaces are configured 
with an IP address, there will be an error message in starting the 
BASP team. When this happens, repeat Step (2) to configure an IP 
address for all the virtual network interfaces. 

Note-1) Forming multiple teams is possible by copying the sample files 
        into "/etc/basp/team-<name>" and modifying this file as 
        described in the sample file. 

Note-2) In order to create more that one virtual interface (VLAN) for 
        each team, refer to the respective description section in the
        sample files. 


8. Configuration - Suse Distribution
------------------------------------

The BASP distribution includes an utility program and several scripts 
for team configuration. Following steps for Suse Linux distributions 
only. Most of the steps are only required to be performed after the first 
time installation. Step 2 "Modify the configuration script" should be 
performed whenever there is any change to the team configuration.

Since Suse distribution does not automatically load driver for the
network device unless the device is configured with IP address, users 
must manually use "yast2" or configure /etc/rc.config file to ensure
proper network drivers loaded during init time.

For users of other Linux distributions, follow instructions in the 
"baspcfg" section.

The configuration process involves the following steps:

(1) Copy a configuration script from "/etc/basp/samples" directory to
    "/etc/basp" directory. Note the configuration script must be 
    prefixed with "team-".

(2) Modify the configuration script to 
    (a) change the team type
    (b) add/delete physical network interfaces
    (c) add/delete virtual network interfaces
    (d) assign IP address to each virtual network interface
    The syntax of the configuration script can be found below. Note that
    when configuring Teaming, at lease one Primary Adapter is required.

(3) Manually start the team for the first time,

    % /etc/init.d/basp start

    Note that this step is only required for the first time installation.
    The team configuration will be automatically started on subsequent
    reboots. 
    
    Note that if not all the virtual network interfaces are configured 
    with IP address, there will be an error message in starting the 
    BASP team. When this happens, repeat step (2) to configure IP 
    address for all the virtual network interfaces.


Note-1) Forming multiple teams is possible by copying the sample files 
        into "/etc/basp/team-<name>" and modifying this file as 
        described in the sample file.

Note-2) In order to create more that one virtual interface (VLAN) for 
        each team, refer to the respective description section in the
        sample files.



9. Configuration and Startup for Other Linux Distribution
---------------------------------------------------------

"baspcfg" is a command line tool to configure the BASP teams, add/remove
NICs and add/remove virtual devices. This tool, may be used to configure 
your own startup script files.  Please read your distribution specific 
documentation for more information on your distributions's startup 
procedure. 

Following is the usage of this tool:

baspcfg v6.2.5  - Broadcom Advanced Server Program Configuration Utility
Copyright (c) 2000-2004 Broadcom Corporation. All rights reserved.
usage: baspcfg <commands>
commands:
    addteam  <tid> <type> <tname>   -- create a team
    addteam  <tid> <type> <tname> <target ip 0-3> <max probe_interval> 
                                       <max retries>
                                    -- create a LiveLink team
    delteam  <tid>                  -- delete a team
    txoffld  <tid> y|n              -- enable or disable tx offload features
    addva    <tid> <vlan_id> <vname> [macaddr]
                                    -- add a virtual adapter to a team
    delva    <tid> <vlan_id>        -- del a virtual adapter from a team
    bind     <tid> <role> <device>  -- bind a physical adapter to a team
    bind     <tid> <role> <device> <nic ip>  -- bind a LiveLink physical 
                                       adapter to a team
    unbind   <tid> <device>         -- unbind a physical adapter from a team
    show     [tid]                  -- display team configurations
    failback <tid>                  -- manual failback on SLB (Auto-Fallback 
                                       Disable) team only
where
    tid      -- An unique ID for each team, starting from 0
    type     -- Team type: 0=SLB, 1=FEC/GEC, 2=802.3ad, 3=SLB(Auto-Fallback 
                Disable)
    tname    -- ASCII string of the team
    vlan_id  -- VLAN ID: from 1 to 4094, 0=untagged or no VLAN
    vname    -- ASCII string of the virtual device
    macaddr  -- MAC address (optional), e.g. 00:10:18:00:11:44
    role     -- Role of the physical device: 0=primary, 1=hot-standby
    device   -- ASCII string of the physical device, e.g. eth0
    target/nic ip   -- probe target ip addres (e.g. 192.168.1.1)
    max probe interval   -- delay between LiveLink probes in seconds (min = 1
                max = 1024).
    max retries   -- number of LiveLink probes retries in seconds (min = 1,
                max = 50).

Following sample startup script should be used to start the BASP after
first time installation and configuration, or in the subsequent reboots.

	#!/bin/bash
	# load basp module
	insmod basp

	# create new team
	baspcfg addteam 0 0 team-one

	# bind physical interfaces / two primary one backup
	baspcfg bind 0 0 eth0
	baspcfg bind 0 0 eth1
	baspcfg bind 0 1 eth2

	# create the virtual interface
	# team-one will support the common set of tx offload
	# features supported by eth0,eth1,eth2
	# use "baspcfg txoffld 0 n" to disable all tx offload features
	baspcfg addva 0 0 sw0

	# bind ip address to virtual interface and initialize
	ifconfig sw0 192.168.0.1 up

Note: Baspcfg command can only be executed in Super User mode only, 
configure with other login mode will yield "Permission Denied.  Must be 
root to manage BASP" error message.

When configuring Teaming, at lease one Primary Adapter is required.


10. Startup scripts - Red Hat distributions
------------------------------------------

(1) basp
This script is intended to be installed in /etc/init.d directory.
After copying the script, run "chkconfig --add basp". This script will 
be executed at runlevel 2, 3, 4 and 5. When "basp" is run, it will 
search the /etc/basp directory to list all the files with "team-" prefix, 
and then it will invoke "baspteam" script to add or delete the teams. It
is normal that each "team-*" file in /etc/basp represents 1 team.

(2) baspteam
This script is called by "basp" script to add or delete a team. 
To install, create "/etc/basp" directory and copy this script over. 

To manually add a team:

    % baspteam team-sample add

To delete a team:

    % baspteam team-sample del

Note that "team-sample" is the configuration script.

(3) team-sample
This script contains a SLB team configuration with 3 NICs: eth0, eth1 
and eth2. The team name is "TeamSample". All 3 NICs are primary. One 
virtual interface is also created for this team and the name of the 
virtual interface  is "sw0". "sw0" is the device that "ifconfig" should 
be run against to set up the IP address. VLANs are not enabled in this 
script.

This script and "team-gec" are intended to be customized. Refer to the 
configuration scripts section for details. This script should be copied
to /etc/basp directory and retain the "team-" prefix.

(4) team-gec
This configuration script creates a GEC team with 3 network interfaces, 
eth0, eth1 and eth2. The team name is "TeamGEC". All 3 NICs are primary. 
One virtual interface is added to the team with the name "sw0" and 
VLANs are not enabled.

This script and "team-sample" are intended to be customized. Refer to 
the configuration scripts section for details. This script should be 
copied to /etc/basp directory and retain the "team-" prefix.



11. Configuration scripts - Red Hat and Suse
-------------------------------------------

Both "team-sample" and "team-gec" are configuration scripts that follow
the same syntax, as following

      TEAM_ID:       This number uniquely identifies a team. 
      TEAM_TYPE:     0 = SLB, 1 = Generic Trunking/GEC/FEC, 2= 802.3ad
                     3 = SLB (Auto-Fallback Disable)
      LIVELINK_ENABLE: 1 = SLB (with LiveLink) 0 = SLB (without LiveLink)  
     		       	 LiveLink is supported for SLB teams only.
                     A value of 1 is ignored when TEAM_TYPE is not 0 (SLB).     
      TEAM_NAME:     ascii name of the team
      TEAM_PAx_NAME: ascii name of the physical interface x,
                     where x can be 0 to 7.
      TEAM_PAx_ROLE: role of the physical interface x
                     0 = Primary, 1 = Hot-standby. This field
                     must be 0 for Generic Trunking/GEC/FEC 
                     and IEEE 802.3ad team.
      TEAM_PAx_IP:   Unique IP to be used by each NIC when LiveLink is enabled.  
                     The format should be x.x.x.x.  
                     If an IP unique to the subnet is not assigned, the NIC will 
                     not be used when LiveLink is enabled.
      TEAM_VAx_NAME: ascii name of the virtual interface x,
                     where x can be 0 to 63
                     TEAM_VAx_VLAN: 802.1p VLAN ID of the virtual interface x.
                     For untagged virtual interface, i.e. without
                     VLAN enable, set it to 0. The valid VLAN ID
                     can be 0 to 4094.
      TEAM_VAx_IP:   IP address of the virtual interface x. The 
                     format should be aa.bb.cc.dd.
      TEAM_VAx_NETMASK: Subnet mask of the virtual interface x.
                    The format should be mm.nn.oo.pp.
      TEAM_VAx_BROADCAST: Optional broadcast address of the virtual
                    interface x. The format should be qq.rr.ss.tt
      TEAM_VAx_GW:  Optional default gateway. The format should
                    be ww.xx.yy.zz. Usually one default gateway
                    is specified for the system and it should
                    be reacheable from one network interface.
      PROBE_TARGET_IPx: Target IP addresses for LiveLink option.  
                    The format shoud be x.x.x.x
                    The first LiveLink target is required when LiveLink is 
                    enabled. 
                    Up to 3 additional targets may be defined.  

      PROBE_INTERVAL: Delay in seconds between sending LiveLink packets.  
                    Minimum value is 1.
	  MAX_RETRY: Number of consecutively missed LiveLink replies before a
                    failover is triggered. Value is "n x 5".  
                    i.e. Setting of 5 = Value of 25 (5 x 5)

Note: Teaming scripts are intended for Red Hat distributions ONLY, 
use with other Linux distribution will causes ERROR.

12. Broadcom NICE patches
-------------------------

Also included in this release are network device drivers patched
with Broadcom NICE support. These drivers are originally taken from 
the Linux 2.4.16 kernel distribution. To install patched drivers:

(1) Copy the Broadcom NICE header file, "nicext.h", to the appropriate
    Linux kernel include directory, e.g.

    % cp /usr/src/nice-2.4.16/nicext.h /usr/src/linux/include/linux

(2) Rename the original network device driver under the Linux kernel 
    source tree, "/usr/src/linux/drivers/net".

(3) Copy the patched drivers to the Linux kernel network driver source
    directory, i.e. "/usr/src/linux/drivers/net".

(4) Follow the kernel rebuild instructions to configure kernel
    support for these drivers, i.e. 

    % cd /usr/src/linux
    % make config

(5) If the patched drivers are configured into the kernel, goto step (7).
    If the patched drivers are configured as modules, goto step (6).

(6) In the case of supporting only the module version of these drivers,
    it is possible to simply run the following to compile patched 
    drivers and to install them into the proper module directory:

    % make modules
    % make modules_install

    There is no need to compile the complete kernel. Goto step (8).

(7) Rebuild the kernel to compile these patched drivers

    % make clean 
    % make dep 
    % make

(8) Either reboot the system or unload/load the patched modules. Run
    configuration scripts to test the patch.



13. Uninstall - RPM package
---------------------------

To uninstall RPM package, 

    % rpm -e basplnx

and to reboot the system,

    % reboot



14. Offload features
--------------------

A load balancing/fault-tolerant team in any mode will automatically
make use of receive checksum offload and pseudo checksum offload
features if they are supported by any NIC in the team. On the
transmit side, the team will only support the common set of transmit
offload features supported by all NICs in the team. These transmit
offload features include scatter-gather, checksum, DMA for highmem, and
TSO (TCP Segmentation Offload). When a virtual adapter is created,
it will by default use the common set of transmit offload features
supported by the NICs that are bound to the team. This set of transmit
offload features will remain fixed until the virtual adapter is deleted.
After the virtual adapter is created, subsequent NICs that are bound to
the team must support all the transmit offload features that are in use
by the virtual adapter, otherwise the bind operation will fail. To deal
with this bind failure, the user has to unbind all NICs, delete the
virtual adapter, and bind all NICs and create the virtual adapter again.
Alternatively, the user can also disable all tranmsit offload features
before creating the virtual adapter. This will gurantee that all bind
oprations after the virtual adapter is created will not fail because
of transmit offload incompatibility. See "baspcfg" section or baspcfg(8)
man page for more details.


15. Removal of physical interface in Generic Trunking and 802.3ad mode
----------------------------------------------------------------------

In Generic Trunking and 802.3 ad mode, all the physical and virtual 
interfaces belonging to a team have the same MAC address. This MAC 
address is the same address as that of the first physical interface 
bounded to the team. In the case that this first physical interface is 
removed dynamically from the team using "baspcfg" tool and bounded to 
the protocol directly, this could lead to a duplicate MAC address 
problem on the network. Note that if the removed physical interface 
does not participate in any traffic, there will not be any problem.

To properly remove a physical interface, follow the steps listed below:

(1) Backup the original team configuration script

    % cp /etc/basp/team-gec /etc/basp/backup-gec

Note-1) "team-gec" is the name of the configuration script.

Note-2) "backup-gec" is the name of the backup script. The name of the
        backup script must NOT be prefixed with "team-".

(2) Modify the team configuration script to remove the physical 
interface

(3) Stop the running team

    % /etc/basp/baspif /etc/basp/backup-gec stop
    % /etc/basp/baspteam /etc/basp/backup-gec del

(4) Re-start the team
    
    % /etc/basp/baspteam /etc/basp/team-gec add
    % /etc/basp/baspif /etc/basp/team-gec start


16. LiveLink
------------------
The LiveLink feature has been added to basp releases 6.3.0 and later.
Without LiveLink, basp only relies on loss of link reported by the NICs 
to perform fail-over and fall-back. With LiveLink each NIC continuously 
sends probes to probe targets and monitor the replies.  LiveLink only 
enables NICs which receive probe replies from the probe target(s) to carry 
network traffic. 

LiveLink can be used if the following requirements are met,

(1) One or more highly available machines to which probes can be sent are 
present on the network. LiveLink currently supports up to four probe
targets. Probe targets should have a static IP address, and should reside 
on the same network than the team. Probe target should also be on vlan 0.

(2) A unique IP is available to assign to each one of the NIC team members.
LiveLink will automatically take a NIC offline if the NIC cannot reach 
any of the assigned probe targets.

The provided team-sample includes more information on how to configure 
LiveLink.

17. Known Problems
------------------

(1) DHCP address problem
We have identified that there is incompatiblity between SLB and IP 
address obtained via DHCP. In the lab, we have seen that the virtual 
network interface lost the IP address after running for a period of 
time.

To solve the problem, always assign static IP address to the virtual
network interfaces.


(2) 802.3ad team member links disconnect and reconnect continuously 
when connected to the HP2524 switch.
This is a 3rd party issue.  It is seen only when configuring an 802.3ad
team with greater than 2 members on the server and connecting an HP2524
switch, with lacp enabled as passive or active. The HP switch will 
show an lacp channel being brought up successfully with only 2 members. 
All other member's links will disconnect and reconnect. This does not 
occur with a Cisco Catalyst 6500.


(3) To avoid fail-over issues when using BASP, make sure that spanning
tree is disabled on the switch that the network adapter is connected to.

(4) While using LiveLink, the Probe Targets must reside on VLAN 0.



