Windows OpenFabrics

User's Manual

Release 2.0

01/06/2009

Overview

The Windows OpenFabrics (WinOF) package is composed of software modules intended for use on Microsoft Windows based computer systems connected via an InfiniBand fabric.

The Windows OpenFabrics software package contains the following:

OpenFabrics Infiniband core drivers and Upper Level Protocols (ULPs):

OpenFabrics utilities:

Documentation

 

WinOF Features

 

 

 

Tools


The OpenFabrics Alliance Windows release contains a set of user mode tools which are designed to faciliate the smooth operation of an Windows OpenFabrics installation. These tools are available from a command window (cmd.exe) as the installation path '%SystemDrive%\Program Files\WinOF' is appended to the system wide search path registry entry. A start menu short-cut 'WinOF Cmd Window' is provided to faciliate correction tool operation.

IPoIB Partition Management

Infiniband Subnet Management

Performance

Diagnostics

<return-to-top>

 

User mode micro-benchmarks


The following user-mode test programs are intended as useful micro-benchmarks for HW or SW tuning and/or functional testing.

Tests use CPU cycle counters to get time stamps without context switch.

Tests measure round-trip time but report half of that as one-way latency
(i.e.. May not be sufficiently accurate for asymmetrical configurations).

Min/Median/Max result is reported.
The median (vs. average) is less sensitive to extreme scores.
Typically the "Max" value is the first value measured.

larger samples only marginally help. The default (1000) is pretty good.
Note that an array of cycles_t (typically unsigned long) is allocated
once to collect samples and again to store the difference between them.
Really big sample sizes (e.g. 1 million) might expose other problems
with the program.

"-H" option will dump the histogram for additional statistical analysis.
See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other
statistical math programs.

Architectures tested: x86, x86_64, ia64


ib_send_lat.exe      - latency test with send transactions

Usage:

ib_send_lat start a server and wait for connection
ib_send_lat <host> connect to server at <host>

Options:

-p, --port=<port> listen on/connect to port <port> (default 18515)
-c, --connection=<RC/UC> connection type RC/UC (default RC)
-m, --mtu=<mtu> mtu size (default 2048)
-d, --ib-dev=<dev> use IB device <dev> (default first device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-s, --size=<size> size of message to exchange (default 1)
-t, --tx-depth=<dep> size of tx queue (default 50)
-l, --signal signal completion on each msg
-a, --all Run sizes from 2 till 2^23
-n, --iters=<iters> number of exchanges (at least 2, default 1000)
-C, --report-cycles report times in cpu cycle units (default microseconds)
-H, --report-histogram print out all results (default print summary only)
-U, --report-unsorted (implies -H) print out unsorted results (default sorted)
-V, --version display version number
-e, --events sleep on CQ events (default poll)


ib_send_bw.exe     - BW (BandWidth) test with send transactions

Usage:

ib_send_bw start a server and wait for connection
ib_send_bw <host> connect to server at <host>

Options:

-p, --port=<port> listen on/connect to port <port> (default 18515)
-d, --ib-dev=<dev> use IB device <dev> (default first device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-c, --connection=<RC/UC> connection type RC/UC/UD (default RC)
-m, --mtu=<mtu> mtu size (default 1024)
-s, --size=<size> size of message to exchange (default 65536)
-a, --all Run sizes from 2 till 2^23
-t, --tx-depth=<dep> size of tx queue (default 300)
-n, --iters=<iters> number of exchanges (at least 2, default 1000)
-b, --bidirectional measure bidirectional bandwidth (default unidirectional)
-V, --version display version number
-e, --events sleep on CQ events (default poll)


ib_write_lat.exe      - latency test with RDMA write transactions

Usage:

ib_write_lat start a server and wait for connection
ib_write_lat <host> connect to server at <host>

Options:

-p, --port=<port> listen on/connect to port <port> (default 18515)
-c, --connection=<RC/UC> connection type RC/UC (default RC)
-m, --mtu=<mtu> mtu size (default 1024)
-d, --ib-dev=<dev> use IB device <dev> (default first device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-s, --size=<size> size of message to exchange (default 1)
-a, --all Run sizes from 2 till 2^23
-t, --tx-depth=<dep> size of tx queue (default 50)
-n, --iters=<iters> number of exchanges (at least 2, default 1000)
-C, --report-cycles report times in cpu cycle units (default microseconds)
-H, --report-histogram print out all results (default print summary only)
-U, --report-unsorted (implies -H) print out unsorted results (default sorted)
-V, --version display version number


ib_write_bw.exe     - BW test with RDMA write transactions

Usage:

ib_write_bw                # start a server and wait for connection
ib_write_bw <host>    # connect to server at <host>

Options:

-p, --port=<port> listen on/connect to port <port> (default 18515)
-d, --ib-dev=<dev> use IB device <dev> (default first device found)
-i, --ib-port=<port> use port <port> of IB device (default 1)
-c, --connection=<RC/UC> connection type RC/UC (default RC)
-m, --mtu=<mtu> mtu size (default 1024)
-g, --post=<num of posts> number of posts for each qp in the chain (default tx_depth)
-q, --qp=<num of qp's> Num of qp's(default 1)
-s, --size=<size> size of message to exchange (default 65536)
-a, --all Run sizes from 2 till 2^23
-t, --tx-depth=<dep> size of tx queue (default 100)
-n, --iters=<iters> number of exchanges (at least 2, default 5000)
-b, --bidirectional measure bidirectional bandwidth (default unidirectional)
-V, --version display version number

<return-to-top>


 


ttcp - Test TCP performance

TTCP accesses the Windows socket layer, hence it does not access IB verbs directly. IPoIB or WSD layers are invoked beneath the socket layer depending on configuration. TTCP is included as a quick baseline performance check.

Usage: ttcp -t [-options] host 
       ttcp -r [-options]
Common options:
	-l ##	length of bufs read from or written to network (default 8192)
	-u	use UDP instead of TCP
	-p ##	port number to send to or listen at (default 5001)
	-A	align the start of buffers to this modulus (default 16384)
	-O	start buffers at this offset from the modulus (default 0)
	-d	set SO_DEBUG socket option
	-b ##	set socket buffer size (if supported)
	-f X	format for rate: k,K = kilo{bit,byte}; m,M = mega; g,G = giga
Options specific to -t:
	-n##	number of source bufs written to network (default 2048)
	-D	don't buffer TCP writes (sets TCP_NODELAY socket option)
Options specific to -r:
	-B	for -s, only output full blocks as specified by -l (for TAR)
	-T	"touch": access each byte as it's read

Requires a receiver (server) side and a transmitter (client) side, host1 and host2 are IPoIB connected hosts.

at host1 (receiver)        ttcp -r -f M -l 4096

at host2 (transmitter)    ttcp -t -f M -l 4096 -n1000 host1

<return-to-top>

 

 

Diagnostics



ibv_devinfo - print CA (Channel Adapter) attributes

usage: ibv_devinfo  [options]

Options:
   -d, --ib-dev=<dev> use IB device <dev> (default: first device found)
    -i, --ib-port=<port> use port <port> of IB device (default: all ports)
    -l, --list print only the IB devices names
    -v, --verbose print all the attributes of the IB device(s)

<return-to-top>
 


ibstat - print InfiniBand stats

usage: ibstat [OPTIONS] <ca_name> [portnum]

Options:
    -d debug
    -l list all IB devices
    -s print short device summary
    -p print port GUIDs
    -V print ibstat version information and exit
    -h print usage

Examples:
    ibstat -l # list all IB devices
    ibstat mthca0 2 # stat port 2 of mthca0

<return-to-top>
 


ib_limits - Infiniband verbs tests

Usage: ib_limits [options]

Options:
-m or --memory
    Direct ib_limits to test memory registration
-c or --cq
    Direct ib_limits to test CQ creation
-r or --resize_cq
    direct ib_limits to test CQ resize
-q or --qp
    Directs ib_limits to test QP creation
-v or --verbose
    Enable verbosity level to debug console.
-h or --help
    Display this usage info then exit.

<return-to-top>

 


cmtest - Connection Manager Tests

Usage: cmtest [options]

    Options:

 -s --server This option directs cmtest to act as a Server
 -l --local This option specifies the local endpoint.
 -r --remote This option specifies the remote endpoint LID as a hex integer 0x; see vstat command for active port LID hex integer.
 -c --connect This option specifies the number of connections to open. Default of 1.
 -m --msize This option specifies the byte size of each message. Default is 100 bytes.
 -n --nmsgs This option specifies the number of messages to send at a time.
 -p --permsg This option indicates if a separate buffer should be used per message. Default is one buffer for all messages.
 -i --iterate This option specifies the number of times to loop through 'nmsgs'. Default of 1.
 -v --verbose This option enables verbosity level to debug console.
 -h --help Display this usage info then exit.

<return-to-top>

 

InfiniBand Partition Management

The part_man.exe application allows creating, deleting and viewing existing host partitions.

Usage : part_man.exe <show|add|rem> <port_guid> <pkey1 pkey2 ...>

show - – shows existing partitions

Expected results after execution part_man.exe show

1.      Output has a format 

port_guid1   pkey1  pkey2  pkey3  pkey4  pkey5  pkey6  pkey7  pkey8

port_guid2   pkey1  pkey2  pkey3  pkey4  pkey5  pkey6  pkey7  pkey8

where port_guid is a port guid in hexadecimal format, pkey – values of partition key (in hex format) for this port.

Default partition key (0xFFFF) is not shown and can not be created by the part_man.exe.

 

add - create new partition(s) on specified port

port_guid  add   <port_guid>  <pkey1>   <pkey2>

creates new partition(s) on port specified by port_guid parameter (in hexadecimal format) and pkey – new partition key value in hexadecimal format (e.g. 0xABCD or ABCD).

Port guid is taken form vstat output and has a following format:

XXXX:XXXX:XXXX:XXXX.

Vstat prints node guid, so user has to add 1 to node guid value to obtain port guid. For example, if node guid is 0008:f104:0397:7ccc, port guid will be

0008:f104:0397:7ccd – for the first port,

0008:f104:0397:7cce – for the second port.

 

Expected results of execution part_man.exe add 0x0D99:9703:04f1:0800 0xABCD

1.      part_man.exe output ends up with …Done message.

2.      A new instance of a Network Adapter named “OpenFabrics IPoIB Adapter Partition” will appear in Device manager window. 
If the new adapter appears with yellow label, manual device driver installation is required.

3.      New adapter name ends with “Partition”, e.g. “OpenFabrics IPoIB Adapter Partition”.

 

rem – removes partition key on specified port.

part_man.exe rem <port_guid> <pkey1>  <pkey2>

Port_guid – in hexadecimal format (same as for add command), identifies port for operation.

Expected results after execution part_man rem <port_guid>  <pkey>

1.      Application prints …Done message.

2.      In device manager window IPoIB network adapter will disappear.

3.      Execution of  part_man.exe show will not show removed adapter.

 

<return-to-top>

 


PrintIP - print ip adapters and their addresses

PrintIP is used to print IP adapters and their addresses, or ARP (Address Resolution Protocol) and IP address.

Usage:
    printip <print_ips>
    printip <remoteip> <ip>        (example printip remoteip 10.10.2.20)

<return-to-top>

 



vstat - HCA Stats and Counters

Display HCA (Host channel Adapter) attributes.

Usage: vstat [-v] [-c]
          -v - verbose mode
          -c - HCA error/statistic counters

Includes Node GUID, Subnet Manager and port LIDs.

<return-to-top>

 

Subnet Management with OpenSM Rev: openib-1.2.0


A single running process (opensm.exe) is required to configure and thus make an Infiniband subnet useable.  For most cases, InfiniBand Subnet Management as a Windows service is sufficient to correctly configure most InfiniBand fabrics.

The Infiniband subnet management process (opensm) may exist on a Windows (WinOF) node or a Linux (OFED) node.

Limit the number of OpenSM processes per IB fabric; one SM is sufficient although redundant SMs are supported. You do not need a Subnet Manager per node/system.

OpenIB Subnet Management as a Windows Service

InfiniBand subnet management (OpenSM), as a Windows service, is installed by default, although it is NOT started by default. There are two ways to enable the InfiniBand Subnet Management service.

  1. Reset the installed OpenSM service "InfiniBand Subnet Management" to start automatically; See
    My Computer->Manage->Services and Applications->Services->InfiniBand subnet Management->Start.
     
  2. Install OpenSM as a 'running' Windows service:
    Request a 'Custom' install, selecting the OpenSM_service install feature. Once the installation has completed, check the running InfiniBand Subnet Management service status via the Windows service manager (see #1).
     
  3. Consult the OpenSM log file @ %SystemRoot%\Temp\osm.log to see what OpenSM thinks is happening.

 

Manual InfiniBand Subnet Management from a command window

Usage: opensm.exe [options]

Options:

-c
--cache-options

Cache the given command line options into the file
/var/cache/osm/opensm.opts for use next invocation
The cache directory can be changed by the environment
variable OSM_CACHE_DIR

-g[=]<GUID in hex>
--guid[=]<GUID in hex>

This option specifies the local port GUID value with which OpenSM should bind. OpenSM may be
bound to 1 port at a time.  If GUID given is 0, OpenSM displays a list of possible port GUIDs and waits for user input. Without -g, OpenSM trys to use the default port.

-l <LMC>
--lmc <LMC>

This option specifies the subnet's LMC value.
The number of LIDs assigned to each port is 2^LMC.
The LMC value must be in the range 0-7.
LMC values > 0 allow multiple paths between ports.
LMC values > 0 should only be used if the subnet
topology actually provides multiple paths between
ports, i.e. multiple interconnects between switches.
Without -l, OpenSM defaults to LMC = 0, which allows
one path between any two ports.

-p <PRIORITY>
--priority <PRIORITY>

This option specifies the SM's PRIORITY.
This will effect the handover cases, where master
is chosen by priority and GUID.
-smkey <SM_Key>
This option specifies the SM's SM_Key (64 bits).
This will effect SM authentication.

-r
--reassign_lids


This option causes OpenSM to reassign LIDs to all end nodes. Specifying -r on a running subnet
may disrupt subnet traffic.  Without -r, OpenSM attempts to preserve existing LID assignments resolving multiple use of same LID.

-u
--updn

This option activate UPDN algorithm instead of Min Hop algorithm (default).

-a
--add_guid_file <path to file>

Set the root nodes for the Up/Down routing algorithm to the guids provided in the given file (one per line)

-o
--once

This option causes OpenSM to configure the subnet once, then exit. Ports remain in the ACTIVE state.

-s <interval>
--sweep <interval>

This option specifies the number of seconds between subnet sweeps. Specifying -s 0 disables sweeping.
Without -s, OpenSM defaults to a sweep interval of 10 seconds.

-t <milliseconds>
--timeout <milliseconds>

This option specifies the time in milliseconds
used for transaction timeouts.
Specifying -t 0 disables timeouts.
Without -t, OpenSM defaults to a timeout value of
200 milliseconds.

-maxsmps <number>

This option specifies the number of VL15 SMP MADs allowed on the wire at any one time.
Specifying -maxsmps 0 allows unlimited outstanding SMPs.
Without -maxsmps, OpenSM defaults to a maximum of one outstanding SMP.

-i <equalize-ignore-guids-file>
-ignore-guids <equalize-ignore-guids-file>

This option provides the means to define a set of ports (by guids) that will be ignored by the link load  equalization algorithm.

-x
--honor_guid2lid

This option forces OpenSM to honor the guid2lid file, when it comes out of Standby state, if such file exists under OSM_CACHE_DIR, and is valid. By default this is FALSE.

-f
--log_file

This option names the OpenSM log file. By default the log goes to %SystemRoot%\Temp\osm.log when started as
a Windows service. When OpenSM.exe is run from a command prompt, the default log file is created as '%TEMP%\osm.log'.
For the log to go to standard output use -f stdout.

-e
--erase_log_file

This option will cause deletion of the log file  (if it previously exists). By default, the log file is accumulative.

-y
--stay_on_fatal

This option will cause SM not to exit on fatal initialization issues: if SM discovers duplicated guids or 12x link with lane reversal badly configured. By default, the SM will exit on these errors.

-v
--verbose

This option increases the log verbosity level. The -v option may be specified multiple times to further increase the verbosity level.  See the -vf option for more information about. log verbosity.

-V

This option sets the maximum verbosity level and forces log flushing.
The -V is equivalent to '-vf 0xFF -d 2'. See the -vf option for more information about log verbosity.

-D <flags>

This option sets the log verbosity level.  A flags field must follow the -D option.
A bit set/clear in the flags enables/disables a specific log level as follows:
BIT LOG LEVEL ENABLED
---- -----------------
0x01 - ERROR (error messages)
0x02 - INFO (basic messages, low volume)
0x04 - VERBOSE (interesting stuff, moderate volume)
0x08 - DEBUG (diagnostic, high volume)
0x10 - FUNCS (function entry/exit, very high volume)
0x20 - FRAMES (dumps all SMP and GMP frames)
0x40 - ROUTING (dump FDB routing information)
0x80 - currently unused.
Without -D, OpenSM defaults to ERROR + INFO (0x3).
Specifying -D 0 disables all messages.
Specifying -D 0xFF enables all messages (see -V).
High verbosity levels may require increasing the transaction timeout with the -t option.

-d <number>
--debug <number>

This option specifies a debug option. These options are not normally needed. The number following -d selects the debug option to enable as follows:
OPT Description
--- -----------------
-d0 - Ignore other SM nodes
-d1 - Force single threaded dispatching
-d2 - Force log flushing after each log message
-d3 - Disable multicast support
-d4 - Put OpenSM in memory tracking mode
-d10 - Put OpenSM in testability mode
Without -d, no debug options are enabled

-h
--help

Display this usage info then exit.

-?

Display this usage info then exit.

<return-to-top>

 


Osmtest - Subnet Management Tests

Invoke open subnet management tests. osmtest currently can not run on the same HCA port which OpenSM is currently using.

 Usage: osmtest [options]

Options:

 -f <c|a|v|s|e|f|m|q|t>
--flow <c|a|v|s|e|f|m|q|t>

This option directs osmtest to run a specific flow:

FLOW DESCRIPTIONS
c = create an inventory file with all nodes, ports & paths.
a = run all validation tests (expecting an input inventory)
v = only validate the given inventory file.
s = run service registration, un-registration and lease.
e = run event forwarding test.
f = flood the SA with queries accoring to the stress mode.
m = multicast flow.
q = QoS info - VLArb and SLtoVL tables.
t = run trap 64/65 flow; requires running an external tool.
(default is all but QoS).

-w <trap_wait_time>
--wait <trap_wait_time>

This option specifies the wait time for trap 64/65 in seconds.
It is used only when running -f t - the trap 64/65 flow
(default to 10 sec).

-d <number>
--debug <number>

This option specifies a debug option. These options are not normally needed.
The number following -d selects the debug option to enable as follows:
OPT Description
--- -----------------
-d0 - Unused.
-d1 - Do not scan/compare path records.
-d2 - Force log flushing after each log message.
-d3 - Use mem tracking.
Without -d, no debug options are enabled.

-m <LID in hex>
--max_lid <LID in hex>

This option specifies the maximal LID number to be searched for during inventory file build (default to 100).

-g <GUID in hex>
--guid <GUID in hex>

This option specifies the local port GUID value with which osmtest should bind. osmtest may be bound to 1 port at a time. Without -g, osmtest displays a menu of possible port GUIDs and waits for user input.

-h
--help

Display this usage info then exit.

-i <filename>
--inventory <filename>

This option specifies the name of the inventory file. Normally, osmtest expects to find an inventory file, which osmtest uses to validate real-time information received from the SA during testing. If -i is not specified, osmtest defaults to the file 'osmtest.dat'.
See the -c option for related information.

-s
--stress

This option runs the specified stress test instead of the normal test suite.
Stress test options are as follows:
OPT Description
--- -----------------
-s1 - Single-MAD response SA queries .
-s2 - Multi-MAD (RMPP) response SA queries.
-s3 - Multi-MAD (RMPP) Path Record SA queries.
Without -s, stress testing is not performed.

-M
--Multicast_Mode

This option specify length of Multicast test :
OPT Description
--- -----------------
-M1 - Short Multicast Flow (default) - single mode.
-M2 - Short Multicast Flow - multiple mode.
-M3 - Long Multicast Flow - single mode.
-M4 - Long Multicast Flow - multiple mode.
Single mode - Osmtest is tested alone , with no other
apps that interact vs. OpenSM MC.
Multiple mode - Could be run with other apps using MC vs.
OpenSM. Without -M, default flow testing is performed.

-t <milliseconds>

This option specifies the time in milliseconds used for transaction timeouts.
Specifying -t 0 disables timeouts.
Without -t, osmtest defaults to a timeout value of 1 second.

-l
--log_file

This option defines the log to be the given file.
By default the log goes to stdout.

-v

This option increases the log verbosity level. The -v option may be specified multiple times
to further increase the verbosity level. See the -vf option for more information about log verbosity.

-V

This option sets the maximum verbosity level and forces log flushing.
The -V is equivalent to '-vf 0xFF -d 2'.
See the -vf option for more information about log verbosity.

-vf <flags>

This option sets the log verbosity level. A flags field must follow the -vf option.
A bit set/clear in the flags enables/disables a specific log level as follows:
BIT LOG LEVEL ENABLED
---- -----------------
0x01 - ERROR (error messages)
0x02 - INFO (basic messages, low volume)
0x04 - VERBOSE (interesting stuff, moderate volume)
0x08 - DEBUG (diagnostic, high volume)
0x10 - FUNCS (function entry/exit, very high volume)
0x20 - FRAMES (dumps all SMP and GMP frames)
0x40 - currently unused.
0x80 - currently unused.
Without -vf, osmtest defaults to ERROR + INFO (0x3).
Specifying -vf 0 disables all messages.
Specifying -vf 0xFF enables all messages (see -V).
High verbosity levels may require increasing
the transaction timeout with the -t option.

<return-to-top>

 



ibtrapgen - Generate Infiniband subnet management traps

Usage: ibtrapgen -t|--trap_num <TRAP_NUM> -n|--number <NUM_TRAP_CREATIONS>
                          -r|--rate <TRAP_RATE> -l|--lid <LIDADDR>
                          -s|--src_port <SOURCE_PORT> -p|--port_num <PORT_NUM>

Options: one of the following optional flows:

-t <TRAP_NUM>
--trap_num <TRAP_NUM>
          This option specifies the number of the trap to generate. Valid values are 128-131.
-n <NUM_TRAP_CREATIONS>
--number <NUM_TRAP_CREATIONS>
          This option specifies the number of times to generate this trap.
          If not specified - default to 1.
-r <TRAP_RATE>
--rate <TRAP_RATE>
          This option specifies the rate of the trap generation.
          What is the time period between one generation and another?
          The value is given in miliseconds.
          If the number of trap creations is 1 - this value is ignored.
-l <LIDADDR>
--lid <LIDADDR>
          This option specifies the lid address from where the trap should be generated.
-s <SOURCE_PORT>
--src_port <SOURCE_PORT>
          This option specifies the port number from which the trap should
          be generated. If trap number is 128 - this value is ignored (since
          trap 128 is not sent with a specific port number)
-p <port num>
--port_num <port num>
          This is the port number used for communicating with the SA.
-h
--help
          Display this usage info then exit.
-o
--out_log_file
          This option defines the log to be the given file.
          By default the log goes to stdout.
-v
          This option increases the log verbosity level.
          The -v option may be specified multiple times to further increase the verbosity level.
          See the -vf option for more information about log verbosity.
-V
          This option sets the maximum verbosity level and forces log flushing.
          The -V is equivalent to '-vf 0xFF -d 2'.
          See the -vf option for more information about. log verbosity.
-x <flags>
          This option sets the log verbosity level.
          A flags field must follow the -vf option.
          A bit set/clear in the flags enables/disables a
          specific log level as follows:

BIT LOG LEVEL ENABLED
---- -----------------
0x01 - ERROR (error messages)
0x02 - INFO (basic messages, low volume)
0x04 - VERBOSE (interesting stuff, moderate volume)
0x08 - DEBUG (diagnostic, high volume)
0x10 - FUNCS (function entry/exit, very high volume)
0x20 - FRAMES (dumps all SMP and GMP frames)
0x40 - currently unused.
0x80 - currently unused.
Without -x, ibtrapgen defaults to ERROR + INFO (0x3).
Specifying -x 0 disables all messages.
Specifying -x 0xFF enables all messages (see -V).

<return-to-top>

 

 

IPoIB - Internet Protocols over InfiniBand


IPoIB enables the use of Internet Protocol utilities (e.g., ftp, telnet) to function correctly over an Infiniband fabric. IPoIB is implemented as an NDIS Miniport driver with a WDM lower edge.

The IPoIB Network adapters are located via 'My Computer->Manage->Device Manager->Network adapters->IPoIB'.
'My Network Places->Properties' will display IPoIB Local Area Connection instances and should be used to configure IP addresses for the IPoIB interfaces; one Local Area Connection instance per HCA port. The IP (Internet Protocol) address bound to the IPoIB adapter instance can be assigned by DHCP or as a static IP addresses via
'My Network Places->Properties->Local Area Connection X->Properties->(General Tab)Internet Protocol(TCP/IP)->Properties'.

When the subnet manager (opensm) configures/sweeps the local Infiniband HCA, the Local Area Connection will become enabled. If you discover the Local Area Connection to be disabled, then likely your subnet manager (opensm) is not running or functioning correctly.

IPoIB Partition Management

<return-to-top>

 

 

Winsock Direct Service Provider


Winsock Direct (WSD) is Microsoft's proprietary protocol that predates SDP (Sockets Direct Protocol) for accelerating TCP/IP applications by using RDMA hardware. Microsoft had a significant role in defining the SDP protocol, hence SDP and WSD are remarkably similar, though unfortunately incompatible.

WSD is made up of two parts, the winsock direct switch and the winsock direct provider. The WSD switch is in the winsock DLL that ships in all editions of Windows Server 2003/2008, and is responsible for routing socket traffic over either the regular TCP/IP stack, or offload it to a WSD provider. The WSD provider is a hardware specific DLL that implements connection management and data transfers over particular RDMA hardware.

WinOF WSD is not supported in the Windows XP environment.

The WSD Protocol seamlessly transports TCP data using Infiniband data packets in 'buffered' mode or Infiniband RDMA in 'direct' mode. Either way the user mode socket application sees no behavioral difference in the standard Internet Protocol socket it created other than reduced data transfer times and increased bandwidth.

The Windows OpenFabrics release includes a WSD provider library that has been extensively tested with Microsoft Windows Server 2003.
During testing, bugs where found in the WSD switch that could lead to hangs, crashes, data corruption, and other unwanted behavior. Microsoft released a hotfix to address these issues which should be installed if using WSD; the Microsoft Windows Server 2003 hotfix can be found here.
Windows Server 2003 (R2) no longer requires this patch, nor does Windows Server 2008.
 

Environment variables can be used to change the behavior of the WSD provider:

IBWSD_NO_READ - Disables RDMA Read operations when set to any value. Note that this variable must be used consistently throughout the cluster or communication will fail.

IBWSD_POLL - Sets the number of times to poll the completion queue after processing completions in response to a CQ event. Reduces latency at the cost of CPU utilization. Default is 500.

IBWSD_SA_RETRY - Sets the number of times to retry SA query requests. Default is 4, can be increased if connection establishment fails.

IBWSD_SA_TIMEOUT - Sets the number of milliseconds to wait before retrying SA query requests. Default is 4, can be increased if connection establishment fails.

IBWSD_NO_IPOIB - SA query timeouts by default allow the connection to be established over IPoIB. Setting this environment variable to any value prevents fall back to IPoIB if SA queries time out.

IBWSD_DBG - Controls debug output when using a debug version of the WSD provider. Takes a hex value, with leading '0x', default value is '0x80000000'

 
0x00000001 DLL
0x00000002 socket info
0x00000004 initialization code
0x00000008 WQ related functions
0x00000010 Enpoints related functions
0x00000020 memory registration
0x00000040 CM (Connection Manager)
0x00000080 connections
0x00000200 socket options
0x00000400 network events
0x00000800 Hardware
0x00001000 Overlapped I/O request
0x00002000 Socket Duplication
0x00004000 Performance Monitoring
0x01000000 More verbose than IBSP_DBG_LEVEL3
0x02000000 More verbose than IBSP_DBG_LEVEL2
0x04000000 More verbose than IBSP_DBG_LEVEL1
0x08000000 Verbose output
0x20000000 Function enter/exit
0x40000000 Warnings
0x80000000 Errors


See https://wiki.openfabrics.org/tiki-index.php?page=Winsock+Direct for the latest WSD status.

Winsock Direct Service Provider Installation

WSD service is automatically installed and started as part of the 'default' installation; except on XP systems - WSD not supported.
Manual control is performed via the \Program Files\WinOF\installsp.exe utility.

usage: installsp [-i | -r | -l]

-i    Install the Winsock Direct (WSD) service provider
-r    Remove the WSD service provider
-r <name>    Remove the specified service provider
-l    List service providers
 

<return-to-top>

 

Network Direct Service Provider


Network Direct Service Provider Installation

ND service is automatically installed and started as part of the 'default' installation for Windows server 2008, Vista or HPC systems.
Manual control is performed via the %windir%\system32\ndinstall.exe utility.

usage: ndinstall [-i | -r | -l]

-i    Install (enable) the Network Direct (ND) service provider
-r    Remove the ND service provider
-r <name>    Remove the specified service provider
-l    List service providers

The Microsoft Network Direct SDK can be downloaded from here.  Once the ND SDK is installed, ND test programs can be located @
%ProgramFiles%\Microsoft HPC Pack 2008 SDK\NetworkDirect\Bin\amd64\ as nd*.exe.

Known working ND test command invocations (loopback or remote host)

svr: ndrpingpong s IPoIB_IPv4_addr 4096 p1
cli: ndrpingpong c IPoIB_IPv4_addr 4096 p1

svr: ndpingpong s IPoIB_IPv4_addr 4096 b1
cli: ndpingpong c IPoIB_IPv4_addr 4096 b1

See ndping.exe /? for details.

<return-to-top>

 

Usermode Direct Access Transport and Direct Access Programming Libraries


The DAT (Direct Access Transport) API is a C programming interface developed by the DAT Collaborative in order provide a set of transport-independent, platform-independent Application Programming Interfaces that exploit the RDMA (remote direct memory access) capabilities of next-generation interconnect technologies such as InfiniBand, and iWARP.

WinOF DAT and DAPL are based on the 1.1 DAT specification. The DAPL (Direct Access Provider Library) which now fully supports Infiniband RDMA and IPoIB.

WinOF 1.0.1, and future WinOF releases, will include DAT/DAPL version 2.0 runtime libraries along with an optional v2.0 application build environment.
DAT 2.0 is configured with InfiniBand extensions enabled. The IB extensions include


How  DAT objects map to equivalent InfiniBand objects:
 
Interface Adapter (IA) HCA (Host Channel Adapter)
Protection Zone (PZ) PD (Protection Domain)
Local Memory Region (LMR) MR (Memory Region)
Remote Memory Region (RMR) MW (Memory Windows)
Event Dispatcher (EVD) CQ (Completion Queue)
Endpoint (EP) QP (Queue Pair)
Public Service Point (PSP) connection identifier
Reserved Service Point (RSP) connection identifier
Connection Request (CR) connection manager event

<return-to-top>

DAT ENVIRONMENT:

DAT/DAPL v1.1 (free-build) runtime libraries are installed into %SystemRoot%, with the v1.1 Debug versions located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.  Debug libraries are identified as datd.dll and dapld.dll.

IA32 (aka, 32-bit) versions of DAT/DAPL 1.1 runtime libraries, found only on 64-bit systems, are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat32.dll and dapl32.dll.

DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as dat2.dll and dapl2.dll.  Debug versions of the v2.0 runtime libraries are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.

IA32 (aka, 32-bit) versions of DAT/DAPL 2.0 runtime libraries, found only on 64-bit systems, are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat232.dll and dapl232.dll.

DAT/DAPL 2.0 (free-build) libraries utilize the IBAL (eye-ball) InfiniBand Access Layer Connection Manager (CM) to establish IB reliable connections to Windows based system. To facilitate DAT v2.0 Windows to Linux InfiniBand communications an interim BSD socket based Connection Manager (sock-cm) is provided. The DAPL socket-CM provider is installed as '%SystemRoot%\dapl2-scm.dll' and listed in the %SystemDrive%\DAT\dat.conf provider file as 'ibnic0v2-scm'. Both nodes must use the same Connection Manager IBAL-CM[ibnic0v2: dapl2.dll] or Socket-CM[ibnicv2-scm: dapl2-scm.dll] in order for connections to be established.

In order for DAT/uDAPL programs to execute correctly, the runtime library files 'dat.dll and dapl.dll' must be present in one of the following folders: current directory, %SystemRoot% or in the library search path.

The default WinOF installation places the runtime library files dat.dll and dapl.dll in the '%SystemRoot%' folder; symbol files (.pdb) are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.

The default DAPL configuration file is defined as '%SystemDrive%\DAT\dat.conf'. This default specification can be overriden by use of the environment variable DAT_OVERRIDE; see following environment variable discussion.

Within the dat.conf file, the DAPL library specification can be located as the 5th whitespace separated line argument. By default the DAPL library file is installed as '%SystemRoot%\dapl.dll'.

Should you choose to relocated the DAPL library file to a path where whitespace appears in the full library path specification, then the full library file specification must be contained within double-quotes. A side effect of the double-quotes is the library specification is treated as a Windows string which implies the '\' (backslash character) is treated as an 'escape' character.  Hence all backslashes in the library path must be duplicated when enclosed in double-quotes (e.g., "C:\\Programs Files\\WinOF\\dapl.dll").

A sample InfiniBand dat.conf file is installed as '\Program Files\WinOF\dat.conf'.  If dat.conf does not exist in the DAT default configuration folder '%SystemDrive%\DAT\', dat.conf will be copied there.
 

DAT application build environment:

DAT library header files are selectively installed in the DAT default configuration folder as
'%SystemDrive%\DAT\v1-1' or '%SystemDrive%\DAT\v2-0'. Your C language based DAT 1.1 application compilation command line should include'/I%SystemDrive%\DAT\v1-1' with C code referencing '#include <DAT\udat.h>'.

The 'default' DAT/DAPL C language calling convention is '__stdcall', not the 'normal' Visual Studio C compiler default. __stdcall was chosen as MS recommended it to be more efficient. An application can freely mix default C compiler linkages '__cdecl' and '__stdcall'.

Visual Studio 2005 command window - (nmake) Makefile Fragments:

DAT_PATH=%SystemDrive%\DAT\v1-1
CC = cl
INC_FLAGS = /I $(DAT_PATH)

CC_FLAGS= /nologo /Gy /W3 /Gm- /GR- /GF /O2 /Oi /Oy- /D_CRT_SECURE_NO_WARNINGS \
            /D_WIN64 /D_X64_ /D_AMD64_ $(INC_FLAGS)

LINK = link
LIBS = ws2_32.lib advapi32.lib User32.lib bufferoverflowU.lib dat.lib

LINK_FLAGS = /nologo /subsystem:console /machine:X64 /libpath:$(DAT_PATH) $(LIBS)


When linking a DEBUG/Checked version make sure to use datd.lib or dat2d.lib for DAT v2.0.

DAT library environment variables:
 

DAT_OVERRIDE
------------
Value used as the static registry configuration file, overriding the
default location, 'C:\DAT\dat.conf'.

Example: set DAT_OVERRIDE=%SystemDrive%\path\to\my\private.conf


DAT_DBG_LEVEL
-------------

Value specifies which parts of the registry will print debugging
information, valid values are 

DAT_OS_DBG_TYPE_ERROR        = 0x1
DAT_OS_DBG_TYPE_GENERIC      = 0x2
DAT_OS_DBG_TYPE_SR           = 0x4
DAT_OS_DBG_TYPE_DR           = 0x8
DAT_OS_DBG_TYPE_PROVIDER_API = 0x10
DAT_OS_DBG_TYPE_CONSUMER_API = 0x20
DAT_OS_DBG_TYPE_ALL          = 0xff

or any combination of these. For example you can use 0xC to get both 
static and dynamic registry output.

Example set DAT_DBG_LEVEL=0xC

DAT_DBG_DEST
------------ 

Value sets the output destination, valid values are 

DAT_OS_DBG_DEST_STDOUT = 0x1
DAT_OS_DBG_DEST_SYSLOG = 0x2 
DAT_OS_DBG_DEST_ALL    = 0x3 

For example, 0x3 will output to both stdout and the syslog. 

DAPL Provider library environment variables


DAPL_DBG_TYPE
-------------

Value specifies which parts of the registry will print debugging information, valid values are

DAPL_DBG_TYPE_ERR          = 0x0001
DAPL_DBG_TYPE_WARN         = 0x0002
DAPL_DBG_TYPE_EVD          = 0x0004
DAPL_DBG_TYPE_CM           = 0x0008
DAPL_DBG_TYPE_EP           = 0x0010
DAPL_DBG_TYPE_UTIL         = 0x0020
DAPL_DBG_TYPE_CALLBACK     = 0x0040
DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080
DAPL_DBG_TYPE_API          = 0x0100
DAPL_DBG_TYPE_RTN          = 0x0200
DAPL_DBG_TYPE_EXCEPTION    = 0x0400

or any combination of these. For example you can use 0xC to get both
EVD and CM output.

Example set DAPL_DBG_TYPE=0xC


DAPL_DBG_DEST
-------------

Value sets the output destination, valid values are

DAPL_DBG_DEST_STDOUT = 0x1
DAPL_DBG_DEST_SYSLOG = 0x2
DAPL_DBG_DEST_ALL    = 0x3

For example, 0x3 will output to both stdout and the syslog.


<return-to-top>


DAPLTEST


    dapltest - test for the Direct Access Provider Library (DAPL)

DESCRIPTION

    Dapltest is a set of tests developed to exercise, characterize,
    and verify the DAPL interfaces during development and porting.
    At least two instantiations of the test must be run.  One acts
    as the server, fielding requests and spawning server-side test
    threads as needed.  Other client invocations connect to the
    server and issue test requests.

    The server side of the test, once invoked, listens continuously
    for client connection requests, until quit or killed.  Upon
    receipt of a connection request, the connection is established,
    the server and client sides swap version numbers to verify that
    they are able to communicate, and the client sends the test
    request to the server.  If the version numbers match, and the
    test request is well-formed, the server spawns the threads
    needed to run the test before awaiting further connections.

USAGE

    dapltest [ -f script_file_name ]
             [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]

    With no arguments, dapltest runs as a server using default values,
    and loops accepting requests from clients.  The -f option allows
    all arguments to be placed in a file, to ease test automation.
    The following arguments are common to all tests:

    [ -T S|Q|T|P|L ]    Test function to be performed:
                            S   - server loop
                            Q   - quit, client requests that server
                                  wait for any outstanding tests to
                                  complete, then clean up and exit
                            T   - transaction test, transfers data between 
                                  client and server
                            P   - performance test, times DTO operations
                            L   - limit test, exhausts various resources,
                                  runs in client w/o server interaction
                        Default: S

    [ -D device_name ]  Specifies the name of the device (interface adapter).
                        Default: host-specific, look for DT_MdepDeviceName
                                 in dapl_mdep.h

    [ -d ]              Enables extra debug verbosity, primarily tracing
			of the various DAPL operations as they progress.
			Repeating this parameter increases debug spew.
			Errors encountered result in the test spewing some
			explanatory text and stopping; this flag provides
			more detail about what lead up to the error.
                        Default: zero

    [ -R BE ]           Indicate the quality of service (QoS) desired.
                        Choices are:
                            HT  - high throughput
                            LL  - low latency
                            EC  - economy (neither HT nor LL)
                            PM  - premium
                            BE  - best effort
                        Default: BE

USAGE - Quit test client

    dapltest [Common_Args] [ -s server_name ]

    Quit testing (-T Q) connects to the server to ask it to clean up and
    exit (after it waits for any outstanding test runs to complete).
    In addition to being more polite than simply killing the server,
    this test exercises the DAPL object teardown code paths.
    There is only one argument other than those supported by all tests:

    -s server_name      Specifies the name of the server interface.
                        No default.


USAGE - Transaction test client

    dapltest [Common_Args] [ -s server_name ]
             [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ] 
             [ -V ] [ -P ] OPclient OPserver [ op3, 

    Transaction testing (-T T) transfers a variable amount of data between 
    client and server.  The data transfer can be described as a sequence of 
    individual operations; that entire sequence is transferred 'iterations' 
    times by each thread over all of its endpoint(s).

    The following parameters determine the behavior of the transaction test:

    -s server_name      Specifies the hostname of the dapltest server.
                        No default.

    [ -t threads ]      Specify the number of threads to be used.
                        Default: 1

    [ -w endpoints ]    Specify the number of connected endpoints per thread.
                        Default: 1

    [ -i iterations ]   Specify the number of times the entire sequence
                        of data transfers will be made over each endpoint.
                        Default: 1000

    [ -Q ]              Funnel completion events into a CNO.
			Default: use EVDs

    [ -V ]              Validate the data being transferred.
			Default: ignore the data

    [ -P ]		Turn on DTO completion polling
			Default: off

    OP1 OP2 [ OP3, ... ]
                        A single transaction (OPx) consists of:

                        server|client   Indicates who initiates the
                                        data transfer.

                        SR|RR|RW        Indicates the type of transfer:
                                        SR  send/recv
                                        RR  RDMA read
                                        RW  RDMA write
                        Defaults: none

                        [ seg_size [ num_segs ] ]
                                        Indicates the amount and format
                                        of the data to be transferred.
                                        Default:  4096  1
                                                  (i.e., 1 4KB buffer)

                        [ -f ]          For SR transfers only, indicates
                                        that a client's send transfer
                                        completion should be reaped when
                                        the next recv completion is reaped.
					Sends and receives must be paired
					(one client, one server, and in that
					order) for this option to be used.

    Restrictions:  
    
    Due to the flow control algorithm used by the transaction test, there 
    must be at least one SR OP for both the client and the server.  

    Requesting data validation (-V) causes the test to automatically append 
    three OPs to those specified. These additional operations provide 
    synchronization points during each iteration, at which all user-specified 
    transaction buffers are checked. These three appended operations satisfy 
    the "one SR in each direction" requirement.

    The transaction OP list is printed out if -d is supplied.

USAGE - Performance test client

    dapltest [Common_Args] -s server_name [ -m p|b ]
             [ -i iterations ] [ -p pipeline ] OP

    Performance testing (-T P) times the transfer of an operation.
    The operation is posted 'iterations' times.

    The following parameters determine the behavior of the transaction test:

    -s server_name      Specifies the hostname of the dapltest server.
                        No default.

    -m b|p		Used to choose either blocking (b) or polling (p)
                        Default: blocking (b)

    [ -i iterations ]   Specify the number of times the entire sequence
                        of data transfers will be made over each endpoint.
                        Default: 1000

    [ -p pipeline ]     Specify the pipline length, valid arguments are in 
                        the range [0,MAX_SEND_DTOS]. If a value greater than 
                        MAX_SEND_DTOS is requested the value will be
                        adjusted down to MAX_SEND_DTOS.
                        Default: MAX_SEND_DTOS

    OP
                        An operation consists of:

                        RR|RW           Indicates the type of transfer:
                                        RR  RDMA read
                                        RW  RDMA write
                        Default: none

                        [ seg_size [ num_segs ] ]
                                        Indicates the amount and format
                                        of the data to be transferred.
                                        Default:  4096  1
                                                  (i.e., 1 4KB buffer)

USAGE - Limit test client

    Limit testing (-T L) neither requires nor connects to any server
    instance.  The client runs one or more tests which attempt to
    exhaust various resources to determine DAPL limits and exercise
    DAPL error paths.  If no arguments are given, all tests are run.

    Limit testing creates the sequence of DAT objects needed to
    move data back and forth, attempting to find the limits supported
    for the DAPL object requested.  For example, if the LMR creation
    limit is being examined, the test will create a set of
    {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to
    failure using that set of DAPL objects.  The 'width' parameter
    can be used to control how many of these parallel DAPL object
    sets are created before beating upon the requested constructor.
    Use of -m limits the number of dat_*_create() calls that will
    be attempted, which can be helpful if the DAPL in use supports
    essentailly unlimited numbers of some objects.

    The limit test arguments are:

    [ -m maximum ]      Specify the maximum number of dapl_*_create()
                        attempts.
                        Default: run to object creation failure

    [ -w width ]        Specify the number of DAPL object sets to
                        create while initializing.
                        Default: 1

    [ limit_ia ]        Attempt to exhaust dat_ia_open()

    [ limit_pz ]        Attempt to exhaust dat_pz_create()

    [ limit_cno ]       Attempt to exhaust dat_cno_create()

    [ limit_evd ]       Attempt to exhaust dat_evd_create()

    [ limit_ep ]        Attempt to exhaust dat_ep_create()

    [ limit_rsp ]       Attempt to exhaust dat_rsp_create()

    [ limit_psp ]       Attempt to exhaust dat_psp_create()

    [ limit_lmr ]       Attempt to exhaust dat_lmr_create(4KB)

    [ limit_rpost ]     Attempt to exhaust dat_ep_post_recv(4KB)

    [ limit_size_lmr ]  Probe maximum size dat_lmr_create()

                        Default: run all tests


EXAMPLES

    dapltest -T S -d -D ibnic0

                        Starts a local dapltest server process with debug verbosity.
                        Server loops (listen for dapltest request, process request).
    
    dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2

                        Runs a transaction test, with both sides
                        sending one buffer with two 4KB segments,
                        one hundred times; dapltest server is on host winIB.

    dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2

                        Runs a performance test, with the client 
                        RDMA writing one buffer with two 4KB segments,
                        one hundred times.

    dapltest -T Q -s winIB -D ibnic0

                        Asks the dapltest server at host 'winIB' to clean up and exit.

    dapltest -T L -D ibnic0 -d -w 16 -m 1000

                        Runs all of the limit tests, setting up
                        16 complete sets of DAPL objects, and
                        creating at most a thousand instances
                        when trying to exhaust resources.

    dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \
       client RW  4096 1    server RW  2048 4    \
       client SR  1024 4    server SR  4096 2    \
       client SR  1024 3 -f server SR  2048 1 -f

                        Runs a more complicated transaction test,
                        with two thread using four EPs each,
                        sending a more complicated buffer pattern
                        for a larger number of iterations,
                        validating the data received.
dt-svr.bat - DAPLtest server script; starts a DAPLtest server on the local node.
	dt-svr [-D [hex-debug-bitmask] ]
dt-cli.bat - DAPLtest client; drives testing by interacting with dt-svr.bat script.
	dt-cli host-IPv4-address testname
		example: dt-cli 10.10.2.20 trans
		         dt-cli -h  # outputs help text.
Verify dt-*.bat script is running same dapltest.exe(v1.1) or dapl2test.exe(v2.0)


BUGS  (and  To Do List)

    Use of CNOs (-Q) is not yet supported.

    Further limit tests could be added.

<return-to-top>

 

 

SRP (SCSI RDMA) Protocol Driver

The SCSI RDMA Protocol  (SRP) is an emerging industry standard protocol for utilizing block storage devices over an InfiniBand™ fabric. SRP is being defined in the ANSI T-10 committee.

WinOF SRP is a storage driver implementation that enables the SCSI RDMA protocol over an InfiniBand fabric.
The implementation conforms to the T10 Working Group draft http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf.

Software Dependencies

The SRP driver depends on the installation of the WinOF stack with a Subnet
Manager running somewhere on the IB fabric.

- Supported Operating Systems and Service Packs:
   o Windows XP SP3 x86 & x64
   o Windows Server 2008/Vista  (x86, x64)
   o Windows Server 2008 HPC (x64)
   o Windows Server 2003 SP2/R2 (x86, x64, IA64)

Testing Levels

The SRP driver has undergone basic testing against Mellanox Technologies' SRP Targets MTD1000 and MTD2000.
Additionally the Linux OFED 1.4 SRP target has been tested.
Testing included SRP target drive format, read, write and dismount/offline operations.
 

Installation

The WinOF installer does not install the SRP driver as part of a default installation.  If the SRP feature is selected in the custom features installation view, an InfiniBand SRP Miniport driver will be installed; see the device manager view under SCSI and RAID controllers.

The system device 'InfiniBand I/O Unit' (IOU) device is required for correct SRP operation.  The WinOF installer will install and load the IOU driver if the SRP feature is selected.  See the device manager view System Devices --> InfiniBand I/O Unit for conformation of correct IOU driver loading.

In order for the SRP miniport driver installation to complete, an SRP target must be detected by a Subnet Manager running somewhere on the InfiniBand fabric; either a local or remote Subnet Manager works.

SRP Driver Uninstall

If the SRP (SCSI RDMA Protocol) driver has been previously installed, then in order to achieve a 'clean' uninstall, the SRP target drive(s) must be released.  Unfortunately the 'offline disk' command is only valid for diskpart (ver 6.0.6001) which is not distributed with Windows Server 2003 or XP.

The consequences of not releasing the SRP target drive(s) are that after the WinOF uninstall reboot there are lingering InfiniBand driver files. These driver files remain because while the SRP target is active they have references, thus when the WinOF uninstall attempts to delete the files the operation fails.

SRP supports WPP tracing tools by using the GUID: '5AF07B3C-D119-4233-9C81-C07EF481CBE6'.  The flags and level of debug can be controlled at load-time or run-time; see ib_srp.inf file for details.

<return-to-top>

 

QLogic VNIC Configuration

The QLogic VNIC (Virtual Network Interface Card) driver in conjunction with the QLogic Ethernet Virtual I/O Controller (EVIC) provides virtual Ethernet interfaces and transport for Ethernet packets over Infiniband.

Users can modify NIC parameters through User Interface icon in Network Connections:
( Properties->"Configure..." button -> "Advanced" Tab).

Parameters available:

Vlan Id (802.1Q) 

  values from 0 to 4094 ( default 0, disabled )
  This specifies if VLAN ID-marked packet transmission is enabled and, if so, specifies the ID.

Priority (802.1P)

  values from 0 to 7 ( default 0, feature disabled)
  This specifies if priority-marked packet transmission is enabled.

Payload MTU size 

  values from 1500 to 9500 (default 1500)
  This specifies the maximum transfer unit size in 100 bytes increments.

Recv ChkSum offload 

  (default enabled)
  This specifies if IP protocols checksum calculations for receive is offloaded.

Send ChkSum offload

  (default enabled)
  This specifies if IP protocols checksum calculations for send is offloaded.
 

Secondary Path 

   (default disabled)
   Enabled - If more than one IB path to IOC exist then secondary IB instance of virtual port will be created and configured with the same parameters as primary one. Failover from Primary to Secondary IB path is transparent for user application sending data through associated NIC.

   Disabled – only one path at a time is allowed. If more than one path to IOC exists then failed path will be destroyed and next available path will be used for new connection. With this scenario there is a possibility new interface instance will be assigned different MAC address when other hosts compete for EVIC resources.
 

LBFO Bundle Id
   (default disabled) Enabling support for OS provided Load Balancing and Fail Over functionality on adapter level.
   If enabled group ID can be selected from predefined names.

 

Heartbeat interval

   configures interval for VNIC protocol heartbeat messages in milliseconds.
   0 – heartbeats disabled.

Note:
   To take advantage of the features supported by these options, ensure that the Ethernet gateway is also configured appropriately.  For example, if the Payload MTU for a VNIC interface is set to 4000, the MTU at the EVIC module must also be set at least 4000 for the setting to take effect.

 <return-to-top>

 

InfiniBand Software Development Kit

If selected during a WinOF install, the IB Software Development Kit will be installed as '%SystemDrive%\IBSDK'. Underneath the IBSDK\ folder you will find an include folder 'Inc\',  library definition files 'Lib\'  along with a 'Samples' folder.

Compilation:

Add the additional include path '%SystemDrive%\IBSDK\Inc'; resource files may also use this path.

Linking:

Add the additional library search path '%SystemDrive%\IBSDK\Lib'.

Include dependent libraries: ibal.lib and complib.lib, or ibal32.lib & complib32.lib for win32 applications on 64-bit platforms.

Samples:

 

<return-to-top>

 

WinVerbs



WinVerbs is a userspace verbs and communication management interface optimized
for the Windows operating system. Its lower interface is designed to support
any RDMA based device, including Infiniband and iWarp. Its upper interface is
capable of providing a low latency verbs interface, plus supports Microsoft's
Network Direct Interface, DAPL, and OFED libibverbs interfaces. It consists of
a userspace library and a kernel filter driver.

The WinVerbs driver loads as an upper filter driver for Infiniband controllers.
(Open source iWarp drivers for Windows are not yet available.) A corresponding
userspace library installs as part of the Winverbs driver installation package.
Additionally, a Windows port of the OFED libibverbs library and several test
programs are also included.

Available libibverbs test programs and their usage are listed
below. Note that not all listed options apply to all applications

ibv_rc_pingpong, ibv_uc_pingpong, ibv_ud_pingpong
no args start a server and wait for connection
-h <host>     connect to server at <host>
-p <port>     listen on/connect to port <port> (default 18515)
-d <dev>     use IB device <dev> (default first device found)
-i <port>      use port <port> of IB device (default 1)
-s <size>      size of message to exchange (default 4096)
-m <size>     path MTU (default 1024)
-r <dep>      number of receives to post at a time (default 500)
-n <iters>     number of exchanges (default 1000)
-l <sl>          service level value
-e                 sleep on CQ events (default poll)

ibv_send_bw, ibv_send_lat
ibv_read_bw, ibv_read_lat
ibv_write_bw, ibv_write_lat
no args start a server and wait for connection
-h <host>              connect to server at <host>
-p <port>              listen on/connect to port <port> (default 18515)
-d <dev>               use IB device <dev> (default first device found)
-i <port>               use port <port> of IB device (default 1)
-c <RC/UC/UD>  connection type RC/UC/UD (default RC)
-m <mtu>              mtu size (256 - 4096. default for hermon is 2048)
-s <size>               size of message to exchange (default 65536)
-a                          Run sizes from 2 till 2^23
-t <dep>                size of tx queue (default 300)
-g                          send messages to multicast group (UD only)
-r <dep>                make rx queue bigger than tx (default 600)
-n <iters>               number of exchanges (at least 2, default 1000)
-I <size>                max size of message to be sent in inline mode (default 400)
-b                          measure bidirectional bandwidth (default unidirectional)
-V                         display version number
-e                          sleep on CQ events (default poll)
-N                         cancel peak-bw calculation (default with peak-bw)

To verify correct WinVerbs and libibverbs installation, run ibv_devinfo. It
should report all RDMA devices in the system, along with limited port
attributes. Because of limitations in the WinOF stack, it is normal for it to
list several values as unknown.

<return-to-top>