NAME

The Network Direct Fabric Provider

OVERVIEW

The Network Direct provider enables applications using OFI to be run over any verbs hardware (Infiniband, iWarp and etc). It uses the Microsoft Network Direct SPI for network transport and provides a translation of OFI calls to appropriate Network Direct API calls. The Network Direct providers allows to OFI-based applications utilize zero-copy data transfers between applications, kernel-bypass I/O generation and one-sided data transfer operations on Microsoft Windows OS. An application is able to use OFI with Network Direct provider enabled on Windows OS to expose the capabilities of the networking devices if the hardware vendors of the devices implemented the Network Direct service provider interface (SPI) for their hardware.

SUPPORTED FEATURES

The Network Direct provider support the following features defined for the libfabric API:

Endpoint types
The provider support the FI_EP_MSG endpoint types.
Memory registration modes
The provider implements the FI_MR_BASIC memory registration mode.
Data transfer operations
The following data transfer interfaces are supported for the following endpoint types: FI_MSG, FI_RMA. See DATA TRANSFER OPERATIONS below for more details.
Modes
The Network Direct provider requires applications to support the following modes:
  • FI_LOCAL_MR for all applications.
Addressing Formats
Supported addressing formats include FI_SOCKADDR, FI_SOCKADDR_IN, FI_SOCKADDR_IN6
Progress
The Network Direct provider supports FI_PROGRESS_AUTO: Asynchronous operations make forward progress automatically.
Operation flags
The provider supports FI_INJECT, FI_COMPLETION, FI_TRANSMIT_COMPLETE, FI_INJECT_COMPLETE, FI_DELIVERY_COMPLETE, FI_SELECTIVE_COMPLETION
Completion ordering
RX/TX contexts: FI_ORDER_STRICT
Other supported features
Multiple input/output vector (IOV) is supported for FI_RMA read/write and FI_MSG receive/transmit operations.

LIMITATIONS

The Network Direct is an experimental provider. The full support of the Network Direct provider will be added to 1.6 release version of libfabric.

Memory Regions
Only FI_MR_BASIC mode is supported. Adding regions via s/g list is supported only upto s/g list size of 1. No support for binding memory regions to a counter.
Wait objects
Wait object and wait sets are not supported.
Resource Management
Application has to make sure CQs are not overrun as this cannot be detected by the provider.
Unsupported Endpoint types
FI_EP_DGRAM, FI_EP_RDM
Other unsupported features
Scalable endpoints, FABRIC_DIRECT
Unsupported features specific to MSG endpoints
FI_SOURCE, FI_TAGGED, FI_CLAIM, fi_ep_alias, shared TX context, operations.

RUNTIME PARAMETERS

The Network Direct provider checks for the following environment variables.

Variables specific to RDM endpoints

FI_NETDIR_INLINETHR
The size of the (default: 8 Kbyte):
  • Transmited data that can be inlined
  • Preposted data for the unexpected receive queue
FI_NETDIR_PREPOSTCNT
The number of pre-registered buffers between the endpoints that are not require internal ACK messages, must be a power of 2 (default: 8).
FI_NETDIR_PREPOSTBUFCNT
The number of preposted arrays of buffers, must be a power of 2 (default: 1).

Environment variables notes

The fi_info utility would give the up-to-date information on environment variables: fi_info -p netdir -e

SEE ALSO

fabric(7), fi_open_ops(3), fi_provider(7), fi_getinfo(3) fi_atomic(3)