fi_gni(7) Libfabric Programmer's Manual
The GNI Fabric Provider
The GNI provider runs on Cray XC (TM) systems utilizing the user-space Generic Network Interface (uGNI) which provides low-level access to the Aries interconnect. The Aries interconnect is designed for low-latency one-sided messaging and also includes direct hardware support for common atomic operations and optimized collectives.
The GNI provider runs on Cray XC systems running CLE 5.2 UP03 or higher using gcc version 4.9 or higher.
The GNI provider supports the following features defined for the libfabric API:
- Endpoint types
- The provider supports the FI_EP_RDM endpoint type.
- Address vectors
- The provider implements both the FI_AV_MAP and FI_AV_TABLE address vector types.
- Data transfer operations
The following data transfer interfaces are supported for a all endpoint types: FI_ATOMIC, FI_MSG, FI_RMA, FI_TAGGED. See DATA TRANSFER OPERATIONS below for more details.
- Completion events
- The GNI provider supports FI_CQ_FORMAT_CONTEXT, FI_CQ_FORMAT_MSG, FI_CQ_FORMAT_DATA and FI_CQ_FORMAT_TAGGED with wait objects of type FI_WAIT_NONE, FI_WAIT_FD, and FI_WAIT_MUTEX_COND.
- The GNI provider does not require any operation modes.
- For control progress, the GNI provider supports both FI_PROGRESS_AUTO and FI_PROGRESS_MANUAL, with a default set to auto. For data progress, FI_PROGRESS_MANUAL is supported. When progress is set to auto, a background thread runs to ensure that progress is made for asynchronous requests.
DATA TRANSFER OPERATIONS
Currently, the GNI provider only supports atomic operations supported directly by the Aries NIC. These include operations on 32- and 64-bit, signed and unsigned integer and floating point values. Specifically,
Basic (fi_atomic, etc.)
- FI_MIN, FI_MAX (no unsigned)
- FI_SUM (no 64-bit floating point)
- FI_BOR, FI_BAND, FI_BXOR (no floating point)
Fetching (fi_fetch_atomic, etc.)
- All of the basic operations as above
Comparison (fi_compare_atomic, etc.)
All FI_MSG operations are supported.
All FI_RMA operations are supported.
All FI_TAGGED operations are supported except
The GNI provider exposes low-level tuning parameters via a domain
fi_open_ops interface named FI_GNI_DOMAIN_OPS_1. The flags
parameter is currently ignored. The fi_open_ops function takes a
struct fi_gni_ops_domain parameter and populates it with the
set_val function sets the value of a given parameter; the
get_val function returns the current value. The currently supported
- Threshold message size at which a rendezvous protocol is used for FI_MSG data transfers. The value is of type uint32_t.
- Threshold message size at which RDMA is used for FI_RMA data transfers. The value is of type uint32_t.
- Initial size of the internal table data structure used to manage connections. The value is of type uint32_t.
- Maximum size of the internal table data structure used to manage connections. The value is of type uint32_t.
- Step size for increasing the size of the internal table data structure used to manage internal GNI connections. The value is of type uint32_t.
- Size of the virtual channel (VC) table used for managing remote connections. The value is of type uint32_t.
- Page size for GNI SMSG mailbox allocations. The value is of type uint32_t.
- Number of GNI SMSG mailboxes per allocation slab. The value is of type uint32_t.
- Maximum number of credits per GNI SMSG mailbox. The value is of type uint32_t.
- Maximum size of GNI SMSG messages. The value is of type uint32_t.
- Recommended GNI receive CQ size. The value is of type uint32_t.
- Recommended GNI transmit CQ size. The value is of type uint32_t.
- Maximum number of message retransmits before failure. The value is of type uint32_t.
- Enable or disable lazy deregistration of memory. The value is of type int32_t.
For more information on uGNI, see Using the GNI and DMAPP APIs (S-2446-3103, Cray Inc.). For more information on the GNI provider, see An Implementation of OFI libfabric in Support of Multithreaded PGAS Solutions (PGAS ‘15).