The NVMe bdev module supports two modes: failover and multipath. In failover mode, only one active connection is maintained and alternate paths are connected only during the switch-over. This can lead to delays and failed I/O reported to upper layers, but it does reduce the number of active connections at any given time. In multipath, active connections are maintained for every path and used based on a policy of either active-passive or active-active. The multipath mode also supports Asymmetric Namespace Access (ANA) and uses that to make policy decisions.
A user may establish connections on multiple independent paths to the same NVMe-oF subsystem for NVMe bdevs by calling the bdev_nvme_attach_controller RPC multiple times with the same NVMe bdev controller name. Additionally, the multipath parameter for this RPC must be set to "multipath" when connecting the second or later paths.
For each path created by the bdev_nvme_attach_controller RPC, an NVMe-oF controller is created. Then the set of namespaces presented by that controller are discovered. For each namespace found, the NVMe bdev module attempts to match it with an existing NVMe bdev. If it finds a match, it adds the given namespace as an alternate path. If it does not find a match, it creates a new NVMe bdev.
I/O and admin qpairs are necessary to access an NVMe-oF controller. A single admin qpair is created and is shared by all SPDK threads. To submit I/O without taking locks, for each SPDK thread, an I/O qpair is created as a dynamic context of an I/O channel for an NVMe-oF controller.
For each SPDK thread, the NVMe bdev module creates an I/O channel for an NVMe bdev and provides it to the upper layer. The I/O channel for the NVMe bdev has an I/O path for each namespace. I/O path is an additional abstraction to submit I/O to a namespace, and consists of an I/O qpair context and a namespace. If an NVMe bdev has multiple namespaces, an I/O channel for the NVMe bdev has a list of multiple I/O paths. The I/O channel for the NVMe bdev has a retry I/O list and has a path selection policy.
If the NVMe driver detects an error on a qpair, it disconnects the qpair and notifies the error to the NVMe bdev module. Then the NVMe bdev module starts resetting the corresponding NVMe-oF controller. The NVMe-oF controller reset consists of the following steps: 1) disconnect and delete all I/O qpairs, 2) disconnect admin qpair, 3) connect admin qpair, 4) configure the NVMe-oF controller, and 5) create and connect all I/O qpairs.
If the step 3, 4, or 5 fails, the reset reverts to the step 3 and then it is retried after reconnect_delay_sec seconds. Then the NVMe-oF controller is deleted automatically if it is not recovered within ctrlr_loss_timeout_sec seconds. If ctrlr_loss_timeout_sec is -1, it retries indefinitely.
By default, error detection on a qpair is very slow for TCP and RDMA transports. For fast error detection, a global option, transport_ack_timeout, is useful.
Multipath mode supports two path selection policies, active-passive or active-active.
For both path selection policies, only ANA optimal I/O paths are used unless there are no ANA optimal I/O paths available.
For active-passive policy, each I/O channel for an NVMe bdev has a cache to store the first found I/O path which is connected and optimal from ANA and use it for I/O submission. Some users may want to specify the preferred I/O path manually. They can dynamically set the preferred I/O path using the bdev_nvme_set_preferred_path RPC. Such assignment is realized naturally by moving the I/O path to the head of the I/O path list. By default, if the preferred I/O path is restored, failback to it is done automatically. The automatic failback can be disabled by a global option disable_auto_failback. In this case, the bdev_nvme_set_preferred_path RPC can be used to do manual failback.
The active-active policy uses the round-robin algorithm or the minimum queue depth algorithm. The round-robin algorithm submits an I/O to each I/O path in circular order. The minimum queue depth algorithm selects an I/O path and submits an I/Os to it according to the number of outstanding I/Os of each I/O qpair. For these path selection algorithms, the number of I/Os routed to the current I/O path before switching to another I/O path is configurable.
The NVMe bdev module has a global option, bdev_retry_count, to control the number of retries when an I/O is returned with error. Each I/O has a retry count. If the retry count of an I/O is less than the bdev_retry_count, the I/O is allowed to retry and the retry count is incremented.
NOTE: The bdev_retry_count is not directly used but is required to be non-zero for the process of multipath mode failing over to a different path because the retry count is checked first always when an I/O is returned with error.
Each I/O has a timer to schedule an I/O retry at a particular time in the future. Each I/O channel for an NVMe bdev has a sorted I/O retry list. Retried I/Os are inserted into the I/O retry list.
If an I/O is returned with error, the I/O completion handler in the NVMe bdev module executes the following steps:
Then the I/O retry poller is scheduled to the closest expiration. If there is no retried I/O, the I/O retry poller is stopped.
When submitting an I/O, there may be no available I/O path. If there is any I/O path which is recovering, the I/O is inserted to the I/O retry list with one second delay. This may result in queueing many I/Os indefinitely. To avoid such indefinite queueing, per NVMe-oF controller option, fast_io_fail_timeout_sec, is added. If the corresponding NVMe-oF controller is not recovered within fast_io_fail_timeout_sec seconds, the I/O is not queued to wait the recovery but returned with an I/O error to the upper layer.
If an I/O is returned with an ANA error or an ANA change notice event is received, the ANA log page may be changed. In this case, the NVMe bdev module reads the ANA log page to check the ANA state changes.
As described before, only ANA optimal I/O paths will be used unless there are no ANA optimal paths available.
If an I/O path is in ANA transition, i.e., its namespace reports the ANA inaccessible state or the ANA change state, the NVMe bdev module queues I/Os to wait until the namespace becomes accessible again. The ANA transition should end within the ANATT (ANA Transition Time) seconds. If the namespace does not report the ANA optimized state or the ANA accessible state within the ANATT seconds, I/Os are returned with an I/O error to the upper layer.
The NVMe driver supports I/O timeout for submitted I/Os. The NVMe bdev module provides three actions when an I/O timeout is notified from the NVMe driver, ABORT, RESET, or NONE. Users can choose one of the actions as a global option, action_on_timeout. Users can set different timeout values for I/O commands and admin commands by global options, timeout_us and timeout_admin_us.
For ABORT, the NVMe bdev module tries aborting the timed out I/O, and if failed, it starts the NVMe-oF controller reset. For RESET, the NVMe bdev module starts the NVMe-oF controller reset.
The following is an example to attach two NVMe-oF controllers and aggregate these into a single NVMe bdev controller Nvme0.
In this example, if these two NVMe-oF controllers have a shared namespace whose namespace ID is 1, a single NVMe bdev Nvme0n1 is created. For the NVMe bdev module, the default value of bdev_retry_count is 3 and I/O retry is enabled by default. ctrlr_loss_timeout_sec is set to -1 and reconnect_delay_sec is set to 20. Hence, NVMe-oF controller reconnect will be retried once per 20 seconds until it succeeds.
To confirm if multipath is configured correctly, two RPCs, bdev_get_bdevs and bdev_nvme_get_controllers are available.
To monitor the current multipath state, a RPC bdev_nvme_get_io_paths are available.
To configure the path selection policy, a RPC bdev_nvme_set_multipath_policy is available. The following is an example for a single NVMe bdev Nvme0n1 to set the path selection policy to active-active, set the path selector to round-robin, and set the number of I/Os routed to the current I/O path before switching to another I/O path to 10.
SPDK NVMe multipath is transport protocol independent. Heterogeneous multipath configuration (e.g., TCP and RDMA) is supported. However, in this type of configuration, memory domain is not available yet because memory domain is supported only by the RDMA transport now.
The RPCs, bdev_get_iostat and bdev_nvme_get_transport_statistics display I/O statistics but both are not aware of multipath.