gVisor supports RDMA (Remote Direct Memory Access) networking, allowing sandboxed applications to use high-performance InfiniBand/RoCE hardware — for example, multi-node NCCL collectives in distributed AI/ML training — while keeping the host isolated from the workload.
To achieve this, gVisor implements a proxy driver inside the sandbox, henceforth
referred to as rdmaproxy. rdmaproxy proxies the application’s interactions
with the host’s RDMA driver, giving the sandboxed application access to the RDMA
verbs devices (/dev/infiniband/uverbs*) declared in its OCI spec. The
application — and libraries such as libibverbs and NCCL — can run unmodified
inside the sandbox and interact transparently with these devices. gVisor also
constructs a faithful view of the host’s RDMA topology under /sys so that
device-discovery and topology-detection logic behaves the same inside the
sandbox as on the host.
RDMA support is enabled with the --rdmaproxy flag and applies to containers
whose OCI spec lists one or more /dev/infiniband/uverbs* devices.
When combined with GPUs (see GPU Support), gVisor supports GPUDirect RDMA, allowing the NIC to transfer data directly to and from GPU memory.
RDMA support is under active development. The following limitations apply:
Mellanox NICs only. Only Mellanox ConnectX (mlx5) adapters are
currently supported. Support for additional vendors is planned.
Host kernel 5.12 or newer. rdmaproxy proxies the modern
RDMA_VERBS_IOCTL interface only; the legacy write(2) command interface
is not supported. The newest uverbs ioctl method it relies on, dma-buf
memory registration for GPUDirect RDMA, landed in Linux 5.12.
GPUDirect uses dma-buf only. GPU memory is registered with the NIC
through the dma-buf mechanism, which is the modern default. The legacy
nvidia-peermem kernel-module path is not supported.
Single-container sandboxes only. The RDMA devices must be declared in the OCI spec of the sandbox’s root container. Deployments where the devices appear only in a sub-container’s spec — such as a Kubernetes pod where the RDMA devices belong to an application container rather than the pod’s root — are not yet supported.
Device lifecycle at sandbox creation. The RDMA network devices must
still reside in the host network namespace when the sandbox is created
(runsc create), and must not be pre-placed into the sandbox’s network
namespace beforehand. They should then be moved, fully configured, into the
sandbox’s network namespace before the application connects RDMA queue pairs
— for example via an OCI createRuntime hook or between runsc create and
runsc start. Such setup is not unusual; Docker seems to be doing this as
well.
RoCE requires shared RDMA namespace mode. For RoCE (RDMA over Converged
Ethernet) devices, the host’s RDMA subsystem must be in the shared
network-namespace mode (rdma system set netns shared), which is the
default on most systems.
No checkpoint/restore. Sandboxes using RDMA devices cannot be checkpointed.