Linux kernel RDMA core (ib_umem / IB_MR_REREG_ACCESS re-registration): An RDMA memory region registered read-only can

Impact

An RDMA memory region registered read-only can be re-registered writable without the kernel ever re-pinning the underlying pages for write. The pin taken at first registration lacked FOLL_WRITE, so the NIC ends up with a writable rkey/lkey pointing at pages the kernel still believes are read-only - copy-on-write mappings, and page-cache pages backing shared files. This is a Dirty-COW-shaped primitive delivered over the RDMA verbs path: a tenant process inside a container with /dev/infiniband/uverbs* mapped in can write to memory it only ever had read access to, including page-cache pages backing binaries shared with other workloads on the node.

Who can reach it

Local to the node, but 'local' in a GPU cluster means any tenant container that was given the RDMA device nodes - which is every container that does RDMA, since GPUDirect and NCCL over IB require /dev/infiniband to be mapped in. No CAP_SYS_ADMIN, no CAP_NET_ADMIN. Sequence is deterministic: register an MR read-only, reregister with IB_MR_REREG_ACCESS adding write, then RDMA-write through it.

What to do

Kernel update carrying the ib_umem_check_rereg() helper and the per-driver call sites (mlx5, mlx4, hns, irdma). There is no configuration mitigation short of removing RDMA device access from tenant containers, which turns off GPUDirect RDMA and collapses collective bandwidth - so on a training fleet this is a drain-and-reboot, not a tunable.

References