NVIDIA Linux GPU Display Driver, UVM driver (nvidia-uvm.ko): A race in the Unified Virtual Memory kernel module lets

Impact

A race in the Unified Virtual Memory kernel module lets a local process wedge or crash the driver. UVM is on the hot path for managed-memory CUDA workloads, so on a Linux training node this is a way for any job with GPU access to take out the whole host's GPUs. Ubuntu shipped it as a security update, so it is real on distro-packaged fleets.

Who can reach it

Any local user or container that has /dev/nvidia-uvm mapped in - which is every GPU container under the standard container toolkit configuration.

What to do

Install the fixed Linux GPU Display Driver branch. nvidia.ko / nvidia-uvm.ko cannot be replaced while any process holds a GPU, so plan a node drain: cordon the node, stop every CUDA job and GPU container, unload the modules or reboot, install, reload. Container runtimes that bind-mount the driver libraries (nvidia-container-toolkit) need restarting so running pods pick up the new userspace. No firmware flash.

References