NVIDIA GPU Display Driver, GPU hardware performance counters: MULTI-TENANT ISOLATION: GPU performance counters

Impact

MULTI-TENANT ISOLATION: GPU performance counters are readable by any local user and leak enough about another process's GPU activity to reconstruct data it is processing - published work recovered neural-network structure and input images this way. On a shared GPU node, one tenant's job can profile a co-tenant's inference or training work through the counters. NVIDIA's answer was not a code fix but an access-control change, and the restriction was not on by default for a long time in several driver branches, so a large installed base ran exposed for years. Treat any node where profiling is unrestricted as offering no side-channel isolation between GPU contexts.

Who can reach it

Any local user or container that can open the GPU device and run a profiling-capable API (CUPTI, nvprof, Nsight) against the same physical GPU as the victim.

What to do

Update to a driver branch that ships the profiling restriction, then verify it is actually enforced - this is the step operators skip. On Linux check the nvidia module parameter NVreg_RestrictProfilingToAdminUsers and set it to 1 in /etc/modprobe.d, then reload the module (node drain) or reboot; on Windows apply the driver update and confirm the developer-mode registry setting is not re-enabling profiling. On a multi-tenant node also stop mapping profiling-capable device nodes into tenant containers. Note this is a mitigation, not a fix: MPS and time-sliced sharing still leave the counters shared at the hardware level, so the only hard isolation is one tenant per physical GPU or MIG.

References