NVIDIA Resiliency Extension: A race condition in the checkpointing core reaches information disclosure, data tampering

Impact

A race condition in the checkpointing core reaches information disclosure, data tampering and privilege escalation. Checkpoints are the training run's crown jewels - a tampering primitive here means silently poisoned model state that survives every restart.

Who can reach it

Local, low privileges. An account on a node participating in the checkpoint write path.

What to do

Update the Resiliency Extension per bulletin 5746 and rebuild training images. Cost: image rebuild and job restart. Verify checkpoint integrity out-of-band if you suspect exposure - a corrupted checkpoint does not announce itself.

References