NVIDIA Triton Inference Server: An invalid request triggers a divide-by-zero and kills the server process

Impact

An invalid request triggers a divide-by-zero and kills the server process. On a shared inference tier this is a noisy-neighbour weapon: one tenant's request kills the server process and takes every co-resident model with it, and the GPU sits idle until the pod restarts.

Who can reach it

Network. Anyone who can reach the Triton HTTP or gRPC endpoint. In most clusters that is anything on the pod network; where ingress is loosely scoped it is the internet. No authentication step exists in Triton itself to stop it.

What to do

Roll to the fixed Triton container image per bulletin 5687. Cost: an ordinary rolling deployment restart - no driver, firmware or node change. Worth pairing with an audit of Triton endpoint exposure, since almost every bug in this component is only interesting because the endpoint is reachable.

References