HTCondor (S3 file transfer, daemon logs and job ClassAds): MULTI-TENANT ISOLATION: Pre-signed S3 URLs for a job's input

Impact

MULTI-TENANT ISOLATION: Pre-signed S3 URLs for a job's input and output are written into daemon logs and into the job ad. Anyone who can read the job queue or the logs gets working credentials to that tenant's private object storage - which on a GPU cluster is the training dataset and the checkpoints.

Who can reach it

Any user who can read job ClassAds (condor_q -l on another user's job in a default pool) or who has access to daemon log files on the access point.

What to do

Upgrade to HTCondor 9.0.10 or 9.5.1 and restart the daemons. Then rotate the S3 credentials used for job transfers and expire any outstanding pre-signed URLs, and purge or restrict the old daemon logs - the leaked URLs stay valid in the log files after you patch.

References