NVIDIA's built-in GPU accounting mode tracks usage even after processes exit
NVIDIA GPUs include a little-known driver-level accounting mode that logs per-process GPU usage, retaining records even after those processes have terminated. The feature, disabled by default, can be enabled via nvidia-smi and provides data on GPU utilization, memory usage, and active time per process. A developer exploring local GPU cost attribution found the tool useful but flagged key pitfalls: accounting state is lost if the driver reloads, and an empty log after a mode reset looks identical to a genuinely idle GPU. Further analysis revealed that the gpu_utilization column is sampled and unreliable, with 97.6% of logged GPU-milliseconds belonging to processes that falsely reported 0% utilization. The developer recommends relying on the time field — a duration accumulated by the driver — rather than utilization figures for accurate cost attribution.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in