Most Kubernetes clusters misattribute GPU costs due to default metric blind spots

A developer building a GPU cost-tracking tool found that production Kubernetes clusters consistently failed to accurately report idle GPU usage, all due to unchanged default configurations. The widely used DCGM_FI_DEV_GPU_UTIL metric measures only whether any kernel is active on a GPU, not whether it is doing meaningful work, masking what the developer calls 'ghost work.' A misconfigured dcgm-exporter setting causes all GPU usage to be attributed to the monitoring tool itself rather than the actual workloads, making cost data appear complete while being entirely fictitious. Profiling counters needed to detect real versus idle GPU activity are disabled by default, so the most costly failure modes go undetected without any error or alert. The developer also warned that falling back to power-draw estimates for cost calculations can produce dangerously confident but inaccurate figures, arguing a cost tool that generates precise numbers from flawed inputs is more harmful than having no tool at all.