Why RunPod checkpoints vanish after a pod stops, and how to keep them

Users running training jobs on RunPod have reported that model checkpoints disappeared after they stopped the pod to save money, often without any error message. The cause is that default output paths and the Hugging Face cache sit on container storage, which is wiped when a pod stops or terminates, while only a network volume mounted at /workspace persists. The article recommends explicitly setting the checkpoint output directory to /workspace and exporting HF_HOME to a persistent path in the launch script. Eight user research sessions with ML engineers found checkpoint loss to be the most consistently reported failure. The author also suggests running a pre-flight storage check, such as the open-source ComputeFence tool, before a job begins.