Retry-and-ledger pattern helps a solo developer run hundreds of asset batches safely

A developer lost roughly forty minutes after a Python batch script crashed on asset 61 of 120, leaving no record of which files had finished. In response, they built a retry-and-log approach centered on a plain text ledger that marks an asset done only after its write succeeds. Failed assets are retried three times with short pauses, then logged with their specific error rather than a full traceback. The pipeline runs one process at a time on a 1-vCPU, 2 GB server, using garbage collection and per-file subprocesses so memory is returned between steps. The author says the setup has processed batches yielding 39 finished products and about 856 MB of output, and that re-runs are designed to produce identical results.