Free-Tier Batch Job Survives Restart by Caching Model Calls as Files
A developer ran an overnight log-classification batch on a free server, only to find that a silent restart caused the job to re-ask questions it had already answered. The root problem was an incorrect definition of completion, along with appending output without any safeguard against duplication. Temperature zero was found not to guarantee identical responses, so the developer shifted focus to durably recording what the model actually returned. The solution borrows record-and-replay techniques from HTTP tooling: each request payload is hashed, and its response is saved under that hash. Repeat requests are then served from disk, and a replay mode allows the whole pipeline to run with the network disabled.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in