Live LLM test calls can cost tens of thousands of requests monthly before launch

An author warns that test suites calling a real model API are billed like user traffic, so repeated CI runs and watchers can add up quickly. Using stated assumptions of 100 calls per run, 30 runs a day and 22 working days, they estimate about 66,000 paid calls a month, or roughly $290 on GPT-5.6 at list price. The article notes that prompt caching does not avoid the billing for each newly written answer. It proposes a record-and-replay approach that saves model responses to files so later identical requests read from disk, with an eleven-line Python example. The author also says that running live calls only nightly drops the cost substantially.