Why 'Free' AI Inference Can Still Cost You Dearly in Pipeline Time
Free-tier AI model access does not mean zero cost — when jobs queue behind others, teams pay in wall-clock time through delayed pipelines, idle CI runners, and repeated retries. The article argues that the critical metric is not token usage but time breakdown: how long a job waited, how long the model actually generated, and how often it had to retry. When queue wait dominates total elapsed time, the problem is scheduling, not model expense, and free shared capacity often makes this worse by absorbing demand through delays rather than scaling workers. The author proposes a lightweight instrumentation schema to log each inference attempt — including failures — capturing queued, started, and finished timestamps per job. Aggregating these spans by percentile rather than average is recommended to surface the outlier jobs that silently block users and inflate real-world costs.
