Four-Stage Pipeline Turns AI Agent Traces Into Fine-Tuning Datasets

Every AI agent running in production continuously generates traces — logs of prompts, tool calls, reasoning steps, and outcomes — that can serve as raw training data, though most teams never use them. A four-stage pipeline converts this telemetry into a curated fine-tuning dataset by first capturing all runs using standardized schemas like OpenTelemetry's GenAI semantic conventions. The second stage involves selective sampling, pulling roughly 500 runs from 50,000 weekly executions by combining random, stratified, and failure-weighted strategies rather than reviewing everything. Labelling and scoring against a defined spec follow, steps the article notes are most commonly skipped by engineering teams. Running this pipeline on a regular weekly cadence, rather than as a one-off export, is recommended to keep the dataset current and production-representative.