Why AI Agents Make Data Lineage Harder — and How to Fix It
As AI agents increasingly write, run, and self-modify data pipeline scripts, tracing the origin of a single changed number becomes far more complex than in traditional human-managed workflows. A data engineer argues that every database row written by an agent must carry four traceable identifiers: the run ID, the exact script commit (via a Git SHA), the original data source, and the key or identity that executed the process. Without these stamps directly on the data, lineage graphs built from configuration alone only describe what a pipeline was supposed to do, not what it actually did. The engineer also highlights that recording denied agent actions is valuable audit evidence, since it reveals what nearly happened. The core argument is that data lineage must reflect real execution history, not just intended architecture, especially as autonomous agents gain the ability to rewrite their own code mid-run.
