Row-Level Data Provenance: Why Tracking Exact Values Matters More Than Column Lineage

A software developer has built a self-contained reference pipeline using dbt-core and DuckDB that captures data provenance at the row and field level, going beyond the schema-level lineage offered by tools like OpenLineage. While column-level lineage is a static fact derived from code and scales with schema size, value-level provenance is dynamic, must be captured at query execution time, and scales with data volume multiplied by source inputs per output. This fundamental difference in complexity creates an entirely new class of engineering challenges, including storage costs that can dwarf the original data if handled naively. The project addresses these challenges by storing predicates that describe contributing data sets rather than recording every individual contributing row. Key use cases include faster debugging of incorrect values, scoping the impact of data corrections, supporting compliance requirements, and surfacing uncertain or unverified data lineage rather than returning silently misleading results.