Why Scheduling LLMs to Run Recurring Data Analysis Is a Silent Risk

A software engineer argues that using large language models to regenerate analytical queries on a recurring schedule introduces a hidden but serious flaw: the measurement itself can change between runs without any warning. Because LLMs may produce subtly different SQL each time — varying in join logic, NULL handling, or date boundaries — a shift in a reported metric could reflect a change in the query rather than a real-world trend. This makes it impossible to distinguish genuine data movement from measurement drift, quietly corrupting time-series data over months. The proposed fix is to use LLMs during an exploratory phase, then freeze the final query as versioned, human-reviewed code that runs deterministically on a schedule. This approach, already adopted by tools like dbt's Semantic Layer, ensures that any change to a metric definition becomes an explicit, auditable event rather than an invisible accident.