FinAutoRubric Uses Multi-Agent Loop to Auto-Generate Financial Evaluation Rubrics

FinAutoRubric is a new framework designed to automate the creation of evaluation rubrics for financial research AI agents, addressing the high cost and inflexibility of manually written benchmarks. The system allows domain experts to write reusable guidance once, which is then applied across queries through a multi-agent pipeline consisting of a writer agent and a reviewer agent. The writer agent researches and drafts expected values for a given query, while the reviewer agent validates those values against sources and expert criteria. If the reviewer's confidence falls below a set threshold, the case is escalated to a human analyst for resolution. By separating reusable expert guidance from query-specific rubric generation, FinAutoRubric aims to encode institution-specific standards — such as compliance rules and data-source preferences — without embedding proprietary information into model training data.