Grading a company-name matcher exposed gaps in its answer key

A team building a lookup that maps company names in financial text to the correct company found that its evaluation depended on an answer key that had not been audited. A second model reviewed 891 items and found the key was rarely wrong but incomplete, with 226 cases where one company had two valid entries and 101 unresolved labels now given answers. After the audit, the lookup scored 0.927 on a held-out half of 629 matches. The authors note that news headlines remain harder to match than SEC filings, and that no human-labelled answer key yet exists for news text. They plan to test whether the score holds beyond SEC filings.