Evaluating AI routers' out-of-scope option with separate recall and false-rejection measures

A comparison of four decision models and five small language models tested how well they route requests to domains, including an explicit out-of-scope option. The test used 1,000 CLINC150 requests, 190 of which were truly out of scope and 810 of which belonged to supported domains. Results varied widely: one model caught 80% of out-of-scope requests but wrongly declined 100 valid ones, while another rejected only 18 valid requests yet missed 144 out-of-scope ones. The author argues that overall accuracy and a single rejection count hide these opposing errors. Recommended metrics include out-of-scope recall, false-rejection rate, and in-scope routing accuracy, with malformed answers tracked separately.