Developer Builds Precision Eval System to Benchmark LLM Smart Contract Auditor

A developer behind spectr-ai, an LLM-powered smart contract auditing tool, built a structured evaluation framework to objectively measure performance across different AI models. The system uses a curated set of Solidity contracts with known vulnerabilities as ground truth, testing whether the model correctly identifies the vulnerability class and affected function rather than matching exact text output. Each evaluation run is scored on two metrics — recall, measuring how many real bugs were caught, and precision, measuring how many flags were legitimate — to avoid both dangerous misses and alert fatigue. The framework revealed a concrete regression when a newer model found a vulnerability but declined to report it due to prompt behavior, a failure that subjective testing would have missed. The developer argues that even a small set of labeled examples with measurable scoring is far more reliable than intuition when model updates ship frequently.