Developer Builds 40-Minute Harness to Test Free AI Endpoints for Secret Detection

A developer created a lightweight evaluation harness to test whether free AI model endpoints can reliably detect secrets in code pull requests before they are merged. The experiment used MonkeyCode's free model endpoint and free server option, running 30 synthetic code diffs — half containing realistic-looking secrets and half clean. The harness measured five metrics across repeated runs: accuracy, JSON validity, latency, variance, and failure modes, labeling each result as a true positive, false positive, false negative, parse error, or timeout. Key failure patterns identified include context loss on long diffs, JSON formatting drift where the model returns prose instead of structured output, and silent false negatives that could allow real secrets to ship undetected. The author emphasizes this is a repeatable template rather than a formal benchmark, intended to help small teams assess free endpoint reliability before integrating AI-based secret scanning into CI pipelines.