Why F1 Score Alone Can Mislead You When Evaluating Security Scanning Tools

F1 score, the metric most commonly cited in static analysis benchmarks, can assign identical scores to tools with completely opposite behaviors. A scanner that flags everything achieves 100% recall but near-zero precision, while one that rarely fires may hit 100% precision but miss almost all vulnerabilities — yet both can score the same F1. The metric is a harmonic mean of precision and recall, which penalizes imbalance but does not reveal which measure has failed. Developers experience poor precision through alert fatigue, while attackers benefit from poor recall since missed vulnerabilities reach production. Experts recommend examining the raw counts — true positives, false positives, and false negatives — rather than relying on a single headline score.