Phish or Legit: Do LLMs Know When NOT to Cry Wolf?

Kaggle Benchmarking Challenge Submission by Anio Joseph Most security benchmarks ask: "Can the model detect the threat?" That's the easy part. The real question in a security operations center is: "Can the model stay quiet when the message is actually safe?" I built Phish or Legit to test exactly that. It's a 40-item cybersecurity triage benchmark with 20 threats and 20 legitimate messages. The twist? Many of the legitimate messages are deliberately suspicious-looking — they have links, urgency, or unknown senders, but are completely benign.