Entropy Gating and Context Scoring Cut DLP False Positives by 80% in Production

Engineers building Spidercob, an enterprise data loss prevention platform, found that regex-based scanners routinely flagged test fixtures, documentation placeholders, and README examples as real secrets. After six months of production tuning, they extracted their detection logic into an open-source library called dlp-patterns. The solution combines three layers: post-regex format validators (such as Luhn checks for credit cards and JSON parsing for JWTs), Shannon entropy gating to reject low-randomness strings unlikely to be real secrets, and context scoring that boosts or penalises findings based on surrounding keywords. Ambiguous patterns like ICD-10 medical codes are silenced entirely unless corroborating domain-specific terms appear nearby. Together, these techniques reduced false positives by 80% compared to naive regex scanning alone.