Why a 99.4% Accurate Fraud Model Can Catch Absolutely No Fraud

A common pitfall in machine learning involves relying solely on accuracy to evaluate model performance, particularly when datasets are heavily imbalanced. In a simulated dataset of 20,000 card transactions where only 0.6% are fraudulent, a model that blindly labels every transaction as legitimate achieves 99.4% accuracy while detecting zero fraud. By contrast, a logistic regression model trained with balanced class weights scores a lower 85.3% accuracy but successfully identifies nearly 78% of actual fraudulent transactions. Metrics like precision and recall offer far more meaningful insight for fraud detection, measuring alert quality and detection coverage respectively. Experts emphasize that choosing the right evaluation threshold is ultimately a business decision, weighing the cost of false alarms against the risk of missed fraud.