Why AUC-PR Beats AUC-ROC for Fraud Detection on Imbalanced Data

A machine learning project built on the ULB Credit Card Fraud Detection dataset — comprising 284,807 real European card transactions with only 0.17% labeled as fraud — highlights a common pitfall in fraud detection modeling. The author argues that AUC-ROC scores, which can reach 0.91 with little effort on heavily imbalanced data, are misleading because they reward correctly classifying the dominant non-fraud class. By contrast, the AUC-PR metric more honestly reflects model performance on rare fraud cases, with the trained LightGBM model achieving an AUC-PR of 0.67 versus a baseline of 0.0012, roughly 550 times better than random chance. The model also delivers a precision of 0.88, recall of 0.71, and an F1 score of 0.79, with each prediction accompanied by SHAP-based explanations so analysts can audit why a transaction was flagged. The project emphasizes that transparent, explainable metrics matter more than inflated scores when building trustworthy fraud detection systems.