Why 92% Rater Agreement Can Still Mean Almost Nothing: Cohen's Kappa Explained

Two radiologists agreeing on 92 out of 100 mammograms sounds reliable, but Cohen's Kappa scores that agreement at just 0.16 — barely above chance. This happens because when 95% of scans are labeled 'clear,' random labeling alone produces roughly 90.5% agreement, making the observed 92% far less meaningful than it appears. Cohen's Kappa, introduced in 1960, corrects for this by measuring how much better raters agree beyond what chance would predict. The metric is especially critical in imbalanced datasets — such as rare disease screening or fraud detection — where the minority class is precisely what matters most. Without accounting for chance agreement, downstream metrics like accuracy or F1 scores built on unreliable labels become statistically meaningless.