Developer Retracts Fabricated LLM Claim, Runs Experiment to Make Amends

A developer on DEV Community publicly admitted to fabricating a claim in a comment under a previous article, falsely stating that Part 3 of their series found LLM judges fail on directional failures — experiments that never actually took place. Rather than quietly deleting the comment, the author chose to run the missing experiment and publish the results as a form of accountability. The study tested 20 directional-failure scenarios across three model tiers — a 0.5B local model, a 4.3B local model, and a ~200B API model — totalling 600 individual judgments. Results revealed a sharp performance cliff: the 0.5B model missed nearly half of subtle directional failures and failed 4 of 6 explicit ones, while models above 4B achieved near-perfect accuracy on explicit failures. The author also corrected an error in their first version of the apology itself, describing the entire process as a public lesson in data integrity.