Why ROUGE and BERTScore Fall Short for Grading Open-Ended AI Answers
Overlap metrics like ROUGE measure how closely a machine reply matches a reference text, but they do not show whether a person would accept it. ROUGE, introduced by Chin-Yew Lin in 2004, counts shared words and word sequences, while BERTScore, introduced in 2019 by researchers at Cornell and ASAPP, uses contextual embeddings to capture paraphrases. Both still measure similarity to a reference, which can miss the right answer when many wordings are acceptable. Examples from Chip Huyen and Vivienne Zhang show a correct-looking reply can still be a poor response, especially when domain terms carry different meanings. The article therefore suggests using an LLM judge with a constrained output format, including an option to say it cannot tell.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in