SShortSingh.
Back to feed

Why ROUGE and BERTScore Fall Short for Grading Open-Ended AI Answers

0
·2 views

Overlap metrics like ROUGE measure how closely a machine reply matches a reference text, but they do not show whether a person would accept it. ROUGE, introduced by Chin-Yew Lin in 2004, counts shared words and word sequences, while BERTScore, introduced in 2019 by researchers at Cornell and ASAPP, uses contextual embeddings to capture paraphrases. Both still measure similarity to a reference, which can miss the right answer when many wordings are acceptable. Examples from Chip Huyen and Vivienne Zhang show a correct-looking reply can still be a poor response, especially when domain terms carry different meanings. The article therefore suggests using an LLM judge with a constrained output format, including an option to say it cannot tell.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why AI models and AI agents are not the same thing, illustrated with a board game

A developer built a strategy game called 6WINS, where a computer opponent uses two separate components. TAEL, a neural network, only estimates how favorable a board position is, producing a value between -1 and +1 without knowing the rules. ORACLE knows the rules, generates legal moves, searches ahead, and calls TAEL to evaluate resulting positions before choosing an action. The author argues this split mirrors the difference between a model, which supplies learned information, and an agent, which combines that information with rules, search, and action. Early versions of the opponent played erratically, and later wins came from ORACLE's search using TAEL's evaluations rather than the network learning to play alone.

0
ProgrammingDEV Community ·

Learn motion graphics in a browser: animate one shape with two keyframes

A beginner tutorial recommends starting motion graphics by moving a single shape from one point to another. Using the SoloMotion browser tool without an account, the reader draws a circle, sets a position keyframe at the start and another about a second later, then scrubs the timeline to watch the movement. The guide explains how to troubleshoot common issues such as the shape jumping, staying still, or moving robotically, and how to adjust easing and keyframe timing. It concludes by suggesting the reader export a short clip or GIF and note any confusing steps, which the author wants to use to improve the interface.

0
ProgrammingDEV Community ·

CISA Adds Puppet Enterprise Report Deserialization Flaw CVE-2026-115731 to KEV Catalog

On 17 November 2026, CISA added a deserialization vulnerability in Perforce Puppet Enterprise, tracked as CVE-2026-115731, to its Known Exploited Vulnerabilities catalog with a remediation deadline of 1 December. Neodyme researcher Alain Rödel reported two defects while preparing a red-team engagement, and Perforce and OpenVox published patches and advisories on 3 November. OpenVox's 4 November advisory included a working malicious report JSON, and Red Hat released Satellite and Capsule updates on 13 November. The flaw, rated CVSS 4.0 9.4 Critical, affects multiple Puppet Enterprise and Puppet Core releases dating back to 2021, and code execution can expose the CA signing key that underpins trust across managed nodes. Organizations are advised to apply the listed fixed versions promptly.

0
ProgrammingDEV Community ·

SoloMotion vs HyperFrames: Visual Canvas Editing or HTML Video Projects?

The article compares two open-source tools for making short launch clips: SoloMotion, a browser-based editor with a canvas and timeline, and HyperFrames, which builds video from HTML compositions rendered to MP4. The author, who makes SoloMotion, frames the choice around whether the user expects to adjust timing and layers visually or to edit components and re-render. HyperFrames also offers a Studio app with a free tier and an agent-friendly workflow built on project files. The piece suggests a test: build a simple animated card, then make three revisions such as shifting timing, changing text, and creating a color variant. It recommends HyperFrames for teams producing many templated versions and SoloMotion for clips needing frequent visual iteration.