Developer's eval harness scores agent output on three axes, with a deterministic rule layer

·1 views

A developer describes an evaluation harness for AI agents that grades runs on quality, faithfulness, and task success, each scored 0-100 with a short justification. Early versions used a single vague quality field, which the developer found insufficient, so the harness was split into three dimensions. Every output passes through a versioned, deterministic rule layer of fourteen checks covering completeness, quality, faithfulness, and honesty, so scores stay traceable to specific rules. The developer argues the rule layer is more trustworthy than the model-based grader, since it checks structure rather than being persuaded by fluent prose. The most valued rule scores the full multi-turn trajectory and tool selection rather than only the final answer.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

ShortSingh: Browser Tool Sorts Lines Numerically or Case-Sensitively, Keeping Text Local

OnePassApps has described its Text Studio's Sort lines tab, which runs entirely in the browser tab so files are not uploaded. The tool offers A-to-Z or Z-to-A ordering, a case-sensitive option, and natural number ordering, which is enabled by default. With natural ordering on, numbers sort by numeric value, so 9 comes before 10, whereas plain text sorting places 10 first. The article notes that case-sensitivity only changes results when two lines differ solely in capitalization, and that natural ordering suits IDs but may be better turned off for version strings.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

High-accuracy classifier can still miss every useful research candidate

A synthetic teaching example shows that a screening classifier scoring 95% accuracy on 100 records can find none of the five positive candidates. The article explains why accuracy alone hides which cases were missed. Precision and recall are compared across two constructed prediction sets, showing that catching more positives can come with more false positives. The author stresses that the labels and predictions are invented for illustration and are not measurements from a real model or laboratory. A short Python audit script is provided that computes confusion-matrix counts, accuracy, precision and recall without external data or dependencies.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How JEE Main converts marks to percentiles and ranks across 19 shifts

About 15 lakh students take JEE Main across 19 shifts, each with a different paper, so raw marks cannot be compared directly. The testing agency NTA converts marks to a percentile calculated within each shift, meaning the same score can yield a higher percentile in a tougher shift. Because NTA does not publish a marks-to-percentile table, predictors must build one from published (marks, percentile) pairs and interpolate between them. Rank is then estimated by applying the share of candidates above a given percentile to the total number of candidates across all shifts. The author notes that such predictions are estimates and recommends displaying a range rather than a single figure.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why '.map is not a function' Errors Occur: The Value Is Not an Array

A common JavaScript error, '.map is not a function', does not signal a problem with the map method itself. It means the value being called is not an array, such as an object, a Set, or an unparsed string. Array-like values such as NodeLists also trigger it, since they lack array methods. Developers can diagnose the issue by logging the value's type and checking it with Array.isArray. Depending on the cause, the fix may involve accessing the nested array, converting the value with spread syntax or Array.from, parsing JSON, or using Object.entries.

0 comments Read more at DEV Community