Study finds AI essay feedback quotes students faithfully in most cases, but filter errors skewed results

·4 views

An independent audit examined the SpecEval dataset of 41,730 LLM feedback pairs on 10,398 student essays, checking whether quoted passages in feedback reproduce the students' actual words. An initial filter kept only 519 quotes, of which 6.17% altered the student's wording, narrowly missing a predicted threshold. The author found the filter, designed for IELTS feedback, had wrongly discarded citation phrases like "for example", which removed most genuine quotes. A revised analysis using 1,546 quotes found a 3.2% alteration rate for cited quotes, with the wider sample showing about 96% overall fidelity. Some comparisons were judged inconclusive, and one directional finding was flagged as exploratory because it was chosen after seeing initial results.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Laravel or Node.js for a Startup Backend: How to Decide

For most startup backends, both Laravel and Node.js can work, and the right pick depends on the product and the team. Laravel tends to let teams build typical SaaS or marketplace features like accounts, payments, emails, and background jobs fastest, since these come built into the framework. Node.js, often paired with NestJS or Express, suits teams that want TypeScript throughout, products with heavy real-time demands, or access to a larger hiring pool. The article compares the two on ecosystem, hiring, build speed, and hosting, noting that Laravel offers first-party tools designed to work together while Node.js requires assembling more components. Survey data cited shows JavaScript and TypeScript have larger developer pools than PHP, though Laravel and NestJS developer shares are comparable.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Experiment Finds Single AI Agent Outperforms Multi-Agent Teams on Coding Task

A 2026 course by Jeremy McEntire tested four ways of organizing AI agents on the same backend-building task, using the same language model. A single agent scored 28 out of 28, a hierarchy scored 18, a leaderless swarm scored 9, and an eleven-agent pipeline produced no code at all. The pipeline exhausted its $50 budget on five planning stages before writing its first function. The swarm delivered five of seven services, but its services used incompatible data-exchange formats. The reviewer notes that the scoring rubric checks conformance to a pre-written specification, so the results reflect that particular kind of work rather than all software tasks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Cadence unveils Maestro, a portfolio of chip design AI models for early access in Q4 2026

Cadence has announced Maestro, a set of domain-specialized AI models for chip and system design that combines post-trained language models, trained decision models and agent skills. The portfolio plugs into Cadence's AI Super Agents and EDA tools and can run on Cadence's cloud, public clouds or customer-controlled on-premises systems. Customers can post-train the models on their own designs and workflows, and Cadence's deterministic solvers score outputs against engineering constraints to guide reinforcement learning. The company reports up to 30% better results and up to 5X greater token efficiency versus a general-purpose foundation model, though it has not detailed the benchmarks behind those figures. First models are slated for select early access customers in the fourth quarter of 2026.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Raw List Parameter in @InvocableMethod Breaks Apex Member Completion in VS Code

A developer found that member completion stopped working across an entire Apex class after writing a raw List parameter in an @InvocableMethod. The Apex language server (jorje) in the Salesforce Extension Pack crashes on collection parameters that lack a type argument, because its semantic checks expect exactly one element type and receive none. The server logs the exception without notifying the user, leaving the file with unresolved types until the signature is corrected. The author measured which signature shapes trigger the failure across 274 cases, finding that raw collections in @InvocableMethod, static @AuraEnabled, global, and webservice methods can crash parsing. A separate parser mechanism also drops whole methods when a raw collection is used as a return type. The author describes how their VS Code extension, Virga, now handles these cases.

0 comments Read more at DEV Community