Study Examines How Much the Test Harness Influences Coding Agent Performance

·1 views

A new research project called HarnessTax investigates the impact that evaluation harnesses have on the measured performance of AI coding agents. The study questions whether benchmark results reflect true agent capability or are significantly shaped by the surrounding test infrastructure. Researchers suggest that harness design choices may introduce bias or variance into coding agent evaluations. The findings aim to help the AI community better interpret and compare coding benchmark results. The project is publicly available online and has begun drawing discussion in developer communities.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Founder Measures AI Brand Visibility After Gemini Misidentifies His New Company

When entrepreneur launched Luxcerta, a platform that tracks how AI models represent brands, he discovered that Gemini with web grounding mistook his company's name for a typo and flagged his domain as a possible phishing site. Running his own product's measurement method on day one, he recorded zero correct mentions or citations across ten queries to Gemini. Notably, Google Search's AI Overview correctly identified Luxcerta from homepage metadata on the very same day, highlighting how different AI surfaces can reach opposite conclusions about the same brand. To improve AI recognition, he added structured schema.org data, a clearer meta description, a disambiguating GitHub profile, and submitted a fresh sitemap via Search Console. One week later, Gemini's grounded responses began correctly identifying Luxcerta, though he cautions the case involves one brand over one week with no controlled variables.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Ships Second Open Redirect Bug in the Same Fix Meant to Patch the First

A developer maintaining the CAS-Authentication-User npm package discovered a second open redirect vulnerability in version 0.3.0, the very release that was meant to fix a previously identified flaw. The new guard function, isSafeReturnTo, correctly rejected malicious returnTo query parameters, but when it fell back to the request path as a default, it inadvertently constructed the same type of protocol-relative URL it was designed to block. The root cause was a modelling error: the request path, though fully controlled by an attacker, was treated as a trusted internal fact rather than as user input requiring validation. Unlike the first bug, this variant required no returnTo parameter, no service ticket, and no CAS server involvement, making it easier to exploit against authenticated users. The developer has since identified the actual sink — the res.redirect calls in index.js — as the correct place to enforce validation, rather than only auditing obvious query string inputs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Headless Tables vs. Full Grid Components: How Developers Choose the Right Fit

Web developers building data tables must choose between headless table libraries and full grid components, a decision that significantly affects design control, speed, and long-term maintainability. Headless libraries give developers complete control over markup and behavior but require manual implementation of features like sorting and filtering, which can lead to complex, hard-to-maintain code. Full grid components offer faster setup by handling most functionality out of the box, but struggle when projects demand custom designs or unsupported behaviors. The right choice depends on project-specific factors such as design strictness, timeline pressure, and anticipated customization needs. The Svelte-based SvGrid project illustrates a dual approach, shipping both a headless core and a ready-to-use component to serve developers with different priorities.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Connect to an Azure Virtual Machine Using SSH

SSH (Secure Shell) is an application-layer protocol that uses asymmetric cryptography to establish encrypted remote connections between devices, operating by default on TCP port 22. To set up SSH access on Microsoft Azure, users must create a virtual machine with SSH public key authentication enabled and ensure port 22 is open in the inbound port rules. During VM creation, Azure generates a key pair and downloads a .pem file, which must be moved to the ~/.ssh directory and set to read-only permissions using chmod 400. Users can then connect to the VM via the terminal using the ssh -i command along with the VM's public IP address, found on the Azure portal. On first connection, Azure prompts users to verify the host fingerprint, which can be cross-checked by running a keygen command directly on the VM through the Azure portal's Run Command feature.

0 comments Read more at DEV Community