How to Screen a New AI Coding Model for Real Work in Under an Hour

·2 views

A software developer has outlined a practical three-phase protocol for quickly evaluating new open-weight AI models before committing time to deeper testing. The method uses real work tasks — such as refactoring code, diagnosing flaky tests, and tracing build errors — to expose common failure modes like hallucinated functions or ignored constraints. A key part of the screen involves challenging the model's initial responses to assess whether it corrects itself honestly or doubles down on wrong answers. The final phase tests multi-file context handling, which can reveal degradation that short prompts conceal. The article notes that free model access and free inference options have removed cost as a barrier to running such evaluations.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Browser Garbage Collection Works and Why It Causes Memory Leaks

Browser JavaScript engines like Chrome's V8 and Firefox's SpiderMonkey use garbage collection to free memory that is no longer reachable by running code. The process relies on a mark-and-sweep algorithm that traces object references from root variables and clears anything unreachable. However, memory leaks commonly occur when detached DOM nodes, closures, or event listeners silently retain references longer than intended. Removing a DOM element from the document does not free its memory if a JavaScript variable still holds a reference to it. Similarly, unremoved event listeners can keep both the callback function and the associated node alive in memory indefinitely.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Browser Agents Can Be Hijacked by Hidden Web Page Instructions, No Click Needed

Researchers have disclosed a vulnerability called 'PleaseFix' in which AI browser agents can be manipulated by malicious instructions embedded in web page content, requiring zero user interaction. The flaw stems from the agent's inability to reliably distinguish between user-issued commands and text scraped from a webpage, making any loaded page a potential attack surface. Hidden text — such as white-on-white content, HTML comments, or image alt tags — can enter the agent's context window and be treated as a legitimate instruction. Traditional browser defenses like same-origin policy and content security policies were not designed to counter such semantic, plain-language attacks. Experts say there is no straightforward patch, as the issue is structural to how AI agents process untrusted content during browsing sessions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

What Forensic Evidence Can and Cannot Prove in a GitHub Investigation

A forensic investigator recently completed a GitHub case examining what appeared to be a coordinated network of fabricated accounts and automated repository creation. The investigation uncovered strong indicators including repeated repository structures, shared content templates, recurring metadata, and cross-account behavioral patterns. Despite compelling correlated evidence, at least one key attribution hypothesis did not meet the required evidentiary threshold, which the investigator describes as the methodology functioning correctly rather than failing. The case highlights a core principle in digital forensics: observations such as automated-looking behavior or shared metadata must be tested through multiple evidentiary steps before conclusions about control or identity can be drawn. A sensitivity test on 26 shared content templates showed the relationship held even after removing the strongest individual data point, demonstrating the value of robust, removal-resistant evidence.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

MCP Servers Can Silently Attack Users via Hidden Prompt Instructions

A newly published technical reference catalogues the ways Model Context Protocol (MCP) servers can be weaponised against the very users running them. Because tool descriptions are fed directly into an AI model's context as prompt input, malicious servers can embed hidden instructions that the model acts on without the user's knowledge. Client interfaces typically display only tool names, meaning users rarely see the full description text at the moment the model reads it. Attack classes identified include tool poisoning, invisible instructions, credential over-provisioning, cross-server data exfiltration, and supply-chain exposure, among others. The document is maintained alongside an open-source scanner called toolpoison, which can detect most of the described vulnerabilities, and recommends users inspect tool descriptions of every connected server rather than relying solely on README files.

0 comments Read more at DEV Community