AI Models Confirm Bugs Regardless of Prompt Framing, Controlled Experiment Finds

·1 views

A developer ran a pre-registered controlled experiment on 200 code samples from the OWASP Benchmark to test whether flagging language in security-scanning prompts inflates AI vulnerability confirmation rates. Three models — gpt-4o-mini, gpt-4o, and Gemma — were each given the same code slices under two prompt conditions: one that mentioned a static-analysis flag and one that did not. The key prediction — that removing the flag would cut gpt-4o-mini's false-confirm rate by at least 15 percentage points — was refuted; the drop was only 6 points and statistically insignificant. Both models confirmed real vulnerabilities at near-identical rates across both prompt arms, suggesting the over-reporting tendency is not driven by sycophantic deference to the flag. The experiment was shaped by reader feedback that pushed for pre-committed predictions and stricter statistical methods before results were collected.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

MeshDrive 2.0 gives AI agents isolated local file storage via MCP on Linux

MeshDrive 2.0 is a local-first Linux storage tool built on JuiceFS and SQLite that provides AI agents with controlled access to a dedicated file system. Released by developer Hardik94 on GitHub, version 2.0 adds a Model Context Protocol (MCP) server enabling AI clients such as Cursor, Claude Code, and Hermes to read, write, and list files through a defined set of tools. Agents are restricted to an isolated root directory by default, preventing them from accessing files outside the designated storage area. The free tier requires no cloud connectivity, keeping all data on local disks, and includes a web-based file browser accessible at localhost. Installation is supported on Ubuntu and Debian amd64 systems running Python 3.10 or later with FUSE support.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Best Practices for Defining AI Agent Skills: Structure, Naming, and Invocation

AI agent skills are used to automate repetitive, precisely describable processes, and must be carefully structured to minimize the risk of a model generating incorrect outputs. Skills can be invoked either explicitly by a developer or implicitly by the agent itself, and each one loaded into a session consumes context, making selective use important. Each skill must reside in its own dedicated directory, with the directory name matching the skill's name, and must contain a SKILL.md file holding its definition. The skill's YAML header requires at minimum a name — formatted in lowercase kebab-case between 1 and 64 characters — and a description of up to 1,024 characters specific enough for the agent to match it to user intent. Optional fields such as license, version, and context can further refine how a skill is managed, deployed, and legally used across different environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Redis Sliding Windows and Token Quotas Protect LLM API Spending

Rate limiting LLM APIs differs fundamentally from conventional APIs because a single model call carries real monetary cost, and agent loops can fire hundreds of requests autonomously. Developers must track both request counts and token usage separately, enforcing whichever limit binds first, since a one-line prompt and a 200-page document summarisation both count as one request but differ vastly in cost. Tenant identifiers used in rate-limit keys must be sourced from authenticated sessions, never from the request body, to prevent callers from manipulating or exhausting other tenants' quotas. Fixed time windows expose a known boundary exploit where a full quota can be consumed twice in rapid succession, making Redis sorted-set sliding windows a more reliable alternative when executed atomically via Lua scripts. Because actual token costs are only known after a model responds, the recommended approach is to reserve an estimated token count before the call and reconcile the difference once the response returns.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

tanstack-fetch: A Typed Fetch Client Designed to Simplify TanStack Query HTTP Layer

A developer has released tanstack-fetch, an unofficial, typed fetch client built specifically to complement TanStack Query in JavaScript and TypeScript applications. The library addresses common pain points that arise as apps scale, including authentication, typed error handling, request cancellation, retries, and server-sent events. Unlike native fetch, which does not throw on HTTP error status codes like 404 or 500, tanstack-fetch automatically converts HTTP failures into structured FetchError objects that integrate cleanly with TanStack Query's error state. The package can be configured with a base URL, token retrieval logic, and status-specific handlers, reducing repetitive boilerplate across query functions. It requires Node 18 or later in server environments and is available via npm alongside @tanstack/react-query.

0 comments Read more at DEV Community