Developer Tests Show Naive LLM Baseline Rivals Custom Provenance Scheme on Gate Decisions

·3 views

A software developer built an experimental benchmarking harness called provenance-compaction-lab to measure how different provenance-tracking strategies affect AI gate decision accuracy. Four arms were tested against a ground-truth oracle across 500-step horizons, varying compaction cadences, degradation profiles, and 20 random seeds per configuration. The custom structural scheme from a prior design iteration was compared against a prose baseline that used an LLM to summarize and re-extract provenance every few steps. Results showed the simple LLM summarization baseline matched or outperformed the hand-crafted scheme on roughly half the gate policy classes tested. The author notes the trajectories are synthetic and the generator component is designed to be swapped out for real-world traces in future testing.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Nylas Agent Accounts enable AI travel assistants to send and update itineraries autonomously

Developers can use Nylas Agent Accounts to build AI travel assistants that manage itineraries through a dedicated mailbox and calendar, without requiring human OAuth authorization. An Agent Account functions as a standard Nylas grant, allowing code to send itinerary emails, read traveler replies, and update calendar events via the Messages, Threads, and Events APIs. When a traveler requests a change — such as pushing a hotel booking by a night — the system classifies the request, retrieves the relevant event ID from the developer's own database, and updates that calendar segment directly. Nylas does not store trip-specific metadata, so developers must maintain their own database mapping trip IDs, traveler emails, thread IDs, and individual segment event IDs. The approach covers the common use case of sending and maintaining a known itinerary, and is distinct from scheduling tools designed for multi-party availability negotiation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PatchFlow: A Six-Agent AI Pipeline That Tests, Diagnoses, and Patches API Bugs

A developer built PatchFlow, an AI-powered tool that stress-tests APIs by injecting 18 real failure modes across network, dependency, data, and resource categories. The system runs a six-agent pipeline — Discovery, Chaos, Analyst, Repo Context, Fix, and Review — to identify vulnerabilities and generate targeted code patches. A key innovation is the Repo Context agent, which traces bugs through multiple files to find the actual source of failure rather than just the surface-level route handler. A dedicated Review agent independently audits every proposed fix and rejects it with structured feedback if issues are found, sending it back to the Fix agent for revision. Built during a hackathon using Qwen's ReAct loop and OpenAI-compatible API endpoint, the tool achieved roughly a 30% fix-rejection rate during testing, reflecting realistic code-review standards.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

WordPress 6.9 Introduces Abilities API to Standardize Site Functionality Exposure

WordPress 6.9 is launching the Abilities API, the first of four AI Building Blocks developed by the WordPress AI Team to address long-standing discoverability problems in the platform. The API lets developers register any unit of functionality as a named, typed, and permissioned 'ability' using a single declarative definition, replacing the fragmented approach of writing separate functions, REST routes, validation logic, and permission checks. Once registered, an ability is automatically accessible via PHP, WP-CLI, the REST API, and AI agents that support the Model Context Protocol. The initiative aims to make WordPress capabilities machine-readable, allowing AI tools and automation systems to query a site directly for what it can do, rather than reverse-engineering its REST routes. Beyond AI use cases, the API also provides developers with automatic input and output validation, colocated permission checks, and a consistent endpoint structure out of the box.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Web Systems Fail Under Real Load: The Benchmark Illusion Explained

Modern web frameworks like Laravel, Spring Boot, and Django spend 70–90% of CPU time on bootstrapping tasks — loading providers, building containers, and initialising connections — before processing a single user request. Benchmarks citing figures like 100,000 Redis operations per second or millions of PHP operations per second are measured in controlled lab conditions that bear little resemblance to production environments. Under real load, threads stall waiting for I/O, connection pools exhaust, and requests queue up, triggering a cascade of contention and system collapse. The stateless nature of HTTP compounds the problem, forcing systems to re-establish context on every request as if the user is entirely new. Architectures like serverless and microservices do not eliminate these overheads — they redistribute or obscure them, often adding cold-start latency and cascading network call delays.

0 comments Read more at DEV Community