Open-source tool distills smaller AI models to cut agent inference costs by 40%

·1 views

Experiential Labs has launched 'wmo serve', an open-source tool designed to reduce the cost of running AI agents by routing repetitive tasks to smaller, distilled models instead of expensive frontier models. The tool ingests existing agent traces and uses them to continuously train specialized smaller models through distillation from open-source alternatives. A built-in router dynamically decides which tasks require a frontier model and which can be handled by the cheaper distilled model, while token compaction further reduces costs by removing noise. Users can run it locally via an OpenAI-compatible endpoint using an OpenRouter key, or opt for a hosted solution that promises equivalent quality at over 40% lower cost. The project is open-source on GitHub, and a hosted waitlist is available at experientiallabs.ai.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Hackathon Judge Shares What Actually Separates Winners From the Rest

A sports-technology builder who judged three hackathons in July 2025 — including MLH x DigitalOcean's AI for Social Good event and the Sports World Cup Hackathon in San Francisco — has shared candid insights on how scores are really determined. He notes that judges evaluate projects comparatively under tight time pressure, not against an abstract ideal, which means first impressions and clarity matter enormously. A live, end-to-end demo consistently outscored more ambitious projects shown only as slides, because it eliminates the doubt judges carry between back-to-back presentations. Teams that opened with a single clear statement of what their product does and who it serves scored better than those that led with market context or lengthy problem framing. Judges also rewarded narrow, well-executed builds over broad platforms, interpreting focused scope as evidence of sound decision-making under the hackathon's constraints.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

GitOps Principles Can Bring Version Control and Stability to AI Agent Configs

Modern AI agents rely on complex configurations spanning tool endpoints, authentication scopes, and memory bindings, but these are often managed informally through dashboards or scattered files, creating serious reliability risks. Untracked changes to such configurations can cause silent failures, security vulnerabilities, and irreproducible agent behavior — a problem analogous to 'configuration drift' in traditional infrastructure. A proposed solution applies established DevOps practices to AI management by storing all agent configurations in version-controlled repositories using a structured format called mcp.jsonc, which serves as a declarative contract defining what an agent can do and how. The GitOps approach for AI requires three core practices: committing configuration files to Git for immutable history, enforcing pull-request-based peer review for any changes, and integrating automated CI validation to catch errors before deployment. Treating AI tool configs and memory as production infrastructure — rather than ad hoc scripts — aims to make AI systems more auditable, reproducible, and secure across development and production environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Fixes Test Suite Gap After Reader Flags Boundary-Testing Blind Spot

A developer building an Electricity Planning Engine received a comment from reader Alex Shev pointing out that energy workflow tests should cover boundary conditions, not just midpoint values. Upon reviewing the existing test suite, the developer found that every peak and off-peak assertion used timestamps comfortably inside their respective windows, none near the actual transition points. The gap mattered because a wraparound time comparison in TimeSlot::contains() could silently return a wrong price at boundary moments without throwing any error. To address this, the developer added four targeted assertions covering the minute before a window opens, the exact opening second, the minute before it closes, and the exact closing second. The fix was validated by intentionally breaking the comparison logic, confirming the new boundary tests failed at precisely the expected points.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds 7ms prompt-injection detector on 78K samples, skips LLM approach

A developer built a prompt-injection detection system trained on over 78,000 attack samples, deliberately avoiding the increasingly common approach of using a large language model as the judge. Instead, the system relies on deterministic regex and classical machine learning — specifically TF-IDF character n-grams fed into logistic regression — completing each check in roughly 7 milliseconds at no per-call cost. On held-out public benchmarks, the detector achieved 0.895 recall on real-world jailbreaks and 0.799 on obfuscated attacks, both at 1.00 precision, but scored only 0.324 recall on subtle roleplay-framed jailbreaks — a gap the developer openly acknowledges. The developer also highlighted that false-positive rates vary significantly by traffic type, ranging from roughly 0.4% on curated benign input to 5% on open conversational logs, warning that single-number FPR claims from vendors are often best-case figures. The core argument is that a cheap, fast, deterministic layer should handle all traffic, with heavier models reserved only for ambiguous edge cases that justify the added latency and cost.

0 comments Read more at DEV Community