Why a Custom Golden Set Beats Public Leaderboards for Evaluating AI Model Routes

·5 views

A developer-focused approach recommends using a curated 'golden set' of 100–300 real request-response pairs to evaluate AI model routes, rather than relying on public benchmark scores. The golden set captures application-specific tool calls, JSON schemas, and edge cases — including prompts designed to deliberately fail — to test behavioral compatibility. An evaluator script compares structured outcomes and latency between a baseline and a candidate model route, flagging semantic mismatches rather than scoring prose quality. Three signals — golden set failure rate, shadow route errors, and actual token costs — guide routing decisions more accurately than leaderboard rankings. The author notes key limitations: golden sets must be regularly rebuilt as prompts evolve, and free-tier model access can introduce cold starts or quota issues that skew latency results.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developers Use Free Sanctions APIs to Embed EU AI Act Compliance in CI/CD Pipelines

Software teams building high-risk AI systems under the EU AI Act face a compliance gap when vendor screening remains in spreadsheets rather than automated pipelines. A developer has shared a Python-based approach that integrates free sanctions-screening APIs directly into CI/CD workflows, automatically checking vendor and supplier names against major sanctions lists. The script fails a build if a vendor returns a HIGH or MEDIUM risk verdict, surfacing which data field triggered the match. The EU AI Act's Articles 9, 10, and 25 require organisations to manage risks across their entire supply chain, including training data providers, cloud API vendors, and payment processors. Annual vendor reviews are considered insufficient in fast-moving development environments where new dependencies can be introduced and shipped within days.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

form.fscss lets developers build modern forms using pure CSS, no JavaScript

A new open-source CSS module called form.fscss, part of the FSCSS ecosystem, enables developers to build fully featured forms without JavaScript UI logic. The module uses native CSS mechanisms such as the :placeholder-shown pseudo-class to power floating labels and the checkbox-hack pattern for custom checkboxes, radio buttons, and toggle switches. Inline validation states like error and success are triggered simply by adding a class to the field wrapper, keeping the presentation layer entirely in CSS. Micro-interactions such as a gradient button press effect are also handled through CSS pseudo-classes rather than event listeners. Full documentation, design tokens, and markup requirements are available on GitHub at github.com/fscss-ttr/form.fscss.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Docker Networking and Volumes Explained: How Containers Communicate and Store Data

Modern applications typically run across multiple containers — such as web servers, databases, and cache layers — which must communicate reliably with one another. Docker addresses this through built-in networking drivers, primarily bridge, host, and none, each suited to different use cases. A key feature of Docker networking is DNS-based container discovery, which lets containers reference each other by name rather than by dynamic IP addresses that can change on restart. Docker volumes complement networking by providing persistent storage that survives container removal, ensuring data is not lost when containers are stopped or recreated. Together, Docker networking and volumes form the foundation for building and managing multi-container applications in production environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Releases ActiveVPN: Open-Source Terminal Tool to Detect VPN and Proxy Use

A developer has released ActiveVPN, an open-source command-line tool designed to verify whether a VPN or proxy is actively protecting a user's connection. The tool checks network interfaces, running processes, external IP addresses, DNS resolvers, and IPv4/IPv6 configurations to detect potential leaks or misconfigurations. It assigns a verdict score from 0 to 100, ranging from 'Clean' to 'VPN Detected,' helping users confirm their actual privacy status. ActiveVPN supports features such as a kill switch, continuous watch mode with alerts, and JSON/CSV export for logging. The project is available on GitHub and is compatible with Python 3.8 through 3.12 across multiple operating systems.

0 comments Read more at DEV Community