Quantiles CLI Lets Developers Run and Compare AI Evaluations Locally

·1 views

Quantiles is an open-source AI evaluation platform licensed under Apache 2.0 that provides a command-line tool called 'qt' for running, analyzing, and comparing AI model evaluations locally. The platform addresses a common developer challenge by consolidating dataset loading, model APIs, scoring logic, result storage, and comparison tooling into a single workflow. It supports macOS and Linux on x86-64 and Arm64 systems and can be installed via a single curl command. As a demonstration, the walkthrough uses the SimpleQA Verified benchmark — a 1,000-prompt factual question set originally developed by Google DeepMind and Google Research — to illustrate the full evaluation pipeline. The CLI supports multiple LLM providers including OpenAI, Anthropic, and Google Gemini, allowing developers to configure and evaluate their own models using the same workflow.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds Brainfuck-to-JVM compiler from scratch using pure Node.js

A developer documenting JVM internals chose Brainfuck, the esoteric programming language created by Urban Müller in 1993, as a minimal test case for building a compiler from scratch. The project, called BrainJuck, aims to compile Brainfuck source code into valid JVM bytecode (.class files) runnable directly with the Java runtime. This first installment in a three-part series covers building an interpreter, which serves as the foundation for the full compiler. The interpreter parses Brainfuck's eight commands into an intermediate representation, handling loop jumps via a stack-based placeholder mechanism. No external dependencies or frameworks are used — only Node.js and its native test runner.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three Useful Action Mailer Features Most Rails Developers Overlook

A developer revisiting the Action Mailer documentation recently discovered three lesser-known built-in features of the Rails mailing framework. The first is an email preview tool that renders messages directly in the browser without sending them, eliminating the need to repeatedly dispatch test emails. The second is an interceptor hook that allows developers to modify outgoing emails just before delivery, commonly used to add environment labels or redirect mail in staging setups. The third feature enables per-email delivery method overrides, useful in multi-tenant apps or when routing different email types through separate providers. Though some of these features have existed in Rails for some time, they remain underused among developers who rely on Action Mailer in production.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How One KNX Config Mistake Led to Better Home Assistant Motion Lighting

A developer building motion-controlled lighting across seven KNX sensors in Home Assistant discovered that assigning a single group address per sensor caused conflicts between lighting and presence/security logic. The root fix required configuring a second group address per physical sensor in ETS — the KNX programming tool — rather than patching conditions inside Home Assistant. This separation ensures lighting automations and security paths respond to motion independently, preventing changes to one from silently affecting the other. The author shares working Home Assistant YAML automations that turn lights on instantly when motion is detected and off after a five-minute delay once motion clears. Key implementation details include using mode: single to prevent automation loops and relying on the for: delay parameter as the most critical tuning variable in any motion-lighting setup.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Dead Man's Switch Can Alert You When Your Monitoring Stack Fails

A monitoring system has a critical blind spot: it cannot reliably detect its own failure, meaning outages can go unreported if the stack itself goes down. The solution is a 'dead man's switch' built using a Prometheus Watchdog alert with a condition that is always true, causing it to fire continuously as long as the pipeline is healthy. This constant alert is routed to an independent external heartbeat service, which expects regular check-ins from Alertmanager. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops refreshing and the external service triggers a notification through a separate channel. The key principle is independence: the system watching for monitoring failure must run on infrastructure that is completely separate from the monitoring stack itself.

0 comments Read more at DEV Community