Changing One Model Pin Left Ten Test Results Stale in agentdrill Field Test

·1 views

During the agentdrill v0.1.0 field test, the agent model pin was switched from Qwen3.5-4B-4bit to Qwen3-4B-Instruct-2507-4bit, leaving nineteen existing test verdicts unclear about which were still valid. Rather than re-running every case, the team compared commit history to see which verdicts predated the switch, finding nine fresh and ten stale results. Only the ten stale cases were re-run, and all came back passing with new captured request-response pairs. Verification confirmed that all nineteen cases' latest captures now show the new model pin. The author also noted that an earlier partial switch had gone unnoticed, which led to wiring the pins through a single source of truth.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Building a Local SQLite Archive of DuckDB Tweets and Their API Costs

A developer built a small standard-library Python script that pages through a Twitter search for DuckDB-related posts and stores them in SQLite, keyed by tweet ID so re-runs avoid duplicates. The query is filtered to English, non-reply posts with at least 20 likes to keep the archive focused on well-engaged content. The author notes that the API bills per tweet returned at 15 credits each, with a 15-credit minimum per request, and that 100,000 credits equal one dollar. A full page of 20 tweets costs about $0.003, and a first run of 15 pages came to roughly $0.045. The writeup also highlights a mistake that made a second run cost as much as the first, which the author explores in the article. The author works on the TwitterAPI.io service being used and says the code and figures come from their own runs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Independent study re-tests Direct Nash Optimization for aligning language models

Researchers independently re-implemented DNO-Prct, a method that iterates on-policy sampling, LLM-based rating against a teacher response, preference-pair selection, and a DPO-style update. The team evaluated the approach on Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1. Independent replication is significant for alignment work because it checks whether Nash-equilibrium preference optimization holds up beyond the original implementation and annotators. The findings point toward a possible shift from fixed reward models to game-theoretic preference equilibria and iterative on-policy methods. The work was reported via arXiv.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Vercel Introduces vercel.ts as Typed Replacement for vercel.json

Vercel now recommends vercel.ts, a TypeScript file that replaces the static vercel.json for configuring projects. The file exports a typed config object covering build settings, routing, headers, and cron jobs, so mistakes like misspelled keys or invalid cron strings fail the build instead of being silently ignored. Because it is executable code evaluated at build or deploy time, it can read environment variables and vary settings per environment, which a pure JSON file cannot do. It does not run per request, so request-time logic still belongs in Next.js Middleware or Routing Middleware. The format is built on the @vercel/config package and is intended to replace the schema-less JSON configuration used since the platform's early days.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

iPhone users can use Screen Time or minulo to curb nighttime Reels and Shorts scrolling

A developer's guide explains how iPhone owners can stop late-night scrolling through Instagram Reels and YouTube Shorts. Apple's built-in Downtime feature can block entire apps on a schedule, but it also cuts off messages and regular videos. In-app limits such as YouTube's Shorts feed setting and Instagram's Sleep mode are free but easy to dismiss or do not hide content. The author's app, minulo, hides Reels and Shorts while keeping messages and standard videos available, and it can lock the official apps at bedtime. The article notes that most of the app's usage data, drawn mainly from Japan, shows social media sessions clustering between 8 pm and 1 am.

0 comments Read more at DEV Community