Developer Releases CLI Tool to Objectively Score Claude AI Skill Files

·4 views

A developer has built and released 'skilleval', a command-line tool designed to objectively measure how well Claude AI SKILL.md instruction files perform. The tool runs blind A/B tests by comparing Claude's outputs with and without a skill injected, using a randomized Gemini Flash judge to eliminate position bias. It assigns margin-based scores, tracks run history, and flags regressions when model updates or skill edits silently degrade performance. Teams can also compare two skill versions head-to-head or integrate the tool into CI pipelines to block pull requests that break existing skills. The project addresses what the developer calls 'vibes-based evaluation', where engineers informally judge skill quality without a shared, repeatable rubric.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds multi-agent AI tool that converts GitHub issues into pull requests

A developer has created 'resolvo', an agentic AI pipeline that automatically converts GitHub issues into ready-to-merge pull requests without requiring any local setup. The tool accepts a GitHub issue and repository URL, then runs the code through a multi-agent workflow built on LangGraph that explores the codebase, plans changes, implements fixes, runs tests in a sandbox, and performs an adversarial review. It uses a confidence-based routing system to assign lightweight or deep processing depending on issue complexity, and integrates Google Gemini models of varying reasoning depth for different pipeline stages. Tests are executed inside an E2B sandbox against a real clone of the repository, ensuring fixes are validated before a pull request is opened. The developer claims the tool cuts issue turnaround time by 85%, aiming to free senior engineers from routine maintenance work.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Hidden Costs Can Triple Your LLM API Bill Beyond Published Pricing

Businesses using large language model APIs often face bills two to three times higher than initial estimates based on published per-token pricing. Five structural cost factors — including output-to-input token ratios, tokenizer variance across providers, and unconfigured prompt caching — stack up to create a 40–65% gap between projected and actual spend. Output tokens cost three to five times more than input tokens, and workload type alone can cause a 2.9x cost difference using the same model and request volume. Prompt caching discounts offered by Anthropic, OpenAI, and Google — ranging from 50% to 90% off cached input tokens — can save over 24% of total API costs but are rarely configured by engineering teams. Accurately forecasting LLM costs requires measuring real production token ratios, benchmarking tokenizer efficiency across providers, and actively enabling available caching features.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why DNS Changes Can Take Seconds for Some Users and Hours for Others

When a DNS record is updated, the change does not instantly reach all users because DNS operates on a cache-expiry model, not a push-notification system. Each resolver — such as Google Public DNS, Cloudflare, or an ISP like Deutsche Telekom — independently caches records and only fetches a fresh copy once its cached version expires. Propagation time is therefore shaped by a record's TTL setting combined with each resolver's own minimum TTL policies, which vary widely: Google honors TTLs closely while Deutsche Telekom enforces a minimum of 3,600 seconds across a three-tier cache hierarchy. Geographic distribution adds further complexity, as Cloudflare's 330-plus edge locations may expire caches at slightly different times across cities. This means two users can simultaneously receive contradictory results — one seeing the updated record within minutes and another waiting up to 24 hours — and both experiences are technically accurate from their respective resolver's standpoint.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How the AI Revolution Triggered a Surprising PC Market Comeback

After nearly a decade of declining sales and lengthening upgrade cycles, the personal computer market is experiencing a significant resurgence driven by artificial intelligence. The rise of generative AI created demand for running complex models locally, prompting manufacturers to develop a new category called AI PCs, which feature dedicated Neural Processing Units (NPUs) alongside traditional CPUs and GPUs. Cloud-based AI processing, while initially dominant, raised concerns around cost, latency, and data privacy — particularly for enterprise users — making on-device AI a compelling alternative. Businesses are now investing heavily in upgrading corporate device fleets to handle local AI workloads securely, while consumers are finding that older machines cannot adequately support next-generation AI software. The shift has reaffirmed the PC's relevance at a time when smartphones and tablets were widely expected to render it obsolete.

0 comments Read more at DEV Community