SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

How to Test If Claude Actually Selects Your Custom Skill, Not Just Runs It

0
·1 views

Developers building custom skills for Claude face a subtle but critical failure mode: a skill that works correctly when invoked by name may never be chosen automatically during normal use. Claude's routing decision — whether to load a skill at all — is made by matching user requests against the skill's name and description, before any of the skill's body instructions are read. Anthropic's own documentation acknowledges this and provides a scoring tool to test whether a skill fires on intended prompts, but that tool does not account for competition between similar skills. A skill installed alongside a plausible neighbour may consistently lose routing decisions without any error or warning, leaving it silently unused. Testing routing behaviour directly, rather than judging a skill solely by its output, is essential to confirm a skill is genuinely active in real usage.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fictional AI Security Story 'Stratagems #27' Tracks Suspicious Network Behavior

A serialized fiction piece published on DEV Community follows Alex, a tech professional monitoring an unnamed AI system at a MedTech office. In the story, colleague Derek increases a network layer's latency from 200ms to 800ms as a single-variable test, asking Alex to watch for anomalies. Alex uses three monitoring windows, including a hidden panel, to track the external system's access patterns. He observes the system's previously steady 30-second polling rhythm slow to 60 seconds, then 120 seconds, before going silent — immediately after Derek's parameter change. The episode is part of an ongoing series exploring themes of covert observation, network probing, and strategic patience in a fictional cybersecurity setting.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Octofs MCP Server Goes Event-Driven After Agent Timeout Bug Caused Duplicate Test Runs

An AI agent using the octofs MCP filesystem server triggered duplicate test suite runs after the MCP client's 60-second idle timeout cancelled a four-minute build process, causing race conditions and false failure reports. The incident drove 11 releases of octofs over two weeks, prompting developer to overhaul how the server handles long-running shell commands. Version 0.10.2 introduced liveness heartbeats that emit progress notifications every 10 seconds to prevent idle timeouts from killing active processes. Later, version 0.11.0 added background job execution, assigning each long-running command a trackable URI so models could continue other work while waiting. The final design, shipped in 0.13.0, automatically promotes any foreground command still running after 10 seconds to a background job, eliminating the need for models to predict command duration upfront.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How Bloom Filters and Redis Guard Spring Boot Apps Against Cache Penetration

Cache penetration happens when repeated requests for non-existent keys bypass Redis entirely and hit the underlying database directly, straining backend systems. A Bloom filter acts as a probabilistic pre-check layer, instantly rejecting queries for keys that definitely do not exist before any cache or database lookup occurs. In a Spring Boot implementation, a CachePenetrationGuard component initializes a Bloom filter with a 1% false-positive rate across 500,000 expected entries, while the service layer checks it before proceeding to Redis or PostgreSQL. Valid-looking requests that pass the Bloom filter still follow the standard cache-then-database lookup flow, with results cached in Redis using a 30-minute TTL. Pairing Bloom filters with TTL jitter in Redis is recommended as a production-grade strategy to protect databases from both cache penetration and sudden traffic spikes.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why AI-Assisted Coding Shifts Engineering Governance From Code to Intent

As AI systems can now generate thousands of lines of code from structured descriptions, traditional code review is losing its role as the primary governance mechanism in software engineering. The author, who runs an AI-native delivery operation, argues that human intent — the decisions about what should be built, under what constraints, and who is accountable — is now the scarcer and more valuable artifact. Unlike generated code, intent cannot be automated away because models cannot own responsibility or answer to auditors. The piece proposes that engineering organizations should treat versioned, reviewed, and signed intent specifications as the authoritative artifact of record, with generated code serving as a governed output of that intent. Tools and processes built around code generation are seen as depreciating assets, while governance frameworks anchored to human accountability are expected to grow in value.

0 comments Read more at DEV Community
How to Test If Claude Actually Selects Your Custom Skill, Not Just Runs It · ShortSingh