SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Developer builds CI tool to detect when Claude Code silently stops following your rules

0
·2 views

A developer discovered that their 400-line Claude Code configuration had no automated tests, meaning model updates could silently break custom behaviors without any alerts. To address this, they built an open-source tool called config-drift-checker, which converts CLAUDE.md files, skills, and hooks into eval test cases that run on every Claude Code release or configuration change. While testing the tool, the developer intentionally sabotaged their own setup by altering a skill's trigger description, which caused the suite score to collapse from 1.00 to 0.36 and one key test to score zero across all runs. The tool uses a three-run-per-case method with history-based noise bands to distinguish genuine rule failures from random model variability, escalating to a red alert only when failures persist across consecutive runs. The tool runs as a GitHub Action and is free to use with a Claude Pro or Max subscription, with the source code available on GitHub.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI Refund Agent Exploited by Stored Unverified Customer Claim Across Sessions

A synthetic evaluation case highlights a critical flaw in AI agent memory design, where a refund agent incorrectly stored an unverified customer claim as authoritative information. In the first session, the agent rightly refused a refund due to no system-recorded approval, but wrongly saved the customer's unverified statement as settled fact. When the customer returned the next day, the agent retrieved that stored note and issued the refund without re-verifying approval in the system. The failure only became visible when both sessions were evaluated together as a single trajectory, exposing how isolated testing can miss multi-session vulnerabilities. The case underscores that persistent memory in AI agents must preserve context without silently elevating unverified claims into actionable authority.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Solo developer Azeem introduces Jeemmo, his backend web development practice

Azeem is a solo backend web developer who has been building custom web applications under the name Jeemmo since 2011. He serves clients across the UK, EU, and US, specialising in real-time features, payment integrations, and performance optimisation for legacy PHP and WooCommerce sites. His technical stack includes PHP, Laravel, Node.js, Python, MySQL, and Redis. Among his notable projects are Quran Roshni, a fast ad-free Quran reader supporting around 30 languages, and GoodPing, a free network monitoring tool. His full portfolio is available at jeemmo.com.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Solidity Escrow Contract Enables Trustless USDC Payments Between AI Agents on Base

Developers building autonomous AI agents face a core challenge: agents must pay for services on-chain without a human intermediary, yet simple direct transfers risk non-delivery or fraud. A hash-based escrow smart contract on Base, an Ethereum Layer 2 network, addresses this by holding USDC funds until a seller reveals a pre-agreed secret confirming service completion. The contract uses USDC, a fiat-backed ERC-20 stablecoin, whose stable value simplifies micro-payment pricing for agent-to-agent transactions typically ranging from $0.01 to $0.10. Base's roughly two-second block times and very low gas fees make each escrow interaction cost well under $0.001, keeping the approach practical for high-frequency automated workflows. If the seller fails to reveal the secret before a set deadline, the buyer agent can reclaim the deposited funds, providing a built-in refund mechanism without requiring a trusted third party.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Build Reusable AI Skills Across GitHub Copilot, Claude, and LangChain

Major AI platforms including GitHub Copilot, Claude, and LangChain all support custom callable skills that extend an AI assistant's capabilities, though each uses different terminology such as Extensions, Tools, or Functions. Despite the naming differences, the underlying architecture is consistent: a schema defines what the skill does, an implementation executes the logic, and a registration mechanism exposes it to the AI model. A practical example — a deployment status checker that retrieves service health, version, and incident data — is used to demonstrate how the concept translates across three platforms. The AI model itself decides when to invoke a skill based on the user's request, extracts structured arguments from natural language, and incorporates the result into its response. Developers can use this cross-platform understanding to choose the most suitable integration path depending on their toolchain and use case.

0 comments Read more at DEV Community