CauterRule v0.3.1 Fixes Silent Semantic Matching Bug, Doubles Agent Recall

·1 views

Open-source tool CauterRule released version 0.3.1, addressing a critical flaw in its replay matcher that had rendered the semantic matching channel effectively inactive since launch. The bug stemmed from two issues: a semantic similarity floor set too high at 0.80, and trajectory embeddings polluted by class labels that artificially suppressed cosine scores. Developers lowered the floor to 0.62 based on measured paraphrase distributions and stripped class labels from signature embeddings to fix both problems. The fixes required no model or prompt changes, yet golden recall nearly doubled — rising from 0.17 to 0.38 on GPT and from 0.23 to 0.43 on LLaMA across 4,742 trajectory runs. The v0.3.1 release is available on GitHub and via pip, with full CLI support, framework adapters, and official rule packs included.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Two Methods to Identify Shopify Stores: Manual Checks and Bulk Detection Tools

Developers and marketers needing to identify Shopify-powered websites can use two main approaches: manual inspection of page source code, response headers, and cookies, or a bulk automated tool for larger lists. Shopify stores leave consistent technical fingerprints, including references to cdn.shopify.com, a powered-by header, and session cookies such as _shopify_y. For bulk detection, a tool called Tech Stack Detector, built on the Apify platform, cross-references sites against over 7,600 open-source technology fingerprints to filter only Shopify stores from a list of domains. The tool also surfaces the broader technology stack of each site, including email, review, and checkout platforms, along with the specific evidence behind each detection. Results can be exported in CSV, Excel, or JSON formats, making it practical for lead generation, agency prospecting, or competitive research.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why LLM Red-Team Reports Need Reproducibility, Not Just Screenshots

A software engineer argues that most AI red-teaming reports are little more than screenshots of bad model outputs, which cannot be independently verified or rerun. Credible reports, the author contends, must include a fixed and versioned probe corpus, raw prompts and model replies, and a SHA-256 checksum so compliance reviewers can verify nothing changed. A meaningful 0–100 score derived from how many probes a model failed should replace vague qualitative assessments, enabling CI pipelines to block deployments that exceed a set risk threshold. The author has built a small toolkit covering 35 probes across 17 attack classes, with a free 8-probe scan available without requiring an account. The core argument is that reproducibility transforms red-teaming from an anecdotal claim into a verifiable, auditable record.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study finds AI-generated code flags zero true error-swallowing cases across 120 samples

A developer ran a controlled experiment generating 120 code samples using Qwen2.5-Coder 1.5B across 12 tasks in Python and TypeScript to test whether small AI models routinely hide or suppress errors. A Semgrep-based static detector flagged four candidates as potential error-swallowers, but manual review confirmed all four were either false positives or legitimate, documented fallbacks. The experiment found that Python generations mostly let exceptions propagate or raised them explicitly, while TypeScript samples favored try/catch blocks with logging or re-throwing. The researcher's key conclusion is that whether code truly swallows an error cannot be determined from syntax alone, as the same pattern can represent either a sound design contract or a genuine hazard depending on context. The study acknowledges significant limitations, including a single small model family, only 12 tasks, and sole-rater adjudication, and makes no broader claims about AI code versus human-written code.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a WebMCP Scoring Tool in Plain HTML With No Dependencies

A developer tutorial published on DEV Community walks through building a WebMCP tool from scratch using only a plain HTML file served via Python's built-in HTTP server. The guide demonstrates how to register a tool called 'score_facts' that evaluates a YAML document against five required fields — name, goal, who, what, and why — and returns a 0–100 score. Chrome supports the document.modelContext API natively behind an experimental flag, while other browsers can use a polyfill as a fallback. The tutorial emphasises defensive coding practices, including input validation and try/catch error handling, so that tool failures return structured error objects rather than raw stack traces. A security section also introduces URL allowlisting to prevent remote-fetch tools from being exploited as open proxies.

0 comments Read more at DEV Community