AI Security Firm Publishes Open Benchmark That Includes Its Own Detection Failures

·1 views

A developer building AI agent security tools has publicly released a benchmark dataset on GitHub, covering 1,669 samples across 22 attack categories, including both successful detections and known failures. The benchmark reports a 99.8% detection rate at a 0.09% false positive rate, but unusually, it also names the specific cases the engine gets wrong. The author argues that unverifiable detection claims — such as '99% detection' with no supporting methodology — are widespread in the industry and undermine informed buyer decisions. The dataset, licensed under CC BY 4.0, includes full methodology and scoring code so that anyone can clone the repository and reproduce every result. The release is framed as a call for greater transparency and reproducibility in AI security benchmarking.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds offline Python tool that converts photos into 3D-printable meshes

A developer has released PolyForge, a standalone Python package for CAD generation and offline photogrammetry that requires no cloud API or language model to function. The tool supports three CAD backends — OpenSCAD, FreeCAD, and Blender — and can generate parametric parts like brackets and enclosures from plain-text descriptions. A newly added feature, reconstruct-from-photos, uses COLMAP for sparse 3D reconstruction and OpenMVS for dense meshing, enabling full offline conversion of overlapping photos into STL files. Building the pipeline required patching OpenMVS source code three times to resolve compatibility issues with a newer OpenCV version on a rolling-release system. Testing revealed that camera coverage pattern matters more than photo count, with a minimum of three elevation rings needed for successful reconstruction.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Developer Kept a Pokémon Generator Fast by Compiling Data at Build Time

A developer building a random Pokémon generator avoided shipping a 2.4MB dataset to users by running a build-time script that compiles only the necessary data from the @pkmn/dex npm package into trimmed static JSON files. The approach reduces what each visitor downloads to just 13 fields per species, cutting out thousands of redundant data points the UI never uses. The build script also handles messy classification logic — such as defining starters, legendaries, and generation ranges — in one place rather than scattering it across UI components. The only exception is the 870KB learnsets file, which is loaded on demand via dynamic import since just one feature requires it. The pattern, described as treating the build step as a data compiler, eliminates runtime API calls entirely and keeps the app fast without depending on a third-party service at runtime.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Bilingual City Guide Models Recurring and One-Off Events Differently

A developer building a bilingual city guide for Santa Cruz de la Sierra documented the challenges of modeling two fundamentally different event types: dated one-off occurrences and open-ended recurring events. Treating both as a single database row with a date column causes recurring events to silently disappear when past-event filters use a date threshold, and workarounds like sentinel future dates corrupt sorting and querying logic. The solution involves splitting the two types with an explicit discriminator, reflected visibly in URL slugs — dated events carry an ISO date and random suffix for uniqueness, while recurring events use a stable marker like '-weekly' with no date. On the structured data side, schema.org's EventSeries with a Schedule node is the correct representation for recurring events, as a plain Event with a fabricated date makes a false machine-readable assertion about a specific occurrence. The guide also maintains fully localized URL path segments for its Spanish mirror, going beyond simple locale prefixes to improve regional search relevance at the cost of a more complex routing table.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Semantic Caching Cuts AI Costs by Matching Meaning, Not Exact Words

A developer analyzing logs for a support bot discovered that its exact-string cache was failing to recognize semantically identical questions phrased differently, triggering a full AI model call every time. Traditional caches rely on character-for-character matches, meaning questions like 'How do I reset my password' and 'I forgot my password, help' never share a cache hit despite asking the same thing. Semantic caching solves this by converting each query into a numerical embedding that represents its meaning, then comparing it against stored embeddings using cosine similarity. If the similarity score exceeds a set threshold — such as 0.92 — the system returns a cached answer instead of making a costly new model call. The approach can significantly reduce redundant AI API calls in customer-facing applications where users routinely ask the same questions in varied wording.

0 comments Read more at DEV Community