Independent Benchmark Finds Jev AI Confident Only When Correct on Agent Risk Tasks

·1 views

A software developer published a 60-case benchmark on September 19, 2026, testing Jev, an AI model launched by TypeSafe AI on September 15, 2026, on its ability to classify agent tool-call risk into four categories. The benchmark was designed to assess accuracy, latency, and the reliability of Jev's confidence scores — the operationally critical claim from the model's launch. Both jev-latest and jev-preview achieved 91.7% overall accuracy, with perfect scores on clear cases but lower performance on ambiguous ones, run from a residential connection in Portland, OR. The most consistent finding across all runs was that every incorrect answer carried a confidence score below 1.000, meaning the model never expressed full certainty on a wrong answer. The full benchmark harness, tasks, and per-call results are publicly available on GitHub for independent verification and re-running.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

GSoC Developer Rebuilds Nominatim's Category Model to Support Multi-Tag Places

Developer Agasta, selected for Google Summer of Code (GSoC), spent the summer overhauling the category model of Nominatim, the geocoder that converts OpenStreetMap place names and addresses into coordinates. The core problem was that Nominatim previously allowed only one class/type pair per place, making it difficult to represent objects with multiple identities, such as a hotel containing a restaurant. The rebuild introduced PostgreSQL's ltree extension to store categories as hierarchical dot-separated paths, enabling filters like 'all descendants of osm.amenity' without requiring explicit prefix storage. Changes spanned the import pipeline, database schema, ranking logic, search indexes, API, and SQLite adaptor, and also uncovered a pre-existing bug in the PostgreSQL array result processor. The updated API now supports advanced filtering with AND/OR semantics, hierarchy matching, and consistent behaviour across all supported backends.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OWASP Core Rule Set Extended to Block LLM Prompt Injection via WAF

A technical guide published on September 20, 2026, demonstrates how to apply the OWASP Core Rule Set (CRS) to large language model traffic using Solo Enterprise's agentgateway and the Coraza rules engine. Unlike traditional web application firewalls that inspect URLs and headers, this approach buffers and parses the full JSON request body so that prompt content and tool call arguments become addressable fields for rule matching. Custom SecLang signatures can be added alongside CRS to detect and block common LLM attack patterns such as prompt injection, jailbreak framing, and system-prompt exfiltration attempts, all returning HTTP 403 responses. The WAFPolicy resource is decoupled from routing, allowing a single policy to be reused across multiple HTTP routes without modifying the underlying model or MCP server. Live cluster tests on version 2026.8.2 confirmed that normal requests pass while all adversarial prompt payloads were blocked.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Securely Verify Telegram Login Widget Payloads in PHP and Yii2

Integrating the Telegram Login Widget into web applications requires strict backend validation to prevent identity spoofing. Developers must verify the incoming authentication payload using HMAC-SHA-256, derived from a SHA-256 hash of the Telegram bot token. The process involves sorting query parameters alphabetically, building a formatted data string, and comparing the computed signature against the received hash using a timing-attack-resistant function. An expiration check on the auth_date parameter is also enforced to guard against replay attacks. In Yii2, a database migration adds a unique telegram_id column to the user table, allowing validated Telegram identities to be mapped to local user records.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Automated Python Webhook Funnel to Drive Votes in Online Competition

A developer participating in the CANEX Create-Thon competition on the Gebeya platform built a zero-cost automated voting funnel to work around the platform's high sign-in friction. The Gebeya voting system requires a full email sign-in before a vote can be cast, which the developer estimates causes a 90% drop-off rate among organic visitors. To counter this, they engineered a three-part system comprising an Astro and Tailwind frontend, a Python FastAPI webhook engine, and an automated direct-message and reward trigger. The developer is currently load-testing the funnel over an eight-day period and has shared the live flow publicly. The project was documented on DEV Community, where the developer invited feedback on optimizing the Python routing logic.

0 comments Read more at DEV Community