Benchmark Tests Jev-1.13.0 Decision Model Across 10 Datasets for AI Agent Use

·1 views

A black-box engineering evaluation of TypeSafe AI's Jev-1.13.0 decision model was conducted in September 2026, spanning 10 public datasets, roughly 22,500 API calls, and 52.2 million input tokens at a total cost of $2.19. Unlike large language models, Jev accepts a state and typed questions, returning typed answers with probabilities and confidence scores without generating free-form text. The model performed strongly on tasks such as indirect prompt-injection detection, document reranking, intent classification, and tool routing, making it a viable fast-decision layer in agent systems. However, it showed no meaningful signal for predicting another model's failure modes, trajectory failure attribution, and performed notably worse on non-English tasks due to its English-primary training. Two key engineering findings emerged: decomposing fuzzy judgments into orthogonal sub-questions and using a two-stage choice-then-verify approach each dramatically improved accuracy in their respective tasks.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Showback vs Chargeback: How Companies Allocate Shared IT Costs to Teams

Showback and chargeback are two standard models for attributing shared IT costs to the teams that incur them, differing primarily in whether money actually moves. Showback reports each team's cost share as information only, leaving the central budget as the payer, while chargeback transfers that share directly onto each team's budget as an internal financial transaction. Showback is lower-risk and easier to implement from the outset, even with imperfect data, since errors result in a conversation rather than a billing dispute. Chargeback creates stronger accountability by making teams responsible for spending from their own budgets, but requires accurate attribution, finance integration, and a formal dispute process. Organizations typically begin with showback to build trust in the data before transitioning to chargeback once methodology and culture are sufficiently mature.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

On-Device AI Pipeline Turns Meeting Speech Into Assigned Tasks Automatically

A team behind the Octic on-device recorder and Octo collaboration platform has built a five-stage pipeline that converts spoken meeting decisions into tracked, assigned tasks without manual input. The system captures audio locally on the Octic device, runs speech recognition and speaker diarization on-device so audio never leaves the room, then passes timestamped transcripts through a natural language understanding model to extract action items, decisions, and open questions. Extracted tasks are automatically turned into structured work units called Loops in Octo, each with an owner, description, and acceptance criteria mapped from the conversation. The project was motivated by internal tracking data showing that roughly 40% of verbally agreed tasks across six or more daily meetings were never entered into any system. The full pipeline runs as a post-meeting batch job, a design choice the team found produces more accurate intent extraction than real-time processing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Finds Double-Counted 21,000 Gas Bug in Smart Contract Refund Logic

A developer building Legwork, a smart contract for recurring on-chain payments on Arc mainnet, discovered a gas refund miscalculation during a post-deployment audit. The contract was incorrectly crediting the 21,000 gas intrinsic transaction cost once per order execution call, rather than once per transaction, causing over-refunds when multiple orders were batched together. For a batch of K orders in a single transaction, executors were being overpaid by 21,000 × (K − 1) gas units, though no funds were exploited and refunds were bounded by each order's reserve. The bug was identified the day after the README claimed refunds were exact, and a fix was implemented by tracking whether the intrinsic cost had already been credited within a transaction. The corrected contract remains live on Arc mainnet, with the full audit trail and receipts preserved in the public MIT-licensed repository.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer turns 1,100 pages of five-year technical notes into seven books with AI nudge

A software developer accumulated over 1,100 pages of technical notes across five years while working on embedded systems projects spanning Linux, QNX, AUTOSAR, and hardware security modules. The notes began as personal study summaries during graduate school and evolved into detailed debugging logs and module documentation during startup internships. Curious about the material, the developer fed the notes into an AI, which assessed the content as covering a rare depth across application, driver, and hardware layers. The AI suggested the notes contained enough coherent technical knowledge to be organized into books that could benefit other engineers. The developer subsequently restructured the material by topic — grouping areas like PTP, UDS, and HSM — and rewrote each section from scratch into seven standalone books grounded in protocol standards, open-source code, and firsthand experience.

0 comments Read more at DEV Community