OpenEval v1.0.0 Proposes a Standard JSON Format to Unify LLM Evaluation Frameworks

·9 views

A new open-source project called OpenEval aims to solve the lack of interoperability among popular LLM evaluation frameworks such as DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness. Currently, each framework uses its own incompatible format for test cases, grader definitions, and results, forcing teams to manually rewrite datasets when switching tools. OpenEval addresses this by defining a portable JSON Schema spec along with TypeScript and Python SDKs, a CLI, and converters for existing frameworks. The project has released version 1.0.0 on both npm and PyPI, making it available to developers immediately. Over 17 framework integration issues are open on the project's GitHub repository, inviting community contributions to expand converter support.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Opinion: Long AI Agent Loops Signal Failure, Not Intelligence, Developer Argues

A developer maintaining an open-source LLM gateway argues that extended agentic loops are typically a sign of model failure rather than effective reasoning. The core claim is that adding more turns, context, and scaffolding to an AI agent usually amplifies errors rather than resolving them, because each step conditions on previous mistakes as if they were facts. Loop costs compound quickly since accumulated context is re-sent with every turn, making long sessions disproportionately expensive. The author also contends that reliability degrades across dependent steps, meaning longer loops increase the chance of an agent drifting further down the wrong path. The proposed fix is not more turns but better task routing to capable models and cheap step-level verification to catch errors early.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CLAUDE.md, Git, and MCP Servers Each Play a Distinct Role in AI Agent Memory

Teams using Claude Code agents have three main tools for preserving context and decisions: a CLAUDE.md file, Git, and an MCP memory server. CLAUDE.md is best suited for stable, repo-specific conventions like coding style and architecture rules, but cannot capture real-time decisions or coordinate across repositories. Git reliably tracks code history and the reasoning behind changes, but is poorly suited for live team coordination since it is poll-based and prone to merge conflicts when used as a shared memory store. MCP memory servers fill the gap by providing a live, shared store that agents can read and write across sessions and machines, capturing decisions the moment they are made. The three tools are designed to complement each other: Git holds the code, CLAUDE.md stores standing instructions, and an MCP layer handles fast-moving team context and cross-agent coordination.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Structured Decomposition Makes AI Agent Workflows Production-Ready

A software engineer at BizFlowAI has outlined a practical framework for building multi-agent AI workflows that remain stable under real-world, unpredictable inputs rather than clean demo conditions. The approach centers on mapping every workflow as a typed sequence of functions before introducing any LLM, clearly separating steps that require judgment from those that can be handled by deterministic code. The engineer's content pipeline assigns LLMs only to creative and analytical decisions, while schema validation, SEO checks, and API calls are handled entirely in code, which he says reduced errors to near zero. Tool selection is identified as a major failure point, with production data suggesting correct tool selection drops from 96–99% with 3–5 tools to below 70% when agents are given more than 20 tools. The recommended fix is a dispatcher-agent architecture where a small routing agent delegates to focused sub-agents, each limited to a narrow, well-described toolset.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

DIY Loop, LangGraph, or Custom Orchestration: How to Pick Your AI Agent Stack

A developer who ships AI agents into production weekly has outlined a practical framework for choosing between three agent-building approaches in 2026: a DIY loop, a framework like LangGraph, and custom orchestration. The decision hinges on three factors — number of steps, concurrency requirements, and crash recovery needs. For simple agents with fewer than five tools and a linear workflow, a plain while loop in roughly 150 lines of Python is preferred and covers about 40% of use cases. LangGraph becomes the better choice when workflows involve significant branching, durable execution over hours or days, or multi-agent coordination. Custom orchestration on tools like Temporal or AWS Step Functions is reserved for the most complex, long-running, or mission-critical workloads.

0 comments Read more at DEV Community