Gemini Tops Structured Document Extraction Benchmark Against Claude and OpenAI

·1 views

A team of engineers benchmarked Gemini, Claude, and OpenAI models on extracting structured data from ten handwritten forms using LangSmith and LangGraph. Models were evaluated on three metrics: partial field accuracy, exact full-record match, and cost efficiency per correct field. Gemini 2.5 Pro and Claude Opus tied on field accuracy at 93.4%, but Gemini produced more perfectly extracted records, completing 7 out of 11 correctly. Among budget-tier models, Gemini Flash Lite outperformed OpenAI's flagship GPT model on field accuracy while costing significantly less. The benchmark highlights that no single model leads across all metrics, and the best choice depends on whether a use case prioritises accuracy, full-record precision, or cost.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

A 41% Failure Rate Stopped Engineers From Shipping a Temporal Knowledge Graph

A software team nearly deployed a temporal knowledge graph (TKG) system designed to give AI agents time-aware memory by storing facts with validity windows instead of relying on flat vector recall. During evaluation, the system failed 41% of the time on a key test: correctly reporting the state of a node at a specific past time T. The root cause was a flawed retrieval query that sorted facts by the most recent start time rather than filtering by the actual reference timestamp, causing the agent to return a later, incorrect fact. Standard static retrieval metrics had shown no problems, masking the issue until a time-specific evaluation test was written. The incident highlights how temporal queries disguised as simple status lookups can silently bypass conventional testing, making targeted evals critical before deployment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Load Testing Your Website Before Launch Can Prevent Costly Outages

Load testing simulates concurrent users hitting a website to measure server performance under expected traffic conditions, helping teams identify breaking points before real users do. Unlike stress testing, which pushes systems to failure, or soak testing, which checks for degradation over time, load testing focuses on whether a site can handle its anticipated peak traffic. Tools like loader.io offer a free, browser-based way to run load tests without any installation, making the practice accessible to small teams. Experts recommend running load tests before every major deployment rather than waiting for a live outage to expose weaknesses. Skipping load testing is particularly risky for startups and growing SaaS businesses, where a crash during a product launch or a critical customer interaction can have lasting consequences.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Next.js App Router CSP with Nonces Forces Every Route into Dynamic Rendering

Implementing a Content Security Policy in Next.js App Router requires a per-request nonce combined with the 'strict-dynamic' directive, as static domain allowlists cannot cover the framework's runtime chunk loading behavior. The nonce must be generated in middleware.ts and passed through both request and response headers so Next.js can apply it to its own bootstrap script tags. A key trade-off is that calling headers() to read the nonce opts every matched route out of static rendering, adding a performance cost developers should plan for. Four directives — object-src, base-uri, form-action, and frame-ancestors — can be safely added via next.config.js without nonces and remain fully cacheable. The 'strict-dynamic' keyword grants inherited trust to scripts loaded by an already-trusted script, eliminating the need for build-time hash enumeration or chunk URL lists.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Fix Data Fetching Race Conditions in Web Apps

Race conditions in data fetching occur when multiple asynchronous HTTP requests are sent in quick succession but return responses out of order, causing the UI to display stale or incorrect data. A common example is an autocomplete search bar where a slow response for an early query overwrites a faster, more accurate response for a later one. This happens because network factors like packet loss, server load, and routing variations mean requests do not resolve in the order they were made. Traditional fixes like disabling UI elements or showing full-screen spinners technically prevent the problem but hurt user experience by making apps feel sluggish. The article argues that developers need more elegant solutions that cancel or ignore outdated responses without blocking user interaction.

0 comments Read more at DEV Community