How to Build a Contract Test Suite for Your LLM Gateway Layer

·1 views

Developers running a proxy in front of AI model providers face bugs not in the models themselves, but in the gateway layer — such as mishandled headers, incorrect error translations, or accidentally buffered streams. A purpose-built contract test suite should verify two core promises: that the gateway speaks an OpenAI-compatible contract downstream and faithfully translates requests upstream. Rather than relying on live model calls, the recommended approach uses a fake local HTTP server to return controlled fixtures, making tests fast, deterministic, and suitable for every pull request. Tools like Mock Service Worker's Node interceptor allow the gateway's real HTTP client, timeout, and retry logic to execute against these stubs. One of the most critical test assertions is confirming that unknown or unrecognized fields from upstream responses are preserved, since a gateway that silently drops them can cause users to lose data like log probabilities or cache metrics.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Build an Offline AI Health Analyst on a MacBook Using Llama-3 and MLX

A new tutorial demonstrates how Mac users can analyze their Apple Health data entirely offline using Meta's Llama-3 language model and Apple's MLX framework, eliminating the need to upload sensitive health information to the cloud. The setup runs a 4-bit quantized Llama-3-8B model locally on Apple Silicon chips, taking advantage of the Unified Memory Architecture for fast on-device inference. Users first export their health data from the Apple Health app, then parse metrics such as heart rate and step count from the XML file using Python and Pandas. The processed data is fed into the local model, which generates actionable health insights without any data leaving the device. The approach requires a Mac with an M1, M2, or M3 chip and a few open-source Python libraries to get started.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Claude Code 2.1.229 adds SSE keepalive pings to fix gateway stream timeouts

Anthropic released Claude Code version 2.1.229 on August 12 UTC, introducing SSE keepalive pings designed to prevent idle-timeout disconnections during extended model-thinking pauses. The fix targets gateway streaming responses routed through Google Vertex AI and Amazon Bedrock upstreams. Teams using enterprise LLM gateways or direct provider connections are advised to upgrade the Claude Code client and verify the fix by testing a workload that previously failed with a stream idle timeout error. Anthropic clarified that the patch addresses only streaming liveness and does not resolve unrelated issues such as expired credentials, provider quotas, or buffering reverse proxies. Platform and developer-experience teams are cautioned to test one variable at a time to accurately attribute any improvements in streaming reliability.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Kanban Was Born in Japan, Yet Japanese Offices Still Swear by Excel

Toyota engineer Taiichi Ohno developed the Kanban system in the late 1940s and 1950s, drawing inspiration from American supermarket restocking methods to create a visual, pull-based workflow that minimised waste on factory floors. The concept was later adopted by the global tech industry and became a cornerstone of modern agile project management. Despite this Japanese origin, many traditional Japanese workplaces continue to rely heavily on Microsoft Excel for nearly all organisational tasks, from scheduling to project tracking. Excel's unstructured, rules-free flexibility makes it a deeply entrenched cultural tool that resists replacement by more opinionated systems like Kanban boards. A firsthand account from an English school in Japan illustrates this tension, where a sprawling, overloaded Excel file managed all student and staff data until the system began to struggle under its own weight.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Handle Speech-to-Text API Timeouts for Large Audio Uploads in EdTech

Developers building edtech support queues face a critical design challenge when handling large audio file transcriptions via speech-to-text APIs. Rather than relying on a single API call, the recommended approach treats transcription as a multi-stage ingestion workflow with distinct checkpoints for file acceptance, upload completion, transcription result, and output quality. Each stage requires its own state and timestamp to avoid ambiguous failures, such as duplicate transcripts caused by retrying with a new job ID after an unclear client-side timeout. An idempotency policy keyed to a stable recording ID is essential to prevent duplicate work from reaching downstream classifiers. File-size limits should be treated as application-level policy decisions, with rejected recordings retained locally and routed through an approved asynchronous path rather than silently compressed.

0 comments Read more at DEV Community