Redis-backed tier-aware queuing cuts LLM p99 latency from 20s to under 2s

·1 views

An AI companion app running on a single 4-core server faced severe latency spikes during peak hours, with paid users experiencing 20-plus-second response times due to free-tier traffic saturating shared LLM call slots. The initial fix using a per-process asyncio.Semaphore failed on two counts: it multiplied slots across eight Uvicorn workers instead of enforcing a true global cap, and its FIFO ordering gave free and paying users equal priority. Engineers replaced it with a Redis-backed slot manager using an atomic Lua script to enforce both a global concurrency ceiling of 30 in-flight LLM calls and separate per-tier caps. The new system also applies different timeout behaviors by tier, letting free and guest requests fail fast while paid users queue patiently, ensuring paying customers always have reserved capacity. The entire implementation runs to roughly 40 lines of code and brought peak p99 latency down from over 20 seconds to under 2 seconds.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AWS Kiro Crew Uses Multi-Agent AI to Cut Code Review Burden for Engineers

Software teams increasingly face a hidden productivity cost called the 'Review Tax,' where senior engineers spend significant time validating low-quality or flawed AI-generated code suggestions. Generic large language models often lack codebase context, hallucinate libraries, and miss subtle security vulnerabilities, making human review of their output inefficient. AWS Kiro, Amazon's AI-powered coding assistant for VS Code and IntelliJ, aims to address this through a feature called Kiro Crew, which coordinates multiple specialized agents — including a Security Auditor, Linter, and Architectural Consistency Checker. Unlike standalone AI chatbots, Kiro integrates directly into the IDE with visibility into project structure, open files, and dependency graphs, giving it the context needed to produce more accurate suggestions. The goal is to shift from single-turn AI prompts to orchestrated multi-agent workflows that filter and validate code before it ever reaches a human reviewer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

What fp-ts teaches TypeScript developers even without using it in production

A developer argues that fp-ts, the functional programming library for TypeScript, is more valuable as a learning tool than as a production dependency. The library's core concepts — Option, Either, and pipe — teach developers to encode uncertainty and failure directly into function signatures rather than relying on implicit nullables or uncaught exceptions. The author illustrates how a silent undefined returned from a database query caused runtime failures across multiple components in a Next.js application, a problem that explicit type modeling could have prevented. These patterns can be applied in vanilla TypeScript alongside tools like Zod and Server Actions without ever installing fp-ts itself. The key takeaway is that fp-ts should be treated as a way to sharpen reasoning about data contracts, not as a framework teams must adopt from the start.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

What fp-ts teaches TypeScript developers even without using it in production

A developer argues that fp-ts, the functional programming library for TypeScript, is more valuable as a learning resource than as a production dependency. The article focuses on three core concepts — Option, Either, and pipe — which can be internalized and applied in plain TypeScript without importing the full library. Option makes nullable return values explicit in a function's signature, while Either models errors as typed values instead of relying on thrown exceptions. The author demonstrates how these patterns improve code clarity in real-world stacks involving Next.js Server Actions, Zod schemas, and Prisma. The central thesis is that fp-ts's true value lies in the mental vocabulary it provides, not in the ecosystem itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Production RAG Chatbot Using Claude, pgvector, and FastAPI

A practical guide outlines how developers can build a retrieval-augmented generation (RAG) chatbot over a weekend using three core tools: PostgreSQL with the pgvector extension, FastAPI, and Anthropic's Claude model. The architecture works by embedding a user's query into a vector, retrieving the most relevant document chunks from the database, and passing that context to Claude to generate a grounded answer. A key point of confusion for beginners is that Claude does not provide an embeddings endpoint, so a separate embedding provider such as Voyage AI or OpenAI must be used. The pgvector column dimensions must match the chosen embedding model exactly, making the embedding provider a critical dependency of the database schema. The guide includes working code for chunking documents, generating embeddings, storing them in Postgres, and exposing the full retrieve-then-generate flow via a FastAPI endpoint.

0 comments Read more at DEV Community