How to Deploy an LLM App That Handles Real Traffic Without Crashing

·1 views

A Dubai-based e-commerce startup nearly derailed its investor demo after a load test with 40 simulated users caused response times to spike from 1.2 seconds to 47 seconds, crashing their AI-powered product search assistant. The root cause was an architectural flaw: the language model, API logic, and session cache all ran in a single process, meaning slow inference blocked every request simultaneously. According to the author, a field engineer who has encountered this problem dozens of times, adding more servers does not fix a prototype that was never designed for production. Four core engineering principles separate a prototype from a production-ready system: statelessness, clear boundaries between layers, backpressure handling, and observability. The article outlines practical deployment options — hosted APIs, self-hosted GPU servers, and quantized CPU inference — with cost and latency tradeoffs to guide teams in making the right infrastructure choice before launch.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Static Config Checks Are Not Enough to Secure AI Agents With Tool Access

As LLM-based agents gain the ability to execute tools — such as shell commands, HTTP fetchers, and cloud SDKs — the risk shifts from harmful text output to harmful real-world actions performed with user credentials. Security researchers highlight four key abuse patterns: remote code execution via shell tools, server-side request forgery through HTTP fetchers, credential exfiltration via environment variables, and unauthorized tool calls triggered by prompt injection. Static configuration scanners, such as the open-source correctover-scan tool, can catch setup-level issues like plain-HTTP transports, hardcoded credentials, and overly broad permissions across common MCP config files. However, experts warn that static analysis cannot evaluate the specific arguments an agent constructs and passes to tools at runtime, which is where many real threats materialize. Runtime verification is therefore presented as a necessary complement to configuration hygiene for teams deploying agents with tool access.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer's Custom AppSec LLM Prompt Found 4 Real Gaps After Filtering AI Noise

A developer nearly filed 23 unverified vulnerability reports against open-source projects after running a self-built AppSec prompt across several repositories, with the LLM flagging XSS, CORS, CSP, and suspicious SVG issues in bulk. The author paused to question whether the volume of findings would be taken seriously by maintainers, which prompted a rethink of how the prompt was structured. The key shift was writing a strict contract before any code review — defining what the model was permitted to assert, anchored to OWASP ASVS 5.0 requirement IDs rather than vague best practices. Discipline rules were added to separate findings from observations and hypotheses, and to prevent the model from mixing discovery with fixes in the same session. The author concludes that static scanners like Semgrep and CodeQL remain essential for pattern-based detection, while LLM-driven hunting adds value only for semantic gaps — absent controls that no regex can identify.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build GDPR-Compliant Consent Revocation Gates in Go Runtime Systems

A software engineer has outlined a technical approach to handling GDPR consent withdrawal in developer tools, warning that simply deleting a database row or toggling a UI flag does not immediately cut off data access. The core recommendation is to model consent withdrawal as an auditable state transition, ensuring every runtime authorization check reads the current consent status before touching personal data. The proposed system uses two API endpoints — one to record withdrawal and one to check per-category consent — with a Go-based authorization function that blocks processing as soon as consent is revoked. The engineer also highlights race conditions where a data operation in flight can complete after a revoke transaction commits, and argues that idempotent deletion workers and conditional database writes are necessary to prevent duplicate or post-withdrawal processing. A trade-off between strongly consistent checks and cached consent state is acknowledged, with the recommendation to favor consistency for personal data despite the added latency cost.

0 comments Read more at DEV Community

ProgrammingHacker News ·

WebFPGA

Article URL: https://webfpga.io/ Comments URL: https://news.ycombinator.com/item?id=49531525 Points: 3 # Comments: 1

0 comments Read more at Hacker News