Developer Loses 48 Hours to Cache Bug That Ignored System Prompts in LLM Wrapper

·2 views

A developer spent nearly two days troubleshooting apparent nondeterminism in an LLM endpoint, initially blaming model instability and sampling noise. The real culprit was a flawed cache key function in a homemade HTTP wrapper that hashed only the user message, ignoring system prompts, temperature, and other parameters that affect model output. As a result, different requests with distinct system prompts collided on the same cache key, returning mismatched stored responses instead of fresh completions. The bug was especially deceptive because the cached answers were fluent and on-topic, making them hard to identify as wrong at a glance. The core lesson is that a cache key must represent the entire request, not just the user-visible text, to avoid silently serving stale or mismatched responses.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Crawlers Read a Site 10,000 Times But Sent Only 2 Human Visitors

A WordPress site called davaonline.net, running a plugin designed to track AI crawler activity rather than human traffic, recorded 13,306 AI crawler requests over 30 days, of which 10,202 targeted actual content pages. Despite this heavy machine-side activity, conventional analytics showed just two human referrals from AI sources — one from ChatGPT and one from DuckDuckGo. The plugin categorises requests by purpose, distinguishing between training data collection, search indexing, and live user-query fetches, with the latter reaching 1,902 requests — a 3,558% increase over the prior 30-day period. The data illustrates a growing disconnect: AI systems are actively retrieving and using web content to answer user questions in real time, but those answers are delivered without routing the user back to the source page. The findings suggest that standard web analytics, which measure human visits, may increasingly undercount how often a site's content is actually being read and used.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Building Production-Grade AI Systems on AWS Goes Far Beyond Calling an LLM API

Modern AI applications require much more than a simple chatbot interface, demanding robust layers including orchestration, memory, guardrails, and observability to function reliably at scale. A production AWS-based AI system typically integrates services such as Amazon Bedrock for model access, OpenSearch or pgvector for retrieval-augmented generation, DynamoDB for agent state, and CloudWatch for monitoring. AI agents add further complexity by planning multi-step actions, calling external tools, and requiring human approval for high-risk decisions. Common production failure points include model timeouts, API rate limits, hallucinations, and stale vector embeddings, each demanding specific mitigation strategies like retries, queuing, and validation guardrails. The article argues that skipping any architectural component — whether security, memory, or observability — can silently degrade system reliability until end users report failures.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Use Python and DoWhy to Find True Causes Behind Your Health Habits

A tutorial published on DEV Community demonstrates how to apply causal inference techniques to personal health data using Python's DoWhy library. The guide addresses a common analytical pitfall where correlation is mistaken for causation, using the relationship between caffeine intake and sleep quality as a practical example. It introduces Directed Acyclic Graphs to model confounding variables, such as work stress, which can independently drive both increased coffee consumption and poor sleep. Using a synthetic dataset of 1,000 simulated days, the tutorial walks through DoWhy's four-step workflow — Model, Identify, Estimate, and Refute — to isolate the true effect of caffeine on sleep. The exercise shows that simple correlation overestimates caffeine's negative impact on sleep because it fails to account for the stress confounder.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three-Character Quality Gate Loophole Taught a Speech Model a Faulty Habit

A developer training a text-to-speech model discovered that a flaw in the corpus quality-filtering logic caused the model to consistently append meaningless sounds at the end of sentences. The corpus pipeline used Whisper to transcribe TTS-generated audio and discarded clips where transcriptions contained more than three inserted characters not found in the original script. Short hallucinations of one or two characters routinely passed this threshold and were included in training data across roughly 200 clips per voice. The model, trained on 12 voices, learned to reproduce this pattern of appending brief extra sounds. The problem went undetected initially because Whisper itself fails to transcribe very short audio artifacts, and only waveform-envelope analysis revealed the defect.

0 comments Read more at DEV Community