Student developer shares hard-learned lessons on building reliable LLM API wrappers

A student developer building LLM-powered projects discovered that infrastructure concerns — not the AI model itself — account for the majority of real-world engineering work. After a friend exposed a timeout flaw in just four minutes, the developer identified four critical gaps: absent request timeouts, indiscriminate retry logic, uncontrolled input token size, and missing per-call cost monitoring. They found that timeouts must be shorter than upstream gateways like nginx to avoid ghost requests that cost money after users receive a 504 error. On retries, the developer warns that blindly retrying 400-level errors wastes money since malformed requests will always fail, and suggests capping retries at two. For RAG pipelines especially, they stress that input token limits matter as much as output caps, since uncontrolled document chunks can silently inflate costs by an order of magnitude.