Treat LLMs as Unreliable Functions to Build Reliable AI-Powered Software

Developers often rely on prompt tweaking to fix AI output issues, but LLMs are probabilistic and cannot guarantee consistent responses, making this approach fragile in production. A more robust strategy involves treating LLM output as untrusted input and validating it immediately against a strict schema using tools like Zod or Pydantic before it reaches any business logic. When validation fails, a retry loop that feeds the specific error back to the model is more effective than blindly resending the original prompt. Restricting outputs to predefined enums rather than open-ended descriptions further reduces unpredictability and simplifies validation. Engineers are also advised to maintain a golden dataset of 50–100 input-output pairs to objectively measure whether prompt changes improve or regress system performance.