Developer argues deterministic code, not LLMs, should drive AI observability decisions

A developer running a managed monitoring service found that letting large language models classify and judge infrastructure alerts produced inconsistent, unreliable results — with the same alert generating different root-cause categories across multiple runs. To fix this, they redesigned the pipeline so that deterministic Python code handles all classification, validation, and aggregation, while the LLM is restricted solely to narrating the outcome in plain English. The approach also incorporates a pre-call metrics lookup to give the model historical baseline context, preventing misleading narratives such as false disk-fill urgency on a stable host. The author notes this design keeps aggregation reliable, prevents hallucinated classifications, and ensures the monitoring pipeline remains functional even if the model fails. The system runs intentionally on modest CPU-only hardware with no GPU, reflecting a deliberate choice to keep all data on-premise rather than sending it to external services.