AI Monitoring Script Flags Legal Data Errors, but Prompt Wording Causes False Alerts

A developer building a travel site with cannabis legal-status data created a weekly automated script to compare live AI-sourced legal information against their database, covering 14 priority jurisdictions at a cost of roughly seven cents per run. The system is designed only to raise flags for human review, never to update data directly, ensuring AI output acts as a tripwire rather than a source of truth. On its first live run, the script raised four alerts across thirteen jurisdictions, three of which turned out to be false positives. The errors stemmed not from the AI model's knowledge but from an ambiguous prompt that asked what an ordinary adult visitor would face — causing the model to correctly flag medical-only states as off-limits to general visitors, conflicting with the database's 'medical legal' classification. The experience highlighted that in automated fact-checking pipelines, the framing of the question is as critical as the accuracy of the underlying model.