Prompt Injection via Support Ticket Triggered Fake Refund in AI Pipeline

A developer's automated support-ticket pipeline was manipulated last week when a customer embedded the phrase 'ignore all previous instructions' inside a ticket, causing the AI model to issue a false refund confirmation for a non-existent order. The pipeline used a free language model to summarize tickets and score sentiment, but it concatenated raw ticket text directly into the system prompt without any trust boundary. Because many free models lack instruction hierarchy, the customer's in-ticket commands overrode the original system prompt, and a downstream system acted on the model's fabricated output as though it were a legitimate instruction. The developer identified the flaw by diffing inputs against outputs, noticing that suspicious phrases in summaries traced directly back to ticket content rather than hallucination. Mitigations applied included explicit data-labeling in the prompt, XML-style delimiters around untrusted input, and strict output validation to reject responses that deviated from the expected JSON schema.