Prompt injection evaded Google's Model Armor, but strict output schema blocked it anyway
Developers building Okimera, a multi-agent maritime sanctions compliance system, discovered that Google's Model Armor prompt-injection filter failed to flag a hidden instruction embedded in a realistic business document, even at its most sensitive detection threshold. The injected text, written to resemble a legitimate commercial clause rather than a typical jailbreak command, went undetected in two languages, while a conventionally worded jailbreak was caught by the same configuration. The team attributed the miss to a fundamental limitation of general-purpose classifiers, which are trained on assistant-style attacks and struggle to identify malicious instructions that blend into domain-specific language. However, the injection caused no harm because the document-reading agent was designed to return only a strictly typed data object with no free-text fields, leaving the malicious instruction no channel through which to travel. The developers concluded that structural output constraints, not probabilistic filters, should bear the primary security burden in agentic systems that process adversarially controlled documents.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in