Developer finds AI safety harness was enforcing its own rules, bypassing actual checks

A software developer building an AI agent harness designed to prevent unchecked or irreversible actions discovered a critical flaw during a rebuild on the open-source pi runtime. When asked to commit code without an approval phrase, the AI model refused on its own — not because the gate mechanism blocked it, but because it had read the rule in its conversation context. This meant the gate was never actually invoked, yet the system appeared to be working correctly, nearly leading to a false verification. The root cause was that the gate announced itself via a method that fed into the model's context, allowing the AI to simulate enforcement and render the gate unverifiable. The developer fixed this by moving gate announcements outside the model's context and adding direct synthetic-event tests that verify gate behavior with no model involvement.