KaliBench: How a Cybersecurity Tool-Use Benchmark Exposes the Gap Between Agent Intent and Executable Commands

Security agents fail in a specific way: they understand what you want but cannot translate that intent into the exact command-line invocation required. KaliBench is a benchmark that measures this translation layer directly, without executing potentially dangerous security tools in a live environment. The gap is not knowledge. It is the boundary between reasoning about a task and generating the precise syntax, flag bindings, and argument order that a CLI tool expects. In cybersecurity workflows, this boundary is unforgiving.