Developers Build First Benchmark to Test AI Agents on Real Cloud Management Tasks

A new benchmark is being developed to evaluate how AI coding agents like Codex and Claude Code perform on real-world cloud management tasks, an area not covered by existing benchmarks. The methodology uses Terraform as a ground truth, deploying known resources on AWS so agent outputs can be scored objectively without human labeling. Test environments vary by account size and history, including messy brownfield setups that better reflect production conditions, and agents run in isolated containers with read-only credentials to ensure controlled, reproducible runs. The first task focuses on waste discovery, asking agents to identify orphaned AWS resources while avoiding false flags on resources still in use. Results, including unflattering ones, along with all code and logs, will be published openly, and the team is inviting community feedback on methodology and future task selection.