Developer Builds Benchmark to Test If AI Models Actually Follow Coding Instructions

A developer has created a custom benchmark to evaluate whether AI models can follow specific instructions while completing coding tasks, not just solve the problem itself. The benchmark presents identical prompts to multiple models and checks compliance with constraints such as avoiding certain methods, using a specific language, or returning output in a required format. Each model is scored separately on task correctness and instruction compliance, allowing patterns to emerge across models. The project was motivated by a common frustration: AI often produces technically correct code while ignoring one or more explicit instructions given by the user. The developer plans to expand the benchmark with more complex scenarios, including conflicting instructions, multi-turn tasks, and self-correction after instruction-following errors.