Coding Agent Pass Rates Hold Across English and Russian Tasks, but Codex Skips Web Search

A developer tested whether coding agents perform differently when tasks are written in Russian versus English, running 1050 trials across seven agent setups. Pass rates without a guide were statistically similar, at 63% for English and 56% for Russian, and guides lifted results by comparable margins in both languages. Translating the guide into Russian made no difference compared with the English original. The notable exception was Codex, which searched the web far less often on Russian tasks and failed a forum-only fact task until a guide supplied the missing information. The author concludes a guide closes the same gap regardless of language, while noting the agent may quietly skip a research step for non-English prompts.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in