Dev team uses DPO fine-tuning to reduce AI coding agent errors on first attempt

A development team running a local VS Code plugin called WUIC Assistant — powered by a 30-billion-parameter Qwen3-Coder model on a single RTX 4090 — found that their AI agent was passing all benchmark tasks but still making repeated wrong tool calls before self-correcting. To address this churn and test true generalization, they built evaluation-only task sets that were never used in training and introduced a harder benchmark of 12 tasks targeting multi-step reasoning, ambiguous prompts, and rare capabilities. Before fine-tuning, the base model scored 69% on the new benchmark and required nine guard-bounced redirects to complete one task correctly. The team also uncovered two hidden test environment contaminations — a faulty database reset and an overwritten shared file — that had been silently skewing results for weeks. These findings led them to apply Direct Preference Optimization (DPO) fine-tuning aimed at getting the model to choose the right tool on the first attempt rather than learning through repeated failures.