Study Finds Half of 330 AI Models Fail to Respond Correctly in Korean

·6 views

A benchmark evaluation of 330 large language models tested Korean language ability across seven axes, including honorifics, terminology, register, and format compliance. A script-based script-contamination check — not an AI judge — automatically failed any response containing less than 25% Hangul, more than three Chinese characters, or any Japanese kana. Using this method, 766 of 2,304 responses (33.2%) were auto-failed before human or AI review, and 54 models failed all seven tasks entirely. Honorifics and knowledge of Korean institutions proved the hardest axes, with AI success rates of just 8.5% and 9.4% respectively. Among vendors with at least eight models tested, Anthropic led with a mean score of 2.29 out of 3, while five models — mostly from OpenAI — achieved a perfect score.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Two-Artifact Freeze Method Keeps AI Patch Reviews Scoped to Real Bugs

Open-source maintainers risk unintended API expansions when coding models generate overreaching diffs in response to bug reports. A proposed workflow requires freezing two artifacts before any model session begins: a git bisect SHA identifying the first bad commit, and a snapshot of existing public API names. These two files act as hard merge gates, restricting model edits strictly to files changed in the offending commit. The bisect must confirm a reproducible test failure before any chat session starts, and a flaky test halts the entire process. The method aims to prevent silent surface-area growth and keep code review focused on the actual defect rather than model-suggested additions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Agents Excel at Exploratory Testing but Fall Short for Regression Suites

AI agents have proven effective for exploratory testing by dynamically navigating interfaces, reacting to unexpected states, and investigating bugs without fully predefined steps. However, regression testing demands repeatable, documented action sequences and consistent checks so that results across multiple runs can be meaningfully compared. When an agent adaptively chooses alternate paths or accepts different signals as proof of success, it undermines the comparability that regression testing requires. Experts argue teams should use AI agents for ambiguous, one-off, or investigative tasks while investing in structured test assets for workflows that must be verified before every release. The key question for testing teams is not whether to use agents or automation, but which work benefits from flexibility and which requires repeatability.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Experiment Reveals Hidden Latency After AI Code Generation Ends

A developer found that the real performance bottleneck in an AI-assisted coding workflow was not the remote language model but the file-writing steps that followed token generation. Standard product dashboards typically highlight first-token latency, masking heavier stalls that occur during the apply phase. To investigate, the developer built a lightweight Python timer using perf_counter to record named spans and render them as an ASCII waterfall chart. The experiment used a small local JSON fixture with no production traffic, deliberately keeping the setup simple and reproducible. The findings showed that apply-phase operations like writing and fsyncing files produced multiple distinct spans, collectively outweighing the generation span in wall-clock time.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Faster Software Development Means Little If Users Still Struggle to Navigate It

Modern tools like agile workflows, component libraries, cloud platforms, and AI-assisted coding have dramatically reduced the time it takes to build software. However, writer Parvej Shah argues that development speed and product usability are fundamentally different metrics, and shipping features quickly does not guarantee those features are easy to use. Shah contends that the true advantage of rapid development lies not in building more features faster, but in shortening the feedback loop — testing assumptions with real users sooner and iterating based on what is observed. While engineering teams measure progress through deployments and passing tests, users only care whether a product helps them accomplish their goal. As competitors also accelerate their development pace, Shah suggests the real strategic question is not how to build faster, but how to use that speed more wisely.

0 comments Read more at DEV Community