Study Finds Half of 330 AI Models Fail to Respond Correctly in Korean

·79 views

A benchmark evaluation of 330 large language models tested Korean language ability across seven axes, including honorifics, terminology, register, and format compliance. A script-based script-contamination check — not an AI judge — automatically failed any response containing less than 25% Hangul, more than three Chinese characters, or any Japanese kana. Using this method, 766 of 2,304 responses (33.2%) were auto-failed before human or AI review, and 54 models failed all seven tasks entirely. Honorifics and knowledge of Korean institutions proved the hardest axes, with AI success rates of just 8.5% and 9.4% respectively. Among vendors with at least eight models tested, Anthropic led with a mean score of 2.29 out of 3, while five models — mostly from OpenAI — achieved a perfect score.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

PocketBase Cloud CLI pbc Lets Developers Run, Deploy and Manage PocketBase Apps

A new command-line tool called pbc, the PocketBase Cloud CLI, aims to handle the full PocketBase workflow from one terminal. It installs on macOS, Linux and Windows and can scaffold a local PocketBase project with the correct binary and folder structure. The article walks through creating a collection, adding a hook that calculates order totals, and verifying it with a test record. Running pbc deploy then publishes the project to a PocketBase Cloud production instance, where the same admin commands work against the hosted app.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OneLifeVibe Uses a Single Hub to Route Users to Separate Single-Purpose Sites

The article describes how the builders of OneLifeVibe chose a hub-and-destinations design, with one front door pointing to standalone websites that each handle a distinct area of everyday life. The author argues that people often struggle to remember where useful tools live, so a single consistent entry point helps them find the right service when a need arises. Each destination keeps its own purpose, community, and data rather than being merged into one large app. The author says the hub is kept intentionally thin, focused on routing, and that clear category names written in users' own terms mattered more than analytics. The piece also notes that the hub must stay fast and current, since a slow or cluttered entrance harms every destination behind it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Scanned PDFs can silently yield empty text, and OCR should be applied selectively

PDFs made from scanned images store no text, so text extraction can return near-empty pages without raising any error. The article explains how to detect this by counting characters extracted from each page and checking whether most pages are blank. Pages that are fully empty need OCR or rejection, while partially empty documents should have only their blank pages OCRed. Rendering at 300 dpi, setting the document language, and recording per-page confidence scores are recommended to improve OCR accuracy. The author also warns that OCR is slow and can misread digits, so it should not be applied to every page by default.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Airline checkout blocks AI booking agents because it cannot verify who they act for

United's stance on AI agents booking flights is better read as a question of identity than of agent competence, the author argues. Booking systems cannot tell a human-delegated agent apart from a scripted bulk booker or fraud traffic, so they block anything that does not resemble a person in a browser. Because every signal the checkout evaluates concerns the client rather than the principal it serves, even a flawless agent is treated like abuse. The author expects scoped, revocable credentials tied to a verified human to become a product requirement, starting with travel and spreading to other sectors. Such credentials would include a verified subject, a key-pair-bound actor, narrow scopes, intent binding, short lifetimes, spend caps, and a revocation mechanism, while the unresolved issue is who bears liability for mistakes.

0 comments Read more at DEV Community