OpenAI Token Costs Vary Sharply by Language: Czech 2x, Chinese Near English Parity

·1 views

A solo developer in Korea tested the same customer-support prompt across 27 languages using OpenAI's tokenizers and found significant cost differences between languages. Simplified Chinese tokenized at nearly the same cost as English (1.03x), while Czech proved the most expensive at 2.0x English token count. The newer o200k_base tokenizer, used since GPT-4o, dramatically reduced costs for Indic, Arabic, and Thai scripts compared to the older cl100k_base encoder — Bengali dropped from 6.1x to 1.68x English tokens. Heavily inflected Latin and Cyrillic-script languages like Polish and Ukrainian now rank among the costlier options due to diacritics splitting into multiple tokens. The developer recommends writing system prompts in English and reserving non-English text only for user-facing content to reduce compounding token costs at scale.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Anthropic Launches Claude Sonnet 5.5 on GitHub Copilot with Lower Task Costs

Anthropic released Claude Sonnet 5.5 on September 28, 2026, positioning it as a faster, more efficient complement to the higher-tier Opus 5.5 model. GitHub simultaneously made it generally available within Copilot, giving developers direct IDE access on launch day. Despite carrying the same per-token list price as its predecessor Sonnet 5, the model is claimed to complete tasks in fewer tokens and steps, reducing actual invoice costs by up to 30%. Anthropic recommends Sonnet 5.5 for well-scoped everyday work such as bug fixes, feature development, and document drafting, while reserving Opus 5.5 for complex, open-ended tasks requiring sustained judgment. The model also supports a native 1-million-token context window and introduces an adjustable effort dial that influences thinking depth, speed, and overall task cost.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds BaZi Birth Chart Engine Using State Machines and Graph Theory

A software developer has spent the past year building a computational engine for BaZi, a traditional Chinese astrology system, by modeling it as a directed graph of five elemental nodes with defined interaction rules. The developer treats BaZi not as a mystical practice but as an ancient modeling system, mapping classical rules onto graph traversal logic and representing personality archetypes as weighted activation dimensions. Significant engineering effort went into handling edge cases such as true solar time at the birthplace, daylight saving transitions, half-hour UTC offsets, and disagreements between schools on day boundaries. Key boundary issues include the fact that the Chinese solar year begins with Lichun in early February rather than Lunar New Year, and that a 30-minute clock error near a solar term transition can shift an entire birth pillar. The developer has published a full derivation on the AcrosStar blog and notes the system is intended as a cultural and cognitive framework, not medical, legal, or financial guidance.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PDF Perfect compresses files entirely in-browser using pdf.js and Web Workers

Developer Arun built PDF Perfect, a browser-based PDF compressor that processes files entirely on the user's device without uploading them to any server. The tool uses Mozilla's pdf.js to render each PDF page onto an OffscreenCanvas, then encodes it as a JPEG using the browser's native canvas API before assembling a new PDF with pdf-lib. All processing runs inside a Web Worker to avoid blocking the main thread, with results transferred back efficiently using ArrayBuffer transfers to prevent memory duplication. Three compression presets pair JPEG quality with render scale, while a Target File Size mode automatically tries progressively smaller settings until the output meets a user-specified size limit. The approach works best on scan-heavy or image-rich PDFs but involves a trade-off for text documents, which lose sharpness due to rasterisation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Tournament Judge for Tabletop Games to Eliminate Rules Disputes

A developer has submitted TableTop Arbiter to the Sanity Challenge, an AI-powered tournament judge designed for competitive tabletop games such as Magic: The Gathering. The tool addresses a common problem in high-stakes play where rules disputes stall matches, and standard AI models give unreliable answers by confusing outdated rulebooks with current official errata. Unlike typical large language models, TableTop Arbiter grounds its responses in a structured content database using GROQ graph queries to resolve rule contradictions deterministically. The system streams its live reasoning process directly in the user interface, allowing players and judges to verify each decision in real time. The project is live on Vercel and its source code is publicly available on GitHub.

0 comments Read more at DEV Community