Same 192 Characters Yield 111 to 172 Tokens Across Three Tokenizers

·1 views

A developer tested one 192-character Chinese code-review request across three tokenizers and got 111 tokens in DeepSeek V4, 133 in o200k_base, and 172 in cl100k_base, a 55% spread. The common rule of four characters per token predicted only 48, so it underestimated the count badly. Byte-level BPE explains the gap: pre-tokenization splits text into chunks that merges cannot cross, and each tokenizer's vocabulary size and training data differ. English text barely varies across the tokenizers, while Chinese and Japanese characters diverge sharply. Code adds its own cost through punctuation, indentation and whitespace. Since Claude and Gemini tokenizers are unpublished, exact counts for them require provider endpoints, and local tokenizers measure only one string, not a full request.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Feed Claude Code axe-core's Violation List, Not a Vague Accessibility Request

A February 2026 WebAIM scan of a million home pages found that 82.7% used ARIA, yet pages with ARIA averaged more detected errors than those without. The author notes that ARIA pages were also more complex, so the link is not established as causal. Asking a coding agent to simply make a page accessible tends to produce large, hard-to-verify diffs, such as blanket aria-label additions that can override existing semantics. The recommended approach is to run axe-core to list violations by rule ID and CSS selector, and to use that list as the agent's task. A Playwright test running the same rules then serves as verification, and it should be written by the developer before the agent begins.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guide Compares Eight AI Knowledge Tools for Air-Gapped Teams

An outdated procedure can keep circulating on air-gapped networks when the current answer is buried in wiki pages or files. A useful knowledge tool must surface trusted material without sending data beyond the security boundary. The guide recommends first deciding whether the source material is maintained wiki pages, a document collection, or both. It then checks whether the application, models, identity setup, updates, and supporting services can run fully offline. Eight options are assessed on shared retrieval, isolated deployment fit, knowledge structure, and offline operating effort.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

ScriptMasterLabs shares lessons from listing 24 MCP servers on icon handling

ScriptMasterLabs runs 24 MCP servers offering 182 tools, listed across the Official MCP Registry, Glama, Smithery, the CDP Bazaar, and its own hub. The company notes that most directories never run a server and instead display whatever icon and metadata are in the published source, so a faulty icon can fail silently across listings. It advises downloading official brand assets from the modelcontextprotocol GitHub organization and committing them locally rather than scraping or hotlinking. The guide recommends treating SVG as the master file, exporting fixed-size PNGs at 16, 32, 128, and 512 pixels, and testing small sizes against light and dark backgrounds. It also recommends using light and dark icon variants selected by media query, so the mark stays visible on any registry theme.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Mercury 2.5 Ranks Near Top for LLM Output Speed, Raising Agent and RAG Design Questions

Artificial Analysis lists Inception's Mercury 2.5 at roughly 1,178 output tokens per second, placing it second among 146 models on speed. The page does not report an Intelligence Index score, so the model's quality remains unverified, and the speed figure varies by provider and measurement snapshot. Most production LLMs generate one token at a time, while Inception is known for diffusion-style models that refine many positions in parallel, reducing sequential steps. The author argues that faster decoding shifts engineering priorities, making extra reasoning, candidate voting, and LLM-based reranking more practical in agent loops and RAG pipelines. The piece also cautions that generation is often not the main latency bottleneck, so teams should profile end-to-end task time before changing models.

0 comments Read more at DEV Community