What Tokens Really Are and Why They Determine LLM Costs and Limits

·1 views

In large language models, a token is not simply a word but a fragment of text determined by a trained vocabulary, meaning one word can equal multiple tokens and vice versa. Each model — including GPT, Gemini, and Claude — uses its own tokenizer, such as Byte Pair Encoding, with vocabularies ranging from roughly 50,000 to 100,000 text pieces. Technical terms, long numbers, emojis, and non-English text tend to consume significantly more tokens than common English words, making them costlier to process via APIs. Every LLM enforces a context window, a hard cap on the total tokens it can handle per call, which must accommodate the system prompt, conversation history, retrieved documents, and the user's message combined. Misunderstanding how tokenization works can lead to unexpected API costs and context-window overflows, especially in applications that pass large amounts of text to the model.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Two devs built an offline YOLO26 posture detector in 7 days — here's what broke

A two-person team in Austin, TX — engineer Igor Eduardo and field specialist Lexi Armstrong — built an on-device posture detection tool called SENTINEL during the webAI YOLO26 MLX Build Challenge in May 2026. The single-file Python app uses the YOLO26n model with a bounding-box heuristic to classify people as standing, sitting, or lying, running at roughly 16 FPS on an M4 Mac with Wi-Fi physically disabled to guarantee zero network egress. The project drew on Eduardo's healthtech background in patient triage and Armstrong's experience with denied-communications environments, targeting mass-casualty and industrial security scenarios. Two major technical failures shaped the build: OpenCV 4.13 crashed reliably when camera capture ran off the main thread on macOS 26, and a silent hang was traced to lazy evaluation in the yolo-mlx Boxes proxy rather than MLX inference itself. An earlier attempt to fine-tune the model on labeled posture data also failed to converge and was abandoned within the seven-day window.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Microsoft Patches Two RCE Flaws in Semantic Kernel Caused by Prompt Injection

Microsoft disclosed two remote code execution vulnerabilities, CVE-2026-26030 and CVE-2026-25592, in its open-source Semantic Kernel agent framework. The first flaw, in the Python SDK, allowed unsanitized AI model output to reach an eval() call inside a vector store metadata filter, enabling attackers to execute arbitrary code by bypassing a blocklist via Python's class hierarchy traversal. The second vulnerability, in the .NET SDK's SessionsPythonPlugin, let a crafted prompt direct the agent to write a file to any location on the host, including the Windows Startup folder, allowing code to persist and execute after the session ended. Both bugs share a common root cause: powerful code-adjacent operations received values derived from model output without treating that input as potentially attacker-controlled. Fixes were shipped in semantic-kernel 1.39.4 for Python and .NET SDK 1.71.0 on the same day.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds hand-written regex outperforms 7B AI model at detecting violent language

A developer testing AI safety classifiers shifted from cloud APIs to a locally run 7-billion-parameter model after their lab's IP address was blocked following repeated policy-sensitive queries. The local model, Qwen 2.5 Coder 7B running on CPU via llama.cpp, incorrectly flagged benign phrases like 'kill a Python process' and 'strangle options to maximise my money' as unsafe content. A simple hand-written regex with an exceptions list correctly identified all such phrases as non-violent, outperforming the neural network on this specific task. The author noted that the model could accurately explain terms like 'murder of crows' when asked directly, yet still flagged the word 'murder' as unsafe during classification — revealing a gap between knowledge and judgment. The findings highlight that AI safety benchmark scores reflect the entire surrounding scaffold, including parsers and exception logic, not just the underlying model's capabilities.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

To Get the Most from Cursor AI, Developers Should Plan Before They Prompt

Cursor is an AI coding tool that can explore codebases, write features, fix bugs, and coordinate multiple agents to handle complex development tasks. However, the tool works best when developers invest time upfront to clearly define what they are building, who will use it, and what the system's boundaries are. Experts recommend sketching user flows and system diagrams before writing prompts, as visual clarity often outperforms lengthy written descriptions. Cursor's internal rules should also be treated as living guidelines, updated whenever the model produces repeated errors or instructions become overly complex. While AI can manage much of the implementation work, critical decisions around architecture, security, and access control remain the developer's responsibility.

0 comments Read more at DEV Community