How Tokenization Converts Raw Text Into Numbers for AI Models

·1 views

Tokenization is the foundational process by which raw text is split into discrete units called tokens, each assigned a unique numerical ID so that neural networks — which can only perform mathematical operations — can process language. Early tokenizers treated whole words as single tokens, but this approach failed whenever a word was absent from the model's fixed vocabulary, replacing unknown words with a generic token and losing all meaningful information. Modern NLP systems use sub-word tokenization, which breaks rare or unseen words into smaller recognizable pieces, allowing virtually any input to be represented. The most widely adopted method, Byte Pair Encoding (BPE), builds its vocabulary by starting with individual characters and repeatedly merging the most frequently occurring adjacent pairs until a target vocabulary size is reached. Models like GPT-2 rely on this approach, operating with a vocabulary of roughly 50,000 tokens to balance coverage and efficiency.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

DeepMind Splits Robot Brain and Body With Three-Model Gemini Robotics 2 Suite

Google DeepMind released Gemini Robotics 2 on July 28, 2026, followed by Gemini Robotics ER 2 on July 30, introducing a modular architecture that separates high-level reasoning from real-time motor control. Rather than a single end-to-end network, the suite uses three specialized models: an embodied reasoning planner, a whole-body vision-language-action controller, and an on-device adaptation model. The reasoning model, ER 2, interprets scenes and natural-language instructions to break multi-minute tasks into sub-goals, while the core VLA model handles physical movement across humanoid bodies and robotic arms at high frequency. A third on-device model runs locally on robot hardware and can adapt to a new robot's mechanics using fewer than 200 demonstration examples and just a few hours of data. The architecture addresses a longstanding robotics trade-off where large context windows needed for reasoning conflict with the low-latency demands of precise physical control.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Shelly Plus 1PM and 2PM Enable Local Smart Switching with Energy Metering

A home automation enthusiast installed Shelly Plus 1PM and 2PM smart relays behind existing wall switches to gain both local circuit control and per-channel energy metering without relying on the cloud. The Gen2 devices communicate via a local RPC API over HTTP and WebSocket, integrating natively with Home Assistant through mDNS discovery and requiring no third-party firmware. By disabling cloud and Bluetooth in the Shelly web UI and assigning fixed IP addresses, the setup remains fully operational even without an internet connection. Physical switches are configured in 'detached' input mode, turning them into event triggers that Home Assistant interprets, allowing a single button to perform multiple actions via automations. Energy data from each channel feeds directly into Home Assistant's Energy Dashboard, eliminating the need for separate metering hardware.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study Finds High Dimensionality, Not Formatting, Is Why LLMs Fail at Tabular Prediction

A new research paper investigates why large language models underperform on tabular prediction tasks compared to classical machine learning methods like gradient-boosted trees. Testing nine methods across 31 benchmark datasets, the authors found that LLM accuracy degrades as the number of input features grows, while traditional baselines remain stable or improve. The study ruled out several common explanations — including poor CSV serialization, bad numeric tokenization, and prompt length — as primary causes of this performance gap. Instead, the researchers identified input dimensionality as the central failure mode, arguing that LLMs struggle to detect useful patterns when many weakly related features accumulate. The findings suggest that reformatting data or tweaking prompts is unlikely to resolve the core limitation in high-dimensional tabular learning scenarios.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Your Phone Probably Isn't Listening — But It Doesn't Need To

Many users report seeing ads for products they only spoke about, never searched, leading to suspicions that their phones are secretly listening. However, experts suggest that microphone surveillance is largely unnecessary given how much behavioral data users already share digitally. Search histories, app usage, location patterns, and browsing habits collectively build a detailed profile that AI can use to predict interests with striking accuracy. The real privacy concern, therefore, lies not in audio eavesdropping but in the vast trail of digital behavior users unknowingly leave behind. Understanding how data profiling works demystifies the phenomenon and shifts the conversation from conspiracy to informed digital awareness.

0 comments Read more at DEV Community