Tokens, Vectors and Embeddings Explained Simply for Developers

A technical explainer published on DEV Community breaks down three foundational AI concepts—tokens, vectors, and embeddings—that frequently appear in discussions about large language models and retrieval-augmented generation. Before any AI model can process text, it must first convert language into numbers through a pipeline that runs from raw text to tokens to embeddings and finally to model output. Tokens are small chunks of text produced by a tokenizer, and they do not always correspond to complete words; modern LLMs typically use subword tokenization methods such as BPE or WordPiece to handle diverse languages, symbols, and new words efficiently. While a vector is simply a list of numbers, an embedding is a vector specifically learned by an AI model to encode meaning, meaning semantically related words like 'cat' and 'dog' end up with numerically closer embeddings than unrelated words like 'car'. These embeddings power practical applications such as semantic search, document recommendations, and RAG systems, where a query can surface relevant content even when the exact wording does not match.