Word Embeddings Explained: How Machines Learn Word Meaning Through Context

Word embeddings represent words as numerical vectors, capturing relationships between words rather than merely counting their occurrences in text. Introduced in 2013 by Mikolov and colleagues, Word2Vec learns these vectors by training a model to predict words from their surrounding context across millions of sentences. A key insight is that words appearing in similar contexts end up with similar vectors, enabling mathematical relationships between words to emerge — though this reflects usage patterns rather than true semantic meaning. However, Word2Vec has notable limitations: it assigns a single fixed vector per word regardless of context, meaning words like 'bank' cannot be distinguished by meaning, a problem later addressed by contextual models such as BERT. Embedding quality also depends heavily on training data, with biased or limited datasets producing vectors that can reflect and reinforce real-world biases.