An embedding is a dense vector representing a token, a word, a sentence, or a document. The transformation from discrete tokens to continuous vectors is where meaning enters the model. Every downstream operation — attention, retrieval, similarity — operates on embeddings.
- 01State what an embedding represents.
- 02Explain why continuous vectors enable gradient-based learning.
- 03Distinguish token embeddings from sentence embeddings.
The transformation
A vocabulary of 100,000 tokens maps into a 768- or 4096-dimensional space. Nearby vectors correspond to related meanings — king − man + woman ≈ queen was the canonical demonstration (Word2Vec, 2013).
Why it enables learning
Discrete tokens have no gradient. Continuous vectors do. Embeddings make the entire pipeline differentiable end-to-end, which is what lets backpropagation train the model.
Levels of embedding
Token embeddings live at the model's input. Sentence embeddings (from CLS tokens or pooling) live at the output and power retrieval. Both are learned; both are geometrically meaningful.
- Embeddings turn discrete symbols into continuous, differentiable vectors.
- Semantic similarity becomes geometric closeness.