What is an embedding? Meaning as numbers

Computers are good at arithmetic and poor at meaning. An embedding is the trick that bridges the two: it turns a word, sentence, image or document into a list of numbers so that things with similar meaning end up with similar numbers. Almost every modern search, recommendation and retrieval system rests on this idea.

From words to coordinates

Picture a map. Cities that are close in the real world sit close together on the page. An embedding does the same for meaning: each item becomes a point in a space, usually with hundreds or thousands of dimensions rather than two. "Invoice" and "bill" land near each other; "invoice" and "volcano" land far apart.

You never choose those numbers by hand. A trained embedding model produces them. It has learned, from vast amounts of text (or images), which contexts go together, and the position it gives an item reflects that.

Measuring "similar"

Once items are points, similarity is geometry. The most common measure is cosine similarity: the angle between two vectors, ignoring their length. Close to 1 means pointing the same way (similar meaning); near 0 means unrelated.

import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# vectors would come from an embedding model
print(cosine(vec("refund policy"), vec("how do I get my money back")))   # high
print(cosine(vec("refund policy"), vec("office opening hours")))          # low

This is why embeddings power semantic search: the query "how do I get my money back" can find a page titled "Refund policy" even though they share no words.

Where embeddings get used

• Search and retrieval. Embed every document once, embed each query at request time, and return the nearest documents. This is the retrieval half of retrieval-augmented generation.
• Recommendations. Items and users in the same space; recommend what is near.
• Clustering and deduplication. Group similar tickets, find near-duplicate records.
• Classification with little data. Embed the text and train a small classifier on top.

Vector databases in one paragraph

Comparing a query against millions of vectors one by one is slow. A vector database (or a vector index inside a normal database) stores the vectors and uses approximate nearest-neighbour structures to find close ones quickly. You give up a tiny amount of exactness for a large gain in speed, and it also stores metadata such as source, date and permissions so you can filter results.

Choosing and using an embedding model

• Match the language and domain. A model trained mostly on English web text may struggle with legal Hindi or medical terminology. Test on your own data.
• Stay consistent. Query and documents must be embedded by the same model and version. Changing the model means re-embedding everything.
• Mind the length limit. Models accept a maximum input size. Long documents are split into chunks first, which is its own design decision.
• Dimensions cost money. More dimensions can capture more nuance, and they also increase storage and search time.

Common mistakes

1. Embedding a whole long document as one vector, so its meaning is averaged into mush.
2. Mixing vectors from two different models in one index.
3. Trusting similarity scores as absolute: a score of 0.78 means different things with different models. Compare relative ranking, and calibrate a cut-off on real examples.
4. Skipping evaluation. Nearest neighbours look sensible even when they are wrong. Build a small set of real questions with known right answers and measure how often the correct item is retrieved.

Try it yourself

Take twenty short sentences on three topics. Embed them with any embedding model, compute all pairwise cosine similarities, and check whether the nearest neighbour of each sentence comes from the same topic. Then reduce the vectors to two dimensions (for example with PCA) and plot them. Seeing topics form clusters makes the abstract idea concrete.