How text becomes tokens (BPE in plain words)

Language models do not read letters or words. They read tokens: chunks of text, each mapped to a number. The way text is split into tokens shapes cost, speed, context limits and even some odd model behaviours. Knowing how it works turns several puzzling errors into predictable ones.

Why not just use words or characters?

• Words are natural, but there are endless new ones: names, typos, technical terms, other languages. A fixed word list would leave out anything unseen.
• Characters never run out, but sequences become very long, and the model must relearn how letters form words.

Subword tokenisation sits in between. Common words become a single token; rare words are split into pieces the model has seen before. Nothing is truly "unknown", because any text can be built from smaller pieces, ultimately bytes.

Byte-pair encoding, in plain words

BPE is a widely used method for building the vocabulary. The idea is greedy and simple:

1. Start with every character (or byte) as its own token.
2. Count how often each adjacent pair appears in the training text.
3. Merge the most frequent pair into a new single token.
4. Repeat until the vocabulary reaches the size you want.

Frequent pairs such as "t" + "h", then "th" + "e", become tokens; a rare name stays in pieces. The list of merges, in order, is the tokeniser.

from collections import Counter

def most_common_pair(tokens):
    pairs = Counter(zip(tokens, tokens[1:]))
    return pairs.most_common(1)[0][0] if pairs else None

def merge(tokens, pair, new_token):
    out, i = [], 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
            out.append(new_token); i += 2
        else:
            out.append(tokens[i]); i += 1
    return out

text = list("low lower lowest")
for step in range(5):
    pair = most_common_pair(text)
    if not pair: break
    text = merge(text, pair, "".join(pair))
print(text)

Run it and watch "lo", then "low", emerge as tokens.

Using the tokeniser at inference time

To tokenise new text, apply the learned merges in the same order to the new characters. The result is a list of token IDs, which the model turns into vectors (embeddings). After generation, IDs are turned back into text.

What it explains in practice

• Cost and limits are counted in tokens, not words. As a rough rule of thumb, an English word is often one to two tokens, but this varies widely by language and content.
• Languages differ. Text in languages that were less represented when the vocabulary was built usually needs more tokens for the same meaning, so it is costlier and fits less in the context window.
• Spelling and counting errors. A model sees "strawberry" as a few chunks, not ten letters, which is why letter-counting and reversing strings can trip it.
• Numbers. Long numbers may be split into arbitrary pieces, which makes exact arithmetic harder.
• Whitespace and case matter. " the", "the" and "The" can be different tokens. Trailing spaces and odd formatting change tokenisation, and therefore behaviour.
• Code and structured text tokenise differently from prose: indentation, punctuation and identifiers all consume tokens.

Practical habits

1. Count tokens with the model's own tokeniser before sending long input; do not estimate from characters.
2. Budget the context window for the prompt, the retrieved text and the answer together.
3. Keep tokenisation consistent between training and inference: the same tokeniser and version, or results will be nonsense.
4. Be careful with special tokens (start, end, padding). Adding or forgetting them changes outputs.
5. Watch prompt formatting. Small changes in whitespace can shift results.

Common mistakes

1. Assuming one word equals one token.
2. Comparing costs across languages by word count.
3. Mixing tokenisers between a model and its fine-tuned version.
4. Cutting text at a fixed character count and splitting a token in a bad place.

Try it yourself

Use a tokeniser library for any open model. Tokenise the same sentence in English and in another language you know, a long number, a URL, and a line of code. Print the tokens as text pieces. Note which inputs use many small tokens, and connect that to cost and context limits.