Chunking documents so retrieval works

In a retrieval system, the model never sees your whole knowledge base. It sees a few pieces of it. How you cut documents into pieces, called chunking, quietly decides whether the right passage can be found at all. Many "the AI gave a wrong answer" reports turn out to be chunking problems.

What chunking is for

Two constraints push you to chunk:

• Embedding models and prompts have length limits.
• A single vector for a long document blurs many topics together, so search matches poorly. A vector for one focused passage is sharper.

The goal is chunks that are small enough to be about one thing and large enough to make sense on their own.

Common strategies

A sensible default is structure-aware splitting with a size cap and a modest overlap: respect headings and paragraphs, and only fall back to fixed-size cuts inside very long sections.

def chunk_by_paragraph(text, max_chars=1200, overlap=150):
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for p in paragraphs:
        if len(current) + len(p) + 2 <= max_chars:
            current = f"{current}\n\n{p}" if current else p
        else:
            if current:
                chunks.append(current)
            tail = current[-overlap:] if current else ""
            current = (tail + "\n\n" + p).strip()
    if current:
        chunks.append(current)
    return chunks

The numbers are starting points, not rules. Measure with your own questions.

Keep the context attached

A chunk that says "It is not refundable after 30 days" is useless if the reader cannot tell what it is. Good pipelines attach context to each chunk:

• The title and section headings above it, prepended to the text or stored as metadata.
• The source, page number and date, so answers can cite and users can verify.
• Permissions or audience labels, so retrieval never returns something the asker should not see.

Special content needs special handling

• Tables. Splitting a table by size destroys it. Keep a table whole where you can, or convert rows into short sentences that repeat the column headers.
• Code. Split on functions or classes, not arbitrary lines.
• Lists and procedures. Keep a numbered procedure in one chunk; half a procedure is dangerous.
• PDF extraction noise. Headers, footers and page numbers repeat on every page and pollute chunks. Strip them.

How to choose sizes

There is no universal best size. Smaller chunks give precise matches but may lack context; larger chunks carry context but dilute the match. The practical method:

1. Write 20 to 50 real questions with the passage that answers each.
2. Try two or three chunking settings.
3. For each, measure how often the correct passage appears in the top few results.
4. Choose the setting that wins, and re-check when your documents change type.

Common mistakes

1. One giant chunk per document.
2. Cutting mid-sentence or mid-table with no overlap.
3. Throwing away headings, so chunks lose their meaning.
4. Changing chunking without re-running the evaluation.
5. Ignoring duplicates: the same paragraph appearing in many chunks crowds out other results.

Try it yourself

Take one real document of a few pages. Chunk it three ways (fixed size, by paragraph, by heading). Write five questions whose answers are in it. For each strategy, check by eye whether the right chunk comes back first for each question. Notice which questions fail and why; the failures teach more than the successes.