Chunking & Ingestion — The Unit You Index

Topics We'll Cover

• Chunking Philosophy — The Unit You Index Is the Unit You Retrieve
• Strategies — Fixed, Semantic, Structural, Contextual (Anthropic)
• Recursive Character Splitting and Where It Breaks
• Layout-Aware Parsing — Unstructured, MinerU, LlamaParse, Docling
• ColPali & Vision-Based Document Parsing
• Metadata Enrichment — Sections, Authors, Dates, Citations
• Deduplication — MinHash, SimHash, Near-Duplicate Detection
• Pipeline Shape — Incremental Ingestion, Embedding Refresh
• What to Take Away

Chunking Philosophy — The Unit You Index Is the Unit You Retrieve

Chunking is the most consequential decision in a RAG pipeline and the most overlooked. The chunk is the atomic unit of retrieval: what you index is what you return to the LLM. A poor chunk boundary splits an answer across two chunks; a poor chunk size under-represents context or wastes token budget.

Strategies — Fixed, Semantic, Structural, Contextual (Anthropic)

Four distinct chunking philosophies, each suited to different document types and retrieval objectives.

Recursive Character Splitting and Where It Breaks

RecursiveCharacterTextSplitter (LangChain) is the most widely used open-source chunker. It tries separators in order (["\n\n", "\n", ". ", " ", ""]) and recursively subdivides sections that exceed the target chunk size.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,           # in characters, not tokens!
    chunk_overlap=64,
    length_function=len,
    # For token-accurate splitting, pass tiktoken counter:
    # length_function=lambda s: len(enc.encode(s))
    separators=["\n\n", "\n", ". ", " ", ""],
    is_separator_regex=False,
)

chunks = splitter.split_text(document_text)

Where it fails:

• Tables in Markdown: splits mid-row, destroying the tabular structure. The cell content is then incoherent without the header row.
• Code blocks: splits mid-function. The resulting chunk has a dangling def with no body.
• PDF extraction artefacts: extracted text from two-column PDFs interleaves columns line by line. No splitter recovers from this without layout-aware parsing upstream.
• Footnotes and asides: footnote text is interleaved with body text after naive PDF extraction; the chunk conflates footnote and main text.

Layout-Aware Parsing — Unstructured, MinerU, LlamaParse, Docling

For PDFs, Word documents, HTML, and other richly structured formats, text extraction must preserve layout semantics. Four tools dominate this space.

ColPali & Vision-Based Document Parsing

ColPali (Faysse et al. 2024) takes a radically different approach: instead of extracting text from PDFs, it embeds each page image directly using a PaliGemma-based vision encoder, then indexes per-patch token embeddings using ColBERT-style late interaction.

Why this matters: ColPali retrieves pages containing charts, complex tables, and diagrams that would be mangled by text extraction. On the DocVQA and SlideVQA benchmarks, ColPali achieves 81.3% NDCG@5 vs 60.1% for the best text-extraction + BGE-M3 pipeline.

from byaldi import RAGMultiModalModel

model = RAGMultiModalModel.from_pretrained("vidore/colpali-v1.3")

# Index a folder of PDFs (converts to page images internally)
model.index(
    input_path="./pdfs/",
    index_name="my_docs",
    store_collection_with_index=True,
    overwrite=True,
)

# Retrieve: returns ranked list of (page_image, score, page_num)
results = model.search("quarterly revenue breakdown by region", k=5)
# Pass page images directly to a VLM (e.g. GPT-4o, Claude 3)

Metadata Enrichment — Sections, Authors, Dates, Citations

Metadata is the invisible infrastructure of effective filtered retrieval. Well-structured metadata turns a slow post-filter into a fast pre-filter and enables faceted search that embedding similarity cannot provide.

Deduplication — MinHash, SimHash, Near-Duplicate Detection

Duplicate and near-duplicate chunks harm retrieval quality: they consume index space, inflate the apparent corpus size, and cause the reranker to return the same information multiple times. Web-scraped and CMS-exported corpora routinely contain 5–30% near-duplicate content.

from datasketch import MinHash, MinHashLSH

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for idx, chunk in enumerate(chunks):
    m = MinHash(num_perm=128)
    for word in chunk.split():
        m.update(word.encode("utf-8"))
    duplicates = lsh.query(m)
    if not duplicates:
        lsh.insert(str(idx), m)   # unique chunk
    # else: skip chunk as near-duplicate

Pipeline Shape — Incremental Ingestion, Embedding Refresh

Most production RAG systems ingest continuously (new documents arrive daily or hourly) and occasionally need to re-embed (new embedding model, chunking strategy change). These are separate operational concerns.

What to Take Away

• The chunk is the unit of retrieval and the unit of context. They need not be the same — use parent–child chunking to retrieve at small granularity and return at large.
• Fixed-size splitting fails on structured documents (tables, code, PDFs). Use structural or semantic chunking for any non-homogeneous corpus.
• Contextual chunking (Anthropic pattern) adds 35% relative recall improvement by prepending a per-chunk context summary at index time. Cost: one LLM call per chunk.
• Docling is the best open-source layout-aware parser as of 2025 for PDFs and DOCX. LlamaParse is the best cloud option for complex layouts.
• ColPali is the right choice for scanned documents, slide decks, and tables that text extraction destroys. It embeds page images directly; storage cost is ~500 KB per page uncompressed.
• Always extract structural metadata and build payload indexes on filter fields. Semantic metadata (entities, topics) should be stored separately and not concatenated into the chunk.
• Incremental ingestion: track document hashes; upsert changed chunks; use shadow index + alias cutover for embedding model migrations.