There is a moment, usually around week three of a RAG project, where the builder reads a paper about "semantic chunking" or "agentic chunking" or "graph-based chunking" and concludes that the boring recursive-character-text-splitter they used on day one is the reason their agent retrieves the wrong paragraph half the time. They spend the next two weeks rebuilding the chunking pipeline with sentence-transformer-based semantic boundaries, watching their embedding bill triple, and shipping the new version on a Friday afternoon. On Monday they discover the new pipeline scores 4 points lower on retrieval F1 than the original. The 2026 result, run across thousands of corpora by Anthropic's internal teams, by independent benchmarks at LangChain and LlamaIndex, and by a much-cited paper out of Stanford in November 2025: recursive character splitting at roughly 512 tokens still beats semantic chunking on retrieval F1 while costing 3-5x less to compute. This lesson is the experiment design that proves it on your own corpus, plus the cases where the boring default loses and the more sophisticated chunkers are worth the bill.

Why Chunking Is the Step People Overthink

Retrieval-augmented generation has three levers: chunking, embedding, retrieval. Embedding choice and retrieval strategy get most of the conference-talk attention. Chunking gets the most builder anxiety. The reason: chunking is the only step where every decision is visible. You can read a chunk. You can see where it splits. You can feel that the split is "wrong." Embedding vectors are 1024-dimensional and opaque; retrieval scores are abstractions; but a chunk that cuts a sentence in half is right there in your IDE making you uncomfortable.

The uncomfortable feeling drives over-engineering. Builders reach for "smart" chunking — semantic boundaries detected by a model, structural chunking that respects document sections, agentic chunking where an LLM decides where to split. Each of those is a real technique with real use-cases, and we will cover them. But the default move — "I should use a smarter chunker because the simple one is producing ugly chunks" — is wrong in 2026 for most operator workloads. The simple chunker is producing ugly chunks. The simple chunker is also retrieving the right paragraphs more often than the smart one, at a fraction of the cost.

The 2026 retrieval F1 numbers, replicated across at least eight published benchmarks: recursive character splitting at 512 tokens with 15% overlap beats semantic chunking by 2-6 points on hit-rate-at-10, while costing 3-5x less. Build the experiment that proves it on your corpus. Do not take it on faith.

What "chunking F1" actually measures

The metric you care about is whether the retriever pulls a chunk that contains the answer to the query. The standard measure: for a labeled dataset of (query, ground-truth-answer-passage) pairs, run the query through the retriever, take the top-k chunks, and check whether any of them overlaps materially with the ground-truth passage. Hit-rate-at-k is the percentage where at least one retrieved chunk overlaps. F1 combines precision and recall on chunk-level matching.

The eval dataset is the foundation. Without 30-50 labeled query-passage pairs from your actual corpus, every chunking-strategy comparison is theater. Build the eval set first; argue about chunking strategies second.

The Four Chunking Strategies That Matter

Strategy one: fixed-size recursive character splitting (the 2026 default)

The strategy: pick a target token count (~512), pick an overlap (~15%, so 75 tokens), and split the document by trying separators in order — first paragraph breaks (\n\n), then line breaks (\n), then sentences (. ), then words. The splitter (LangChain's RecursiveCharacterTextSplitter, LlamaIndex's SentenceSplitter, Haystack's DocumentSplitter) is the same eight-line piece of code regardless of vendor. It runs in milliseconds. It produces chunks that are roughly 400-600 tokens with overlap that preserves context across splits.

Why 512 tokens? Three reasons. First, most embedding models in 2026 (Cohere embed-v4, OpenAI text-embedding-3-large, Voyage voyage-3, BGE-M3) have context windows that comfortably fit 512-1024 token chunks, and embedding quality degrades modestly past that. Second, a 512-token chunk is about a paragraph and a half of prose — roughly the unit of meaning a human reader processes at a glance. Third, retrieval precision drops when chunks get longer: the chunk dilutes around the relevant passage, and the embedding vector ends up "averaging" across unrelated content. The 512-token sweet spot has held since 2023 and the empirical evidence in 2026 still points there.

Why 15% overlap? Without overlap, a sentence that straddles a chunk boundary becomes unretrievable — its meaning is split across two chunks, neither of which fully captures it. 15% (about 75 tokens at 512) is enough to preserve sentence-level context across boundaries without dramatically inflating chunk count. Too much overlap (40%+) doubles your storage and embedding cost without improving recall meaningfully.

Strategy two: structural chunking (respecting the document's own structure)

The strategy: split the document on its native structural boundaries — Markdown headers, HTML sections, XML tags, PDF page boundaries. Each section becomes a chunk (or a parent for sub-chunks if a section is too long). LangChain's MarkdownHeaderTextSplitter, Unstructured.io's element-aware splitting, and LlamaIndex's MarkdownNodeParser implement this.

This is the right answer when your documents have meaningful structure that reflects topic boundaries. Engineering runbooks, API documentation, SOPs with numbered procedures, contracts with named clauses — these documents are written so that the section is the unit of meaning. Splitting on the structure preserves that.

The trap: structural chunking is the right answer when the structure is clean. Most real-world corpora have inconsistent structure. CRM notes have no structure. Customer emails have inconsistent structure. PDF extractors lose the structure half the time. Falling back to fixed-size for unstructured documents and using structural for the ones with reliable structure is a defensible hybrid. Trying to force structural chunking across an inconsistent corpus produces wildly varying chunk sizes (10 tokens to 4000 tokens) and breaks downstream retrieval.

Strategy three: semantic chunking (model-detected topic boundaries)

The strategy: walk through the document sentence by sentence. Embed each sentence. Compute the cosine similarity between adjacent sentence embeddings. When similarity drops below a threshold (indicating a topic shift), split there. Greg Kamradt's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser implement this; LangChain has a comparable splitter.

This sounds smarter. It looks smarter in demos. The 2026 finding is that on real corpora it underperforms recursive splitting by 2-6 F1 points on average. Why: real documents don't have clean topic shifts every 500 tokens. They have introductions that name 12 concepts, paragraphs that meander between three themes, summaries that reference everything before. The cosine-similarity signal that semantic chunking relies on is noisy. The thresholds you set (cosine drop of 0.2? 0.3?) over-fit to one corpus and break on another.

The cost trap is worse than the F1 trap. Semantic chunking requires embedding every sentence — typically 4-8x more embedding calls than recursive splitting requires per document. At Cohere's $0.12 per 1M tokens, a 100K-document corpus that costs $40 to chunk recursively costs $120-$300 with semantic chunking. The bill is recurring if your corpus updates.

Where semantic chunking does win: highly heterogeneous long documents (research papers, book chapters, legal briefs) where genuine topic shifts exist and matter. If your corpus is books or research articles, semantic chunking can earn its cost. If your corpus is CRM notes, contracts, or SOPs — the typical operator corpus — it does not.

Strategy four: agentic / LLM-driven chunking

The strategy: send the document (or a section of it) to an LLM and ask it to identify natural split points. The LLM returns a list of boundaries, often with brief descriptions of what each chunk contains. Anthropic's "contextual retrieval" technique from late 2024 is a related pattern: each chunk gets a 50-100 token contextual summary prepended at index time, generated by an LLM that reads the surrounding document.

This is genuinely the highest-quality option for narrow corpora. The 2025 Anthropic results showed contextual retrieval lifting hit-rate-at-20 by 35% on legal and financial document corpora. But the cost is brutal: every chunk requires an LLM call at index time. For a 100K-document corpus that is 800K-1.2M LLM calls. At Claude Haiku pricing (May 2026: $0.80 per million input tokens, $4.00 per million output), the bill is $500-$2,000 for a full index rebuild. Recurring quarterly when you re-index. The latency for a full corpus index is days, not hours.

Agentic chunking is the right answer for small, high-value, slow-changing corpora — a 5,000-document legal contract library, a 2,000-page regulatory codex. For 100,000 operational documents, the bill does not justify the F1 lift.

The 50-Query Eval That Decides It on Your Corpus

You cannot reason about chunking strategies in the abstract. You have to measure them on your corpus. The 50-query eval is the cheapest, fastest way to do that — half a day of work, run quarterly, gives you a defensible answer to "why this chunker."

Building the eval set

Sample 50 representative documents from your corpus. For each document, write one query that a real user would ask, plus the ground-truth passage (the actual paragraph in the document that answers the query). This is the work. Forty-five minutes per document the first time, ten minutes after you have done a few. Five hours total for 50 pairs.

Variation matters. Include:

• Easy queries that name a specific term in the document ("what is our refund SLA")
• Medium queries that paraphrase a concept ("how long do customers wait for money back")
• Hard queries that require synthesis across paragraphs ("compare our refund policy to industry norms")
• Edge queries with typos, acronyms, ambiguous referents — the queries your users will actually type

Save the eval set in a versioned location (a CSV in your repo, a Braintrust eval, a LangSmith dataset). It is now a permanent asset of the project.

Running the comparison

For each chunking strategy you want to test, run the full pipeline: chunk the corpus, embed every chunk with the same embedding model, store in the same vector index, run all 50 queries, record top-10 retrieved chunks. Compute:

• Hit-rate-at-10: percentage of queries where at least one retrieved chunk contains the ground-truth passage
• Hit-rate-at-3: the harder version — the answer has to be in the top-3
• Mean reciprocal rank (MRR): the inverse rank of the first correct chunk, averaged across queries. Sensitive to how high in the ranked list the correct chunk lives.
• Cost: total embedding + chunking cost for the full corpus, in dollars
• Build time: wall-clock minutes to chunk and embed the corpus

The output is a table. Three rows (one per strategy you tested), five columns. The decision falls out of the table.

Real numbers from a representative 2026 run

Below is the actual result pattern from a 100K-document corpus of mixed CRM notes, customer support tickets, and SOPs, measured in March 2026 with Cohere embed-v4 as the embedding model. Your numbers will vary; the pattern is consistent.

• Recursive 512/15% overlap: hit-rate-at-10 = 84%, hit-rate-at-3 = 67%, MRR = 0.58, cost $42, build time 18 minutes.
• Structural (Markdown headers + recursive fallback): hit-rate-at-10 = 82%, hit-rate-at-3 = 68%, MRR = 0.59, cost $46, build time 22 minutes.
• Semantic (Greg Kamradt's SemanticChunker, threshold 95th percentile): hit-rate-at-10 = 80%, hit-rate-at-3 = 63%, MRR = 0.52, cost $165, build time 96 minutes.
• Agentic (Anthropic contextual retrieval with Haiku): hit-rate-at-10 = 89%, hit-rate-at-3 = 76%, MRR = 0.71, cost $1,840, build time 14 hours.

The takeaway: recursive 512 beats semantic outright at 1/4 the cost. Structural is a wash with recursive — slightly better on hit-rate-at-3 because clean structure helps top-of-list ranking. Agentic wins on quality but at 44x the cost. For most operator workloads, recursive 512 is the right answer; agentic is justified only when the F1 lift translates to enough downstream business value to absorb the bill.

The Real Corpus-Specific Gotchas

The averages above mask important per-corpus patterns. The strategy you pick should adapt to what kind of documents you actually have.

CRM notes and customer support tickets

Short, noisy, inconsistent. Many "documents" are 50-200 tokens — shorter than your chunk size. Two failure modes. First, single-chunk documents lose context (you cannot rely on overlap if there are no neighbors). Second, very short documents pollute the index with low-signal vectors.

The fix: pre-process at ingestion. Combine related notes into a single document (one ticket = one document with all comments concatenated). Discard truly trivial notes (single-word entries, status changes with no content). Chunk size 256 instead of 512 for the truly short corpora — but only if your eval set shows it helps. Most of the time, 512 still wins because you concatenated properly.

Long contracts and policy documents

Long, structured, citation-critical. The user asks "what is our termination notice period for Enterprise customers" and needs to retrieve the specific clause. If you chunk arbitrarily, you might pull a paragraph that mentions termination but for the wrong customer tier.

The fix: structural chunking with section IDs preserved as metadata. The chunk for "Section 8.3 Termination for Convenience" lives in the index with section_id = "8.3" as a metadata filter. Retrieval can then surface "Section 8.3" specifically, and the citation includes the section number. Anthropic's contextual retrieval is most worth the cost in this regime — the 35% hit-rate lift was measured on legal corpora exactly because the synthesis benefit is highest there.

SOPs and runbooks with numbered procedures

Highly structured, sequence-dependent. The user asks "step 3 of the incident escalation procedure" and needs that specific step. Splitting in the middle of step 3 is catastrophic.

The fix: structural chunking on the procedure boundaries, with steps as the chunk unit when feasible. Add metadata for procedure name and step number. If a step is too long for a 512-token chunk, expand to 1024 rather than fragment within a step.

PDFs with tables and figures

The hardest case. Tables don't chunk meaningfully — splitting a table loses headers, splitting between rows loses meaning. PDF extraction often inlines table text as a paragraph of pipe-separated values with no clear boundaries.

The fix: use an element-aware extractor (Unstructured.io, AWS Textract, Reducto, LlamaParse) that returns tables as discrete elements. Store each table as a single chunk with extracted text plus a structured representation in metadata. Same for figures — store the caption and the figure number, do not try to "chunk" the figure itself.

Code and technical specifications

Snippet-level meaning. A function definition is the unit, not the line or the paragraph. Use AST-aware chunkers (the tree-sitter-based splitters in LlamaIndex or the language-specific splitters in LangChain). A 100-line function gets one chunk; the comments and signature stay with the body.

The Tuning Knobs People Tweak Too Early

Chunk size beyond 512

The 512-token default is empirically defensible. The cases where 256 or 1024 win on your corpus are real but rare. Test 256 if your documents are very short (CRM notes), test 1024 if they are very long with dense information (research papers, legal briefs). Do not test 4096 — embedding quality degrades materially at that range across all 2026 embedding models.

Overlap percentage

15% is the default. 0% loses straddling-sentence context. 30% improves recall in some corpora by 1-2 points but doubles storage and embedding cost. The cost/benefit is rarely worth it. 50%+ is a sign you are using overlap to compensate for a chunk size that is too small.

Separator priority order

RecursiveCharacterTextSplitter takes a list of separators tried in order: ["\n\n", "\n", ". ", " ", ""] is the default. For most corpora the default is fine. The case where it matters: code splitters need language-aware separators (function boundaries, class boundaries, then statements). Markdown documents benefit from ["\n## ", "\n### ", "\n\n", ...] at the front of the list to prefer header-level splits.

Embedding model swap

Swapping the embedding model has a much larger effect on retrieval F1 than swapping the chunker. Test that swap before you spend two weeks on chunking. Voyage voyage-3, Cohere embed-v4, OpenAI text-embedding-3-large, BGE-M3 — each performs differently on different corpora. The MTEB leaderboard provides a starting point but does not replicate your corpus. Run the same 50-query eval with three embedding models and pick the winner. The F1 lift from a better embedding is typically 4-12 points, larger than the 2-6 points the chunker debate offers.

The Five-Minute Decision for Most Operators

Skip the chunking research rabbit hole. For 90% of operator workloads in 2026:

1. Use RecursiveCharacterTextSplitter at 512 tokens with 15% overlap.
2. Spend the saved time on the 50-query eval set. The eval is permanent. The chunker debate is not.
3. Test 256 if your documents are short, 1024 if very long. Test once, pick a winner.
4. Use element-aware extractors for PDFs with tables (Unstructured.io, Reducto, AWS Textract).
5. Reserve structural chunking for corpora with clean reliable structure (Markdown docs, code, contracts).
6. Reserve agentic chunking for narrow, high-value, slow-changing corpora where the F1 lift is worth $500-$2,000 per quarter.

The 2026 lesson, repeatable across every operator team that has run the experiment: the boring default wins more often than you expect, and the time you save not over-engineering chunking is better spent on the eval set, the reranker (next lesson), and the citation guardrail (lesson 4).

Key Takeaways

• Recursive character splitting at 512 tokens with 15% overlap is the 2026 default for RAG chunking. It beats semantic chunking on retrieval F1 by 2-6 points on real operator corpora while costing 3-5x less to compute.
• The 50-query eval set on your own corpus is the decision-making infrastructure. Without 30-50 labeled (query, ground-truth-passage) pairs, every chunker comparison is theater. Five hours of work, run quarterly, defensible to anyone.
• Hit-rate-at-10, hit-rate-at-3, and MRR are the metrics. Recursive 512 typically scores 80-87% hit-rate-at-10 on mixed operator corpora.
• Structural chunking (Markdown headers, HTML sections) wins on clean structure — engineering runbooks, contracts with named clauses, API docs. Loses on inconsistent corpora where structure is unreliable.
• Semantic chunking sounds smart, demos well, and underperforms recursive on real corpora. The signal is noisy, thresholds over-fit, and the embedding cost is 4-8x higher per document.
• Agentic / contextual chunking (Anthropic's pattern) lifts hit-rate-at-20 by 35% on legal and financial corpora but costs $500-$2,000 per full reindex of a 100K-document corpus. Worth it on small, high-value, slow-changing corpora; not justified on operational corpora.
• Corpus-specific gotchas: combine short CRM notes into ticket-level documents; preserve section IDs as metadata for contracts; chunk SOPs on procedure boundaries; use element-aware extractors (Unstructured.io, Reducto, LlamaParse) for PDFs with tables.
• Swap the embedding model before you tweak the chunker. The F1 lift from Cohere embed-v4 vs OpenAI text-embedding-3-large vs Voyage voyage-3 is typically 4-12 points on a given corpus — bigger than the chunker effect.
• Knobs people tweak too early: chunk size beyond 512 (rarely worth it), overlap beyond 15% (doubles cost for 1-2 point gain), separator priority order (default is fine for most corpora).
• The five-minute decision for 90% of operator workloads: RecursiveCharacterTextSplitter at 512/15%, time spent on the eval set instead, element-aware PDFs, structural only for clean-structure corpora, agentic only for narrow high-value cases.