Day 1: Stop Letting Naive Chunking Ruin Your RAG Pipelines

Part 1 of 7 — RAG Systems in Practice

Stop Letting Naive Chunking Ruin Your RAG Pipelines

When a RAG system answers a query with confidence and gets it totally wrong, the initial reaction is usually to blame the model's temperature or swap vector databases. Most of the time, the real bug happened hours earlier when you ingested the raw PDF.

---

What actually breaks when you use fixed character splitting?

Standard tutorials tell you to take your document and run a RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50). It works fine for demo apps, but here is what happens on enterprise data:

• Tables turn into nonsense: A 10-column financial or spec table gets chopped right in half. Row entries lose their column headers, turning structured data into random numbers.
• Headers detach from paragraphs: The section heading ### Rate Limits lands in Chunk A, while the actual rate limit numbers end up at the top of Chunk B without any context.
• Code blocks get sliced mid-function: Python or JSON payloads split mid-syntax, producing broken code blocks that confuse the LLM during retrieval generation.

Fixing Ingestion: AST Parsing + Context Injection

Instead of slicing blindly every 500 characters, a production ingestion worker should split along the document’s natural Abstract Syntax Tree (AST)—respecting Markdown headers, code block boundaries, and table structures—while injecting parent metadata directly into the vector payload.

ingest_pipeline.py

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

# 1. Define explicit header rules so section hierarchy is preserved
headers_to_split = [
    ("#", "section_h1"),
    ("##", "section_h2"),
    ("###", "section_h3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split,
    strip_headers=False
)

# 2. Split by logical sections first
header_docs = markdown_splitter.split_text(raw_markdown_content)

# 3. Apply secondary character splitter ONLY to long sections
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", " ", ""]
)

final_chunks = text_splitter.split_documents(header_docs)

# Each chunk now retains 'section_h1' and 'section_h2' in its metadata payload
print(final_chunks[0].metadata)

The Key Rule for Clean Retrieval

Rule of thumb: Never store raw text in a vector database without metadata context. If your vector chunk cannot be uniquely understood by a human reading it in isolation without seeing the rest of the document, your LLM won't understand it either.

Next up in this series:

Day 2: Hybrid Search & Reranking—Why vector search alone misses exact keyword matches and how to combine BM25 with Cross-Encoders.