Posts

Day 7: How to Evaluate RAG Systems: The RAG Triad, Ragas, TruLens, and CI/CD Gates

Image
Production RAG Masterclass · Day 7 of 7 Shipping a new embedding model without an eval suite is a vibe check. Production teams need numbers that fail the build when quality slips. Figure 1. Treat evaluation as product code — versioned and blocking. Thesis You cannot optimize a RAG system without automated evaluation metrics. Human spot-checks do not survive embedding model swaps. The RAG Triad (no gold labels required) Figure 2. Three LLM-as-judge scores catch most production regressions. Context Relevance — Did we retrieve the right chunks for this question? Groundedness / Faithfulness — Is every claim supported by those chunks? Answer Relevance — Does the answer address the user prompt? Score each 0–1 with an LLM judge. Track means and failure cases over a fixed set of 50–200 real queries — including the ACL leak cases from Day 6. Hands-on with Ragas / TruLens from ragas import evaluate from ragas.metrics import ( context_precision, # ~ con...
Post a Comment
Continue reading: Day 7: How to Evaluate RAG Systems: The RAG Triad, Ragas, TruLens, and CI/CD Gates

Day 6: Enterprise RAG Security: RBAC, Document-Level Access Control, and Prompt Injection

Image
Production RAG Masterclass · Day 6 of 7 The most expensive RAG bug is not a wrong answer. It is a correct answer pulled from a document the user was never allowed to see. Figure 1. Permissions belong in retrieval — not only in the UI. Thesis Ingesting confidential enterprise documents without strict permission controls creates severe security leaks. Retrieval must respect identity. Document-Level Security (DLS) Every chunk carries ACL metadata: roles, groups, tenant IDs, classification. If the chunk does not know who may read it, the vector index will happily serve it to everyone. chunk.metadata = { "doc_id": "hr-comp-2025", "acl_roles": ["hr_admin", "comp_committee"], "tenant_id": "acme", "classification": "confidential" } Figure 2. Enforce ACLs on the retrieval path. Pre-filter vs post-trim Metadata pre-filtering (preferred) — push ACL filters into the vector q...
Post a Comment
Continue reading: Day 6: Enterprise RAG Security: RBAC, Document-Level Access Control, and Prompt Injection

Day 5: Agentic RAG: Query Routing, Self-Reflection, and Corrective RAG (CRAG)

Image
Production RAG Masterclass · Day 5 of 7 A linear pipeline assumes the first retrieval was good enough. Production questions often need a second look, a different tool, or an honest “we do not have evidence.” Figure 1. Plan, retrieve, critique, then answer — or try again. Thesis Static, single-shot retrieval fails when a system must verify sources or gather multi-step evidence. Move from a pipeline to a bounded loop. From pipeline to loop Classic RAG: query → retrieve → generate → done . Agentic RAG: plan → route → retrieve → critique → (retry | answer) with a hard stop on iterations. Figure 2. Cap loops so latency stays predictable. Query routing Not every question belongs in the vector DB: Vector index — policies, runbooks, wiki. SQL / warehouse — metrics, counts, “how many…”. External API — live status, tickets, prices. Graph — multi-hop entity questions (Day 4). route = llm_json(f"""Choose tools for this question. Options: ...
Post a Comment
Continue reading: Day 5: Agentic RAG: Query Routing, Self-Reflection, and Corrective RAG (CRAG)

Day 4: GraphRAG Explained: Combining Knowledge Graphs with Vector Search

Image
Production RAG Masterclass · Day 4 of 7 Some questions are not “find the nearest paragraph.” They are “walk a path across three documents and explain the join.” Flat vector search was never designed for that. Figure 1. Vectors find seeds. Graphs find relationships. Thesis Cosine similarity cannot infer multi-hop entity relationships across separate documents. Pair vectors with an entity–relation graph. Where flat vector RAG breaks Ask: “Which vendors touch the same data pipeline as Team Atlas, and who owns the SLA?” The facts live in an org chart, a vendor contract, and a runbook. No single chunk holds the full path. Similarity returns locally relevant paragraphs — not the chain. Build the graph at ingest Extract triples while you chunk: (Entity) -[RELATION]-> (Entity) (Team Atlas)-[OWNS]->(Billing Pipeline) (Billing Pipeline)-[DEPENDS_ON]->(Vendor Stripe) (Vendor Stripe)-[HAS_SLA]->(99.95%) Store nodes and edges in Neo4j (or similar). Keep chunk t...
Post a Comment
Continue reading: Day 4: GraphRAG Explained: Combining Knowledge Graphs with Vector Search

Day 3: Query Optimization for RAG: HyDE, Multi-Query Expansion, and Sub-Query Decomposition

Image
Production RAG Masterclass · Day 3 of 7 Most RAG post-mortems blame the index. In practice, the query that hit the index was a vague chat message that never resembled the documents you stored. Figure 1. Fix the query shape before you retune the vector store. Thesis Raw user queries are short, vague, and poorly formatted for semantic matching. Optimize the query — then retrieve. The query–document mismatch User: “why is billing failing?” Doc: “Invoice reconciliation jobs abort when Stripe webhook signature validation returns HTTP 400…” Those strings barely share tokens. Embeddings help. A deliberate bridge helps more. Strip conversational noise first Before expansion, normalize the prompt: Drop politeness filler (“please”, “can you”, “thanks”). Drop chat history that is not about the current ask. Keep entities, IDs, product names, and constraints. def clean_query(q: str) -> str: return llm(f"Rewrite as a concise search query:\n{q}") ...
Post a Comment
Continue reading: Day 3: Query Optimization for RAG: HyDE, Multi-Query Expansion, and Sub-Query Decomposition

Day 2: Hybrid Search and Reranking for Production RAG (Dense + BM25 + Cross-Encoders)

Image
Production RAG Masterclass · Day 2 of 7 If your RAG system “almost” finds the right answer when users paste an error code, a part number, or an acronym — the problem is rarely the LLM. It is almost always retrieval that smoothed away the exact tokens that mattered. Figure 1. Sparse and dense retrieval merged before a final rerank pass. Thesis Vector distance is great at meaning and weak at literals. Hybrid search closes that gap; a cross-encoder decides what actually belongs in the prompt. Where dense-only retrieval quietly fails Embedding models shine on paraphrases. Production users do not. They paste strings the model was never trained to treat as sacred: ERR_TIMEOUT_5042 PN-88421-B Domain shorthand: SOX, MTTR, OIDC, SKU families Cosine similarity softens those tokens into “something nearby.” BM25 keeps them exact. You need both lists, then a principled way to merge them. Stage 1: run sparse and dense in parallel Dense path — embed the query, ne...
Post a Comment
Continue reading: Day 2: Hybrid Search and Reranking for Production RAG (Dense + BM25 + Cross-Encoders)

Day 1: Stop Letting Naive Chunking Ruin Your RAG Pipelines

Image
Part 1 of 7 — RAG Systems in Practice Stop Letting Naive Chunking Ruin Your RAG Pipelines When a RAG system answers a query with confidence and gets it totally wrong, the initial reaction is usually to blame the model's temperature or swap vector databases. Most of the time, the real bug happened hours earlier when you ingested the raw PDF. What actually breaks when you use fixed character splitting? Standard tutorials tell you to take your document and run a RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50) . It works fine for demo apps, but here is what happens on enterprise data: Tables turn into nonsense: A 10-column financial or spec table gets chopped right in half. Row entries lose their column headers, turning structured data into random numbers. Headers detach from paragraphs: The section heading ### Rate Limits lands in Chunk A, while the actual rate l...
Post a Comment
Continue reading: Day 1: Stop Letting Naive Chunking Ruin Your RAG Pipelines