TecWorms – Python, AI & Developer Tutorials

Production RAG Masterclass · Day 7 of 7 Shipping a new embedding model without an eval suite is a vibe check. Production teams need numbers that fail the build when quality slips. Figure 1. Treat evaluation as product code — versioned and blocking. Thesis You cannot optimize a RAG system without automated evaluation metrics. Human spot-checks do not survive embedding model swaps. The RAG Triad (no gold labels required) Figure 2. Three LLM-as-judge scores catch most production regressions. Context Relevance — Did we retrieve the right chunks for this question? Groundedness / Faithfulness — Is every claim supported by those chunks? Answer Relevance — Does the answer address the user prompt? Score each 0–1 with an LLM judge. Track means and failure cases over a fixed set of 50–200 real queries — including the ACL leak cases from Day 6. Hands-on with Ragas / TruLens from ragas import evaluate from ragas.metrics import ( context_precision, # ~ con...

Production RAG Masterclass · Day 6 of 7 The most expensive RAG bug is not a wrong answer. It is a correct answer pulled from a document the user was never allowed to see. Figure 1. Permissions belong in retrieval — not only in the UI. Thesis Ingesting confidential enterprise documents without strict permission controls creates severe security leaks. Retrieval must respect identity. Document-Level Security (DLS) Every chunk carries ACL metadata: roles, groups, tenant IDs, classification. If the chunk does not know who may read it, the vector index will happily serve it to everyone. chunk.metadata = { "doc_id": "hr-comp-2025", "acl_roles": ["hr_admin", "comp_committee"], "tenant_id": "acme", "classification": "confidential" } Figure 2. Enforce ACLs on the retrieval path. Pre-filter vs post-trim Metadata pre-filtering (preferred) — push ACL filters into the vector q...

Production RAG Masterclass · Day 5 of 7 A linear pipeline assumes the first retrieval was good enough. Production questions often need a second look, a different tool, or an honest “we do not have evidence.” Figure 1. Plan, retrieve, critique, then answer — or try again. Thesis Static, single-shot retrieval fails when a system must verify sources or gather multi-step evidence. Move from a pipeline to a bounded loop. From pipeline to loop Classic RAG: query → retrieve → generate → done . Agentic RAG: plan → route → retrieve → critique → (retry | answer) with a hard stop on iterations. Figure 2. Cap loops so latency stays predictable. Query routing Not every question belongs in the vector DB: Vector index — policies, runbooks, wiki. SQL / warehouse — metrics, counts, “how many…”. External API — live status, tickets, prices. Graph — multi-hop entity questions (Day 4). route = llm_json(f"""Choose tools for this question. Options: ...