Day 7: How to Evaluate RAG Systems: The RAG Triad, Ragas, TruLens, and CI/CD Gates
Production RAG Masterclass · Day 7 of 7 Shipping a new embedding model without an eval suite is a vibe check. Production teams need numbers that fail the build when quality slips. Figure 1. Treat evaluation as product code — versioned and blocking. Thesis You cannot optimize a RAG system without automated evaluation metrics. Human spot-checks do not survive embedding model swaps. The RAG Triad (no gold labels required) Figure 2. Three LLM-as-judge scores catch most production regressions. Context Relevance — Did we retrieve the right chunks for this question? Groundedness / Faithfulness — Is every claim supported by those chunks? Answer Relevance — Does the answer address the user prompt? Score each 0–1 with an LLM judge. Track means and failure cases over a fixed set of 50–200 real queries — including the ACL leak cases from Day 6. Hands-on with Ragas / TruLens from ragas import evaluate from ragas.metrics import ( context_precision, # ~ con...