Day 7: How to Evaluate RAG Systems: The RAG Triad, Ragas, TruLens, and CI/CD Gates

Production RAG Masterclass · Day 7 of 7

Shipping a new embedding model without an eval suite is a vibe check. Production teams need numbers that fail the build when quality slips.

Figure 1. Treat evaluation as product code — versioned and blocking.

Thesis You cannot optimize a RAG system without automated evaluation metrics. Human spot-checks do not survive embedding model swaps.

The RAG Triad (no gold labels required)

Figure 2. Three LLM-as-judge scores catch most production regressions.

1. Context Relevance — Did we retrieve the right chunks for this question?
2. Groundedness / Faithfulness — Is every claim supported by those chunks?
3. Answer Relevance — Does the answer address the user prompt?

Score each 0–1 with an LLM judge. Track means and failure cases over a fixed set of 50–200 real queries — including the ACL leak cases from Day 6.

Hands-on with Ragas / TruLens

from ragas import evaluate
from ragas.metrics import (
    context_precision,   # ~ context relevance
    faithfulness,
    answer_relevancy,
)

dataset = load_eval_set()  # question, contexts, answer
report = evaluate(
    dataset,
    metrics=[context_precision, faithfulness, answer_relevancy],
)
assert report["faithfulness"] >= 0.85
assert report["answer_relevancy"] >= 0.80

TruLens adds tracing and feedback functions you can attach to live traffic samples after CI gates pass.

What else to measure

• Retrieval: Recall@K, MRR (when you have labeled relevant docs).
• Latency: p50 / p95 end-to-end and rerank stage.
• Security: ACL leak tests — forbidden docs must never appear (Day 6).
• Cost: tokens per answer (HyDE, multi-query, and agents add up).

CI/CD regression gate

On every change to chunking, embeddings, prompts, or rerankers:

1. Rebuild a staging index from a frozen corpus snapshot.
2. Run the eval suite.
3. Fail the pipeline if any triad metric drops past your threshold (for example, 3 points).
4. Publish a report as a PR artifact.

# .github/workflows/rag-eval.yml (conceptual)
- run: python evals/run_rag_triad.py --baseline metrics/main.json
- run: python evals/check_acl_leaks.py

Closing the series

Days 1–6 covered how to build. Day 7 is how you know you did not break it. Treat evals as product code — reviewed, versioned, and blocking merges.

Key takeaways

1. Use the RAG Triad when gold labels are scarce.
2. Automate with Ragas / TruLens on a fixed query set.
3. Gate merges on metric regressions and ACL leak tests.
4. Track latency and cost alongside quality.

Series: Production RAG Masterclass
Previous: Day 6 — Enterprise Security
Next: — End of series —

Also in this series: Day 1 · Day 2 · Day 3 · Day 4 · Day 5 · Day 6 · Day 7