Does Your RAG Actually Work? Evaluating AI Systems in Production

The Question You Cannot Answer

Let me be direct: most teams running a RAG system or an AI agent in production cannot answer a simple question. Is it good? They have a demo that impressed someone, a few cherry-picked examples, and a vague sense that users seem happy. That is not evaluation, that is hope. And hope does not tell you whether last week's change made things better or quietly worse.

Evaluating AI is harder than evaluating traditional software because there is rarely one correct output. A retrieval can be partly right, an answer can be plausible but wrong, and the same input can produce different outputs. But harder does not mean impossible. You measure AI quality the same way you measure anything you cannot eyeball: build a test set, define metrics, score against them, and watch the numbers over time. This guide is how.

If you cannot say whether your AI got better or worse after a change, you are not operating a system, you are hoping about one. Evaluation is what turns hope into engineering.

We build measurable AI, and we run confidence scoring and a review loop in our own Exfinity platform. This guide is that discipline. For the retrieval architecture itself, see our enterprise RAG systems guide, and this is core AI services work.

Who Cares About What

Two Kinds of Evaluation

There are two modes, and you need both.

Offline evaluation runs against a fixed test set, before you ship. You have known questions and known good answers, and you score the system against them. This is your regression gate: it tells you whether a change helped or hurt before real users feel it.

Online evaluation runs against live traffic, after you ship. Real users ask unexpected things, and you measure how the system does in the wild through confidence scores, feedback, and outcomes. This catches what your test set did not anticipate.

The mistake is doing only one. Offline-only misses what real users actually ask. Online-only means you find regressions after they hit customers. Our AI observability guide covers the online side in depth.

Build the Test Set First

Everything starts with a test set: representative questions paired with what a good answer looks like. This is the single highest-leverage thing you can build, because without it every evaluation is subjective.

A good test set is not a hundred easy questions. It is a deliberate mix.

1. Common cases. The questions users actually ask most.
2. Hard cases. Ambiguous, multi-part, or edge questions where the system is likely to slip.
3. Known failures. Every bug you have ever found, added as a permanent regression check.
4. Adversarial cases. Questions designed to trigger hallucination or leakage.

Keep it version-controlled, grow it every time you find a new failure, and run against it on every change. This turns "I think it is better" into "recall went up two points and nothing regressed." Our prototype to production guide covers building this from the start.

Measure Retrieval and Generation Separately

A RAG answer can be bad for two very different reasons: retrieval brought back the wrong documents, or retrieval was fine and the model wrote a bad answer from good documents. If you only measure the final answer, you cannot tell which, and you cannot fix what you cannot locate.

Measure the two stages separately.

When quality drops, this split tells you where to look. Retrieval metrics down means fix chunking, embeddings, or hybrid search, covered in our vector search architecture guide. Generation metrics down with good retrieval means fix the prompt or the model. Our RAG production reliability guide covers the failure modes.

A Worked Example: One Test-Set Entry

Make evaluation concrete with a single test case, because the whole discipline is built from these.

A test-set entry is a question, the documents that should be retrieved, and what a good answer contains.

Question:        "What is the cancellation policy for same-day tours?"
Should retrieve: policy-doc-cancellations (chunk 3)
Good answer:     states same-day tours are non-refundable, cites the policy

Now run a change, say a new chunking strategy, and score the system against it.

Retrieval:   was the right chunk in the top k?        yes  -> recall ok
Rank:        where did it land?                        rank 1 -> good
Faithfulness: did the answer stick to the chunk?       yes -> no invention
Relevance:   did it actually answer the question?      yes
Attribution: did it cite the source?                   yes

One green case proves nothing. A hundred of these, spanning common questions, hard ones, every past bug, and adversarial attempts, give you a number: this change moved retrieval recall up two points and regressed nothing, or it quietly broke the same-day-tour case that used to pass. That is the difference between "it feels better" and "here is what changed." Every bug you ever find becomes a permanent entry here, so the same mistake cannot come back unnoticed. Our enterprise RAG systems guide covers tuning the retrieval these metrics measure.

Confidence Scoring: Evaluation That Runs Live

Offline metrics need known answers. In production you do not have them, so you need a signal the system can compute on every response without a ground truth. That signal is a confidence score.

A practical confidence score is a composite of factors you can measure at runtime. In Exfinity, every response is scored on four: whether retrieval returned useful results, whether the response passed policy checks, whether it is substantive rather than a stub, and whether there was enough conversation depth to ground it.

confidence = retrieval * 0.4      // did tools return useful results
           + policy * 0.2         // any validation warnings
           + completeness * 0.2   // substantive, not a stub
           + memory * 0.2         // enough context to be grounded

A response below a threshold is flagged as low confidence and routed to human review before it is trusted, and the system classifies why it was low, a retrieval miss, a policy block, a hallucination signal like a past date presented as current. This gives you a live quality signal on every single response, not just on your test set. It is evaluation that runs in production, continuously.

Close the Loop: Review That Improves the System

A confidence score that only flags problems is a warning light. A confidence score wired to a review loop is a system that improves.

When a response is low confidence, a human reviews it. If the answer should have been good, the approved answer is folded back into the knowledge base, so the same question retrieves a good answer next time. In Exfinity, this auto-learn loop even tunes its own threshold: if reviewers approve almost everything flagged, the threshold lowers to catch more; if they reject a lot, it raises to cut noise. The system gets measurably better from being used, instead of just accumulating. This is the human-in-the-loop pattern applied to quality itself, and it connects to the memory patterns in our agent memory guide.

Make It a Gate, Not a Report

Evaluation that produces a report nobody reads changes nothing. Evaluation that gates a deploy changes everything. Wire your offline test set into the pipeline so a change that drops quality below the bar fails the build, the same way a failing unit test does.

This is the shift from "we evaluate sometimes" to "we cannot ship a regression." It is the single practice that most separates teams whose AI quality improves from teams whose quality drifts. Our AI observability guide covers wiring the online metrics into alerting the same way.

Getting Started

1. Build a test set. Common, hard, known-failure, and adversarial cases. Version it.
2. Split the metrics. Measure retrieval and generation separately so you know where to look.
3. Add a confidence score. A live signal on every response, with a review threshold.
4. Close the loop. Route low confidence to human review, fold good answers back in.
5. Gate the deploy. Fail the build on a quality regression, not just a broken test.

This is staged and fits our methodology. Our custom software and consulting teams build measurable AI. Start at contact or get a scoped quote.

Common Ways This Goes Wrong

1. No test set. Every evaluation is subjective. Build one and version it.
2. Only measuring the final answer. You cannot tell if retrieval or generation failed. Split them.
3. Offline or online only. Offline misses real questions, online finds regressions late. Do both.
4. A confidence score with no loop. Flagging without fixing changes nothing. Route to review, fold back.
5. A report nobody reads. Gate the deploy on quality, so a regression cannot ship.

Who Builds This

Oronts is a founder-led software company in Munich. Refaat Al Ktifan, our founder and solution architect, leads a senior team that builds AI you can measure. Our own Exfinity platform scores every response, routes low confidence to review, and improves from the loop. We build the test set, the metrics, the confidence scoring, and the review loop, and hand you a system whose quality you can actually see and defend. See our services and solutions pages.

Takeaways

• If you cannot say whether your AI got better or worse after a change, you are hoping, not engineering.
• Build a versioned test set of common, hard, known-failure, and adversarial cases.
• Measure retrieval and generation separately so you know which one to fix.
• Add a live confidence score with a review threshold, and close the loop so the system improves.
• Gate the deploy on quality so a regression cannot ship.

Evaluation is not a phase you do once. It is the instrument panel you fly the system by. Build it early, gate on it always, and your AI improves instead of drifting.

Not sure whether your AI actually works? Tell us what you are running. Start at contact or get a scoped quote.