Why Answer Accuracy Is Not Retrieval Accuracy

By S. Teka · September 2026 · Series: Retrieval Quality & AI Evaluation

A retrieval-augmented generation system answers a test question correctly. The evaluation says PASS. It is tempting to conclude that the RAG pipeline worked. But what if the retriever selected the wrong evidence?

What if the retrieved context did not support the answer at all?

A large language model can still produce the expected answer. In a real system, that could happen for several reasons: information already represented in model parameters, clues in the question, other useful context, inference from incomplete evidence, or simply a fortunate output.

A correct answer tells us that the answer was correct. It does not prove that retrieval was correct.

This article explores that distinction experimentally. The companion repository uses deterministic synthetic cases to evaluate retrieval, a deterministic evidence-support check, answer correctness, and the overall RAG contract independently.

The experiment deliberately creates the condition that an answer-only evaluation can miss:

wrong retrieval
      +
unsupported evidence
      +
correct answer
      =
hidden RAG failure

The objective is not to benchmark a particular LLM. It is to make the evaluation failure visible and reproducible.

The test passed. But what actually passed?

Consider a simplified RAG evaluation. A user asks a synthetic question about a fictional plan. The expected answer is 30 days, and the generated answer is also 30 days.

Expected:  30 days
Generated: 30 days

Answer correctness: PASS

If this is the only signal being measured, the test passes.

Now inspect the retrieval trace. Suppose the system retrieved evidence describing a different plan whose cancellation period is 14 days. The generated answer is still 30 days.

Answer correctness:    PASS
Retrieval correctness: FAIL
Evidence support:      FAIL

The answer matches the expected answer, but the retrieved evidence does not support the expected fact. Calling the complete RAG execution successful would hide the retrieval failure.

A RAG pipeline has multiple evaluation boundaries

Question
   │
   ▼
Retriever ─────────────► Retrieval Evaluation
   │
   ▼
Retrieved Evidence ────► Evidence-Support Evaluation
   │
   ▼
Generator
   │
   ▼
Generated Answer ──────► Answer Evaluation

Retrieval evaluation asks whether the system retrieved the evidence required for the question.

Evidence-support evaluation asks whether the supplied evidence contains support for the expected fact. In this companion experiment, that check is deliberately deterministic and narrow; it is not a general semantic grounding judge.

Answer evaluation asks whether the generated answer matches the expected answer.

Collapsing these questions into a single answer-accuracy score removes diagnostic information. RAG evaluation research reflects the same broader separation: RAGAS evaluates retrieval and generation dimensions; ARES separates context relevance, answer faithfulness, and answer relevance; RAGChecker focuses on fine-grained diagnosis across retrieval and generation components.

Four experiments, not one accuracy score

The Article 02 companion experiment contains four controlled cases. It uses synthetic data, deterministic fixtures, no external model API, no cloud credentials, and no LLM judge.

Case A — Correct retrieval, correct answer

Correct Evidence
      │
      ▼
Correct Answer

Retrieval       PASS
Evidence        PASS
Answer          PASS
Overall         PASS

HEALTHY_RAG

This is the expected healthy path. The required evidence is present, the evidence supports the expected fact, and the answer is correct.

Case B — Correct retrieval, wrong answer

Correct Evidence
      │
      ▼
Incorrect Answer

Retrieval       PASS
Evidence        PASS
Answer          FAIL
Overall         FAIL

GENERATION_FAILURE

This distinction matters operationally. If retrieval already found the right evidence, changing chunk size, embedding strategy, top-K, or ranking may not address the failure. The evidence reached the generation stage; the observed failure is downstream.

Case C — Wrong retrieval, wrong answer

Wrong Evidence
      │
      ▼
Wrong Answer

Retrieval       FAIL
Evidence        FAIL
Answer          FAIL
Overall         FAIL

RETRIEVAL_FAILURE

This condition is easier to notice because the incorrect final answer exposes a problem. The retrieval signal adds diagnostic information by showing that the evidence path had already failed.

Case D — Wrong retrieval, correct answer

Wrong Evidence
      │
      ▼
Correct Answer

Retrieval       FAIL
Evidence        FAIL
Answer          PASS
Overall         FAIL

CORRECT_ANSWER_WRONG_EVIDENCE

This is the centerpiece of Article 02. An answer-only evaluator sees PASS. The retrieval-aware contract sees FAIL.

The correct final answer has not repaired the evidence path. It has only made the broken path easier to miss.

How can the answer be right when retrieval is wrong?

There is no single explanation that should automatically be assumed in a real RAG system. The model may already contain the fact, the question may leak useful clues, another chunk may provide sufficient evidence, incomplete context may still permit inference, or the output may simply be fortunate.

The companion experiment intentionally does not claim to determine why a real model produced a particular answer. It uses deterministic answer fixtures instead of pretending to reproduce model behavior.

The experiment asks a narrower engineering question:

The retrieval-aware evaluation contract

RAG PASS =
    Retrieval PASS
    AND
    Evidence Support PASS
    AND
    Answer PASS

Not simply:

RAG PASS = Answer PASS

And not a weighted average that permits one successful component to hide a failed critical component.

{
  "retrieval": { "pass": false },
  "grounding": { "pass": false },
  "answer": { "pass": true },
  "overall": {
    "pass": false,
    "failure_mode": "CORRECT_ANSWER_WRONG_EVIDENCE"
  }
}

The JSON shape is less important than the contract behind it: do not let a successful output erase evidence that an upstream component failed.

Retrieval metrics still matter

Article 01 evaluated retrieval using Recall@K, Mean Reciprocal Rank, and NDCG@K. Article 02 does not replace those metrics. It adds a per-case diagnostic layer.

Dataset-level retrieval evaluation
        │
        ├── Recall@K
        ├── MRR
        └── NDCG@K

Per-case diagnostic evaluation
        │
        ├── Retrieval PASS/FAIL
        ├── Evidence Support PASS/FAIL
        ├── Answer PASS/FAIL
        └── Failure Mode

The first view measures retrieval quality across an evaluation set. The second helps explain what happened in an individual controlled execution.

Remove the evidence and run the test again

The second Article 02 experiment holds the deterministic answer fixture fixed while changing only the evidence condition: correct evidence, wrong evidence, or no evidence.

From the perspective of answer correctness, all three fixtures look successful. From the perspective of the RAG evidence path, they are very different.

What evidence ablation actually tells us

If a controlled case remains answer-correct without retrieval, that does not prove retrieval is globally unnecessary. It tells us something narrower: answer correctness in that test case is not sufficient evidence that retrieval contributed to the successful result.

Other questions may require retrieval. Facts may change. Private or domain-specific information may not exist in model parameters. Multi-hop questions may require several sources. Evidence ablation is therefore a diagnostic technique, not an argument against RAG.

Give failures names

Binary pass/fail signals are useful. Diagnostic categories make them more actionable. The Article 02 implementation includes explicit failure modes such as:

HEALTHY_RAG
RETRIEVAL_FAILURE
GENERATION_FAILURE
GROUNDING_FAILURE
CORRECT_ANSWER_WRONG_EVIDENCE
INSUFFICIENT_EVIDENCE
UNSUPPORTED_CORRECT_ANSWER

Instead of seeing only a global accuracy percentage, an engineer can ask which component failed and why. Retrieval failures point toward one family of investigation; correct evidence followed by a wrong answer points toward another.

A note about “grounding” in this experiment

The companion repository deliberately keeps Article 02 deterministic. It does not ask an LLM to judge whether arbitrary natural-language output is semantically grounded.

For these synthetic fixtures, the implementation performs a deterministic evidence-support check: it verifies whether the retrieved evidence contains the expected answer needed by the controlled case.

That design makes the experiment offline, reproducible, inspectable, free of model nondeterminism, and suitable for CI. But it can false-negative on paraphrases and should not be confused with a complete production faithfulness evaluator.

Production systems may need to handle paraphrases, multiple supporting passages, contradictions, partial support, numerical reasoning, and ambiguous evidence. Those requirements may justify deterministic rules, specialized evaluators, model-based judges, human review, or a combination.

LLM judges do not eliminate the evaluation problem

Using another language model as an evaluator can be useful. It also introduces another measurement component whose behavior must be understood and validated.

Deterministic labels
        +
Retrieval metrics
        +
Automated evaluators
        +
Human review where necessary

The right combination depends on the application. The important principle is that the evaluation architecture should make clear what is being measured and what uncertainty each evaluator introduces.

Why this matters in CI/CD

Suppose an engineering change modifies chunk size, overlap, embedding strategy, metadata filters, top-K, reranking, query rewriting, document preprocessing, or retrieval strategy.

Final answer accuracy might remain unchanged on a small evaluation set even while retrieval quality regresses. An answer-only regression suite can therefore miss an upstream change that the generator happens to mask.

Pull Request
     │
     ▼
Retrieval Evaluation
     │
     ├── regression? ──► FAIL
     ▼
Evidence-Support Evaluation
     │
     ├── regression? ──► FAIL
     ▼
Answer Evaluation
     │
     ├── regression? ──► FAIL
     ▼
Release Gate

Article 01 argued that retrieval evaluation belongs in engineering release discipline. Article 02 adds another reason: generation can sometimes hide retrieval regressions.

The experiment is intentionally small

The companion repository does not claim to reproduce every behavior of production RAG. Its Article 02 experiment uses synthetic deterministic fixtures. It does not benchmark a real LLM, estimate production accuracy, infer why a real model produced an answer, solve semantic grounding, or replace human evaluation.

Those limits are part of the experiment design. The objective is to isolate one evaluation principle so readers can reproduce it without infrastructure or model variability.

At the Article 02 implementation checkpoint, the repository reports 41 passing tests—the original Article 01 tests plus the Article 02 coverage—and the Article 01 baseline remains unchanged.

Companion Repository

terkeka-rag-eval article02

What should we measure?

1. Retrieve: Did we retrieve the evidence required for the question?
2. Support: Does the supplied evidence support the expected fact?
3. Answer: Did the generator produce the expected answer?

Those questions distinguish healthy retrieval and generation from upstream retrieval failures, downstream generation failures, and the especially important condition where bad retrieval is hidden behind an apparently correct answer.

Retrieval is not merely an implementation detail in a RAG system. It determines what external evidence reaches the model. If the application depends on evidence being current, private, authoritative, traceable, or auditable, validating only the final answer leaves part of the system untested.

The engineering principle

A correct answer tells you that the answer was correct. It does not tell you why it was correct.

For a RAG system, the evidence path is part of the product. Test retrieval separately. Test evidence support separately. Test the answer separately. And when a critical component fails, do not let a fluent—or even correct—final answer hide the failure.

References & further reading

1. Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. EACL 2024 System Demonstrations. ACL Anthology.
2. Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. NAACL 2024. ACL Anthology.
3. Ru, D., et al. (2024). RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation. arXiv.
4. Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2025). Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation. NAACL 2025. ACL Anthology.