Why RAG Evaluation Is Hard

A retrieval-augmented generation system can produce a correct answer for the wrong reason, a wrong answer from the right evidence, or a perfectly grounded answer to the wrong question. All three may look equally fluent. That is why evaluating RAG is harder than checking whether the final sentence resembles a reference answer.

The central mistake is treating RAG as one model. It is a causal chain: a corpus is selected, documents are split, a query is interpreted, evidence is retrieved and ranked, context is packed into a prompt, an LLM generates claims, and citations are attached. A failure at any link changes the answer. One aggregate score can tell us that the system failed; it usually cannot tell us why.

This note builds a practical evaluation map from primary research, including RAGAS, ARES, RAGChecker, RAGTruth, ALCE, RGB, and Lost in the Middle. It also develops a worked regulatory-compliance example to show, with concrete measurements, why two systems with the same answer accuracy can have very different reliability.

RAG Is a Pipeline, Not a Model

The original RAG formulation combines parametric memory in a sequence-to-sequence model with non-parametric memory in a retrieved document index (Lewis et al., 2020). Modern production systems expose even more moving parts:

$$q\to D\to C\to R_k\to \pi (R_k)\to P\to y\to A$$

Here, $q$ is the query, $D$ the available corpus, $C$ its chunks, $R_k$ the top-$k$ retrieval result, $\pi$ a reranking and packing operation, $P$ the final prompt context, $y$ the generated answer, and $A$ its citations or attributions.

A diagnostic view of RAG. Each component can independently produce the same visible outcome: a wrong answer. Original diagram.

Layer	Example failure	What the final answer cannot reveal
Corpus	The current regulation was never indexed	Whether retrieval was even capable of succeeding
Chunking	A rule and its exception were split apart	Whether the source document contained both facts
Retrieval	A topical but non-answering passage outranked the rule	Whether semantic similarity matched answer utility
Reranking	Correct evidence fell below the context cutoff	Whether first-stage retrieval had already succeeded
Context packing	Correct evidence was buried among distractors	Whether the LLM could use its nominal context window
Generation	The model ignored a retrieved exception	Whether the error belongs to retrieval or reasoning
Citation	The cited document was relevant but did not support the claim	Whether the answer is verifiable
Abstention	The corpus had no answer, but the model guessed	Whether the system knows when evidence is insufficient

This modularity creates an attribution problem. Suppose a response is wrong. If the required evidence was absent from $D$, changing the generator cannot fix it. If the evidence was retrieved but ignored, tuning the retriever may improve Recall@k while leaving the user-facing error untouched. Evaluation must therefore serve two different people:

• A user needs an end-to-end measure: did the system complete the task safely and correctly?
• A developer needs diagnostic measures: which component caused the error, and what change is likely to fix it?

RAGChecker formalizes this distinction with overall, retriever, and generator metrics computed at claim level (Ru et al., 2024).

RAGChecker's claim-level decomposition of overall, retriever, and generator quality. Image: Ru et al. (2024), RAGChecker, CC BY 4.0.

One Correctness Score Hides Different Failures

Consider the question: "What retention period applies to transaction logs under the current policy, and what exception permits earlier deletion?" Assume the complete gold answer contains two atomic claims:

1. Transaction logs must normally be retained for five years.
2. Earlier deletion is permitted only after an approved legal-hold release and documented authorization.

Now compare four outputs:

Output	Looks plausible?	Correct?	Faithful to context?	Complete?
"Five years." The retrieved context contains both claims.	Yes	Partly	Yes	No
"Five years, with early deletion after manager approval." The context states legal-hold release, not manager approval.	Yes	No	No	No
Both gold claims, produced from model memory after retrieval missed the policy	Yes	Yes	No	Yes
"The available sources do not establish the retention period." No authoritative policy was indexed.	Cautious	Correct behavior	Yes	N/A

An exact-match metric may fail the first, second, and fourth outputs while passing the third. Yet the third is dangerous in a source-grounded system: its answer happened to be right, but the retrieval path cannot justify it and may fail as soon as the policy changes. Conversely, the fourth response is not a failed answer. It is a correct abstention.

This motivates separate definitions:

• Answer correctness: are the claims true relative to the task's accepted answer?
• Faithfulness or groundedness: can every generated claim be inferred from the retrieved context?
• Answer relevance: does the response actually address the query without irrelevant material?
• Completeness: does the response include every claim required by the query?
• Context relevance: how much of the retrieved context is useful for answering the query?
• Citation correctness: does each cited source entail the claim attached to it?
• Citation completeness: does every externally verifiable claim have evidence?
• Abstention correctness: does the system refuse when the corpus cannot support an answer, while answering when it can?

Correctness and faithfulness are deliberately different. An answer can be correct but ungrounded, or faithfully repeat a false, outdated, or malicious retrieved document. RAG does not turn retrieved text into truth. It changes the provenance of a generation.

Evaluating Retrieval

Classical information-retrieval metrics remain useful, provided we know exactly what they measure. Let $G_q$ be the set of gold-relevant chunks for query $q$, and let $R_k(q)$ be the top-$k$ retrieved chunks.

$$Recall@k(q)=\frac{|R_k(q)\cap G_q|}{|G_q|},Precision@k(q)=\frac{|R_k(q)\cap G_q|}{k}$$

Mean Reciprocal Rank rewards placing the first relevant result early:

$$\mathrm{MRR}=\frac{1}{|Q|}\sum _{q\in Q}\frac{1}{{\mathrm{rank}}_q}$$

Normalized Discounted Cumulative Gain is more appropriate when relevance is graded rather than binary:

$$DCG@k=\sum _{i=1}^k\frac{2^{{\mathrm{rel}}_i}-1}{{\log }_2(i+1)},NDCG@k=\frac{DCG@k}{IDCG@k}$$

These metrics are valuable for comparing retrievers on a fixed benchmark. Dense Passage Retrieval, for example, reported gains of 9–19 percentage points over a strong BM25 baseline on top-20 passage retrieval accuracy across several open-domain QA datasets (Karpukhin et al., 2020). That is real evidence that retrieval models matter. But a RAG application introduces complications that passage-retrieval benchmarks often hide.

A Gold Chunk Is Not the Same as Gold Evidence

Chunk labels depend on a particular preprocessing decision. Change a 300-token chunk into two 150-token chunks and the identities of the "relevant documents" change even though the underlying evidence does not. RAGChecker explicitly criticizes Recall@k and MRR for depending on annotated chunks and a rigid chunking scheme.

Evidence-level evaluation is more stable. Decompose the gold answer into atomic claims $C^{(g)}=\{c_1,\dots ,c_n\}$. Retriever claim recall asks how many required claims are entailed by at least one retrieved chunk:

$$\mathrm{ClaimRecall}=\frac{|\{c\in C^{(g)}:R_k\models c\}|}{|C^{(g)}|}$$

RAGChecker defines a retrieved chunk as relevant when it entails at least one gold claim. Its context precision is then:

$$\mathrm{ContextPrecision}=\frac{\text{number of retrieved chunks entailing a gold claim}}{k}$$

This answers a more useful pair of questions: did retrieval cover the facts needed for the answer, and how much distracting material came with them?

Relevance Is Not Usefulness

A passage can be topically relevant and still fail to answer the question. For the retention example, a chunk describing the history of transaction-logging policy is semantically close to the query but cannot establish the current duration. Likewise, a passage containing "five years" may refer to customer identity records rather than transaction logs. Dense similarity sees overlapping meaning; evaluation must verify whether the passage entails a required claim.

The opposite problem also occurs. An unannotated passage may be a valid alternative source. Treating the original gold chunk as the only acceptable result creates false retrieval errors. A defensible benchmark stores evidence spans and acceptable alternative sources, not only one document identifier.

More Context Trades Recall for Noise

RAGChecker provides unusually concrete evidence for this trade-off. When the number of retrieved chunks increased from 5 to 20, reported claim recall rose from 61.5 to 77.6 and generator faithfulness from 88.1 to 92.2. But noise sensitivity also rose from 34.0 to 35.4, while overall F1 moved only from 51.7 to 53.4 (Ru et al., 2024).

This is why "increase $k$" is not a free improvement. It expands the opportunity set for the correct claim and for a misleading claim. The optimum depends on the reader model, context budget, reranking quality, and cost of each error.

Evaluating Generation at Claim Level

BLEU and ROUGE compare surface overlap with a reference. They can penalize a correct paraphrase and reward a fluent answer that copies reference wording while attaching it to the wrong source. BERTScore improves semantic matching through contextual embeddings (Zhang et al., 2020), but response-level similarity still hides which individual claims are supported.

Long-form evaluation therefore benefits from atomic claims. Let $C^{(y)}$ be claims extracted from the model response and $C^{(g)}$ claims from the gold answer. If $\models$ denotes semantic entailment, claim precision and recall are:

$$P_{\text{claim}}=\frac{|\{c\in C^{(y)}:C^{(g)}\models c\}|}{|C^{(y)}|},R_{\text{claim}}=\frac{|\{c\in C^{(g)}:C^{(y)}\models c\}|}{|C^{(g)}|}$$ $$F{1}_{\text{claim}}=\frac{2P_{\text{claim}}{R}_{\text{claim}}}{P_{\text{claim}}+R_{\text{claim}}}$$

Faithfulness instead compares response claims to retrieved context $R_k$:

$$\mathrm{Faithfulness}=\frac{|\{c\in C^{(y)}:R_k\models c\}|}{|C^{(y)}|}$$

RAGAS introduced a related reference-free strategy: decompose an answer into statements, ask an LLM whether each is inferable from context, and score the supported fraction (Es et al., 2023/2025). On WikiEval, RAGAS reported pairwise agreement with human preferences of 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance. Context relevance was the hardest dimension, particularly for longer context.

RAGChecker goes further by separating three sources of incorrect response claims:

• Relevant-noise sensitivity: the generator copies an incorrect claim from a chunk that also contains useful evidence.
• Irrelevant-noise sensitivity: it copies an incorrect claim from an irrelevant chunk.
• Hallucination: it produces an incorrect claim that appears in no retrieved chunk.

That distinction is actionable. Hallucination suggests generation or prompting work. Irrelevant-noise sensitivity suggests retrieval precision or stronger context filtering. Relevant-noise sensitivity suggests finer chunking, evidence extraction, or a generator that can distinguish supported spans from neighboring noise.

Two more diagnostics matter:

• Context utilization: of the gold claims available in retrieved context, how many did the generator actually use?
• Self-knowledge reliance: how many correct response claims were unsupported by retrieved context and therefore came from parametric memory?

A system can be highly faithful and still incomplete because it conservatively uses only one of three available facts. Faithfulness is not a substitute for utilization or recall.

Citation Quality Is Its Own Target

Adding [1] after a sentence does not make the sentence supported. Citation evaluation has at least two dimensions:

$$\mathrm{CitationCorrectness}=\frac{\text{citations that entail their attached claims}}{\text{all citations}}$$ $$\mathrm{CitationCompleteness}=\frac{\text{verifiable claims with supporting citations}}{\text{all verifiable claims}}$$

ALCE evaluates fluency, correctness, and citation quality in end-to-end systems. Its authors found that, on the ELI5 dataset, even the best evaluated models lacked complete citation support about 50% of the time (Gao et al., 2023). This is proof that good answer quality and good attribution do not automatically move together.

Consider the claim: "Logs must be retained for five years unless a legal-hold release authorizes deletion [1]." Citation [1] may:

• establish five years but say nothing about the exception;
• describe the exception but apply to a superseded policy;
• be topically relevant without containing either requirement;
• support both clauses, but only in a table that the extraction pipeline dropped.

Evaluation should split compound claims before checking entailment and record document version, effective date, and exact evidence span. In a regulatory system, a citation to an outdated rule is not merely low relevance; it is a potentially forbidden fact.

A Long Context Window Does Not Guarantee Context Use

The Lost in the Middle study controlled the position of an answer-containing document while keeping the question, documents, and desired answer otherwise unchanged. Performance followed a U-shaped curve: models used evidence better near the beginning or end and worse in the middle (Liu et al., 2023).

Conceptual reproduction of the reported positional pattern; this is not a copy of the paper's figure. Based on Liu et al. (2023). Original diagram.

The controlled design is important: reordering distractors does not change what the correct answer should be, so a performance change isolates context position as a cause. The paper reported that GPT-3.5-Turbo could drop by more than 20 percentage points; in some 20- and 30-document settings, performance with the answer in the middle fell below its 56.1% closed-book accuracy. Extended-context variants were not necessarily better at using context that fit both windows.

More documents also showed diminishing downstream value. On NaturalQuestions-Open, moving from 20 to 50 retrieved documents improved answer accuracy by only about 1.5% for GPT-3.5-Turbo and 1% for Claude-1.3, even as retriever recall continued to rise. The reader saturated before the retriever.

This yields two evaluation requirements that average Recall@k misses:

1. Randomize or systematically sweep the position of supporting evidence in the packed prompt.
2. Plot answer quality, retrieval coverage, latency, and token cost together as $k$ and chunk size change.

Robustness Tests That Average QA Misses

The RGB benchmark decomposes RAG-reader behavior into four abilities (Chen et al., 2023/2024):

• Noise robustness: use correct evidence despite irrelevant context.
• Negative rejection: decline to answer when no supporting evidence exists.
• Information integration: combine facts distributed across documents.
• Counterfactual robustness: resist false retrieved information that conflicts with correct knowledge.

Their evaluation of six LLMs found some noise robustness, but substantial difficulty with rejection, multi-document integration, and false information. RAGTruth independently confirms that retrieval does not eliminate hallucination: its corpus contains nearly 18,000 naturally generated RAG responses, manually annotated at case and word level for unsupported or contradictory content (Niu et al., 2024).

A practical stress suite should also include:

Slice	What it exposes
Unanswerable query	Whether the system guesses instead of abstaining
Current + superseded document	Temporal reasoning and source authority
Contradictory sources	Conflict handling and provenance
Near-duplicate chunks	Ranking stability and context waste
Ambiguous entity	Entity resolution rather than keyword overlap
Multi-hop relation	Ability to combine distributed evidence
Answer-bearing middle chunk	Positional context utilization
Retrieved prompt injection	Whether document text can override system policy
Missing corpus document	Whether evaluation distinguishes corpus from retriever failure

LLM-as-a-Judge: Useful, Not Ground Truth

Human evaluation is expensive and slow, especially when annotators need legal, medical, or technical expertise. LLM judges make iteration practical, but they add another model whose behavior must be evaluated.

RAGAS uses prompted models to estimate faithfulness, answer relevance, and context relevance without reference answers. ARES instead generates in-domain synthetic data, fine-tunes lightweight judges for the three dimensions, and uses a small human-labeled set with prediction-powered inference to correct estimates and produce confidence intervals (Saad-Falcon et al., 2024).

ARES trains domain-oriented judges from synthetic examples, then calibrates system scores with human labels and prediction-powered inference. Image: Saad-Falcon et al. (2024), ARES, CC BY 4.0.

ARES reported that it required 78% fewer annotations than its sampled-annotation baseline while producing more accurate rankings. It found about 150 human annotations to be the minimum useful validation size in its tested settings. However, its limitations matter: specialized domains still need expert annotators, and drastic shifts such as English-to-other-languages or text-to-code required new in-domain data.

RAGChecker performed a meta-evaluation against human pairwise preferences. For overall assessment, its metric reached Pearson/Spearman correlations of 0.6193/0.6090, compared with 0.4831/0.5723 for the strongest reported RAGAS baseline in that comparison. Human annotator correlation was still higher at 0.7009/0.6889. The paper displays these values on a 0–100 scale; I normalize them here to 0–1. Automated evaluation improved, but did not reach the human ceiling.

More generally, MT-Bench found that strong LLM judges could exceed 80% agreement with human preference, roughly matching inter-human agreement in its setting. The same work documented position, verbosity, and self-enhancement biases (Zheng et al., 2023). Therefore:

• swap answer order in pairwise judging and measure preference flips;
• separate style from factuality in the rubric;
• require claim-level evidence, not a single holistic score;
• calibrate against domain-human labels;
• pin judge model, version, prompt, temperature, and decoding settings;
• report agreement and confidence intervals, not just judge scores.

Worked Example: A Regulatory RAG Evaluation

The following policy excerpts are synthetic. They are designed to expose evaluation behavior, not to state real law.

Corpus Snapshot

ID	Status	Excerpt
C1	Current, authoritative	"Covered institutions retain transaction logs for five years from transaction completion."
C2	Current, authoritative	"Earlier deletion requires release of all active legal holds and documented authorization from the records officer."
C3	Superseded	"Transaction logs shall be retained for three years."
C4	Current but irrelevant	"Customer contact records must be corrected upon a verified request."
C5	Commentary, non-authoritative	"Many teams obtain manager approval before routine deletion."

Query: What is the current transaction-log retention period, and when may logs be deleted earlier?

Gold claims:

• G1: Normal retention is five years.
• G2: Early deletion requires release of all active legal holds.
• G3: Early deletion requires documented records-officer authorization.

Forbidden claims: three-year retention; manager approval is sufficient; legal-hold release alone is sufficient.

Four Runs

Run	Retrieved top-3	Generated answer	Main failure
A	C3, C5, C4	"Three years; manager approval permits early deletion."	Retrieval selected superseded and non-authoritative text
B	C1, C2, C4	"Five years; legal-hold release permits early deletion."	Generator omitted the authorization condition
C	C1, C2, C3	Full gold answer with citations to C1 and C2	Correct despite a dangerous distractor
D	C4, C5; C1 and C2 absent from corpus snapshot	"The available sources do not establish the rule."	Correct abstention

For retrieval, a chunk is relevant only if it entails a gold claim. The calculations are:

Run	Claim recall	Context precision	Why
A	$0/3=0.00$	$0/3=0.00$	None of the current gold claims is supported
B	$3/3=1.00$	$2/3=0.67$	C1 and C2 jointly cover all claims
C	$3/3=1.00$	$2/3=0.67$	Same coverage, plus a superseded distractor
D	N/A	N/A	Corpus coverage failed before retrieval could succeed

Now evaluate generated claims:

Run	Claim precision	Claim recall	Faithfulness	Forbidden hit	Correct behavior?
A	$0/2=0.00$	$0/3=0.00$	$2/2=1.00$	Yes	No
B	$2/2=1.00$	$2/3=0.67$	$2/2=1.00$	Yes	No
C	$3/3=1.00$	$3/3=1.00$	$3/3=1.00$	No	Yes
D	N/A	N/A	1.00 to available context	No	Yes

Run A is the critical counterexample: faithfulness is perfect while correctness is zero. The generator faithfully repeated bad retrieval. Run B proves that perfect retrieval claim recall does not imply a complete answer. Run D proves that an answer-only metric can punish the safest behavior when evidence is genuinely absent.

For high-risk applications, define strict end-to-end acceptance:

$$\mathrm{Pass}(q)=1\left[G_q\subseteq Y_q\wedge Y_q\cap F_q=\varnothing \wedge \mathrm{CitationsValid}(Y_q)\right]$$

where $G_q$ is the required claim set, $Y_q$ the generated claim set, and $F_q$ forbidden or superseded claims. For unanswerable cases, the pass condition changes to correct abstention with no unsupported substantive claim.

Build the Evaluation Set Before Choosing the Metric

The query set determines what a score means. Questions generated directly from source chunks often leak the source vocabulary, overrepresent simple lookups, and guarantee that an answer exists. They make retrieval look easier than production.

A stronger benchmark combines real usage with controlled stress cases. Store enough structure to diagnose failures:

query
corpus_snapshot_id
answerability
gold_claims
supporting_evidence_spans
acceptable_alternative_sources
forbidden_or_outdated_claims
query_type
difficulty
temporal_scope
risk_level

Recommended slices include lookup, synthesis, comparison, multi-hop, temporal, ambiguous-entity, unanswerable, hard-negative, noisy, and adversarial questions. Report each slice separately. A system can improve average F1 by getting easy lookups right while regressing on the rare high-risk queries that motivated the system.

Three test modes serve different purposes:

• Production-derived queries estimate real utility and expose authentic vocabulary.
• Expert-authored cases cover safety boundaries, forbidden facts, and domain nuances.
• Synthetic perturbations isolate causal factors such as context order, document deletion, contradiction, and hard-negative similarity.

The benchmark must also freeze its corpus snapshot. A gold answer evaluated against a newer or older corpus silently changes answerability and makes model comparisons invalid.

Measure Uncertainty, Cost, and Drift

A one-point improvement is not evidence unless it is larger than measurement noise. Evaluate candidate systems on the same queries, bootstrap the paired score differences, and report a confidence interval:

$${\Delta }_i=s_i^{(B)}-s_i^{(A)},\bar{\Delta }=\frac{1}{n}\sum _{i=1}^n{\Delta }_i$$

If a 95% bootstrap interval for $\bar{\Delta }$ spans zero, the experiment does not establish that B is better. This matters even more when an LLM judge adds prompt and decoding variance. Repeat a calibration subset across judge prompts or runs and report agreement.

Quality also lives on a cost frontier. For each configuration, log:

• retrieval, reranking, and generation latency separately;
• input and output tokens;
• monetary cost per query;
• context size and number of sources;
• timeouts and fallback frequency;
• abstention, escalation, correction, and incident rates.

Version the corpus, chunker, embedding model, index, reranker, prompt, generator, and judge. Otherwise an evaluation cannot be reproduced and a regression cannot be localized.

A Practical Evaluation Matrix

Layer	Offline metric	Controlled proof test	Production signal
Corpus	answer coverage, freshness	delete or supersede the gold source	unresolved-query rate
Chunking	evidence-span containment	split a rule from its exception	source-inspection failures
Retriever	claim recall, context precision, NDCG	hard negative and multi-hop query	source opens and downstream use
Reranker	evidence position, NDCG	move gold evidence across ranks	useful-context rate
Generator	claim F1, faithfulness, utilization	contradictory and noisy context	corrections and escalations
Citation	correctness and completeness	relevant-but-non-entailing source	citation opens and disputes
Abstention	selective accuracy, false refusal	remove all answer evidence	unsupported-answer incidents
System	strict pass rate, task success	end-to-end frozen benchmark	latency, cost, user resolution

A minimum serious evaluation should therefore contain:

1. An end-to-end task metric aligned with actual acceptance criteria.
2. Claim-level retrieval coverage and context precision.
3. Claim correctness, completeness, and faithfulness.
4. Citation entailment and coverage.
5. Unanswerable, contradictory, temporal, multi-hop, and hard-negative slices.
6. Human calibration of automated judges.
7. Paired confidence intervals plus latency and cost.

Takeaways

• RAG evaluation is difficult because the final response is generated by a chain of interacting components.
• Correctness, faithfulness, completeness, relevance, citation quality, and abstention measure different properties; none safely substitutes for the others.
• Traditional Recall@k and MRR are useful but inherit assumptions about chunk boundaries and gold documents. Claim-level evidence gives a more stable target.
• More retrieved context increases coverage and can increase faithfulness, but also introduces noise, positional failure, latency, and cost.
• A long context window is capacity, not proof of utilization. Sweep evidence position and context length.
• LLM judges make iteration feasible, but they require human calibration, bias checks, versioning, and uncertainty estimates.
• The best benchmark starts from real usage, then adds controlled perturbations that expose the failures production logs have not yet shown.

The goal is not to produce the largest dashboard. It is to create an evaluation in which every important failure changes at least one metric, and every metric points to an engineering action.

References

1. P. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS, 2020. arXiv
2. V. Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering," EMNLP, 2020. arXiv
3. S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, "Ragas: Automated Evaluation of Retrieval Augmented Generation," 2023, revised 2025. arXiv
4. J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia, "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems," NAACL, 2024. arXiv
5. D. Ru et al., "RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation," 2024. arXiv
6. J. Chen, H. Lin, X. Han, and L. Sun, "Benchmarking Large Language Models in Retrieval-Augmented Generation," AAAI, 2024. arXiv
7. C. Niu et al., "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models," 2024. arXiv
8. T. Gao, H. Yen, J. Yu, and D. Chen, "Enabling Large Language Models to Generate Text with Citations," EMNLP, 2023. arXiv
9. N. F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," TACL, 2023. arXiv
10. L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS Datasets and Benchmarks, 2023. arXiv
11. S. Min et al., "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation," EMNLP, 2023. arXiv
12. T. Zhang et al., "BERTScore: Evaluating Text Generation with BERT," ICLR, 2020. OpenReview
13. C. D. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval. Cambridge University Press, 2008. Online
14. S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling, "Corrective Retrieval Augmented Generation," 2024. arXiv

Citation

If you want to reference this note:

Ma'ruf, Muhammad Rifqi. "Why RAG Evaluation Is Hard." rifqimaruf.dev (2026). https://rifqimaruf.dev/writing/why-rag-evaluation-is-hard/