Agentic RAG Evaluation — More Questions Beat More Repeats

With a fixed token budget for evaluating an agentic RAG system, should you ask more questions, sample more search trajectories, or re-grade answers? This paper finds that more questions win, cutting standard error by up to 33%.

Key takeaways:

• Split the noise. Generalizability theory separates evaluation variance into questions, search trajectories and repeated reads, tested on HotpotQA and MuSiQue.
• Go wide. At about 34 million tokens, broader question coverage lowered standard error by 33% compared with five reads, and by 12.6% compared with three trajectories.
• One read is usually enough. Reading each answer once cost only 0–9.9% in variance relative to the optimal allocation.
• Set the judge to temperature zero. It cut answer disagreement from 14.3% to 3.4%, with similar comparison precision.
• Plan ahead. Forecasts from a small pilot predicted the allocations within 3.5–4.0%, so you can budget before spending. Results are on academic benchmarks.

Chapters:

00:00 Wasted evaluation budgets

01:31 Why agentic RAG evals are noisy

03:05 Brute-force repeats

03:52 Generalizability theory

05:24 Judge disagreement and temperature

06:56 Spending 34 million tokens

08:30 Why one read is enough

10:01 Borrowing from A/B power analysis

11:32 Benchmark caveats

13:07 Planning your token spend

14:37 Recap

Sources:

• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (Jingjie Ning, Xueqi Li and Yibo Kong; arXiv, Oct 2026)

Related episode: E33: Token-Budgeted Escalation — Predicting Cost Is Easy, Predicting Benefit Is Hard

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.