Token-Budgeted Escalation — Predicting Cost Is Easy, Predicting Benefit Is Hard

Token budgets for RAG: which questions deserve extra retrieval? On FinanceBench, the cost of an extra call is easy to predict (R² 0.93) but its benefit isn't (AUROC 0.60). Gain-per-token escalation still uses 46.6% fewer tokens than top-5 everywhere.

Key takeaways:

• Escalate selectively. Every question gets a cheap top-1 answer. Only the calls with the best predicted gain per token get a top-5 retry.
• Cost is predictable; benefit isn't. Extra-call cost is predicted with R² 0.93. Spotting which questions will improve reaches only AUROC 0.60, where 0.5 is a coin flip.
• It still pays off. At a 10% budget, adjudicated quality is 0.034 higher than with gain-only ranking, using 46.6% fewer tokens than top-5 on every question.
• It's uplift targeting. Spend on “persuadable” queries that top-5 can fix. Skip the sure things and the lost causes.
• Our caveats. The test uses only 150 questions, and a weak predictor will starve some queries that needed more context.

Chapters:

00:00 Why every retrieved token has a price

01:26 The paper, and the cost of top-5 everywhere

04:01 FinanceBench setup and gain per token

06:10 Cost R² 0.93 vs. benefit AUROC 0.60

08:32 Results: +0.034 quality, 46.6% fewer tokens

09:43 What's new: uplift targeting

11:49 Related work on agentic RAG budgets

14:14 Limitations, and when triage makes sense

16:42 Recap: should we pay for uplift?

Sources:

• Token-Budgeted Escalation for Financial Document QA: Cost Is Predictable, Benefit Is the Bottleneck (Zhu, Yang, Ding, Qi; arXiv, Oct 2026)
• FinanceBench (benchmark)
• Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads (related)
• Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks (related)

Related episode: E32: Agentic RAG Evaluation — More Questions Beat More Repeats

Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.

Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.

Made with AI-generated voices; curated by Peter Liu from the sources above.