Google — Beating Your Own Teacher at Tool Use
Note sull'episodio
Google's ToolGrad flips tool-use data generation: run real API calls first, then write the question they answer. A 12B model trained on just 500 examples scored 83.1 on the Berkeley Function Calling Leaderboard, beating GPT-5's 74.4.
Key takeaways:
- Answer first. Writing a query first and hunting for an API chain that solves it often fails. ToolGrad builds a valid chain of executed calls and writes the user query afterward, reaching a 99.8% annotation pass rate.
- Textual “gradients.” An API proposer suggests next calls, they run in parallel, and an LLM critic picks a branch with plain-language feedback that steers the next step.
- Small data, strong model. ToolGrad-500, built from ToolBench's 16k+ APIs, fine-tuned Gemma-3 into ToolGrad-12B. It scored 83.1, close to Gemini 2.5 Pro (83.2) and Claude 4.5 Opus (82.8).
- The student beats the teacher. The data came from Gemini 2.5 Flash-Lite, yet the fine-tuned 12B model outperformed it.
- Our take. Generation is grounded in real execution, so the data is clean by construction.
Chapters:
00:00 Shooting arrows blindfolded
00:47 From query-first to answer-first
02:19 Why query-first fails
03:50 Execution as ground truth
04:36 The API proposer
05:22 Parallel execution
06:07 Textual gradients and the critic
08:24 ToolGrad-500 and ToolGrad-12B
09:55 Benchmark results
11:28 Beating the teacher
13:01 Related work
14:34 Our take
16:07 Recap
Sources:
- ToolGrad: Efficient tool-use dataset generation with textual “gradients” (Google Research blog, 2026)
Related episode: E8: Airbnb — The LLM Eval Loop That Went From Weeks to a Day
Enjoyed it? Follow Data Science in the Wild (one real production ML/AI system or paper per episode, hosted by Peter Liu) and rate it on Apple Podcasts or Spotify.
Also from Peter: The GenAI Evolution Atlas, Wednesday AI deep dives and Friday news roundups.
Made with AI-generated voices; curated by Peter Liu from the sources above.