A deep dive into how Shopify rebuilt the improvement loop for Sidekick, its AI agent, as a four-stage flywheel that keeps getting better without perpetually scaling frontier-model spend. We cover the business context (quality that gets expensive fast when you only buy it by upgrading models), and the technique: a human-calibrated quality rubric (validated with Cohen's kappa), an LLM-as-judge tuned with DSPy/GEPA/ACE and — crucially — validated against real production A/B outcomes before it's trusted, automated prompt/tool research, and continual learning via SFT plus GRPO fine-tuning that uses the calibrated judge itself as the reward signal. The payoff: serving cost for a GraphQL agent handling 2,000 requests/min fell roughly  ...