This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Mat Ryer of Grafana Labs about AI observability, production agents, evals, telemetry cost, guardrails, and what changes once AI moves beyond demos and into systems teams actually depend on.

Mat is Senior Director of AI at Grafana Labs, where he focuses on how AI fits into observability and production systems.

We talk about Grafana Assistant, why AI observability is not just logs, latency, and HTTP 200s, and how teams can measure whether agents are actually helping. Mat gets into evals, LLM-as-judge patterns, traces as a way to think about conversations, user feedback, tool choice, model changes, and the cost of collecting new telemetry.

We also dig into UX and tru ...