How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)
Dave Ebbelaar. A working AI engineer walking through his actual eval ladder — assert-style unit tests, reference-free metrics, LLM-as-judge alignment with humans, and the analyze/measure/improve loop. The structure is the closest match on video to the article's argument that evals are a regression-catching system, not a leaderboard.
AI Expert note
Some tool choices will age, but the ladder is sound: deterministic checks first, then model-graded checks validated against humans. Do not skip calibration just because an LLM judge is easy to add.
What you should get from this
Design an eval ladder that catches regressions before prompt or model changes reach users.
Watch or know first
Experience shipping or maintaining an AI workflow with known failure examples.
Watch next
Continue through the same learning path with the next curated companion videos.
154 minutesInstrumenting & Evaluating LLMs
Connect tracing, evaluation, feedback and production review into an operating loop for LLM systems.
Watch next
104 minutesBuilding Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic
Understand how MCP fits into agentic systems beyond a local demo server.
Watch next
29 minutesPrompting for Agents | Code w/ Claude
Decide which behavior belongs in the system prompt, tool description or tool precondition.
Watch nextRelated videos
4 minutesIntroducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings
32 minutesHow to Build Human-Centered AI Workflows in Localization with Shashi Bhushan
59 minutesFrom Hype to Habit: How Tech Companies Are Scaling AI Beyond the Experimental
41 minutesPrivate AI vs. Cloud: How Enterprise Leaders Can Make Smarter Build-or-Buy Decisions
Take it further
Hand-picked external courses that go deeper on this topic.
Coursera · DeepLearning.AIAI for Everyone
Six years after it launched, still the cleanest starting point for anyone who needs to understand AI without learning to code. No math, no jargon, no hype — you'll finish able to have an informed conversation about AI projects.
Coursera · The Wharton SchoolAI Strategy and Governance
Wharton's rigorous framing for executives making build-vs-buy decisions. Cuts through vendor pitches by focusing on the economics of AI deployment, algorithmic bias in hiring and operations, and the governance practices that survive an audit. Best taken before, not after, your next major AI procurement decision.