Loading
154 minutesInstrumenting & Evaluating LLMs
Hamel Husain, Eugene Yan, Brian Bischof, Harrison Chase, and Shreya Shankar working through tracing, log analysis, LLM-as-judge, and the workflow around looking at real production data.
Watch nextDave Ebbelaar. A working AI engineer walking through his actual eval ladder — assert-style unit tests, reference-free metrics, LLM-as-judge alignment with humans, and the analyze/measure/improve loop. The structure is the closest match on video to the article's argument that evals are a regression-catching system, not a leaderboard.
Some tool choices will age, but the ladder is sound: deterministic checks first, then model-graded checks validated against humans. Do not skip calibration just because an LLM judge is easy to add.
Design an eval ladder that catches regressions before prompt or model changes reach users.
Experience shipping or maintaining an AI workflow with known failure examples.
Continue through the same learning path with the next curated companion videos.
