Loading
154 minutesInstrumenting & Evaluating LLMs
Hamel Husain, Eugene Yan, Brian Bischof, Harrison Chase, and Shreya Shankar working through tracing, log analysis, LLM-as-judge, and the workflow around looking at real production data.
Watch nextStanford Online. Methodical pass through rule-based metrics, LLM-as-judge biases, factuality and agent evaluation, and the failure modes of static benchmarks. Use it as the theory companion to the article's section on choosing what to measure and why most off-the-shelf metrics under-predict real regressions.
Use this for evaluation theory, not as a product checklist. Translate the ideas into small tests tied to your own risk categories, user tasks and release process.
Understand why benchmark scores, model judges and factuality metrics can fail to predict product regressions.
Comfortable with technical lecture material and basic LLM evaluation terminology.
Continue through the same learning path with the next curated companion videos.
