Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation
Stanford Online. Methodical pass through rule-based metrics, LLM-as-judge biases, factuality and agent evaluation, and the failure modes of static benchmarks. Use it as the theory companion to the article's section on choosing what to measure and why most off-the-shelf metrics under-predict real regressions.
AI Expert note
Use this for evaluation theory, not as a product checklist. Translate the ideas into small tests tied to your own risk categories, user tasks and release process.
What you should get from this
Understand why benchmark scores, model judges and factuality metrics can fail to predict product regressions.
Watch or know first
Comfortable with technical lecture material and basic LLM evaluation terminology.
Watch next
Continue through the same learning path with the next curated companion videos.
154 minutesInstrumenting & Evaluating LLMs
Connect tracing, evaluation, feedback and production review into an operating loop for LLM systems.
Watch next
104 minutesBuilding Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic
Understand how MCP fits into agentic systems beyond a local demo server.
Watch next
29 minutesPrompting for Agents | Code w/ Claude
Decide which behavior belongs in the system prompt, tool description or tool precondition.
Watch nextRelated videos
4 minutesIntroducing EmbeddingGemma: The Best-in-Class Open Model for On-Device Embeddings
32 minutesHow to Build Human-Centered AI Workflows in Localization with Shashi Bhushan
59 minutesFrom Hype to Habit: How Tech Companies Are Scaling AI Beyond the Experimental
41 minutesPrivate AI vs. Cloud: How Enterprise Leaders Can Make Smarter Build-or-Buy Decisions
Take it further
Hand-picked external courses that go deeper on this topic.
Coursera · DeepLearning.AIAI for Everyone
Six years after it launched, still the cleanest starting point for anyone who needs to understand AI without learning to code. No math, no jargon, no hype — you'll finish able to have an informed conversation about AI projects.
Coursera · The Wharton SchoolAI Strategy and Governance
Wharton's rigorous framing for executives making build-vs-buy decisions. Cuts through vendor pitches by focusing on the economics of AI deployment, algorithmic bias in hiring and operations, and the governance practices that survive an audit. Best taken before, not after, your next major AI procurement decision.