Topic
Evals, Observability & Quality
Measure AI behavior, catch regressions, trace cost and latency, and keep workflows improving.
12 stories (5 articles · 7 videos)
Start here
A few good first pieces before you browse the full feed.
10 min readArticle
Evals for non-engineers: know if your AI workflow is getting better or worse
Evals — systematic measurement of AI output quality — are usually treated as an engineering concern. But every team running AI workflows needs them, and the basics are accessible without code. The how-to.
Measure whether an AI workflow is improving by using examples, rubrics, and regression checks.
Intermediate
13 min readArticle
Building evals that actually catch regressions
Most eval suites look impressive but miss real regressions. Building evals that catch what matters requires careful dataset construction, sensitive metrics, judge calibration, and a culture of trust. The patterns from teams that get this right.
Evaluate the implementation pattern, failure modes, and guardrails before building.
Advanced
12 min readArticle
Observability for LLM apps: tracing, costs, latency, quality drift
LLM applications fail in unique ways that traditional observability misses. The patterns for tracing multi-step flows, tracking costs that vary 100x per call, monitoring quality drift, and debugging hallucinations at production scale.
Evaluate the implementation pattern, failure modes, and guardrails before building.
Advanced
More in this topic
48 minutesVideo
How to Build Reliable AI Agents (Context + Evals Explained) | Tobias Leong, Axium
Arize AI. Explains why production agents fail when the system lacks the right context, evaluation data, tracing and domain expertise. It maps well to the article's failure-mode register because it makes reliability an engineering loop: separate retrieval from reasoning, define expected outcomes, evaluate tool calls, and trace failures before changing models.
Advanced
10 min readArticle
Production AI failure modes: what breaks after the demo
AI systems usually fail in predictable ways: hallucination, stale context, sycophancy, prompt injection, unsafe tool use, schema drift, and weak fallbacks. A production failure-mode register for teams shipping real workflows.
Build a production AI failure-mode register with controls for hallucination, stale context, prompt injection, unsafe tool use, and weak fallbacks.
Advanced
11 min readArticle
Chunking, reranking, and hybrid search: make RAG actually work
Most RAG implementations work poorly because they get three things wrong. A practical guide to chunking documents, reranking results, and combining keyword with semantic search — without becoming a search engineer.
Evaluate the implementation pattern, failure modes, and guardrails before building.
Intermediate
154 minutesVideo
Instrumenting & Evaluating LLMs
Hamel Husain. Hamel Husain, Eugene Yan, Brian Bischof, Harrison Chase, and Shreya Shankar working through tracing, log analysis, LLM-as-judge, and the workflow around looking at real production data. Sit with it the same way you would a long podcast — it is the single best deep treatment of the article's "look at your traces" thesis on YouTube.
Advanced
9 minutesVideo
LangSmith in 10 Minutes
LangChain. A guided tour of an LLM trace, project, and dataset by LangChain's co-founder — token cost, latency, error rate, feedback aggregation, drilling into a single retrieval-step span. It's the closest visual analogue to what the article describes when it talks about "every call is a span" and why structured traces beat print logging.
Advanced
109 minutesVideo
Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation
Stanford Online. Methodical pass through rule-based metrics, LLM-as-judge biases, factuality and agent evaluation, and the failure modes of static benchmarks. Use it as the theory companion to the article's section on choosing what to measure and why most off-the-shelf metrics under-predict real regressions.
Advanced
55 minutesVideo
How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)
Dave Ebbelaar. A working AI engineer walking through his actual eval ladder — assert-style unit tests, reference-free metrics, LLM-as-judge alignment with humans, and the analyze/measure/improve loop. The structure is the closest match on video to the article's argument that evals are a regression-catching system, not a leaderboard.
Advanced
3 minutesVideo
Evaluate prompts in the Anthropic Console
Anthropic. A three-minute Anthropic walkthrough of running a real eval inside the Workbench — auto-generating realistic test cases, grading outputs, tweaking the prompt, and re-running the same suite side-by-side. The view count sits below the usual bar, but for "how do I actually do this without writing code" this is the cleanest official demo and slots neatly under the more strategic Husain/Shankar conversation.
Intermediate
107 minutesVideo
Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Lenny's Podcast. Hamel Husain and Shreya Shankar walk through the entire eval workflow on a real property-management AI assistant — looking at traces, open and axial coding of errors, deciding when to stop, building an LLM-as-judge, and validating it against human judgment. This is the rare long-form conversation that is genuinely aimed at PMs and team leads rather than ML engineers, and it covers the same "30 minutes a week after setup" rhythm the article recommends.
Intermediate