Evals, Observability & Quality

48 minutes

Video

How to Build Reliable AI Agents (Context + Evals Explained) | Tobias Leong, Axium

Arize AI. Explains why production agents fail when the system lacks the right context, evaluation data, tracing and domain expertise. It maps well to the article's failure-mode register because it makes reliability an engineering loop: separate retrieval from reasoning, define expected outcomes, evaluate tool calls, and trace failures before changing models.

Advanced

10 min read

Article

Production AI failure modes: what breaks after the demo

AI systems usually fail in predictable ways: hallucination, stale context, sycophancy, prompt injection, unsafe tool use, schema drift, and weak fallbacks. A production failure-mode register for teams shipping real workflows.

Build a production AI failure-mode register with controls for hallucination, stale context, prompt injection, unsafe tool use, and weak fallbacks.

Advanced

11 min read

Article

Chunking, reranking, and hybrid search: make RAG actually work

Most RAG implementations work poorly because they get three things wrong. A practical guide to chunking documents, reranking results, and combining keyword with semantic search — without becoming a search engineer.

Evaluate the implementation pattern, failure modes, and guardrails before building.

Intermediate

154 minutes

Video

Instrumenting & Evaluating LLMs

Hamel Husain. Hamel Husain, Eugene Yan, Brian Bischof, Harrison Chase, and Shreya Shankar working through tracing, log analysis, LLM-as-judge, and the workflow around looking at real production data. Sit with it the same way you would a long podcast — it is the single best deep treatment of the article's "look at your traces" thesis on YouTube.

Advanced

9 minutes

Video

LangSmith in 10 Minutes

LangChain. A guided tour of an LLM trace, project, and dataset by LangChain's co-founder — token cost, latency, error rate, feedback aggregation, drilling into a single retrieval-step span. It's the closest visual analogue to what the article describes when it talks about "every call is a span" and why structured traces beat print logging.

Advanced

109 minutes

Video

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation

Stanford Online. Methodical pass through rule-based metrics, LLM-as-judge biases, factuality and agent evaluation, and the failure modes of static benchmarks. Use it as the theory companion to the article's section on choosing what to measure and why most off-the-shelf metrics under-predict real regressions.

Advanced

55 minutes

Video

How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)

Dave Ebbelaar. A working AI engineer walking through his actual eval ladder — assert-style unit tests, reference-free metrics, LLM-as-judge alignment with humans, and the analyze/measure/improve loop. The structure is the closest match on video to the article's argument that evals are a regression-catching system, not a leaderboard.

Advanced

3 minutes

Video

Evaluate prompts in the Anthropic Console

Anthropic. A three-minute Anthropic walkthrough of running a real eval inside the Workbench — auto-generating realistic test cases, grading outputs, tweaking the prompt, and re-running the same suite side-by-side. The view count sits below the usual bar, but for "how do I actually do this without writing code" this is the cleanest official demo and slots neatly under the more strategic Husain/Shankar conversation.

Intermediate

107 minutes

Video

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

Lenny's Podcast. Hamel Husain and Shreya Shankar walk through the entire eval workflow on a real property-management AI assistant — looking at traces, open and axial coding of errors, deciding when to stop, building an LLM-as-judge, and validating it against human judgment. This is the rare long-form conversation that is genuinely aimed at PMs and team leads rather than ML engineers, and it covers the same "30 minutes a week after setup" rhythm the article recommends.

Intermediate

Evals, Observability & Quality

Start here

Evals for non-engineers: know if your AI workflow is getting better or worse

Building evals that actually catch regressions

Observability for LLM apps: tracing, costs, latency, quality drift

More in this topic

How to Build Reliable AI Agents (Context + Evals Explained) | Tobias Leong, Axium

Production AI failure modes: what breaks after the demo

Chunking, reranking, and hybrid search: make RAG actually work

Instrumenting & Evaluating LLMs

LangSmith in 10 Minutes

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation

How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)

Evaluate prompts in the Anthropic Console

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar