1:46:33Lenny's Podcast
Hamel Husain and Shreya Shankar walk through the entire eval workflow on a real property-management AI assistant — looking at traces, open and axial coding of errors, deciding when to stop, building an LLM-as-judge, and validating it against human judgment. This is the rare long-form conversation that is genuinely aimed at PMs and team leads rather than ML engineers, and complements the article's requirement for an owned review cadence tied to changes, incidents and observed failures.
What you should get from this: Learn the product-builder eval loop: inspect traces, label failures, define criteria, test changes and compare against human judgment.
Watch or know first: Have at least one AI workflow where quality can get better or worse over time.
AI Expert note: This remains one of the best non-engineer eval explanations because it focuses on workflow discipline rather than a specific tool. Keep the method, then choose tooling that fits your privacy and observability constraints.
