Companion videos

Cost-optimizing inference — companion videos

The article is about the small set of techniques that actually move LLM bills — prompt caching, model routing, output-length control, batching — and how to combine them without quietly breaking quality. Cost-optimization talks on YouTube are mostly product demos, but these two cover the core mechanic (prompt caching) and the full lever set with enough technical depth to be useful in a production review.

Primary pick

18:50

Is This the End of RAG? Anthropic's NEW Prompt Caching

Prompt Engineering

Walks through Anthropic's prompt caching against Gemini's context caching with concrete latency-and-cost reductions per use case (long-document chat, few-shot, multi-turn). The breakdown of cache-write surcharge vs. cache-read discount is exactly what the article assumes when it talks about when caching pays off.

What you should get from this: You can estimate when prompt caching pays off by weighing cache-write surcharges against read savings for your real workloads.

Watch or know first: Working knowledge of LLM API pricing and long-prompt workloads.

AI Expert note: Recorded in August 2024, so the model names and per-token prices are dated. The cache-write-surcharge vs. cache-read-discount mechanic still holds; check current provider pricing pages before running the numbers for your workload.

Open video page

Also worth watching

56:03

Build Hour: Prompt Caching

OpenAI

OpenAI's own Build Hour on prompt caching — the 1024-token threshold, the prefix-stability requirement, steep audio-caching discounts for realtime (check the current pricing page for the exact rate), time-to-first-token impacts at long inputs. Useful when you are sizing the engineering effort to actually hit the cache reliably on your production prompts.

What you should get from this: Use prompt caching only when stable prefixes, latency and cost behavior match the workload.

Watch or know first: You should already be shipping prompts against the OpenAI API — the session is about hitting the cache reliably in production.

AI Expert note: Well under our view bar but it's the official deep dive; the 1024-token threshold and prefix-stability rules decide whether caching works for you at all, and current rates live on the pricing page.

Open video page