Companion videos

Self-hosted vs hosted inference — companion videos

The article works through the actual break-even math between API providers and self-hosted vLLM/TGI — utilisation, KV-cache memory, GPU economics, on-call overhead — and tries to kill the cargo-cult version of "we'll save money by self-hosting." The vLLM authors' own talk is the cleanest technical foundation for why a serving engine matters at all.

Primary pick

32:07
Fast LLM Serving with vLLM and PagedAttention

Anyscale

Walks through why naive LLM serving wastes 60–80% of GPU memory, how PagedAttention borrows OS-style paging to fix that, and how continuous batching changes throughput. This provides background for the article's quality-equivalent cost model. Measure capacity and utilisation for your own workload instead of transferring a benchmark figure.

What you should get from this: Understand why serving engines, batching and KV-cache memory dominate self-hosted inference economics.

Watch or know first: Basic knowledge of transformer inference, GPU memory limits and API-vs-self-hosting tradeoffs.

AI Expert note: The PagedAttention mental model remains important, but throughput numbers and serving-engine defaults age quickly. Use this for fundamentals, then benchmark your own model, hardware, batch shape and uptime requirements before making a hosting decision.

Open video page