Self-hosted vs hosted inference — companion videos

The article works through the actual break-even math between API providers and self-hosted vLLM/TGI — utilisation, KV-cache memory, GPU economics, on-call overhead — and tries to kill the cargo-cult version of "we'll save money by self-hosting." The vLLM authors' own talk is the cleanest technical foundation for why a serving engine matters at all.

Primary pick

32:07

Fast LLM Serving with vLLM and PagedAttention

Anyscale

Walks through why naive LLM serving wastes 60–80% of GPU memory, how PagedAttention borrows OS-style paging to fix that, and why continuous batching produces the 24× throughput numbers the article uses in its math. After this, the article's "you'll be lucky to hit 50% utilisation" line stops feeling abstract.

What you should get from this: Understand why serving engines, batching and KV-cache memory dominate self-hosted inference economics.

Watch or know first: Basic knowledge of transformer inference, GPU memory limits and API-vs-self-hosting tradeoffs.

AI Expert note: The PagedAttention mental model remains important, but throughput numbers and serving-engine defaults age quickly. Use this for fundamentals, then benchmark your own model, hardware, batch shape and uptime requirements before making a hosting decision.

Open video page