English

Topic

Production LLM Apps

Architect, ship, observe, and operate LLM applications after the demo stage.

34 stories (23 articles · 11 videos)

Start here

A few good first pieces before you browse the full feed.

The 2026 LLM stack: models, inference, tooling, and trade-offs

A working architect's view of the 2026 LLM stack — the model tiers, inference providers, orchestration layers, evaluation tooling, and the trade-offs that actually matter when shipping production AI. Everything you wish someone had laid out before you started.

Designing prompts for production: system, developer, and user layers

A governance pattern for separating trusted instructions, runtime data, and user input, then versioning, evaluating, deploying, and observing prompts according to risk.

Production AI failure modes: what breaks after the demo

Build a failure-mode register for hallucination, stale context, prompt injection, unsafe tool use, schema drift, weak fallback, and observability gaps.

More in this topic

Hermes vs n8n: choose by the job (and when to use both)

A decision framework for Hermes Agent versus n8n: deterministic plumbing stays in n8n, while a bearer-authenticated Hermes API step or a separate event-webhook path handles bounded agent work.

OpenClaw vs Hermes: choose by the job, not the brand

OpenClaw and Hermes are overlapping self-hosted agent systems with different operational strengths. Compare channel routing, webhooks, tools, and automation boundaries before choosing either or both.

Experimental dual-DGX Spark + DeepSeek-V4-Flash + n8n + Hermes stack

How to evaluate an experimental community dual-DGX Spark path for DeepSeek-V4-Flash, with n8n on the deterministic edge and Hermes on the judgment path.

The Agent Landscape - Lessons Learned Putting Agents Into Production

MLOps.community. Prosus's VP of AI and an AI engineer report what actually broke when they deployed agents across the group's portfolio companies: prompt-injection pen-testing before launch, an unsafe write when a Jira agent choked on human shorthand, stale context handled by making agents surface their assumptions, and fallback design that merged or killed agents once they added cognitive load. It reads like the article's failure-mode register replayed as a live postmortem.

Choosing an agent framework in 2026: a reproducible bake-off

Compare agent frameworks against one representative workflow, explicit operational requirements, and an exit-cost review instead of relying on popularity or opinion.

Designing agents that don't loop forever

Infinite or pseudo-infinite loops are a costly agent failure mode. This guide shows how to bound work, detect lack of progress, and terminate safely.

Computer-use agents: a production-readiness evaluation

Evaluate a narrow computer-use workflow with runtime-enforced scope, human approval, independent result checks, security testing, and measured unit economics.

Context engineering: testing long-context systems without guesswork

Large context windows are capacity limits, not quality guarantees. Build position, distractor, retrieval, latency, and cost tests for the workload you actually run.

Cost-optimizing inference: prompt caching, routing, and output control

Build a trace-based inference cost model, optimize the largest measured contributors, and prove that each change preserves task quality.

Building evals that actually catch regressions

Most eval suites look impressive but miss real regressions. Building evals that catch what matters requires careful dataset construction, sensitive metrics, judge calibration, and a culture of trust. The patterns from teams that get this right.

Fine-tuning in 2026: an evidence-first LoRA and QLoRA experiment

Decide whether parameter-efficient tuning is justified, govern the data, pin a reproducible experiment, compare held-out and safety results, and benchmark serving before deployment.

MCP server design in TypeScript: from minimal server to production review

Build a minimal Model Context Protocol server, then review the design for schemas, authorization, idempotency, observability, deployment, and security before calling it production-ready.

Designing MCP tools that LLMs actually use correctly

Most MCP tools we see are technically correct and practically useless. LLMs ignore them, misuse them, or call them in unhelpful ways. The principles for designing tools LLMs adopt naturally, with examples of common failures and their fixes.

Building memory for long-running agents

Long-running agents need an owned persistence design: provenance, confirmation, tenant isolation, retrieval tests, retention, correction, and verifiable deletion.

Multi-model orchestration: routing by cost, latency, and quality

Routing different tasks to different models can reduce cost or latency, but only workload-specific evaluation can show whether the added complexity pays off. The patterns, measurements, and failure modes.

Observability for LLM apps: tracing, costs, latency, quality drift

Extend ordinary observability with multi-step traces, attributable cost, prompt and model versions, evaluated quality signals, privacy controls, and workload-derived alerts.

Production RAG design: ingestion, retrieval, reranking, and evaluation

A production RAG pipeline is six stages, each with specific patterns that determine quality. The architecture, the choices at each stage, and the iterative evaluation discipline that distinguishes RAG that works from RAG that disappoints.

Prompt injection and LLM security: threat models and defense-in-depth

Prompt injection is a permanent LLM security class, not a prompt-writing mistake. A production guide to threat models, data boundaries, tool permissions, regression tests, monitoring, and incident response.

Choosing between prompting, RAG, and fine-tuning (and when to combine)

Prompting, RAG, and fine-tuning are the three big levers for adapting LLMs to your problem. Each is right for some problems and wrong for others. A framework for choosing, the realistic costs of each, and the production patterns where combining them shines.

LLM product unit economics: an evidence-first pricing worksheet

Model usage distribution, contribution margin, failure handling, support, and retention before choosing a price. This worksheet replaces unsupported market ranges with auditable inputs.

Structured outputs and function calling: the production patterns

Structured outputs and function calling are the bridge from 'LLM that generates text' to 'system that does work'. In production, the patterns that matter are about schemas, error handling, idempotency, and graceful degradation — not just JSON mode.

Deep Dive into LLMs like ChatGPT

Andrej Karpathy. This is the clearest end-to-end explanation on YouTube of what an LLM actually is — pretraining, tokenization, SFT, RLHF, reasoning RL, tool use, hallucinations — at the level of detail an engineer needs to reason about model trade-offs. Watch it once and the "GPT-class vs. open-weights vs. reasoning model" decisions in the article stop feeling like brand choices and start feeling like training-recipe choices.

Andrej Karpathy: Software Is Changing (Again)

Y Combinator. Karpathy's AI Startup School keynote frames LLMs as a new kind of computer — utility, fab, and OS rolled together — and argues for "partial autonomy" products with a human-controlled leash. It is the cleanest articulation of the stack-level mental model the article assumes: that you are picking inference vendors and tooling for a programmable substrate, not a chatbot.

LangSmith in 10 Minutes

LangChain. A guided tour of an LLM trace, project, and dataset by LangChain's co-founder — token cost, latency, error rate, feedback aggregation, drilling into a single retrieval-step span. It's the closest visual analogue to what the article describes when it talks about "every call is a span" and why structured traces beat print logging.

Instrumenting & Evaluating LLMs

Hamel Husain. Hamel Husain, Eugene Yan, Brian Bischof, Harrison Chase, and Shreya Shankar working through tracing, log analysis, LLM-as-judge, and the workflow around looking at real production data. Sit with it the same way you would a long podcast — it is the single best deep treatment of the article's "look at your traces" thesis on YouTube.

AI prompt engineering: A deep dive

Anthropic. Four Anthropic prompt engineers (research, alignment, applied, developer relations) talking at length about what they actually do day to day — how they edit prompts under pressure, how they think about "honesty" in instructions, when XML scaffolds help, when they don't. The article's layered model maps cleanly onto how they describe the work; this is the best way to hear that mental model out loud.

Prompting 101 | Code w/ Claude

Anthropic. A live build session by Anthropic's Applied AI team on an insurance-claims prompt — they start with a vague instruction and iterate to something a developer would actually ship, showing the kind of revisions the article describes for the system and developer layers. Watch this before re-reading the article's checklist on examples, output structure, and refusal handling.

Vertical AI Agents Could Be 10X Bigger Than SaaS

Y Combinator. The Lightcone hosts work through why vertical AI agents — not horizontal wrappers — are the defensible shape for application-layer companies, with concrete examples and a clear-eyed take on which categories the model providers will eat. That is the anti-moat trap the article warns about, expressed as a positive playbook.

How AI is Reinventing Software Business Models ft. Bret Taylor of Sierra

Sequoia Capital. Bret Taylor walks through the shift from per-seat SaaS to outcomes-based pricing — what to anchor on (resolution, CSAT, NPS), why incumbents struggle to follow, and how vertical specialisation creates pricing power. It directly mirrors the article's pricing and margin sections.

OpenAI DevDay 2024 | Structured outputs for reliable applications

OpenAI. Walks through `strict: true`, the difference from old JSON mode, refusal handling, and how function calling and response-format schemas compose. Useful precisely because it describes the contract the API gives you, which is what the article's production patterns are built on top of.

Pydantic is all you need: Jason Liu

AI Engineer. The talk that crystallised the modern "define a Pydantic model, hand it to the LLM, let validation do the rest" pattern, with concrete examples of nested objects, validators that catch hallucinated URLs, and Chain-of-Thought as a typed field. Watch it before re-reading the article's section on validators and you will recognise where its retry and refusal rules come from.