The pitch is compelling: rent or own accelerators, serve an open-weight model, and replace a variable API bill. But models are not automatically equivalent, GPUs do not stay fully utilized, and the serving stack becomes your security and reliability responsibility.
This is a cost-modeling guide, not a benchmark. Review the current vLLM documentation, vLLM security guidance, SGLang documentation, and Hugging Face TGI documentation before selecting a server. Benchmark supported versions on the target hardware.
The reality is more complex. Self-hosting genuinely wins at certain scales. At others, the operational cost dwarfs the inference savings. The break-even point varies by workload, model size, latency requirements, and team capability.
This article goes deep on the math, the operational realities, and the patterns that distinguish teams who should self-host from teams who shouldn’t. We assume you’re considering this seriously and want concrete numbers.
When self-hosting makes sense
Some characteristics that favor self-hosting:
Scale and utilization. Sustained, predictable demand can amortize reserved capacity. Use the measured hourly load distribution; monthly API spend alone is not a break-even test.
Predictable workload. Steady, predictable usage. Self-hosting requires capacity planning; spiky workloads waste capacity (under-utilization) or fail (over-saturation).
Privacy / compliance requirements. Data that can’t be sent to cloud providers (regulated industries, certain government contracts, internal-only data).
Custom models. Fine-tunes, custom architectures, or specialized variants that managed providers don’t offer.
Latency control. Hosting closer to users and controlling batching may help latency, but network, queueing, model size, prompt length, and load still dominate. Benchmark the target percentile.
Cost per call below break-even. When you’re doing the math and self-hosting genuinely wins.
When most of these are true, self-hosting is worth serious consideration.
When self-hosting doesn’t make sense
The other side. Characteristics that favor managed APIs:
Low or variable scale. Idle capacity and peak provisioning can erase apparent token-price savings. Managed inference may fit better, but run both models.
Spiky workloads. Large differences between peak and quiet periods can leave reserved capacity idle or require costly peak provisioning.
Need a closed-model capability. Some current models, modalities, safety systems, or hosted tools are available only through a provider service. If a workload evaluation requires one of them, include the provider path and its contract constraints rather than substituting an unevaluated open model.
Small team. Self-hosted inference requires operational expertise. Without dedicated capacity, things break.
Rapid iteration. Trying many different models, configurations, providers. APIs make this easy; self-hosting makes each change a deployment.
Multi-region / global users. Self-hosting may require regional capacity, routing, data-transfer, and recovery work. Managed APIs can reduce some infrastructure ownership, but region availability, residency, failover, and network latency still require verification.
For these cases, managed APIs may remain the lower-risk or lower-ownership candidate even when their direct usage cost is higher.
The cost math (carefully)
Build three current, quality-equivalent scenarios: closed-model API, managed open-weight serving, and self-hosting. Do not compare models until they pass the same task evaluation.
For each scenario, calculate:
monthly total = metered inference or dedicated capacity + storage + network + observability + support + security/compliance + engineering + expected incident loss
For metered APIs, inference comes from billed uncached input, cache writes/reads, output/reasoning tokens, tools, batches, and retries. For self-hosted or managed dedicated capacity, the accelerator cost pool must include utilized, idle, headroom, rollout, and failure/fallback capacity; do not add a second “inference” charge unless the contract has a separate mutually exclusive meter. Use benchmarked requests per accelerator-hour at the required latency and availability to allocate that capacity cost to workload units—not the vendor’s peak throughput.
Run sensitivity analysis on volume, peak-to-average ratio, model quality, accelerator price, utilization, staff time, and migration cost. The break-even point is where the quality-equivalent net costs cross under a plausible range of assumptions.
The operational cost
Beyond raw inference cost, the operational cost of self-hosting.
Initial setup:
- Picking the right inference server (vLLM, TGI, SGLang).
- Configuring for your model and hardware.
- Setting up GPU infrastructure (cloud or owned).
- Networking, security, observability.
- Quantization and optimization.
Estimate initial delivery from a scoped work breakdown, hardware/provider lead time, security review, benchmark matrix, availability design, and the team’s observed throughput. This article does not assert a transferable engineer-week range.
Ongoing operations:
- Monitoring (latency, throughput, errors, GPU utilization).
- Capacity planning.
- Upgrades (new model versions, inference server updates, security patches).
- Incident response (GPU failures, OOM crashes, software bugs).
- Scaling (more GPUs as load grows).
Record actual ownership across platform, ML, security, and on-call work. A fractional-FTE assumption is not portable across organizations.
Hidden costs:
- GPU price volatility.
- Cloud egress costs if hybrid.
- Specialty expertise (CUDA, quantization, optimization).
- Replacement / failure costs for owned hardware.
Use the organization’s fully loaded labor rates and opportunity cost. Token-price savings are not net savings until operational ownership is included.
The inference servers
If you’re going to self-host, the main options:
vLLM. An open-source serving engine with continuous batching and broad, version-dependent model support. Treat its security guidance as required reading.
TGI (Text Generation Inference). Hugging Face’s serving project. Verify current maintenance status and model support rather than relying on this article’s snapshot.
SGLang. A serving and programming stack under active development. Benchmark the required models, structured-output path, and operational tooling.
LMDeploy. Another serving candidate with version-dependent model, quantization, and hardware support; benchmark it under the same acceptance tests.
llama.cpp / Ollama. Candidates for local and some server workloads. Supported hardware, concurrency, security boundary, operational controls, and production fit must be tested; neither name guarantees lower throughput or production readiness.
Managed dedicated endpoints. Hugging Face and other providers offer managed endpoint products whose engines, billing, isolation, and operational split change over time. Verify the current service rather than assuming it runs TGI or uses one billing model.
Managed GPU or inference platforms. These can reduce some capacity and serving work while retaining integration, security, evaluation, and provider dependencies. Compare current quotes and responsibilities; do not assume a fixed price relationship with self-operation.
Shortlist only projects that support the exact model, accelerator, quantization, API contract, and security controls. A reproducible load test decides among them.
Hardware choices
The GPU question:
Accelerator generations, memory capacities, and rental prices change quickly. Obtain current quotes for the required region and commitment term.
Compare memory capacity and bandwidth, supported numeric formats, interconnect, software compatibility, quota, regional availability, failure behavior, and price. Spot or preemptible capacity belongs in the model only with interruption handling and a measured recovery path.
Quantization
Quantization is one capacity/performance option. Its trade-offs depend on model, format, kernel, hardware, and task:
FP16/BF16-class reference precision. Often used as a comparison baseline for supported models and hardware; it is not automatically the model’s original or “full-quality” format.
INT8 / FP8 (8-bit). May reduce weight memory or improve supported execution paths; quality and speed effects vary.
INT4 (4-bit). Can reduce weight memory further; measure quality and kernel performance for the exact artifact.
AWQ, GPTQ, GGUF. Different quantization formats with different trade-offs.
Raw parameter-count arithmetic is only a lower-bound estimate; runtime memory also includes KV cache, activations, workspaces, fragmentation, and replicas. Use the serving engine’s profiler and a load test. Evaluate output quality on the production task, not only a general benchmark.
Throughput and capacity planning
A key planning question: how many tokens/second do you need?
Measure time to first token, inter-token latency, end-to-end latency, throughput, queue time, error rate, and memory headroom across prompt/output lengths and concurrency levels.
For capacity planning:
- Estimate peak concurrent requests.
- Estimate average request length.
- Calculate total tokens/second needed.
- Add headroom derived from burst, failure, and rollout requirements.
Reliability and fallback
Self-hosting means you own reliability.
Health checks. Continuous health monitoring. Restart unhealthy instances.
Overload behavior. Use bounded queues, admission control, backpressure, and a tested degradation or rejection policy. Letting requests wait indefinitely can worsen an overload and breach latency objectives.
Fallback to APIs. A managed fallback can absorb some outages or peaks, but only if model quality, data policy, contracts, rate limits, state, and failover behavior are compatible and tested. It adds complexity and can fail concurrently.
Failure capacity. Size spare capacity or an alternative path from the availability objective and tested failure modes; every accelerator can fail, but dedicated idle hardware is not the only design.
Multi-region. For global users, replicate. Or use managed APIs for distant regions.
Update strategy. New model versions, server upgrades. Blue-green deployments to avoid downtime.
Each of these is engineering work that managed APIs absorb for you.
Two decision records to produce
Do not invent an anonymized outcome. Produce auditable decision records from current quotes and benchmark artifacts.
Self-host candidate record:
- model artifact, revision, license, quantization, serving version, accelerator, region, and deployment topology,
- workload distribution and quality evaluation against the current hosted baseline,
- load-test command, dataset, latency/throughput results, saturation point, and recovery behavior,
- capital or rental cost, utilization, engineering effort, security work, and expected incident cost,
- break-even range with sensitivity analysis and an exit criterion.
Managed candidate record:
- provider, model/revision behavior, region, price sheet date, quotas, and contract terms,
- equivalent quality, latency, rate-limit, outage, and data-boundary evidence,
- migration effort and vendor concentration risk,
- conditions that would trigger another self-hosting evaluation.
When to revisit the decision
The decision isn’t permanent. Periodically revisit:
Volume changes. Up significantly: self-hosting more attractive. Down significantly: less attractive.
Pricing changes. Closed APIs getting cheaper or more expensive. Managed open getting cheaper. Hardware getting cheaper.
Model improvements. New open-weight or source-available candidates that meet the workload target, or new closed models that change the quality comparison. Verify licenses and actual availability.
Operational capacity. Team grew or shrunk in ML/ops capability.
Privacy / compliance changes. New requirements that mandate self-hosting.
Set a review cadence from contract, price, model, workload, security, and capacity volatility, and add event-driven triggers. A quarterly review is an example, not a universal default.
Common mistakes
Patterns we see in self-hosting decisions:
Mistake 1: Cost math without operational cost. Token or accelerator savings are reported while engineering, on-call, security, and incident costs are omitted.
Mistake 2: Self-hosting too early. Spending engineering effort on self-hosting when the workload is small. Optimization premature.
Mistake 3: Comparing non-equivalent quality. A lower-cost model is selected without demonstrating that it meets the workload’s quality, safety, and latency requirements.
Mistake 4: No failure plan. Self-hosted infrastructure goes down without a tested degradation, queue, rejection, or fallback path. Managed APIs can also fail; compare both architectures against the same availability objective.
Mistake 5: Assuming the first serving configuration is efficient. No reproducible sweep is run across supported quantization, batching, concurrency, prompt lengths, and server settings, so the capacity model rests on an unverified configuration.
Mistake 6: Ignoring quality drift. Self-hosted model has degraded vs current closed. Customers notice; team doesn’t.
Mistake 7: Not reconsidering. Once self-hosting, never re-evaluating. The decision might have been right two years ago and wrong now.
Mistake 8: Spot/preemptible without graceful handling. Discounted capacity is modeled without interruption frequency, recovery time, duplicate work, or fallback cost.
A decision checklist
To make the decision deliberately:
- Quality-equivalent hosted and self-hosted candidates have been benchmarked?
- Workload is steady and predictable?
- Team has or can hire MLOps/inference expertise?
- An appropriately licensed open-weight/source-available candidate exists and meets the workload’s quality and safety target?
- Latency requirements compatible with self-hosting?
- Have done detailed cost math including operational costs?
- Have a fallback plan?
- Compliance/privacy requirements don’t mandate one path?
- Is there a documented review cadence and event-driven triggers for material model, price, contract, workload, security, or capacity changes?
Do not reduce this to a checkbox count. Security, model quality, or operational ownership can be a veto even when every financial input looks favorable.
Hybrid patterns
It is not all-or-nothing. Candidate hybrid patterns include:
Self-hosted for the bulk; APIs for the hard cases. Classification, simple generation on self-hosted; complex reasoning on closed APIs.
Self-hosted for steady; APIs for spikes. Self-hosted handles base load; APIs absorb peaks.
Self-hosted for sensitive; APIs for general. Sensitive data through self-hosted; general queries through APIs.
Self-hosted for fine-tunes; APIs for base. Custom models run yourself; off-the-shelf models from APIs.
Hybrid adds routing, data-policy, evaluation, observability, contract, and failure complexity. Adopt it only when tests show that the split improves an explicit objective.
Decide on current evidence
Self-hosting is a viable architecture when a supported model meets the workload’s quality target and the organization can own the serving lifecycle.
The break-even point is not a universal monthly spend. It changes with model quality, demand shape, utilization, accelerator and provider prices, availability, data-boundary requirements, and staff cost.
Evidence supporting a self-host decision should show:
- Have done the math carefully, including operational costs.
- Have or can build MLOps capability.
- Run at sufficient scale to justify the investment.
- Have steady workloads.
- Don’t need frontier-only capabilities.
- Plan for reliability, monitoring, and updates.
Evidence favoring a managed-service decision may include:
- Lower scale.
- Spiky workloads.
- Need rapid iteration.
- Small teams without ops capacity.
- Need frontier closed capabilities.
The right answer is specific to the workload. Run quality, load, failure, security, and cost comparisons and assess operational capacity. Prefer the lowest-ownership option that satisfies the mandatory requirements; that may be managed, self-hosted, or hybrid.
When self-hosting is selected, record the measured benefit, assumptions, owner, exit criteria, and next review trigger. Do the same for managed inference; neither path is correct without current evidence.
