Self-Hosted Models: Cost, Infrastructure, and When to Run Your Own

Running self-hosted models means managing an end-to-end inference stack yourself — the GPU hardware, the engine that executes the weights, and the memory budget both live on — instead of paying a third party per token.

You move from buying consumption to buying capacity.

You should make that move when your organization's sensitive data cannot legally leave its jurisdiction, or when utilization gets high enough that owning a token pool around the clock beats per-token billing. The teams that succeed at this are not the ones chasing a cheaper invoice. They are the ones whose compliance requirements left no alternative, or whose utilization is genuinely high enough to justify the hardware.

A large language model running on a GPU cluster inside a company's own infrastructure, set against the alternative of calling a third-party API over the internet

What does self-hosting an LLM actually mean?

Self-hosting a large language model is the operational practice of managing an end-to-end inference stack, which means taking on full responsibility for weight matrices, static and dynamic memory allocation, and the physical constraints of VRAM — the memory that lives on the GPU itself. Unlike an API that abstracts compute into a service, running your own model means managing a finite VRAM "container". If the static model weights and the dynamic cache exceed the physical High Bandwidth Memory (HBM) capacity, the system triggers an Out-of-Memory (OOM) error and crashes.

The performance of this stack is defined by two phases in a request's lifecycle: prefill and decode. During prefill, the engine reads the entire input prompt in parallel. This phase is primarily memory-bandwidth bound; Time-To-First-Token (TTFT) is determined by how fast HBM can stream weights to the compute cores. The decode phase is an autoregressive loop that generates tokens one at a time. While single-sequence decoding is memory-bound because you must load the entire model for a single token, modern engines use continuous batching to interleave requests, which turns decode into a compute-bound operation limited by the raw TFLOPS of the GPU.

To prevent fragmentation from killing concurrency, you need PagedAttention, which treats VRAM like virtual memory and uses block tables to map tokens to non-contiguous physical blocks. Without it, the key-value (KV) cache that holds conversation history grows linearly until it exhausts VRAM, producing a concurrency collapse where throughput nose-dives despite available compute.

Standard infrastructure maintenance for a self-hosted stack includes:

Updating GPU drivers and CUDA kernels to maintain compatibility with new inference engines.
Tuning the engine's scheduler for first-come-first-served or priority-based policies.
Managing model quantization (FP8, INT4, AWQ) to optimize the weight-to-cache ratio.
Monitoring thermal throttling and HBM saturation during high-concurrency bursts.

Why do engineering teams take on running a model?

The primary driver is absolute legal and operational control over data. You must distinguish between three tiers: data residency (where data sits), data localization (legal mandates for data to stay in-country), and data sovereignty (which nation's laws apply). An American cloud provider's data center in Frankfurt offers residency but not sovereignty. Under the US CLOUD Act, US authorities can compel American-incorporated companies to hand over data stored anywhere in the world, which conflicts directly with GDPR Article 48.

Three concepts people confuse: data residency, data localization and data sovereignty, with the control spectrum running from hyperscaler to sovereign cloud to on-premise

Regulatory frameworks add weight to the same argument. EU AI Act Article 10 mandates documented data governance for high-risk AI systems, including those used in HR, credit scoring, and critical infrastructure, and enforcement for these rules began on 2 August 2026. Breaches of the high-risk obligations that Article 10 falls under carry administrative fines of up to €15 million or 3% of global annual turnover; the higher €35 million or 7% tier is reserved for the prohibited practices set out in Article 5. Self-hosting on sovereign infrastructure — a market that reached $80 billion in 2026, growing 35.6% year over year — removes foreign legal authority over your inference traffic entirely.

Operationally, self-hosting handles the spiky demand that comes with coding work. Developer token usage typically peaks mid-afternoon and drops to near zero overnight. Relying on external APIs means hitting rate limits during those peaks. Owning the hardware removes the ceiling, and it lets you eliminate stranded silicon by scheduling automated agents to soak up idle capacity at 3:00 AM for tasks like bug-report triage or re-indexing a vector database.

There is a sharper version of the privacy argument too. Inference is not data storage, it is active data processing, so every API call is a restricted international transfer under GDPR Chapter V. In RAG and agentic systems, context accumulates continuously: long multi-turn conversations and thousands of embedded internal documents mean the volume of sensitive material leaving your perimeter grows with usage rather than staying flat.

What layers make up a self-hosted stack?

A self-hosted inference stack begins with the hardware layer, ranging from deskside development units to rack-scale nodes. At the entry level, a DGX Spark workstation provides enough HBM for single-user work or testing small models. For production scale, the standard is a high-end accelerator like the NVIDIA H200 (141GB HBM3e) or the 8×B200 rack node (1.5TB total HBM). Next-generation models like Kimi K3, which carries 1.4TB of weights, need the higher capacity of an 8×B300 node (2.3TB HBM) to leave headroom for the KV cache.

The four layers of a self-hosted stack: GPU hardware, the inference engine with its scheduler and KV cache manager, the serving layer, and the AI gateway, alongside the prefill and decode phases

The second layer is the inference engine, such as vLLM or SGLang, which manages the model executor and the scheduler. The model executor handles forward passes on the GPU, while the scheduler determines which requests enter the next engine step. You choose between first-come-first-served policies for fair access and priority-based policies for critical production traffic. The engine uses PagedAttention block tables to map logical token sequences to physical VRAM blocks, keeping the KV cache manageable even as context windows expand toward 128k tokens.

The third layer is the serving layer, the web scaffolding for online access. This consists of API servers, often exposing OpenAI-compatible endpoints, and load balancers that distribute requests across replicas. In a distributed setting, a data-parallel coordination layer tracks which engine core has the lowest load, calculated from waiting and running requests. As one GPU reaches its cache limit, new traffic routes to available silicon elsewhere in the cluster.

The final layer is the AI gateway, the policy enforcement point between the application and the model. It handles redaction of personally identifiable information (PII), zero-data-retention enforcement, and the consolidated audit logs that regulatory compliance requires. Placing the gateway in front of the engine means that when model weights change or a new engine is deployed, your security and sovereignty standards stay consistent across every AI-driven workflow.

How much GPU and VRAM does serving a model need?

Sizing begins with the static weight footprint: parameters in billions multiplied by bytes per parameter. Precision is the primary variable, with FP16 at 2 bytes, FP8 at 1 byte, and INT4 at 0.5 bytes. A 70B model in FP16 requires 140GB of VRAM just to load the weights. Since an H100 offers 80GB, the model physically cannot fit on a single device at that precision, so tensor parallelism across at least two GPUs becomes a feasibility requirement rather than a performance choice.

The VRAM budget for a 70B model: static memory for weights at FP16, FP8 and INT4, plus the dynamic KV cache that grows with context length and concurrency, plus hidden runtime overhead

Beyond weights, you must calculate the dynamic KV cache. For a 70B model the field rule of thumb is roughly 0.35 MB per token at FP16. Supporting 10 concurrent users with a 32k context window works out to 32,000 × 10 × 0.35 MB, or 112GB of VRAM. Added to the 140GB for weights, the total requirement reaches 252GB. Failing to account for this dynamic growth is the most common cause of cache traps, where a model loads successfully and then crashes the moment a user submits a long prompt.

You must also reserve for the hidden VRAM tax. The CUDA context, activation buffers, and runtime overhead typically consume 4–5 GB, and if your calculation leaves less headroom than that the system will crash during intermediate tensor calculations. Enabling prefix caching relieves some of the pressure by stopping the engine from recomputing identical prompt prefixes, such as a shared system instruction, across requests.

When weights exceed a single GPU's capacity, tensor parallelism shards the weight matrices across devices. This pools VRAM but introduces a communication tax, because GPUs must synchronize after every layer. If a model can be quantized to fit on a single GPU, it will generally outperform a multi-GPU setup of the same weights, because there is no interconnect latency to pay. You are balancing the precision loss of quantization against the latency penalty of distributed execution.

Which engine: vLLM, SGLang, or Ollama?

The engine determines the throughput ceiling and concurrency limits of your stack. vLLM is the production standard, using PagedAttention and continuous batching to interleave prefill and decode. It supports speculative decoding, where a smaller draft model guesses tokens that the larger model verifies, and it comfortably handles 32 to 48 concurrent users before KV-cache exhaustion forces a plateau.

Comparing three inference engines: vLLM for production serving at high concurrency, SGLang for new-generation models on large clusters, and Ollama for local development and quick testing

SGLang is the leading alternative, optimized for high-performance models like GLM-5.2 and Kimi K3. Where vLLM is a general-purpose workhorse, SGLang leans on speculative decoding methods like EAGLE to cut inter-token latency, which pays off most on prefix-heavy traffic where the same system prompt or retrieved context is reused across requests.

Absolute throughput matters less than matching the engine to the model and the workload. GLM-5.2 on an 8×B200 rack under SGLang saturates at roughly 175 tokens/sec across 16 concurrent users, because a near-frontier model produces a much smaller token pool than its hardware suggests. DeepSeek-V4-Flash on 4×H200 under vLLM exceeds 1,700 tokens/sec at high concurrency on considerably cheaper hardware. Bigger models come with smaller pools, and the hardware does not fully compensate.

Ollama is excellent for local development and single-user testing, but it is not built for production concurrency. It lacks the scheduler logic and distributed orchestration needed to serve a team at once. On entry-level hardware, a DGX Spark running Qwen3.6-35B struggles past one or two concurrent users and returns a stream of timeouts. Once more than about 30 people depend on the endpoint, vLLM or SGLang is the only realistic choice.

Is self-hosting really cheaper than an API?

The economics are governed by utilization thresholds, and the answer is usually no. A 4×H200 node must stay roughly 89% busy, around the clock for five years, before it beats the token pricing of an efficient API like DeepSeek-V4-Flash. You size for the peak and pay for it 24/7, and a GPU sitting idle bills exactly the same as a saturated one.

Break-even by utilization: a 4xH200 node must stay extremely busy to beat a low-cost API, while an 8xB200 rack needs only low utilization to beat frontier APIs

The picture inverts at the top of the quality range. A B200 rack needs only about 15% utilization to beat frontier API pricing, because frontier APIs charge a steep premium for reasoning quality while the hardware cost of running a near-frontier open-weight model is fixed. At rental rates, that rack delivers frontier-quality task completions for roughly $1.11 each — cheaper than either API invoice. The break-even depends far more on which model tier you need than on your raw token count.

Cost volatility is its own argument, separate from the average. Median annual AI spend per employee sits near $140, but the 90th percentile approaches $7,300 and the 99th nears $90,000. Teams have reported per-developer costs jumping from $29 to $750 a month as agentic workflows ramped up. Fixed hardware converts that tail risk into a predictable line item.

The size of the win also depends heavily on which model you run, and most cost calculators leave that out. Renting a GPU came out around 35 times cheaper than the API for a small model like Qwen3.6, while for DeepSeek-V4-Flash the same comparison ran the other way — coding agents cache up to 98% of input tokens, and that efficiency is already priced into the API you would be replacing.

When is self-hosting the wrong call?

Self-hosting is wrong for organizations with low aggregate demand or without the engineering capacity to manage a kernel and driver stack. If median utilization is under 15%, it will cost more than an API. An improperly tuned vLLM instance delivers worse inter-token latency than a commercial endpoint, which turns an infrastructure win into a developer-experience loss.

Concurrency collapse is the failure mode that surprises people. Around 48 to 64 concurrent users the KV cache exhausts and throughput nose-dives rather than plateauing, as the scheduler struggles to balance incoming prefill prompts against the memory needs of running decode sequences. If you need high-concurrency access without the overhead of multi-node load balancing, a managed API is the safer choice.

Quality regressions are the second trap. Aggressive INT4 quantization breaks reasoning on math, finance, and code generation, so those workloads belong at FP16 or FP8 — which puts the hardware requirement straight back up. Speed is the third: a self-hosted Kimi K3 runs about 8 times slower than a Claude Code baseline, and GLM-5.2 on an 8×B200 rack is already 3 times slower at just 8 concurrent sessions.

Finally, there is release cadence. Open-weight models trail the frontier by several months, and the gap is narrower than it used to be — GLM-5.2 resolves 62.5% of SWEBench Pro tasks, matching Claude Opus 4.8 exactly on that set — but a benchmark tie is not parity across every reasoning nuance your workload will hit. If you need the best available reasoning on day zero, an open-weight model you host yourself will not deliver it.

Where should you start?

Do not buy hardware first. Measure your actual workload through cheap open-weight APIs to establish which model quality you genuinely need and what your real token volume looks like, then check that number against your peak-hour concurrency rather than your monthly total. Renting a GPU before buying one gives you the same measurement with none of the capital risk.

Buy GPUs for what an invoice cannot price — control, privacy, and latency — not for cost alone. If your data cannot legally leave your jurisdiction, that decision is already made for you and the economics are secondary. If it can, and you cannot keep the hardware busy, the honest answer is that an API is still cheaper.