Self-host vs API cost calculator for LLMs
Should you run your own model or pay per token? Enter your traffic, model, hardware and costs. See the monthly bill for each option, where they cross, how it plays out over time and which assumptions to verify first.
Start from a scenario, then change anything
1. Your workloadHow much traffic, how big each request is
Prompt, system message and retrieved context.
How much busier is your busiest hour than the average?
Share of input tokens served from a prefix cache, on both options.
Work nobody waits for. Gets API batch discounts and fills quiet GPU hours.
Used in the multi-year projection.
2. Model and latencyWhat you would run, and how fast it must feel
Weight precision
KV cache precision
Maximum time per output token for each user (50 ms is about 20 tokens per second). Tighter targets shrink batches and raise cost.
3. API pricingWhat you pay today or would pay
Many providers charge a fraction for cached prompt tokens.
Support plan, committed spend or platform fee.
4. Self-host hardware and pricingGPU, how you pay for it, and how well it is used
Sets prefill speed.
GPUs per node link
How you pay
Illustrative default; use your quote.
Share of peak bandwidth real engines achieve during decode.
Share of peak FLOPS achieved in prefill.
5. Operations and riskPeople, redundancy and everything around the GPUs
Capacity
Redundancy
Spare capacity so queues stay short.
Fraction of a person running, upgrading and being on call.
Build, load test, security review.
Must data stay in your own environment?
Verdict
Too close to call on cost alone
The two options are within about 11% of each other. Decide on control, privacy, latency and the effort you are willing to carry.
Self-host
$13k
per month · $1.42 per M tokens
API
$12k
per month · $1.27 per M tokens
Break-even
≈ 5.8M req/mo
Payback
Month 9
Monthly tokens
9.4B
What the money buys
Self-host $13k
API $12k
Things to check
- watchAverage GPU utilisation is only 24%. You pay for idle capacity: autoscale, batch more work, share GPUs between models, or consider the API.
- noteThis assumes an open model you host reaches the same quality as the API model on your task. Test that with your own evaluation before comparing prices.
Where the lines cross
Monthly cost against volume. The self-host line steps up each time another GPU replica is needed.
■ self-host■ API
36-month outlook with 5% monthly growth
Cumulative spend, with setup paid up front.
Total over the period: self-host $645k, API $1.15M.
What moves the answer most
Each assumption changed one at a time, on its own.
API prices
Monthly requests
Quantisation
Output tokens per request
Engineering time
Input tokens per request
GPU price
Prompt cache hit rate
Bars show how the monthly gap (self-host minus API) moves from today's +$1,378 when one assumption changes. Longest bars are the assumptions to verify first.
How the self-hosted setup was sized
Decode is memory-bound, prefill is compute-bound; see the method below.
- GPUs per replica
- 2
- Replicas at peak
- 4 (includes 1 spare)
- GPUs at peak
- 8
- GPUs billed on average
- 4.0
- Weights in memory
- 66 GiB
- KV cache per request
- 563 MiB
- Memory pool per replica
- 144 GiB
- Concurrent requests that fit
- 135
- Batch used under latency target
- 135
- Time per token per user
- 35.4 ms (28 tok/s)
- Decode throughput per replica
- 3,818 tok/s
- Prefill throughput per replica
- 6,308 tok/s
- Requests per second per replica
- 4.08
- Average and peak load
- 2.00 and 7.99 req/s
- Average utilisation
- 24%
- Hours billed per month
- 730
How to use it
- Start from a scenario that looks like your product, or enter your own requests per month and tokens per request.
- Replace the prices. The API and GPU prices are illustrative placeholders. Enter your provider's current rates and your real GPU quote.
- Set the latency target your users need. Tighter targets mean smaller batches and more GPUs.
- Read the verdict, then the details: the break-even chart, the multi-year outlook and the list of assumptions that move the result most.
- Share it. The link keeps your inputs, so you can send a scenario to your team.
How the model works
- Memory: weights at your chosen precision, plus KV cache for each concurrent request, decide how many GPUs one replica needs ( VRAM math).
- Decode speed: each step reads the weights and the KV cache from memory, so bandwidth sets tokens per second. The batch grows until your latency target is reached ( roofline). Mixture-of-experts models read only the experts a batch touches ( MoE).
- Prefill speed: compute-bound; prompt caching skips the repeated part.
- Replicas: requests per second at peak, with headroom and optional redundancy, give the GPU count. Autoscaling bills the average, fixed capacity bills the peak.
- Cost: GPUs (on-demand, reserved or bought and amortised with power and hosting), engineering time, tooling, network, and setup spread over a horizon.
- API cost: fresh input, cached input and output tokens at your prices, with batch discounts on the offline share.
When self-hosting tends to win, and when an API does
Self-hosting tends to win when
- Volume is high and steady, so GPUs stay busy.
- A good open model meets your quality bar, ideally a small or specialised one.
- Much of the work is offline and can fill the GPUs flat out.
- Data must stay in your environment, or you need control over latency and versions.
- You already run GPU infrastructure and have people who can operate it.
An API tends to win when
- Volume is low, uncertain or very spiky.
- You need a frontier model that no open model matches on your task.
- You are still changing the product, and speed to market matters more than unit cost.
- You have no one to run and be on call for GPU infrastructure.
- The provider's caching and batch discounts already cut your effective price.
Many teams use both: an API for hard or rare requests and a self-hosted small model for the high-volume, narrow ones ( gateways and routing).
Frequently asked questions
Is it cheaper to self-host an LLM or use an API?
It depends on volume and on how busy you can keep the GPUs. At low or spiky volume an API is usually cheaper, because you pay only for tokens and nothing for idle hardware or the people running it. As volume grows and GPUs stay well used, the cost of a GPU is spread over more tokens and self-hosting can win. This calculator finds the break-even volume for your own numbers.
At what volume does self-hosting an LLM become cheaper?
There is no single number. The break-even point depends on the API price per million tokens, the model you would host, the GPU price, how much traffic arrives at peak, your latency target and how many engineer-hours you count. The chart in the calculator shows both costs across a wide range of volumes, so you can see where the lines cross and how the answer changes when you change one assumption.
What hidden costs does self-hosting have?
Engineering time to deploy, upgrade and be on call; spare capacity for redundancy and traffic peaks; monitoring and logging; storage and network egress; load testing and security review; and the cost of idle GPUs at night and on weekends. The calculator includes each of these as an explicit input so they are not forgotten.
How does GPU utilisation change the cost per token?
Cost per token is the hourly cost of the hardware divided by the tokens it actually produces. If a GPU is busy a third of the time, each token costs about three times as much as at full use. Batching, autoscaling, sharing GPUs between models and moving non-urgent work to an offline queue are the main ways to raise utilisation.
How do prompt caching and batch APIs affect the comparison?
Both sides can benefit. Many API providers charge less for cached input tokens and offer discounts for batch jobs that can wait. A self-hosted engine with prefix caching skips repeated prefill work, and offline jobs can fill quiet GPU hours. The calculator lets you set a cache hit rate, an offline share and the provider's discounts.
Do I need several GPUs to run a 70B model?
Often yes. A 70B model needs about 140 GB of weights in 16-bit, about 70 GB in 8-bit and about 35 to 40 GB in 4-bit, plus memory for the KV cache that grows with every concurrent request. On 80 GB GPUs an 8-bit 70B model usually needs two GPUs per replica to leave room for batching. The calculator sizes this from the model, the precision and your context lengths.
How accurate is the calculator?
It is a planning tool, not a quote. It uses physical models (decode is limited by memory bandwidth, prefill by compute) with efficiency factors you can edit, and prices that are illustrative placeholders you should replace with real quotes. Always benchmark your own model and traffic before committing. The sensitivity chart shows which assumptions matter most, so you know what to verify first.
Does the calculator send my numbers anywhere?
No. Everything runs in your browser. The page address updates with your inputs so you can share a scenario by copying the link, and nothing is stored on a server.
Estimates only. Prices are illustrative placeholders and change often; confirm current prices, benchmark your own workload and test quality before deciding. Specs are approximate public figures.