Llama 3 70B vs Claude: how to choose an LLM for your product

“Should we self-host an open model or call a hosted API?” is one of the first real decisions on any product that leans on an LLM, and it gets argued badly — usually as open-versus-closed ideology, or as a cost spreadsheet that quietly ignores the salary of whoever keeps the GPUs alive. I want to lay out the tradeoff the way I actually weigh it, using Llama 3 70B as the stand-in for open-weights-you-host and Claude as the stand-in for a hosted frontier model. The categories generalise to whatever the current model names happen to be.

What you are actually choosing between

These are not two products competing on quality alone. They are two different operating models:

• Open weights, self-hosted (Llama 3 70B). You download the weights and run inference on your own hardware or a rented GPU. You own the model, the data path, and every operational problem that comes with serving it.
• Hosted frontier model (Claude). You send requests to an API and get responses. Someone else owns the hardware, the scaling, the upgrades, and the uptime. You rent capability by the token.

Almost every meaningful difference falls out of that one distinction. Let me walk the five dimensions I score every time.

1. Quality and capability

On raw capability, the frontier hosted models still lead — particularly on hard reasoning, long-context reliability, instruction following under pressure, and tool use, where the model has to decide when to call a function and thread the results back coherently. If your product depends on an agent orchestrating multiple tools without falling apart, that gap is real and it shows up in production.

But “which model is smarter” is often the wrong question. A 70B open model is more than good enough for a large share of real tasks: classification, extraction, summarization, routing, structured generation, retrieval-augmented answers over your own documents. If your job is bounded and well-specified, the marginal quality of a frontier model may buy you nothing a user would notice. Match the model to the task, not to the leaderboard.

2. Privacy and control

This is where self-hosting genuinely shines and where I have seen it decide the whole question. When you run the weights, prompts and completions never leave your boundary. For regulated data — health, finance, anything under strict residency or contractual constraints — “the data physically does not go to a third party” can be non-negotiable, and no amount of a vendor’s data-handling policy substitutes for it. Control also means version stability: a hosted model can change underneath you, while a self-hosted checkpoint is frozen until you choose to move.

3. Cost — where the naive math misleads

Per token, self-hosting looks dramatically cheaper, and at high, steady volume it genuinely is. But the honest comparison is not price-per-token against zero. It is the API bill against the fully loaded cost of running your own inference.

Hosted cost   = tokens x price_per_token         # scales linearly, near-zero fixed

Self-host cost = GPU_hours x hourly_rate         # you pay for capacity, not usage
               + idle_capacity_waste             # GPUs cost the same at 3am
               + engineer_time (serving, scaling, upgrades)
               + reliability + on-call

Crossover: self-hosting wins once sustained volume keeps
those GPUs busy enough to beat the API bill. Below that line,
you are paying full price for mostly-idle hardware.

The shape that matters: hosted cost is almost purely variable — you pay for exactly what you use and nothing at 3am. Self-hosted cost is mostly fixed — you rent capacity whether or not requests are flowing. At low or spiky volume the API wins comfortably. At high, predictable volume, a well-utilised GPU fleet wins, sometimes by a lot. The mistake is assuming you are on the winning side of that crossover before you have the traffic to be there.

4. Latency

Self-hosting gives you control over latency: no shared multi-tenant queue, the option to co-locate the model next to your application, and freedom to tune batching, quantization, and hardware for your specific traffic. Hosted APIs add a network hop and can vary under load, though in practice they are fast and streaming hides much of it. Latency rarely decides the question alone, but for tight interactive loops or on-prem or edge deployments, owning the serving path is a real lever.

5. Operational burden — the one that gets underestimated

This is the cost that never makes the pitch deck. Self-hosting a 70B model means GPU provisioning, an inference server (vLLM, TGI, and friends), autoscaling for spiky load, monitoring, model and dependency upgrades, and someone on call when it falls over at 2am. That is a standing commitment, not a one-time setup. A hosted API collapses all of it into an HTTP call and an SDK. For a small team, that difference can be the entire product timeline.

How I actually decide

My default for a new product is to start hosted. It gets you to a working, high-quality experience fastest, with no infra to babysit, and it lets you learn what your real workload even looks like — volume, prompt shapes, latency needs — which you rarely know up front. You cannot make a good self-hosting decision without that data.

Then I revisit. Move toward self-hosting when a concrete forcing function appears: a privacy or residency requirement the API cannot satisfy, sustained volume that pushes you past the cost crossover, or a need for control (custom fine-tunes, frozen versions, on-prem) that hosting cannot give. Absent one of those, the operational simplicity of a hosted model is usually worth more than the token-price savings — and a mature team often ends up routing: cheap self-hosted models for the bulk of simple, high-volume calls, and a frontier API reserved for the hard requests that actually need it. The real answer is rarely “pick one forever.”