Choose a hosted API or a self-hosted model

A hosted API runs the model and the hardware and charges per token. Self-hosting runs open weights on nodes you rent or own, paid per node-hour, busy or not. Managed open-weight endpoints sit in between.

Decide per workload: a live feature and an overnight job can land on different sides. First write down each workload's goal, traffic shape, latency target or deadline, data rules, and quality bar. When the options use different models, the choice bundles which model and who runs it, so settle quality before cost.

Run both models on your own suite with the same prompts and grading, and report every slice (a group of cases that matters on its own, such as a language family) as well as the overall rate. An option that fails the gate for a workload is out for it, whatever it saves. A gap of a few cases on a slice of a few hundred can be noise: confirm a narrow pass with a larger paired evaluation or a shadow run, while a large gap on a small slice is still evidence.

Hosted cost follows use: requests x tokens x price. Self-hosted cost follows provisioned time: node-hours x price, busy or not, plus people. Utilization is average load over the measured capacity you pay for: a $3.60-per-hour node serving 3,600 requests per hour costs $0.001 per request fully busy, $0.002 half busy.

An interactive pool is sized for its peak plus headroom and a spare and runs all day, so the average-to-peak ratio caps its utilization. A batch job that can wait runs nodes only while there is work: its cost tracks the work, and the deadline sets how many nodes run at once.

Hosted, the provider's shared capacity shapes your tail latency and rate limits cap your peak; you can ask for a higher limit or buy reserved or priority capacity at a price, but you do not control the provider's shared pool.

Self-hosted, you own the queue and admission, so you can protect a latency target, but only with capacity already running: a node takes minutes to start, more quota can take weeks. Take throughput per node from a pilot on your own prompts and runtime.

A data rule asks two separate questions: where processing happens, and who keeps copies of inputs and outputs, for how long. Providers often keep API traffic for a limited time for abuse monitoring, grant zero retention only on approval, and process in-region in some regions only (as of November 2026, OpenAI's data controls page describes all three), so check your exact endpoint and model.

Self-hosting is not compliant by itself: logs, traces, caches, backups, and evaluation copies must follow the same rule, and the open-weight license must allow your use.

Hosted models retire on the provider's schedule (as of November 2026, Anthropic's deprecations page, for example, gives at least 60 days' notice for publicly released models). Each retirement is a migration: rerun your suite on the successor, recheck price, latency, and limits, then roll out. Self-hosted weights stay pinned, but runtime, driver, and security updates are yours.

Customization (adapters, quantization, hosted fine-tuning) is a lever, not a verdict. For exit, keep prompts, output contracts, and the suite provider-neutral; OpenAI-compatible servers such as vLLM narrow interface changes, not behavior differences.

Self-hosting adds upgrades, patches, capacity planning, monitoring, incidents, and on-call. Google's SRE book calls manual, repetitive work that grows with the service "toil" and aims to keep it below 50% of each engineer's time. Count people time as a fixed monthly line, often shared by every self-hosted workload.

A rotation needs several trained people with runbooks (the book's minimum single-site on-call team is eight), so one experienced person is a single point of failure. Hosted APIs need less operations work.

Workloads with different traffic shapes, quality results, or data rules can land on different options; each extra path adds a suite, monitoring, and a runbook. A fallback in either direction may take only traffic it is allowed and able to take: same quality gate, same data rule, capacity running when needed. Route by data class before falling back.

Put both options for one workload in one table with a row per dimension: quality gate by slice, cost at real utilization and the breakeven, latency and capacity, data handling for every copy, lifecycle and operations, and exit and fallback. Fill each row with the scenario's figures, then give a recommendation per workload, the evidence that would settle what is uncertain, and the change that would reverse it.

Work every figure before you compare.

Use the district rule and Option H's terms.

Start from your own recommendation per workload.