Choose a hosted API or a self-hosted model

The team asks whether each workload should stay on the hosted API, move to the self-hosted open-weight model, or split. For the tutor and for essay scoring, state the workload's goal and constraints, compare both options on the same dimensions (quality gate, monthly cost at real utilization and the breakeven, latency and capacity, the district rule, lifecycle, and operations), and recommend one option per workload. Name the evidence that would settle what is still uncertain and the change that would reverse each recommendation.

All figures are synthetic; the provider, models, and prices are fictional. Months have 30 days (720 hours).

Suppose a language-learning app runs two AI workloads on a hosted model API, and a school district's contract starts in 10 weeks. Should each stay hosted or move to an open-weight model on the company's own GPU nodes?

District rule. 40% of the essays come from the district. They must be processed in its region, no model provider may keep an essay or its output after the request, and the company's own copies (logs, traces, backups) must stay in that region. The tutor is not covered.

Option H: hosted API, pinned model version H-M

• $0.20 per million input tokens, $2.60 per million output tokens.
• Account limit for H-M: 1,500,000 tokens per minute (input plus output), shared by both workloads; higher limits on request, no committed date.
• Standard endpoint: any of the provider's regions. Regional endpoint: the chosen region only, at 15% more per token.
• Both endpoints keep inputs and outputs up to 30 days for abuse monitoring. Zero retention needs the provider's approval; the team applied two weeks ago and has no decision date.
• Versions retire with notice and a named successor.
• Tutor, last 30 days: daily p95 TTFT 500 to 650 ms. On 3 days the busiest hour's p95 reached 1.2 s, with all the extra time before the provider's first byte.

Option S: self-hosted open-weight model O

• O's license allows commercial use. Nodes in the district's region: $7.20 per node-hour, billed per minute while running; about 20 minutes to start one and load O; quota 6 nodes, about two weeks to raise.
• Two-week pilot on the company's prompts, per node: 10 tutor turns per second with p95 TTFT at or below 800 ms (450 ms at 7.5 per second); 5 essays per second with large batches.
• Tutor pool rule: the fewest nodes that keep the peak at or below 75% of their combined measured rate, plus one spare, running around the clock (a burst starts faster than a node can be added).
• Essay batch rule: plan at 80% of the measured rate (this covers start-up, retries, and uneven essay lengths); nodes run only during the batch, enough to finish within 6 of the 8 hours.
• People: any self-hosting takes about 0.4 of an engineer ($6,500 a month, one workload or both). One platform engineer has run GPU serving; the six-person on-call rotation would take serving pages after training.

Quality (the company's suites; same prompts and grading for both models)

Tutor pass: teachers rate the conversation 4 or 5 of 5. Essay pass: within one point of the teachers' score (6-point scale).

Quality gate. A model may replace H-M for a workload only if its pass rate is at most 2.0 percentage points below H-M's overall and at most 5.0 points below on every slice of that workload's suite.