Design an A/B test with cost guardrails

Write the experiment contract for the staged-hints test, including the sample size and how long the test must run. Implement the per-arm summary so it validates one outcome and cost record per assigned learner and reproduces fixtures A and B. Then explain the data checks, uncertainty estimates, and quality and cost guardrails that must pass before the lift can support a ship decision, and review the teammate's draft analysis plan.

Suppose a coding-practice platform answers a hint request with one full explanation from a language model it calls. The team has built staged hints on the same model: a nudge, then a targeted hint, then one worked step. They want to know whether staged hints help learners solve problems on their own without raising cost or answer disclosure beyond agreed limits. All figures are synthetic.

Model cost is recorded per learner in integer microunits (millionths) of one declared currency, including retries, escalations, and failed attempts.

Write the contract before launch: hypothesis, eligibility, assignment unit, variants, primary outcome, guardrails with tolerances, observation window, statistical method, stopping rules, and decision owner. The smallest effect worth acting on sets the sample size and separates a real but trivial change from a useful one.

Randomize before the treatment can influence who is included, and keep each learner's assignment stable across sessions and devices.

Version the model, prompt, retrieval policy, and UI. If the treatment bundles several changes, the experiment estimates the bundle's effect and cannot isolate each part without additional design.

The function takes one row per assigned learner whose observation window has closed. success is true when the primary outcome occurred within the window. A learner who never came back has an observed outcome of false once the window closes; one whose events were lost, or whose window is still open, has a missing outcome, which is resolved upstream and never coded as false.

costMicrounits is the learner's total attributable model cost as an integer, including retries and failed attempts, and 0 for a learner who never asked for a hint. Integers avoid floating-point drift in currency totals.

type UserRecord = {
  userId: string;
  variant: "control" | "treatment";
  success: boolean;
  costMicrounits: number;
};

type ArmSummary = {
  users: number;
  successRate: number; // successes / users
  meanCostMicrounits: number; // total cost / users
  costPerSuccessMicrounits: number | null; // total cost / successes, null with 0 successes
};

declare function summarizeVariants(rows: UserRecord[]): {
  control: ArmSummary;
  treatment: ArmSummary;
  absoluteLiftPercentagePoints: number; // 100 * (treatment rate - control rate)
  relativeLift: number | null; // (treatment rate - control rate) / control rate, null when the control rate is 0
};

Python uses summarize_variants(rows) and snake_case keys (user_id, cost_microunits, success_rate, and so on).

Validate in this order and stop at the first failure (JavaScript throws an Error, Python raises ValueError):

Two fixtures, each listing user (success, cost in microunits), exercise the function:

IDs are compared exactly as given (no trimming or case folding), and keys outside the record are ignored.

Four learners per fixture check arithmetic and validation only, not whether staged hints work.