Design an A/B test with cost guardrails

Write the contract before launch: hypothesis, eligibility, assignment unit, variants, primary outcome, guardrails with tolerances, observation window, statistical method, stopping rules, and decision owner. The smallest effect worth acting on sets the sample size and separates a real but trivial change from a useful one.

Randomize before the treatment can influence who is included, and keep each learner's assignment stable across sessions and devices.

Version the model, prompt, retrieval policy, and UI. If the treatment bundles several changes, the experiment estimates the bundle's effect and cannot isolate each part without additional design.

An intent-to-treat analysis compares eligible units by assigned variant, including those who never received the treatment. It estimates the effect of offering the experience under the real delivery process, failures included.

Exposure-based analyses can be useful, but conditioning on successful loading or later engagement may select different users in each arm, so treat them as secondary unless the causal design justifies them.

For delayed outcomes, wait until each cohort's observation window is complete, or use an appropriate time-to-event method. Never compare mature control users with newly enrolled treatment users.

Use distinct events for assignment, exposure, request completion, feedback, and learning outcomes. Each carries the experiment ID, a pseudonymous unit ID, the variant, an event ID, a timestamp, and the relevant artifact and configuration revision.

Deduplicate events by their stable event ID. Preserve failures and missingness instead of turning them into successful no-op events. Check time zones, late arrivals, retries, cross-device identity, and reassignment after sign-in.

Join outcomes to the authoritative assignment record. A variant label sent by the client is not a source of truth on its own. Collect only the fields the analysis and operational investigation need.

The function takes one row per assigned learner whose observation window has closed. success is true when the primary outcome occurred within the window. A learner who never came back has an observed outcome of false once the window closes; one whose events were lost, or whose window is still open, has a missing outcome, which is resolved upstream and never coded as false.

costMicrounits is the learner's total attributable model cost as an integer, including retries and failed attempts, and 0 for a learner who never asked for a hint. Integers avoid floating-point drift in currency totals.

type UserRecord = {
  userId: string;
  variant: "control" | "treatment";
  success: boolean;
  costMicrounits: number;
};

type ArmSummary = {
  users: number;
  successRate: number; // successes / users
  meanCostMicrounits: number; // total cost / users
  costPerSuccessMicrounits: number | null; // total cost / successes, null with 0 successes
};

declare function summarizeVariants(rows: UserRecord[]): {
  control: ArmSummary;
  treatment: ArmSummary;
  absoluteLiftPercentagePoints: number; // 100 * (treatment rate - control rate)
  relativeLift: number | null; // (treatment rate - control rate) / control rate, null when the control rate is 0
};

Python uses summarize_variants(rows) and snake_case keys (user_id, cost_microunits, success_rate, and so on).

Validate in this order and stop at the first failure (JavaScript throws an Error, Python raises ValueError):

Two fixtures, each listing user (success, cost in microunits), exercise the function:

IDs are compared exactly as given (no trimming or case folding), and keys outside the record are ignored.

Four learners per fixture check arithmetic and validation only, not whether staged hints work.

Before reading any effect, compare observed assignment counts with the planned allocation. A sample ratio mismatch (SRM) is a gap that chance does not explain, usually checked with a chi-square test; the split need not be exactly 50/50. An SRM points to an assignment, logging, join, or filtering problem, and results are untrustworthy until the cause is found.

A/A checks, where both arms get the same experience, test instrumentation and false-positive behavior, though one passing A/A test does not prove the pipeline correct. Pre-treatment balance and invariant metrics are further diagnostics.

Compare assignment, exposure, and outcome counts per arm: differential missingness can bias results even after balanced assignment. Never delete inconvenient rows until the arms look similar.

Analyze at the assignment unit. Repeated hint requests from one learner are correlated, so treating each request as an independent unit understates uncertainty; aggregate to one row per learner or use a clustered method.

Size the test from the baseline rate, the smallest effect worth acting on, the significance level, and the power. For a binary outcome with two equal arms, a common normal-approximation planning estimate is:

Round n up, divide by the learners each arm receives per week, and add the observation window, since the last learner enrolled still needs a complete window. Formula variants (pooled variance, continuity correction) shift the result slightly; clustered or skewed metrics need more units.

Report an interval for every effect, not only a point estimate. Predefine the primary metric and how multiple comparisons are handled, and treat surprising subgroup results as hypotheses to confirm. Checking a fixed-horizon test daily and stopping at the first significant result inflates false positives: run to the planned size or use a sequential method chosen in advance.

A guardrail needs a tolerance and evidence that rules out exceeding it. A non-significant increase in incorrect hints is not proof that quality held, because the interval may still include meaningful harm; the guardrail passes when the upper end of that interval sits inside the tolerance.

Judge sampled quality with blinded human review or independent execution checks, keep the sampling design so easy cases are not overrepresented, and track reviewer disagreement.

Count every attributable cost: retries, escalation, tools, and review. Report mean cost per assigned learner, the heavy-user distribution, and total spend. Cost per success is a ratio of totals, not the causal cost of one more success, and it is unstable when successes are rare.

Read results in order: data validity, primary effect, guardrails, operational feasibility, then predefined slices. Distinguish "failed to show benefit" from "showed no meaningful benefit" and "showed harm".

A decision memo gives effect estimates with uncertainty, absolute counts, delivery rates, limitations, and the action, and says whether it covers all users or a supported segment. Do not change the hypothesis after seeing results; write a new one and test it.

After a successful test, roll out gradually with rollback criteria, monitoring, and a holdout when longer-term effects matter. Population shifts and model updates can alter the effect, so carry the experiment's limitations into the release record.

Apply the population, event, statistics, and guardrail sections to a teammate's draft before any data arrives.