AI Product & Experimentation · 3 to 4 hours
Decide whether to ship on conflicting evidence
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
4 h
Overview
Decide whether to ship an AI feature when offline scores, immediate metrics, matured outcomes, and cost disagree. The level is practiced on one synthetic coach experiment: check the design, compute every rate and interval by hand, and write a decision memo that states its limits. Offline evaluation mechanics belong to Evaluation & Observability.
Study time: 3 to 4 hours
Interview format: System design discussion with calculations and a decision memo
By the end you can
- State an experiment's estimand per assigned unit and explain what its outcome does and does not measure
- Check the assignment ratio, exposure, outcome maturity, and attrition before reading an effect
- Compute rates, differences, and 95% intervals and read them against benefit and harm thresholds
- Explain why offline scores, immediate metrics, and matured online outcomes can disagree
- Price an effect as incremental cost per additional success, with its uncertainty
- Write a decision memo with scope, owner, rollback trigger, next experiment, and reversal evidence
Prerequisites
Helpful background: Comfort with proportions, standard errors, and chi-square tests helps.
This topic covers user outcomes, baselines, online experiments, cost guardrails, and ship decisions. Evaluation & Observability covers offline evaluation mechanics.
In this guide
- Define the causal question
- Resolve offline and online disagreement
- Assignment, exposure, and sample ratio
- Interference and experiment infrastructure
- Delayed outcomes and attrition
- Intervals and decision thresholds
- Interim looks, multiplicity, and segments
- Economics of the decision
- Scenario: the coach experiment
- Work the evidence table
- Failure-injection checks
- Decision memo and rollout
- Write the decision memo
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
AI Engineering
Keeps immature labels apart from matured metrics and ends in a decision memo, for a model whose outcomes arrive weeks late.
System Design
Explains why offline metrics often fail to predict online results, and why business outcomes need longer tests than proxies.
Behavioral Interview
Teaches calibrated confidence and weighing reversibility when the evidence is incomplete and an answer is wanted now.
System Design
Walks through shadow, canary, and A/B stages with one-step rollback, useful when planning exposure and rollback triggers.
Community article
A memo structure with options considered, one recommendation, an owner, and rollback criteria, handy for the decision memo.
Practice the interview question
Answer it in your own words before you look at the reference answer.