AI Product & Experimentation · 3 to 4 hours
Decide whether to ship on conflicting evidence
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
4 h
1
Step 1: The question
Using the coach experiment, decide what the platform should do at the end of day 14 and defend it. Check the assignment split, exposure, maturity, and attrition; compute every rate, difference, and 95% interval for the primary outcome and guardrails, overall and for each predeclared segment; and compute cost per assigned learner and per additional success. Explain how the offline ratings, the completion gain, and the dashboard's learning metric relate to the primary outcome, then state the action, its scope and owner, the rollback trigger, the next experiment, and what would change the decision.
Constraints
- Use the pre-registered outcomes, thresholds, and guardrails; do not redefine them after seeing results
- Use 95% Wald intervals for differences and a Wilson interval for the disclosure rate, unless you justify another method
- Only the two predeclared segments are confirmatory
- Any restricted rollout must be enforceable from facts known at assignment
Deliverables
- The causal question and estimand, including what the primary outcome does and does not measure
- Validity checks: assignment split, exposure, maturity, attrition, and interference
- Rates, differences, and 95% intervals read against the benefit and harm thresholds, overall and by segment
- Cost per assigned learner, cost per success, and cost per additional success with its range
- A decision memo: action, scope, owner, rollback trigger, next experiment, and what would change it
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.