AI Product & Experimentation · 90 minutes
Define a user outcome and baseline
Explain why longer sessions do not justify shipping AI hints, then use a fictional pilot's per-arm figures to propose an evaluation with a real baseline, an independent learning outcome, and guardrails.
1.5 h
1
Step 1: The question
In the hint pilot, the feature team wants to ship staged AI hints because median session length rose from 19 to 26 minutes. Explain why that rise is not enough to ship, and what the pilot's other figures do and do not show. Then propose a stronger evaluation: the population, the baseline, a primary outcome with numerator, denominator, and window, guardrails with limits, the assignment unit, and the decision rule you would fix in advance.
Worked scenario: a hint pilot
The platform, pilot, and figures are fictional.
Suppose a coding-practice platform piloted AI hints on its beginner exercises. Each learner who opened a supported beginner exercise for the first time during a two-week enrollment period was assigned by account, with a configured 50/50 split, to one arm:
- Control: the current curated hint, one human-written hint per exercise, shown on request.
- Treatment: staged AI hints, up to three per exercise on request, from a nudge to a near-complete solution.
Every assigned learner was followed for 7 days after assignment. A related exercise is a different exercise that practices the same concept. Before the pilot, the team agreed two tolerances for reviewed hint responses: at most 5% may reveal the full solution, and at most 3% may contain an incorrect statement. Reviewers scored a random sample of 200 hint responses per arm.
| ID | Metric | Control | Treatment |
|---|---|---|---|
| M1 | Median session length | 19 min | 26 min |
| M2 | Hint requests per assigned learner (mean) | 1.2 | 3.4 |
| M3 | Assigned learners who solved a related exercise without hints within 7 days | 405 of 2,024 | 415 of 1,976 |
| M4 | Sampled hint responses that revealed the full solution | 2 of 200 | 18 of 200 |
| M5 | Sampled hint responses with an incorrect statement | 1 of 200 | 5 of 200 |
| M6 | p95 time to first hint text | 0.2 s | 2.6 s |
| M7 | Model cost per assigned learner | $0.00 | $0.05 |
| M8 | Learners assigned | 2,024 | 1,976 |
The feature team's summary reads: "AI hints raised median session length from 19 to 26 minutes. Learners are clearly more engaged. Let's ship to everyone."
Constraints
- Use only the pilot figures shown, and do not assume results that were not measured
- Keep the answer to about three minutes spoken
Deliverables
- Why session length cannot carry the decision, and the user outcome it stands in for
- The baseline arm and why it is the real alternative
- A primary outcome with numerator, denominator, unit, and window, and guardrails with limits
- What the pilot figures can and cannot establish, and the assignment unit and analysis population for the next test
- A decision rule fixed in advance, and what a guardrail breach or an inconclusive result leads to
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Outcome and metric definitions or Problem framing
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can connect a feature to a measurable user benefit, against a fair baseline and with guardrails fixed in advance, without relying on a proxy alone.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why can offline quality improve while product value falls?
Is 10% relative lift the same as 10 percentage points?
Why randomize?
Why predefine guardrails?
Does non-significance mean no effect?
7