Define a user outcome and baseline

A model capability is not a product outcome. "We can generate explanations" describes a capability. "Learners can identify their mistake and solve a similar exercise independently" describes an outcome.

Suppose a coding-practice platform adds AI hints. They should help learners progress without simply revealing the answer. The product question is whether the experience improves learning under acceptable correctness, latency, accessibility, and cost constraints.

Define the target population and situation. A beginner learning loops and an experienced developer rehearsing system design may need different help. Do not assume one average result applies to both.

Possible baselines include no feature, existing human-written hints, a search tool, a deterministic rule, or the current AI implementation. Choose the baseline representing the real alternative users would have.

A comparison with a deliberately weak prompt exaggerates the value of a new model. Compare a viable incumbent and include the effort needed to maintain each approach.

AI may be unnecessary when a fixed rubric or curated hint solves the problem reliably. The experiment should test usefulness rather than justify an implementation chosen in advance.

A useful hypothesis names a change, a population, an expected outcome, and important limits.

Example: "For learners starting a supported beginner exercise, staged AI hints will increase the probability of solving a related unassisted exercise within seven days, without increasing answer disclosure beyond the agreed tolerance."

This is an illustrative hypothesis, not a recommended universal metric or duration. The observation window must fit the behavior you want to measure and the available traffic.

Choose definitions before looking at results. Specify the numerator, denominator, unit of analysis, and time window. "Completion improved" is ambiguous if one variant changes which users are counted.

More hint requests may mean engagement or confusion. More time spent may mean interest or difficulty. Interpret a metric in relation to the user outcome, not its direction alone.

Offline evaluation uses a fixed or sampled dataset to assess outputs before broad exposure. It can check correctness, instruction following, schema validity, unsupported claims, and important task slices. It is useful for finding obvious regressions and comparing candidates under controlled inputs, but it cannot fully predict whether users notice, trust, understand, or benefit from the feature. If a model is repeatedly tuned to a benchmark, that benchmark becomes less informative about unseen cases, so keep evaluation examples independent of development choices where practical.

An online controlled experiment (an A/B test) assigns eligible units at random to a control or a treatment and compares outcomes. Random assignment makes the groups comparable in expectation, so a difference can be attributed to the tested change under the experiment's assumptions.

Choose an assignment unit that matches the experience. User-level assignment often avoids showing one learner inconsistent hint behavior across sessions. Team or classroom settings may need different units because people influence one another.

An experiment is not a casual before/after comparison. Traffic, seasonality, marketing, and other releases change over time and can create misleading differences.

Assignment means a unit belongs to a variant. Exposure means the unit actually encountered the relevant feature. Outcome means the defined result was observed during the window.

These are different events. If treatment makes the feature harder to load, analyzing only users who successfully saw it may hide the harm. Start with the planned analysis population and document missing data and delivery failures.

Check that observed assignment counts are consistent with the configured split. A substantial unexpected mismatch, called a sample ratio mismatch, can indicate assignment or logging problems and makes causal conclusions unsafe until investigated. A chi-square test of the observed counts against the configured split is the usual check.

Observed differences fluctuate because the experiment sees a sample of possible users and outcomes. A confidence interval expresses uncertainty under its statistical assumptions; it is not a guarantee about every learner.

Plan sample size and duration around a minimum effect worth detecting, expected baseline rate, variability, and desired error control. A small experiment can be inconclusive even when the observed result looks promising.

Repeatedly checking an ordinary fixed-horizon test and stopping as soon as it looks significant increases false positives. Use the planned analysis or an appropriate sequential method. Safety monitoring can still stop harmful exposure; that does not automatically establish efficacy.

Measure cost at the workflow level: model calls, retries, retrieval, tools, review, and support. A cheap request that often fails may cost more per useful outcome than a more expensive reliable one.

Report per-user or per-task spending as well as total cost. Include users who receive failures so the metric does not reward dropping difficult requests.

For a learning product, the value of a successful outcome is not identical to time spent or immediate revenue. Decide which outcome matters and whether the operational cost is sustainable.

Possible decisions are ship within limits, iterate, stop, or collect more evidence. An inconclusive result is not proof of no effect. A statistically detectable change can still be too small to justify complexity or cost.

A positive primary metric does not override an unacceptable correctness regression. Examine guardrails, important slices, and data quality before interpreting success.

After release, monitor whether the effect persists. Users may react differently after novelty wears off or the population expands.

Use the pilot table above and the five metric types in "Build a metric hierarchy".

This exercise changes the baseline. Assume the same platform and traffic as the hint pilot.