Product Sense
product-sense
AI Engineering
Define a user outcome and baseline
Explain why longer sessions do not justify shipping AI hints, then use a fictional pilot's per-arm figures to propose an evaluation with a real baseline, an independent learning outcome, and guardrails.
Design an A/B test with cost guardrails
Write the experiment contract and sample-size plan for a staged-hints A/B test, implement a validated per-arm summary of outcomes and costs, and explain which checks and guardrails must pass before a lift can justify shipping.
Decide whether to ship on conflicting evidence
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
