Experiment Outcome Summary

Experiment Outcome Summary

A learning app runs an A/B test of a staged-hint feature built on a language model. Each learner is randomly assigned to control (the current hints) or treatment (staged hints). After every learner's observation window has closed, an upstream job writes exactly one record per assigned learner: whether the predefined outcome occurred, and the learner's total attributable model cost, including retries and failed attempts (0 for a learner who never asked for a hint). Costs are whole numbers of microunits, millionths of one declared currency unit, so no floating-point money is ever summed.

Write the descriptive summary an analyst reads before any statistical test. It validates the records, then reports each arm's success rate and costs and the difference between the arms. It does not estimate uncertainty or decide whether to ship.

Implement summarizeVariants(rows).

Input

rows is an array of records. Records from the two arms arrive mixed together, in any order.

An outcome that was never observed (a lost event, for example) is invalid input. It must never be counted as false. The variant field may hold any value, even an array; anything other than the two exact strings is rejected. Keys outside this table are ignored.

Validation

Check the input, then each record in order, and stop at the first failure. Within one record, run the checks in the order below. Throw an Error with exactly the message shown:

IDs are compared exactly as given, with no trimming or case folding: "t1" and "T1" are different learners. The variant match is exact and case-sensitive.

Report

Return an object with exactly these keys:

Each arm summary has exactly these keys:

• Mean cost divides by every learner in the arm. Cost per success divides the same total by successes only, so it still includes what failed learners cost.
• A ratio with a zero denominator is null (None in Python), never 0, NaN, or Infinity.
• Do not round. The tests round floats to 4 decimal places before comparing.

Example 1

summarizeVariants([
    { userId: 'c1', variant: 'control', success: true, costMicrounits: 150 },
    { userId: 't1', variant: 'treatment', success: true, costMicrounits: 60 },
    { userId: 'c2', variant: 'control', success: false, costMicrounits: 50 },
    { userId: 't2', variant: 'treatment', success: true, costMicrounits: 90 },
    { userId: 't3', variant: 'treatment', success: false, costMicrounits: 60 },
    { userId: 't4', variant: 'treatment', success: true, costMicrounits: 90 },
]);
// {
//     control: { users: 2, successRate: 0.5, meanCostMicrounits: 100, costPerSuccessMicrounits: 200 },
//     treatment: { users: 4, successRate: 0.75, meanCostMicrounits: 75, costPerSuccessMicrounits: 100 },
//     absoluteLiftPercentagePoints: 25,
//     relativeLift: 0.5,
// }

Control succeeds for 1 of 2 learners and treatment for 3 of 4. The difference is 0.25, which is 25 percentage points, and 0.25 / 0.5 = 0.5 relative to control. Treatment spends 300 microunits in total: 75 per assigned learner and 100 per success.

Example 2

summarizeVariants([
    { userId: 'c1', variant: 'control', success: 1, costMicrounits: 100 },
    { userId: 't1', variant: 'treatment', success: true, costMicrounits: 80 },
]);
// throws Error: Outcome must be observed boolean

1 is a number, not a boolean. Coercing it would hide an upstream encoding bug, and coercing a missing outcome to false would quietly lower one arm's success rate.

Fixtures

The tests compare the whole summary for these two fixtures. Each cell lists learner (success, cost in microunits).

They also run these validation rows. Each one changes fixture A, whose rows are u1, u2, u3, u4 in that order:

Constraints

• At most 100,000 records.
• Valid costs are at most 1,000,000,000 microunits per learner, so every total stays an exact integer.
• User IDs are at most 64 characters.

Expected Complexity

• Time: O(n) for n records.
• Space: O(n) for the set of seen user IDs.

Python

Implement summarize_variants(rows). rows must be a list and each record a dict with the Python keys above. Return a dict with the Python keys, use None for null, and raise ValueError with the same messages in the same order. success must be a bool. The cost must be an int that is not a bool (Python treats True as the integer 1), and a float such as 100.0 is rejected even though it has no fractional part.