Diagnose leakage and misleading metrics

Implement the evaluation audit and predict its findings and metrics for E1 to E5. Then diagnose the launch model's failure: separate the causes that broke the model in service from the causes that only inflated the offline estimate, and for each give the evidence, a confirming check, a minimal fix, and a regression check. Correct the dashboard's accuracy claim with the majority-class baseline, precision, recall, and the share flagged, each with its denominator, and name the leaks in the team's notes that the audit cannot detect, with a check for each.

Suppose a meal-kit subscription service scores every active subscriber each week with a model that predicts whether they will cancel in the next 30 days. The model was trained on six months of history, looked excellent offline, and failed in its first month of service. All figures are synthetic.

A flag is a prediction of 1. The first month ran in shadow: flags were logged and no one was contacted, so outreach did not change any outcome.

Timing. A row is scored at the end of its cutoff day. Its label is 1 when the subscriber cancels on any of the next labelWindowDays days, so the label becomes final at the end of day cutoffDay + labelWindowDays. A model can serve only after its last training label is final, so an honest evaluation split starts on a later day.

Function. Implement auditEvaluation(experiment) in TypeScript or JavaScript, or audit_evaluation(experiment) in Python. It returns { findings, metrics }. Inputs are well-formed (the practice problem lists every guarantee): each split has at least one row and the report total is at least 1, so validate nothing and throw nothing.

Python uses snake_case for these keys: label_window_days, cutoff_day, window_end_offset_days, derived_from_label, fit_splits, threshold_split, model_split, and baseline_accuracy. Every other key and every finding string is the same. The practice problem has the full mapping.

Rules, in output order. Run every rule; one finding never stops the others.

1. For each feature, in list order: label-derived:<name> when derivedFromLabel is true; otherwise after-cutoff:<name> when windowEndOffsetDays is greater than 0. At most one finding per feature.
2. For each preprocessing step, in list order: fit-beyond-train:<name> when any entry of fitSplits is not "train". An empty list is clean, and so is ["train", "train"].
3. Let L be the largest cutoffDay among train rows plus labelWindowDays. For validation, then test (always this order, whatever the row order): time-overlap:<split> when the smallest cutoffDay among that split's rows is less than or equal to L. Each evaluation split is compared with train only.
4. selected-on-test, once, when thresholdSplit or modelSplit (or both) is "test".
5. population-mismatch, once, when tp + fp + tn + fn differs from the number of rows whose split equals report.split, or when tp + fn differs from that split's number of label-1 rows.
6. recall-undefined when tp + fn is 0; then precision-undefined when tp + fp is 0.
7. accuracy-not-above-baseline when tp + tn is less than or equal to max(tp + fn, tn + fp). Compare these integer counts; that is the same as comparing the two ratios, which share a denominator.

Metrics, from the report's counts with total = tp + fp + tn + fn. They are computed even when the population is flagged, and then describe the reported rows, not the split.

Do not round inside the function. A precision or recall of 0 is 0, never null.

type Split = "train" | "validation" | "test";
type EvalSplit = "validation" | "test";
type Experiment = {
  labelWindowDays: number;
  rows: { id: string; split: Split; cutoffDay: number; label: 0 | 1 }[];
  features: {
    name: string;
    windowEndOffsetDays: number; // last day of the window relative to the cutoff (0 = cutoff day)
    derivedFromLabel: boolean;
  }[];
  preprocessing: {
    name: string;
    fitSplits: Split[]; // splits the step was fitted on; [] fits nothing
  }[];
  selection: { thresholdSplit: EvalSplit; modelSplit: EvalSplit };
  report: { split: EvalSplit; tp: number; fp: number; tn: number; fn: number };
};
type AuditResult = {
  findings: string[];
  metrics: { accuracy: number; baselineAccuracy: number; precision: number | null; recall: number | null };
};
declare function auditEvaluation(experiment: Experiment): AuditResult;

E1 mirrors the launch configuration on a 22-row sample so you can trace it by hand; E2 to E5 are later experiments by the same team.

Fixture rows. Every experiment uses labelWindowDays 30. Each cell lists a split's cutoff days in manifest order, then its label-1 rows in parentheses. Row IDs are the split prefix plus the position in the list (tr1, tr2, and so on; va1 onward; te1 onward). In rows, each experiment lists its train rows, then validation, then test, in the order shown.

Fixture configuration. A feature is written as its name (windowEndOffsetDays, derivedFromLabel) and a step as its name (fitSplits), where (none) is an empty list.