Diagnose leakage and misleading metrics

In machine learning, leakage means a model is trained or evaluated with information that will not be available when it makes real predictions. Kaufman and colleagues call it information about the target that should not be legitimately available; scikit-learn's pitfalls guide describes it as information that would not be available at prediction time being used to build the model. It is not the security meaning, where data reaches someone who should not see it (Security & Privacy covers that).

Leakage enters in two places:

• Features: a column computed after the prediction time, or from the outcome itself.
• Procedure: fitting, splitting, selecting, and reporting so that evaluation rows shape the model or the reported number.

It causes two separate harms: an inflated offline estimate, so a team ships on a number that will not hold, and a model that relies on a signal the scoring job cannot reproduce (training-serving skew, in the terms of Google's Rules of ML). Leakage usually makes offline results look better than production will be, so an unusually strong result is the first thing to check.

Every feature must be computable from data that exists at the row's cutoff. Describe each feature by the last day its data window covers, relative to the cutoff: 0 is the cutoff day, negative is earlier, and positive reads the future.

• Point-in-time joins. Build each training row from tables as they were at its cutoff (an as-of join). Joining to today's table reads later updates; Rules of ML #31 warns that a joined table may change between training and serving.
• Label-derived features. Anything created by the outcome, or by the process that leads to it (an exit survey, a refund, a cancellation reason), is a copy of the label whatever its timestamp says. The lineage in a feature registry is how you find them.
• Backfills. A value computed later for historical rows can cover a wider window than the scoring job will see. Log the values the scoring job actually used and compare them with training values for the same rows; Rules of ML #29 recommends logging served features, at least for a sample, to check that training and serving agree.

Fitting. Imputers, scalers, encoders, and feature selection learn statistics from the rows they are fitted on. Fit them on the training split only and apply them to the others; scikit-learn's pitfalls guide says never to include test data in fit or fit_transform. A target encoder is the sharpest case: it stores a label rate per category, so fitting it on evaluation rows copies their labels into a feature. Under cross-validation, a pipeline refits the step inside each training fold.

Time. With a forward-looking label, a random split lets training labels describe the same weeks the model is evaluated on. Split by time and leave a gap. For example (not a fixture value), with a 30-day label window and a last training cutoff on day 200, the last training label is final at the end of day 230, so the first evaluation cutoff can be day 231. scikit-learn's TimeSeriesSplit has a gap parameter that excludes samples from the end of each training set before the test set. It counts samples, not days: with one row per day, a gap of at least the label window plays this role.

Selection. Every choice made by looking at a split (model, hyperparameters, features, threshold) turns it into a validation set. Choose on validation and run the test split once. scikit-learn says test data should never be used to make choices about the model, and Rules of ML #33 says to test on data gathered after the training data.

Write each metric as a fraction with a named denominator.

Report the share flagged at the deployed threshold: outreach capacity is spent on flagged rows.

Majority-class baseline. Always predicting the more common actual class scores max(tp + fn, tn + fp) / total. On imbalanced data, accuracy near this baseline shows nothing. For example (not a fixture value), on 1,000 rows with 50 positives, a model that flags no one scores 950 / 1,000 = 95%, exactly the baseline, with precision 0 / 0 (undefined) and recall 0 / 50 = 0.

Undefined ratios. Precision is undefined when nothing is flagged, and recall when the population has no actual positives; the crash course calls the result NaN. scikit-learn's precision_score returns 0 with an UndefinedMetricWarning by default and lets zero_division set the value, so the choice must be deliberate. This question's audit contract deliberately reports null.

Population. A metric describes the rows it was computed on. Filtering rows (missing features, survivors, only contacted customers) changes the population and usually the base rate, so check that a report's row and positive counts match its split.

Timing. A row is scored at the end of its cutoff day. Its label is 1 when the subscriber cancels on any of the next labelWindowDays days, so the label becomes final at the end of day cutoffDay + labelWindowDays. A model can serve only after its last training label is final, so an honest evaluation split starts on a later day.

Function. Implement auditEvaluation(experiment) in TypeScript or JavaScript, or audit_evaluation(experiment) in Python. It returns { findings, metrics }. Inputs are well-formed (the practice problem lists every guarantee): each split has at least one row and the report total is at least 1, so validate nothing and throw nothing.

Python uses snake_case for these keys: label_window_days, cutoff_day, window_end_offset_days, derived_from_label, fit_splits, threshold_split, model_split, and baseline_accuracy. Every other key and every finding string is the same. The practice problem has the full mapping.

Rules, in output order. Run every rule; one finding never stops the others.

1. For each feature, in list order: label-derived:<name> when derivedFromLabel is true; otherwise after-cutoff:<name> when windowEndOffsetDays is greater than 0. At most one finding per feature.
2. For each preprocessing step, in list order: fit-beyond-train:<name> when any entry of fitSplits is not "train". An empty list is clean, and so is ["train", "train"].
3. Let L be the largest cutoffDay among train rows plus labelWindowDays. For validation, then test (always this order, whatever the row order): time-overlap:<split> when the smallest cutoffDay among that split's rows is less than or equal to L. Each evaluation split is compared with train only.
4. selected-on-test, once, when thresholdSplit or modelSplit (or both) is "test".
5. population-mismatch, once, when tp + fp + tn + fn differs from the number of rows whose split equals report.split, or when tp + fn differs from that split's number of label-1 rows.
6. recall-undefined when tp + fn is 0; then precision-undefined when tp + fp is 0.
7. accuracy-not-above-baseline when tp + tn is less than or equal to max(tp + fn, tn + fp). Compare these integer counts; that is the same as comparing the two ratios, which share a denominator.

Metrics, from the report's counts with total = tp + fp + tn + fn. They are computed even when the population is flagged, and then describe the reported rows, not the split.

Do not round inside the function. A precision or recall of 0 is 0, never null.

type Split = "train" | "validation" | "test";
type EvalSplit = "validation" | "test";
type Experiment = {
  labelWindowDays: number;
  rows: { id: string; split: Split; cutoffDay: number; label: 0 | 1 }[];
  features: {
    name: string;
    windowEndOffsetDays: number; // last day of the window relative to the cutoff (0 = cutoff day)
    derivedFromLabel: boolean;
  }[];
  preprocessing: {
    name: string;
    fitSplits: Split[]; // splits the step was fitted on; [] fits nothing
  }[];
  selection: { thresholdSplit: EvalSplit; modelSplit: EvalSplit };
  report: { split: EvalSplit; tp: number; fp: number; tn: number; fn: number };
};
type AuditResult = {
  findings: string[];
  metrics: { accuracy: number; baselineAccuracy: number; precision: number | null; recall: number | null };
};
declare function auditEvaluation(experiment: Experiment): AuditResult;

Trace the seven rules by hand for each experiment, and write every metric as a fraction, before you run any code.

An audit like this one checks declared fields only: registry metadata, split assignments, cutoffs, labels, and reported counts. Wrong or missing metadata passes, and several kinds of leakage need checks on the data itself.

Fix each cause at its source, then add a check that fails if the cause returns.

• Compute point-in-time features in one code path that both training and scoring call, and compare logged served values with training values for the same rows.
• Fit preprocessing inside the training pipeline, on train rows only.
• Split by time with a gap of at least the label window.
• Choose the model, features, and threshold on validation, and run the test split once.
• Report on the whole split, with the baseline, precision, recall, the share flagged, and their denominators.

Turn every bug you find into an audit rule or a fixture, and run the audit on every experiment manifest in CI. Expect the honest number after a fix to be lower: that is the fix working, not a regression.

The audit sees only the manifest, the feature registry, the selection record, and the report. The team's notes below add what the manifest does not record; use both, with the figures in the churn model scenario.