Serving Timing Summary

Serving Timing Summary

A company self-hosts an open-weight model behind the code-review assistant in its internal developer portal. Before changing the server's batching settings, the team runs a load test and logs one timing record per request. Your job is the analysis step: turn the records into a summary that cannot make an overloaded server look healthy.

Every time is in seconds on one monotonic clock. A request starts when the client sends it and ends when it reaches its terminal status. For a successful request, firstToken is when its first output token arrived and outputTokens is how many tokens it generated.

Implement summarize(records, windowStart, windowEnd). The observation window [windowStart, windowEnd] is the load test's whole measurement period, and it must contain every record.

Records

A non-success record needs only start, end, and status. Its firstToken and outputTokens are ignored, even when they are present or malformed: a timed-out request may have streamed part of an answer, but that output is not useful work.

Validation

Check in this order and throw an Error with the given message at the first failure. Check the records in array order, and finish each record before moving on to the next one.

A finite number is a value of type number that is not NaN, Infinity, or -Infinity. Never convert: null, true, and "2.5" are not numbers. Every bound is inclusive: a request may start at windowStart, end at windowEnd, and receive its first token at its own start or end.

Times come before status on purpose. A record with broken timestamps is reported as a timing problem even when its status is also unknown.

Summary

Return an object with exactly these keys. Only successful requests contribute to latency, tokens, and throughput.

• Throughput uses the whole window. Divide by windowEnd - windowStart, not by the time between the first and last record, and not by the sum of latencies.
• One-token responses have no TPOT. A single-token response has no interval after its first token, so it is left out of the TPOT values. It still counts in latency, TTFT, and tokens.
• p95 is the nearest rank. Sort the n values in ascending numeric order and take the value at 1-based rank ceil(0.95 * n), which is sorted[max(0, ceil(0.95 * n) - 1)]. There is no interpolation: the p95 of the 20 values 1, 2, ..., 20 is 19, not 19.05 and not 20.
• No values means null. When a metric has no values (no successes, or no success with more than one token), its p95 is null, never 0 or NaN.
• Do not modify records or the record objects.

Fixture

The tests summarize this load test over the window 0 to 10, variants of it, and the edge cases above.

Example

summarize(
    [
        { start: 0, end: 3, status: 'success', firstToken: 1, outputTokens: 5 },
        { start: 1, end: 2, status: 'failed' },
    ],
    0,
    4,
);
// {
//     requests: 2, successes: 1, nonSuccessFraction: 0.5,
//     successfulRequestsPerSecond: 0.25, successfulOutputTokensPerSecond: 1.25,
//     successLatencyP95S: 3, successTtftP95S: 1, successRequestTpotP95S: 0.5,
// }

The only success took 3 seconds, and its first token arrived after 1 second. Its other 4 tokens took the remaining 2 seconds, so its TPOT is 0.5. Both rates divide by the 4-second window, and the failed request counts only in requests and nonSuccessFraction.

Python

Implement summarize(records, window_start, window_end) with lists and dicts, and raise ValueError with the same messages. The keys are renamed:

In Python, records must be a list and each record a dict. A finite number is an int or float that is not a bool (because True == 1), and output_tokens must be an int that is not a bool.

Constraints

• A valid call has 1 <= records.length <= 10^5, but any argument or field may hold any value or be missing. Validation decides.
• Valid times and token counts are small enough that the arithmetic never overflows.

Expected Complexity

O(n log n) time for n records, from sorting three lists of at most n values, and O(n) extra space.