Separate retrieval, packing, and generation failures

An evaluation gives a system an input, observes its behavior or outcome, and applies a defined scoring rule. “The answer sounds good” is difficult to reproduce. “The answer states the current return window, includes the exception, and cites the supporting policy” can be checked.

Define success before inspecting candidate outputs. Otherwise, a team can unconsciously change the rubric to favor a result it already likes. Include behavior for ambiguous requests, missing information, and unsuccessful execution.

For an agent that closes a ticket, the final database state matters. A fluent message claiming that the ticket was closed does not establish task completion.

A case can be run multiple times because model output may vary. Keep the case count and trial count distinct. Ten executions of the same question provide different evidence from ten different questions.

Retrieval quality concerns whether useful, eligible evidence is found and ranked appropriately. Answer quality concerns whether the response correctly uses available evidence and satisfies the user's task.

An assistant can retrieve the correct exception clause and still ignore it. It can also fail to retrieve the clause and then produce an unsupported generalization. These failures may look identical in the final answer but require different corrections.

Inspect the actual context sent to the model, not only the top retrieval results. Context packing can remove a relevant passage after retrieval succeeds.

These dimensions can disagree. An answer may be grounded in an outdated document and therefore wrong for a current-policy question. A valid JSON response can contain fabricated values. A complete answer may be too slow for the product's requirement.

Use deterministic checks when expected behavior is precise, such as JSON schema validity, a permitted category, or a database state. Use human review when meaning, ambiguity, or domain judgment matters.

A model-based grader can help assess large sets of free-text answers, but it can make mistakes and favor style over substance. Give it an explicit rubric and compare its judgments with reviewed examples. Treat grader errors as part of the measurement system.

Exact text matching is appropriate only when one exact representation is required. Two different sentences may express the same correct answer. Conversely, a sentence can contain an expected keyword while reversing its meaning.

A positive case exercises a task the system should complete. A negative case exercises behavior such as refusing an unauthorized action or acknowledging that evidence is missing. Define these terms for each dataset because “negative” can also mean an incorrect candidate response.

Boundary cases probe distinctions that commonly fail. In a policy assistant, a standard product and a custom product may have different rules. Include both so a system cannot pass by applying the common rule to everything.

Synthetic cases make edge conditions explicit. Representative real-world cases reveal the actual language and task mix. Use permitted, minimized data and retain the source of each case so evaluation limitations are visible.

Suppose ten attempted cases produce eight successful outcomes, one incorrect answer, and one timeout. Overall task success is 8/10, or 80%. Excluding the timeout would report 8/9 and hide a user-visible failure.

Now suppose a different run of the same ten cases gives six substantive answers, five of them correct. Accuracy among answered cases is 5/6, about 83.3%, while answer coverage is 6/10, or 60%. These metrics answer different questions and should be reported together.

Always name the denominator. Distinguish attempted tasks, completed generations, answered cases, and scored outputs. If a judge fails to grade an output, report the missing judgment rather than silently treating it as a pass.

Metrics summarize measurements, such as error count or latency distributions. Logs record individual events. A trace connects steps belonging to one request, often through spans representing operations such as retrieval, model generation, and tool execution.

For an AI request, useful metadata includes configuration versions, candidate IDs, selected evidence IDs, tool-call outcomes, token usage, and elapsed times. This helps connect a quality failure to the stage that produced it.

Telemetry does not automatically explain correctness. A 200 response in 300 ms can still contain a wrong answer. Quality evaluation and operational signals need to be interpreted together.

Prompts, documents, and tool results can contain private information. Store only what is needed for diagnosis, restrict access, and define retention. Stable IDs and configuration metadata can often support investigation without storing complete content.

A request ID belongs in a trace or log. Using every request ID as a metric label creates extremely high cardinality and can make the metric system expensive or unusable. Aggregate metrics by bounded categories such as outcome and configuration version.