Diagnose a missing document

Record stable identifiers at each boundary. A useful trace connects source revision, parsing run, chunk set, indexing acknowledgement, publication state, retrieval candidates, selected context, and final citation references.

Each stage answers a different question. Did the parser extract the relevant sentence? Did chunking preserve it? Was the chunk indexed? Was its revision published? Did filtering exclude it? Did retrieval return it? Did context packing keep it? Did generation use it correctly?

Trace content only when necessary and permitted. IDs, statuses, lengths, revision hashes, and failure codes can often identify the issue without copying private text into general logs.

Inspect the text actually indexed. A parser may preserve the words but reorder a table. A chunk may include “within 7 days” without the phrase “manufacturing defects.” The resulting fragment can support a misleading answer.

Compare chunking alternatives with fixtures that require headings, exceptions, tables, and adjacent paragraphs. Keep original-source locators so you can inspect the evidence at its origin. A generated contextual summary can help retrieval, but it introduces derived text that may itself be inaccurate. Preserve the original passage and distinguish any added summary from source evidence.

Measure token counts with the selected tokenizer when budgets are token-based. Character counts are useful diagnostics but do not establish actual model input length.

A retry should not create duplicate published chunks. A chunk key can include tenant, document, revision, pipeline configuration, and chunk position. Content hashes help detect repeated work, while a pipeline version captures changes to parsing or splitting behavior.

Build a complete revision in a staging state. Validate its expected chunk manifest before making it eligible for current queries. A partially indexed new revision should not silently replace a complete older one.

If processing the new revision fails, retain an explicit failure state and a documented freshness policy. Whether old content remains usable depends on the product and source requirements. Do not label an older revision as current merely because it is available.

The following fixture uses a trusted access context and an authoritative published-revision map. The department list represents access grants for the exercise. Production authorization should be enforced by trusted services or database policies as appropriate.

The fixture types and the six chunks are below. Each chunk differs from C1 in one field.

type Chunk = {
  id: string;
  tenant: string;
  document: string;
  revision: number;
  department: string;
  state: "published" | "staged";
  deleted: boolean;
  tokens: number;
};

type Access = {
  tenant: string;
  departments: ReadonlySet<string>;
};

// Returns the first exclusion reason, checked in this order:
// "tenant", "deleted", "unpublished", "revision", "department".
// Returns null when the chunk is eligible.
declare function exclusionReason(
  chunk: Chunk,
  access: Access,
  publishedRevision: ReadonlyMap<string, number>
): string | null;

const access: Access = { tenant: "alpha", departments: new Set(["support"]) };
const publishedRevision = new Map([["returns", 3]]);
const base: Chunk = {
  id: "C1", tenant: "alpha", document: "returns", revision: 3,
  department: "support", state: "published", deleted: false, tokens: 120
};
const chunks: Chunk[] = [
  base,
  { ...base, id: "C2", department: "procurement", tokens: 100 },
  { ...base, id: "C3", tenant: "beta" },
  { ...base, id: "C4", revision: 2 },
  { ...base, id: "C5", state: "staged" },
  { ...base, id: "C6", deleted: true }
];

Build a request-local map from source labels to selected chunks and source revisions. The generator can reference those labels. Check that every returned label belongs to the map and resolve links through trusted application metadata.

For example, if the context contains S1 and S2, a reference to S9 is invalid even if a document with a similar title exists elsewhere. A reference to S1 may be structurally valid but semantically wrong for a claim about the custom-product exception in S2.

Define whether every factual claim needs support and how the UI associates claims with citations. Evaluate support and coverage. Counting citation markers alone can reward unsupported answers with many links.

A useful experiment substitutes a known-good context for the retrieved context. If the answer improves, investigate the upstream path. If it remains wrong, inspect generation behavior and the task instructions. Keep evaluation cases separate from prompt tuning cases.