Prove a context selector cannot leak data

Step 1: The question

Implement selectContext and design tests proving that D2 to D5 and D1's internal note do not reach the model-bound payload for the alpha support user. Extend the test plan to revocation, cache reuse, indirect injection, and outbound tool calls.

Define the protection boundary

The fictional assistant may answer from active documents in the authenticated user's tenant when at least one of the user's trusted groups is allowed by the document. An empty allowedGroups list denies access. Internal ingestion notes must never be included in model context, even when the document itself is readable.

The trusted identity comes from the application's authentication layer. Document access metadata comes from an authoritative record, not from text inside the document. In production, retrieval should enforce eligibility early and validate current state before exposing content downstream.

A leakage-safe context selector

The selector receives the search candidates, the trusted identity from the authentication layer, the untrusted client request, and a server-configured number of context slots. Each document's access metadata comes from an authoritative record. A real selector also ranks candidates for the query and fits them into a fixed number of slots, which is where a forbidden record most easily slips in, so this version takes the query and the slot limit too.

type Identity = { tenant: string; groups: ReadonlySet<string> };
type Doc = {
  id: string;
  tenant: string;
  allowedGroups: string[];
  deleted: boolean;
  title: string;
  body: string;
  internalNote: string;
};
type ContextItem = { id: string; title: string; body: string };
// From the client, so untrusted: only query may be read.
// tenant and groups model a tampered request and must change nothing.
type ClientRequest = { query: string; tenant?: string; groups?: string[] };

// 1. Eligible: same tenant as identity, not deleted, and at least one of
//    allowedGroups in identity.groups. An empty list denies.
// 2. Score: distinct lowercase query terms found in title or body.
//    internalNote is never searched. Documents that score 0 are left out.
// 3. Order: highest score first; equal scores keep the input order.
// 4. Return at most maxItems items with only id, title, and body.
declare function selectContext(
  documents: Doc[],
  identity: Identity,
  request: ClientRequest,
  maxItems: number
): ContextItem[];

const documents: Doc[] = [
  { id: "D1", tenant: "alpha", allowedGroups: ["support"], deleted: false,
    title: "Returns", body: "Standard returns within 30 days. PUBLIC_TEST_A.",
    internalNote: "INTERNAL_TEST_A" },
  { id: "D2", tenant: "beta", allowedGroups: ["support"], deleted: false,
    title: "Private beta policy", body: "FORBIDDEN_TENANT_TEST_B",
    internalNote: "INTERNAL_TEST_B" },
  { id: "D3", tenant: "alpha", allowedGroups: ["payroll"], deleted: false,
    title: "Payroll", body: "FORBIDDEN_GROUP_TEST_C", internalNote: "internal" },
  { id: "D4", tenant: "alpha", allowedGroups: ["support"], deleted: true,
    title: "Retired policy", body: "DELETED_TEST_D", internalNote: "internal" },
  { id: "D5", tenant: "alpha", allowedGroups: [], deleted: false,
    title: "Unassigned", body: "DENY_DEFAULT_TEST_E", internalNote: "internal" }
];
const identity: Identity = { tenant: "alpha", groups: new Set(["support"]) };
selectContext(documents, identity, { query: "test" }, 5);

Every body carries a synthetic marker containing TEST, so the query matches all five documents and only the access rules decide the result.

Constraints

Tenant and groups come only from the trusted identity, never from the request or from document text
Use only synthetic documents and markers in an owned test environment
Every regression run includes an allowed-user control, not only denial tests

Deliverables

A selectContext implementation that reproduces the fixture result
Tests for D2 to D5 and the D1 internal note, asserted on the serialized model-bound payload
Test plan extensions for revocation, cache reuse, indirect injection, and outbound tool calls
A statement of what the marker scan cannot prove

Step 2: Hints

Optional. Each hint gives a little more away.

Step 3: Work through it

Code exercise

Leakage-safe context selection

Implement selectContext so that, for the alpha support user and the query "test", the context holds only D1, projected to id, title, and body. Decide eligibility from the trusted identity before ranking, score only the title and body, and return at most maxItems items.

Expected

Only D1 is selected, as id, title, and body. D2 fails tenant scope, D3 fails group access, D4 is deleted, and D5 has no granting group. D1's internalNote never appears, and a request that names tenant beta or adds the payroll group gets the same result.

Open coding workspace

Limitations

The selector assumes trusted, typed records and a fixed snapshot. It does not authenticate callers, validate external JSON, or handle permission changes after selection. In a real service, reject malformed access metadata, obtain current policy from trusted storage, and never let a client supply its own identity object.

Answer draft

Kept in this browser. Saving your score below keeps a copy with it.

0 / 5000

Step 4: Reference answer

Step 5: Score yourself

Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.

Step 6: Follow-up questions

An interviewer may ask these next.

Why inspect reranker input?

Why is tenant-only isolation insufficient?

Does redaction replace authorization?

What does a clean marker scan establish?

Why test retries?