Prompting & Structured Output · 60 to 90 minutes
Validate a model's JSON review summary
Write a pure function that accepts a model's JSON review summary or returns every contract violation as a stable error code, predict its result for eight replies, and say what a valid summary still cannot prove.
Live coding
1.5 h
1
Step 1: The question
Implement validateReviewSummary(raw) (Python validate_review_summary(raw)) for the review summary contract, and predict its result for replies R1 to R8 before you run it. List the tests you would write first and the unclear cases you would confirm before coding. Then say what the dashboard should do with a rejected reply, and name two ways an accepted summary can still be wrong.
The review summary feature
Suppose an online bookshop that sells new and secondhand books asks a language model to summarize each written customer review as JSON. A moderation dashboard shows the summary next to the review, lets moderators filter reviews by sentiment and topic, and averages the rating per book. A program reads every reply, so a reply that breaks the contract must never reach the database or the dashboard.
The replies are untrusted text: most follow the contract and some do not. The three fields mean:
rating: the model's estimate, from 1 to 5, of how satisfied the reviewer issentiment: the overall tone of the reviewtopics: up to three things the review discusses, fromprice,delivery,condition(the physical copy), andstory(the book's content)
For example, the review "Arrived two weeks late and the spine was cracked." gets this reply, and the contract accepts it:
{"rating": 2, "sentiment": "negative", "topics": ["delivery", "condition"]}The contract and the result
A reply is accepted only when it is a JSON object with exactly these three keys:
| Field | Rule |
|---|---|
rating | An integer from 1 to 5 inclusive. An integer is a JSON number with no fractional part, so 4.0 counts as 4. Booleans, strings, null, and fractions are not integers. |
sentiment | Exactly one of "positive", "mixed", "negative" (case-sensitive, no trimming) |
topics | An array of 0 to 3 items, each exactly one of "price", "delivery", "condition", "story", with no item repeated |
| Any other key | Not allowed |
The validator runs these checks in this order:
| Step | Code | Fails when |
|---|---|---|
| 1 | reply:parse | The text is not valid JSON |
| 2 | reply:type | The parsed value is not a plain object (an array, null, a string, a number, or a boolean) |
| 3a | rating:missing | The key is absent (a key present with the value null is not missing) |
| 3b | rating:type | The value is not an integer as defined above |
| 3c | rating:range | The integer is below 1 or above 5 |
| 4a | sentiment:missing | The key is absent |
| 4b | sentiment:type | The value is not a string |
| 4c | sentiment:value | The string is not exactly one of the three values |
| 5a | topics:missing | The key is absent |
| 5b | topics:type | The value is not an array |
| 5c | topics:too-many | The array has more than 3 items |
| 5d | topics:value | Any item is not exactly one of the four topics (one code, however many items fail) |
| 5e | topics:duplicate | Every item is allowed, but one appears more than once |
| 6 | extra:<key> | One code per key the contract does not name, with the key exactly as written |
- Steps 1 and 2 end validation at once: the result holds only that code.
- Steps 3, 4, and 5 each add at most one code: the first rule in the listed order that fails for that field. All three fields are always checked.
- Field codes come first in contract order (rating, sentiment, topics), then the extra codes, sorted by plain string comparison (JavaScript's default
sort(), Python'ssorted()). Only the extra codes are sorted. - No code means the reply is accepted.
The result, in TypeScript terms:
type ReviewSummary = {
rating: number; // an integer from 1 to 5
sentiment: "positive" | "mixed" | "negative";
topics: ("price" | "delivery" | "condition" | "story")[];
};
type ValidationResult =
| { ok: true; value: ReviewSummary }
| { ok: false; errors: string[] };
declare function validateReviewSummary(raw: string): ValidationResult;Python's validate_review_summary(raw) returns a dict with the same keys, using lists, never tuples. An accepted value is a new object with its keys in contract order, rating as an integer (a Python int, also for an input such as 4.0), and topics in the reply's order. A failed result has no value key, and an accepted one has no errors key.
Constraints
- A pure function: no network, clocks, randomness, or model calls, and no exception for bad model output
- Exactly the three contract keys; do not repair values (no trimming, case folding, or converting strings to numbers)
- One code per failing field (its first failing rule) in contract order, then one code per extra key, sorted
- Answer in JavaScript or Python; both languages must agree on booleans, 4.0, and the order of keys
Deliverables
- The contract restated, with its unclear cases settled before coding: null versus missing, unknown keys, letter case, and 4.0
- A validator that passes the practice problem's tests
- The result for each of R1 to R8: the accepted value or the ordered error codes
- A test list covering valid, invalid, boundary, and cross-language replies, grouped by rule
- What happens to a rejected reply, and two things the validator cannot prove
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Predict the outcome
Predict the validator's results
Apply the contract to each reply. Write the accepted value, or the error codes in order, before you reveal the answers.
| Reply | Raw reply text | Expected result | Deciding rules |
|---|---|---|---|
| R1 | {"rating": 4, "sentiment": "positive", "topics": ["story", "condition"]} | ? | ? |
| R2 | The object {"rating": 5, "sentiment": "positive", "topics": ["delivery"]} inside a Markdown code fence: the line ```json before it and the line ``` after it | ? | ? |
| R3 | [{"rating": 3, "sentiment": "mixed", "topics": []}] | ? | ? |
| R4 | {"rating": "4", "sentiment": "Positive", "topics": ["price"]} | ? | ? |
| R5 | {"rating": 4.5, "sentiment": null, "topics": "delivery"} | ? | ? |
| R6 | {"rating": true, "topics": ["price", "delivery", "condition", "story"]} | ? | ? |
| R7 | {"rating": 6, "sentiment": "negative", "topics": ["delivery", "shipping", "delivery"], "priority": "high", "flagged": true} | ? | ? |
| R8 | {"rating": 5, "sentiment": "negative", "topics": []} | ? | ? |
Code exercise
Review summary validator
Implement validateReviewSummary(raw) (Python validate_review_summary(raw)) for the review summary contract. Return the accepted value, or every failing field's first failing code in contract order followed by the sorted extra keys. Never throw for bad model output.
Expected
The linked practice problem calls validateReviewSummary(raw) in JavaScript and validate_review_summary(raw) in Python; the result keys are the same in both languages. An accepted reply returns { ok: true, value }, where value is a new object with exactly rating, sentiment, and topics in that order, rating is an integer (a Python int, also for an input such as 4.0), and topics keeps the reply's order. A rejected reply returns { ok: false, errors } with the codes in the stated order. The function never throws. The six tests per language check: replies R1 to R8 in one comparison; text that is not JSON (empty, fenced, cut off, prose) and JSON that is not an object (an array, null, a string, a number, true); rating and sentiment edges (fractions, numeric strings, booleans, null, a missing key, both sides of each bound, letter case, a trailing space); topics edges (an empty array, three and four items, duplicates, unknown, wrongly cased, and non-string items, a string, null, an object, and a missing key); extra keys sorted after the field codes, including integer-like and wrongly cased names, and {} with all three fields missing; and the result shapes (the accepted value's key order, 4.0, a reply padded with whitespace, results that later calls do not share, and a failed result with only ok and errors).
Limitations
This validator checks structure only. It does not read provider response states such as a refusal or a truncated reply, does not reject repeated keys (both parsers keep the last value), and does not treat NaN and Infinity the same way in both languages (Python's json.loads accepts them by default; pass parse_constant to reject them). It cannot judge whether the rating, sentiment, or topics match the review. A larger contract is easier to keep correct with a maintained JSON Schema validator whose errors are mapped to these codes.
Diagnose
Review two ports of the validator
Two teammates ported the validator before the contract was final. Port A (JavaScript) and Port B (Python) match the contract except for the checks below. has(key) checks for an own key, and TOPICS is the list of four allowed topics.
// Port A (JavaScript): rating and topics checks
if (!has('rating')) errors.push('rating:missing');
else if (typeof data.rating !== 'number') errors.push('rating:type');
else if (data.rating < 1 || data.rating > 5) errors.push('rating:range');
const topics = data.topics;
if (!has('topics')) errors.push('topics:missing');
else if (topics.length > 3) errors.push('topics:too-many');
else if (!topics.every((topic) => TOPICS.includes(topic))) errors.push('topics:value');
else if (new Set(topics).size !== topics.length) errors.push('topics:duplicate');# Port B (Python): rating check
if "rating" not in data:
errors.append("rating:missing")
elif not isinstance(data["rating"], int):
errors.append("rating:type")
elif not 1 <= data["rating"] <= 5:
errors.append("rating:range")- Which of R1 to R8 does each port get wrong, and what does it return instead?
- What does each port do with
{"rating": 3, "sentiment": "mixed", "topics": null}and with{"rating": 4.0, "sentiment": "mixed", "topics": []}? - Give a one-line fix for each defect and a test that would have caught it.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: Restates the three fields and both result shapes, and settles null versus missing, unknown keys, letter case, and whole-number floats before writing code.
What earns a 2: Checks each field's runtime type in the chosen language and shows, with a test, how a numeric string, a fraction, a boolean, null, and an array where an object is expected are handled.
What earns a 2: Returns at most one code per field in the stated order, sorts only the extra keys, never throws on bad output, and ties every R1 to R8 prediction to the rule that decides it.
What earns a 2: Lists table-driven tests with expected results for a valid reply, one failure per rule, both sides of every bound, combined failures, and cross-language replies.
What earns a 2: Says what happens to a rejected reply and how its codes are used, and names checks beyond structure that an accepted summary still needs, with where each would run.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Runtime type checks or Error codes and order or Limits and handling
Kept in this browser until you sign up, then it moves to your account.
Complete this level when your validator passes the practice problem's tests, your R1 to R8 predictions match the reveal with the rule behind each code, and you can explain what the validator cannot prove.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why is `typeof value === "object"` not enough to accept a parsed JSON object in JavaScript?
Why does an integer check in Python need a separate test for booleans?
How does a missing field differ from a field set to null?
Why reject keys the contract does not name instead of ignoring them?
Why return stable error codes instead of prose messages?
7