AI Engineering
ai-engineering
Practice Problems
Validate a Review Summary
Check a model's JSON review summary against a small contract and return the accepted value, or one error code per failing field followed by the sorted unknown keys.
Fix a Webhook Reply Sender
Debug an SMS reply webhook handler that sent duplicate texts, left some messages unanswered, and multiplied model calls under rate limits, timeouts, and redeliveries: the tests replay each failure with scripted providers and a fake clock, and you fix each defect with the smallest change.
Bounded Retry Policy
Decide whether to retry a failed provider call: stop for unsafe replay, a permanent failure, used-up attempts, or a deadline that cannot fit the wait, and otherwise return a capped, jittered backoff delay that honors retry-after.
Evaluation Scoring Harness
Score an AI evaluation set: task success, answer coverage, and answered accuracy with their own denominators, per-slice results, and missing outputs counted instead of hidden.
Evidence Region Validator
Validate a model-proposed page-region citation against trusted document metadata: the revision, a zero-based page index, and a normalized rectangle of finite numbers, then return a new minimal record or the exact error of the first failed check.
Experiment Outcome Summary
Validate one outcome and cost record per randomized learner, then report each A/B arm's success rate, mean cost, and cost per success, plus absolute and relative lift, with null wherever a denominator is 0.
Leakage-Safe Context Selection
Select the documents a user may see from trusted identity alone, rank them by query terms in the title and body, and return at most maxItems minimized context items without ever exposing an internal note.
Lexical and Dense Ranking
Rank help articles by shared words and by cosine similarity with a deterministic tie rule, then score each ranking with precision@k and recall@k.
Handle MCP Tool Requests with Schema Validation
Implement the request handler of a small read-only MCP server for the 2026-07-28 revision: check the JSON-RPC envelope and per-request _meta, list tools with cache hints, validate arguments from each tool's declared schema, and put every failure in the right channel.
RAG Chunk Eligibility
Return the first reason a retrieved chunk is excluded for a user (tenant, deletion, publication state, revision, or department), or null when the chunk is eligible.
Serving Timing Summary
Summarize an inference load test: validate the window and every timing record, count throughput from successful work over the whole window, and report nearest-rank p95 latency, time to first token, and per-request time per output token, with null when a metric has no values.
Fine-Tuning Dataset Validator
Validate a supervised fine-tuning dataset before training: exactly three splits, unique IDs, chat rows that end in an assistant target, and no exercise family or whitespace-normalized conversation shared between splits.
Streaming State Reducer
Write the pure reducer behind a streamed assistant answer: accept only contiguous events from the active run, ignore replays and stale runs, mark gaps as interrupted without losing text, keep finished runs final, and never mutate the input.
Audit a Model Evaluation for Leakage and Misleading Metrics
Audit a churn model's experiment manifest for leaky features, preprocessing fitted beyond train, evaluation splits that overlap the training label window, choices made on test, and reports on the wrong population, then compare accuracy with the majority-class baseline next to precision and recall.
Validate Ticket Output
Validate a model's JSON reply for a support-ticket contract: return the accepted ticket, or the first failing layer (parse, schema, or source).
Validated Tool Adapter
Validate a model-proposed get_ticket call (tool name, exact arguments, ID format), take the tenant and read permission from trusted context, and return a copy of only that tenant's ticket or a controlled error code.
AI Engineering
Correct common LLM assumptions
Explain why a context window is not a word count, why a model does not remember earlier chats on its own, and why temperature zero does not make an answer true.
Diagnose incomplete model output
Work out why a model's answer looks cut off or incomplete by reading termination metadata, token budgets, and stream evidence before changing any setting.
Choose long context or retrieval
Compare sending whole documents in a large context window with retrieving a few passages for two features of one product, using window fit, cost with prefix caching, latency, access, and a sliced pilot, and say what would change each choice.
Select a model under quality, cost, and latency limits
Recommend a model or routing policy from measured quality, cost, and latency, say what evidence is still missing, and revise the decision when the traffic mix changes.
Write an extraction prompt with examples
Rewrite a vague request into a testable extraction prompt with category rules, a missing-data policy, consistent examples, and a JSON output contract, then explain why parseable JSON can still be wrong.
Validate a model's JSON review summary
Write a pure function that accepts a model's JSON review summary or returns every contract violation as a stable error code, predict its result for eight replies, and say what a valid summary still cannot prove.
Validate JSON output and recover from failures
Implement a dependency-free validator for a ticket contract, classify each fixture failure by layer, and design a two-attempt recovery that never commits invalid output.
Design a versioned output contract and migration
Upgrade a ticket classifier to a versioned contract with multiple IDs, evidence, and review states, keep two v1-only consumers working, and decide from evaluation data whether a candidate prompt should ship.
Explain semantic and exact-match retrieval
Explain how embeddings and exact-term matching find candidates for a query, and why the top result is a ranking signal rather than a verified answer.
Compare lexical and dense retrieval on fixtures
Rank four help articles by shared words and by cosine similarity, score both rankings against labels, and explain a dense miss and an accidental lexical success.
Design hybrid retrieval within a latency budget
Design hybrid retrieval with reranking for a multi-tenant help center, choose a configuration from measured quality and latency gates, and keep access rules intact through timeouts and model migrations.
Explain retrieval vs. generation
Explain how documents become evidence for a generated answer, and why an answer can be wrong even when it cites a real document.
Retrieve evidence for a follow-up question
Turn a follow-up message into a standalone retrieval query, keep access filters tied to the signed-in session on every turn, and decide which turns should not retrieve at all.
Diagnose a missing document
Trace a missing policy exception through ingestion, eligibility, and context packing without weakening access rules.
Design a cited support assistant with measurable quality
Design a support assistant that answers from help articles with citations the customer may see, or hands off to a human. Turn its requirements into metrics, plan its evaluation set, keep it fresh through updates and deletion, and check its token, cost, and latency budgets.
Design multi-tenant RAG with deletion
Design a private multi-tenant policy assistant whose answers and citations stay correct through asynchronous updates, retries, permission changes, and deletion.
Debug a partially failed re-embedding backfill
Find why a partly failed re-embedding backfill left a help-center index missing, duplicating, and mixing vectors, then plan a repair and a cutover gate you could defend in a release review.
Trace a tool call from request to answer
Follow one ticket lookup from tool declaration to answer: what the model proposes, what the application validates and executes, and what the assistant says when the result is unavailable.
Implement a validated, tenant-scoped tool adapter
Implement a get_ticket adapter that validates model-proposed arguments, takes tenant scope from trusted context, and passes a fixture table, then explain how a bounded orchestrator handles timeouts, invalid calls, and parallel lookups.
Extend a tool adapter with a search tool
Extend the validated ticket adapter with a paginated search_tickets tool behind a tool registry under a 3-hour cap, hand in tests and a README, then review a sample submission as the interviewer would.
Choose a fixed workflow or an agent loop
Compare a fixed workflow and an agent loop for a parcel-delivery assistant's rescheduling and stuck-parcel tasks using a pilot's success, cost, and latency figures, then recommend a design per task and say what would change it.
Design an agent with idempotent side effects
Design an agent that closes a ticket and sends one approved closure email safely across duplicate jobs, crashes, timeouts, and concurrent workers, and say exactly what the design can and cannot guarantee.
Separate retrieval, packing, and generation failures
Use a request trace to tell retrieval, context-packing, and generation failures apart, and report answer quality with a named denominator next to operational signals.
Build and score a small evaluation set
Score a six-case evaluation set with a deterministic harness, report success, coverage, and answered accuracy with named denominators, and keep missing outputs visible in the totals.
Design a release gate with slices and uncertainty
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
Contain an indirect prompt injection
Explain why a retrieved document cannot grant itself authority, and name the controls in trusted code that still hold when the model follows an injected instruction.
Prove a context selector cannot leak data
Implement a context selector that enforces tenant, group, and deletion rules before ranking, then design tests proving that forbidden records and internal notes never reach the model-bound payload.
Threat-model an agent with private data
Threat-model a multi-tenant assistant that reads private documents, keeps conversation summaries, and sends approved notifications, naming where each control is enforced and what risk remains.
Explain latency, throughput, and rate limits
Explain why a slow model provider raises concurrency and retry load, using latency, throughput, rate limits, deadlines, and a bounded response strategy.
Implement a bounded retry policy
Implement a pure, deterministic retry policy with attempt limits, capped jittered backoff, retry-after, and a deadline check, then fit it into an executor that handles cancellation, partial streams, and uncertain writes.
Design SLOs, rollout, and cost controls
Design reliability controls for a multi-tenant AI assistant: SLOs and error budgets, quota-aware admission, fallback and cost reservations, staged rollout with a canary decision, rollback, and recovery.
Debug an integration with partial failures and retries
Find why an SMS reply webhook sent duplicate texts, left some messages unanswered, and multiplied model calls under rate limits, timeouts, and redeliveries. Fix each defect with a regression test, then declare the release check for the fix.
Explain train, validation, and test splits
Explain what training, validation, and test splits are each for, then repair a fictional marketplace's evaluation plan by choosing a split that matches how its model will be used.
Diagnose leakage and misleading metrics
Implement an audit that flags leaky features, preprocessing, time splits, selection, and mismatched reports, predict its findings on five fixture experiments, and diagnose why a churn model that looked excellent offline failed after launch.
Design time-aware evaluation and drift response
Design rolling-origin backtests, capacity-bound metrics, label-delay handling, and drift monitoring for a synthetic card-fraud model whose chargebacks arrive weeks late, then decide whether to promote, shadow, or hold a retrained candidate.
Stream, stop, and replace a response
Explain how a streaming feedback UI shows partial output, what Stop really cancels, and how run IDs keep late events from an old attempt out of a new answer.
Implement a streaming state reducer
Implement a pure reducer that keeps a streamed answer consistent through stale runs, replays, gaps, and stops, and predict its transitions. Then review a fetch integration for decoding, framing, cancellation, and end-of-stream defects.
Design a resilient streaming chat
Design a streaming interview-practice chat that keeps one answer correct across reconnects, several open tabs, and cancellation races, checks citations against current access, and never repeats its booking tool during recovery.
Decide whether to fine-tune
Decide whether a feedback assistant's failures call for prompting, retrieval, or fine-tuning, and explain the reviewed data, grouped split, and baseline comparison that fine-tuning would need.
Validate a fine-tuning dataset and plan SFT
Implement a split validator for a fine-tuning dataset, predict its results on fixture cases, find the leaks it cannot catch, and plan a supervised fine-tuning experiment against a prompted baseline.
Design a model specialization program
Design a specialization program for a fictional hint model within GPU-hour and expert-review budgets: choose learning signals and parameterization, protect splits and evaluation, and plan a release with rollback. Then investigate a candidate that won a preference comparison while experts found wrong explanations.
Handle an uncertain screenshot
Route a blurry code screenshot and a mixed PDF to suitable extraction methods, and explain the result with evidence the user can check and honest uncertainty.
Validate page-region citations
Implement a validator for page-region citations and predict its result on fixed cases. Then design extraction for a mixed PDF with a rotated scan and a table split across two pages.
Design a verifiable document and media pipeline
Design a pipeline that turns a partly scanned course pack and a lesson recording into cited study material, tracking coverage, transforms, and cross-modal conflicts, and decide what it may answer while processing is still partial.
Diagnose a slow first token
Split a slow first token into queue wait, prefill, and emit from a server trace, and choose a serving configuration per workload instead of by throughput alone.
Profile TTFT, throughput, and failures
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
Choose a hosted API or a self-hosted model
Compare a hosted model API with a self-hosted open-weight model for a live tutor and a nightly scoring job: apply a quality gate, compute monthly cost at real utilization and the breakeven volume, check a data-location rule, and say what would reverse the choice.
Choose a serving strategy under tail-latency limits
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.
Make human oversight meaningful
Examine a fictional AI coach whose feedback passes a human Approve button, calculate what careful review would cost, and propose the purpose, data, review, and ownership controls that make the oversight meaningful.
Specify a review and escalation workflow
Specify how disputed AI feedback is re-reviewed, corrected, and escalated, with server-side authorization, artifact-bound approval, deadlines, and concurrency rules, then prove the rules with acceptance fixtures.
Govern AI outputs that may be repurposed
Decide what happens when a partner repurposes formative AI practice ratings for candidate screening, and design the authority, enforceable controls, evidence, and remedy that make the decision hold.
Define a user outcome and baseline
Explain why longer sessions do not justify shipping AI hints, then use a fictional pilot's per-arm figures to propose an evaluation with a real baseline, an independent learning outcome, and guardrails.
Design an A/B test with cost guardrails
Write the experiment contract and sample-size plan for a staged-hints A/B test, implement a validated per-arm summary of outcomes and costs, and explain which checks and guardrails must pass before a lift can justify shipping.
Decide whether to ship on conflicting evidence
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
Explain how a coding assistant gets context and tools
Explain how a coding assistant assembles its context, which component decides whether each proposed action runs, and why an instruction found inside a file cannot grant itself permission.
Build a small MCP server with validated tools
Implement the request handler of a small read-only MCP server that validates tool arguments from their declared schemas and puts each failure in the right channel, then review the trust model of adding a shell tool and installing a third-party server.
Design an agent harness for a team codebase
Design one agent harness for a fictional payroll company's developer machines and its CI agent, covering permissions, sandboxing, extensions, hooks, evaluation, and budgets. Then decide from predeclared rules which configuration rolls out first, where, and with which permissions.
Choose which team workflows to hand to AI
Judge six workflows at a fictional home-services company by volume, error cost, checking effort, data, and the people affected, set an automation level and a human checkpoint for each, and choose the first pilot.
Set team rules for AI tools and company data
Write the rules a 30-person accounting team follows when it uses AI tools with client data. Classify the data, match each class to a tool by its data terms, give alternatives and an exception path, and handle client data that staff already pasted.
Turn an executive's AI mandate into a scoped pilot
Answer a COO who wants AI on every customer email and support costs down by a third: find the goal behind the request, check both targets against the team's figures and rules, and commit to a pilot you can measure.
Measure an AI pilot's quality, cost, and return
Judge a six-week AI order-entry pilot against a team that did not use the tool, count its full cost and quality per order channel, and recommend whether to expand, fix, or stop it with a budget cap and a review date.
Decide whether to build or buy an AI tool
Choose between the AI feature in your finance suite, a specialist vendor, and an in-house build: apply hard gates first, read a pilot on your own invoices, compare three-year cost on one basis, and recommend one option with contract conditions and an exit plan.
Review a team-built AI automation before launch
Review an office manager's inbox automation step by step after a two-week test, price normal running and a runaway mail loop, and decide what launches, under which conditions, and what, if anything, waits.
Judge whether coding assistants helped your team
Use three quarters of team delivery data to separate what coding assistants changed from a holiday freeze and a migration, check whether code review became the bottleneck, and recommend which seats to renew and whether and how to try agent mode.
Plan an AI rollout across a support team
Plan a staged rollout of AI drafting, knowledge suggestions, and auto-tagging across a fictional retailer's 64-agent support team before peak season. Set the automation level per queue, size each stage against training, QA, cost, and calendar limits, write gates before results, and lead the people side honestly.
Lead the response to AI errors that reached customers
Lead the response when a fictional retailer's website chat assistant told customers the wrong return window and some were refused at the returns desk. Contain it in the first hour, scope it from records, decide and cost the remedy, plan each message, run a blameless review, and set the conditions to turn auto-send back on.