LLM Evaluation
llm-evaluation
Practice Problems
Evaluation Scoring Harness
Score an AI evaluation set: task success, answer coverage, and answered accuracy with their own denominators, per-slice results, and missing outputs counted instead of hidden.
AI Engineering
Select a model under quality, cost, and latency limits
Recommend a model or routing policy from measured quality, cost, and latency, say what evidence is still missing, and revise the decision when the traffic mix changes.
Validate JSON output and recover from failures
Implement a dependency-free validator for a ticket contract, classify each fixture failure by layer, and design a two-attempt recovery that never commits invalid output.
Design a versioned output contract and migration
Upgrade a ticket classifier to a versioned contract with multiple IDs, evidence, and review states, keep two v1-only consumers working, and decide from evaluation data whether a candidate prompt should ship.
Design a cited support assistant with measurable quality
Design a support assistant that answers from help articles with citations the customer may see, or hands off to a human. Turn its requirements into metrics, plan its evaluation set, keep it fresh through updates and deletion, and check its token, cost, and latency budgets.
Choose a fixed workflow or an agent loop
Compare a fixed workflow and an agent loop for a parcel-delivery assistant's rescheduling and stuck-parcel tasks using a pilot's success, cost, and latency figures, then recommend a design per task and say what would change it.
Separate retrieval, packing, and generation failures
Use a request trace to tell retrieval, context-packing, and generation failures apart, and report answer quality with a named denominator next to operational signals.
Build and score a small evaluation set
Score a six-case evaluation set with a deterministic harness, report success, coverage, and answered accuracy with named denominators, and keep missing outputs visible in the totals.
Design a release gate with slices and uncertainty
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
Decide whether to fine-tune
Decide whether a feedback assistant's failures call for prompting, retrieval, or fine-tuning, and explain the reviewed data, grouped split, and baseline comparison that fine-tuning would need.
Validate a fine-tuning dataset and plan SFT
Implement a split validator for a fine-tuning dataset, predict its results on fixture cases, find the leaks it cannot catch, and plan a supervised fine-tuning experiment against a prompted baseline.
Design a model specialization program
Design a specialization program for a fictional hint model within GPU-hour and expert-review budgets: choose learning signals and parameterization, protect splits and evaluation, and plan a release with rollback. Then investigate a candidate that won a preference comparison while experts found wrong explanations.
Define a user outcome and baseline
Explain why longer sessions do not justify shipping AI hints, then use a fictional pilot's per-arm figures to propose an evaluation with a real baseline, an independent learning outcome, and guardrails.
Design an A/B test with cost guardrails
Write the experiment contract and sample-size plan for a staged-hints A/B test, implement a validated per-arm summary of outcomes and costs, and explain which checks and guardrails must pass before a lift can justify shipping.
Decide whether to ship on conflicting evidence
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
Design an agent harness for a team codebase
Design one agent harness for a fictional payroll company's developer machines and its CI agent, covering permissions, sandboxing, extensions, hooks, evaluation, and budgets. Then decide from predeclared rules which configuration rolls out first, where, and with which permissions.
Measure an AI pilot's quality, cost, and return
Judge a six-week AI order-entry pilot against a team that did not use the tool, count its full cost and quality per order channel, and recommend whether to expand, fix, or stop it with a budget cap and a review date.