AI Developer Tooling · 3 to 4 hours
Design an agent harness for a team codebase
Design one agent harness for a fictional payroll company's developer machines and its CI agent, covering permissions, sandboxing, extensions, hooks, evaluation, and budgets. Then decide from predeclared rules which configuration rolls out first, where, and with which permissions.
4 h
1
Step 1: The question
Using the payroll company's scenario and the evaluation results, design one agent harness for its developer machines and its CI agent: the loop and stopping rules, context, permissions and sandboxing, MCP servers and skills, hooks, evaluation, budgets, audit logging, and incident response. Compute both configurations' per-area accepted rates with 95% Wilson intervals, the paired comparison, cost per accepted change, daily spend, and peak concurrency. Then apply the rollout rules to decide which configuration the CI agent uses first, in which repositories, and with which permissions.
Constraints
- Production payroll data and production credentials never reach developer machines or the CI environment
- The CI agent produces draft pull requests only; no agent pushes to a protected branch, approves, or merges
- Apply rollout rules R1 to R5 as written and in order; a different rule is a recorded change for a later stage
- Prices are fictional; ignore prompt caching, and assume each configuration's mean cost and wall-clock time hold in every area
- Name products only as examples; the design must work with any harness that offers equivalent controls
Deliverables
- Harness architecture: the loop, its stopping conditions, and the context plan
- Permission and sandbox policy for each use: tools and paths with allow, ask, or deny, network egress, credentials, and branch and pull request rules
- Extension and hook policy: which MCP servers and skills are admitted and how, and what each hook enforces
- Evaluation and budget figures for both configurations, with the arithmetic
- The rollout decision (configuration, repositories, permissions, stages), audit logging, and the response to a leaked secret and to injected issue text
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.