AI Developer Tooling · 3 to 4 hours
Design an agent harness for a team codebase
Design one agent harness for a fictional payroll company's developer machines and its CI agent, covering permissions, sandboxing, extensions, hooks, evaluation, and budgets. Then decide from predeclared rules which configuration rolls out first, where, and with which permissions.
4 h
Overview
Design the harness around a coding agent: the loop, context, permissions, sandbox, extensions, hooks, evaluation, budgets, and operations. A strong answer names the control that enforces each rule, evaluates configurations on tasks from the team's own history, and makes a scoped rollout decision from rules written in advance.
Study time: 3 to 4 hours
Interview format: System design discussion with an evaluation and budget review
By the end you can
- Describe the agent loop and the stopping conditions that a harness, not the model, enforces
- Plan context for a large monorepo with layered instruction files, search, read-only subagents, and compaction
- Write allow, ask, and deny rules per tool and path for a local and a CI agent, and back them with an OS-level sandbox, an egress allowlist, and scoped credentials
- Admit MCP servers, skills, and hooks under a reviewed, pinned, least-privilege policy, and use hooks for checks that must run every time
- Evaluate harness configurations on a task suite with hidden tests, intervals, paired comparisons, and budgets, and make a scoped rollout decision from predeclared rules
- Plan audit logging, staged rollout, and the response to a leaked secret and to injected issue text
Prerequisites
Helpful background: Comfort with git branches and pull requests, CI pipelines, and binomial proportions helps.
This topic covers the tools an engineer uses in their own AI-assisted development: coding assistants, MCP servers, skills, hooks, repository instructions, and agent harnesses. Tool Calling & Agents covers the tools a model calls inside your application. AI for Team and Tech Leads covers whether a team adopts these tools, the rules staff follow, and measuring the effect on the team's work.
In this guide
- Scenario: one harness for two uses
- The loop, and who decides when it ends
- Context for a large codebase
- Permissions, sandbox, and credentials
- MCP servers and skills are code you install
- Hooks for checks that must run every time
- Design the harness
- Evaluate agent output on your own history
- Budgets and capacity
- Evaluation results and rollout rules
- Work the rollout numbers
- Operations and incidents
- Incident drill
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
AI Engineering
Works a release decision from paired comparisons, protected slices, and zero-failure bounds under criteria fixed in advance.
AI Engineering
Works error budgets, fallbacks, and a canary decision for a multi-tenant AI assistant, a companion to the staged rollout here.
System Design
Its secure development section covers secrets managers, rotation, egress filtering, and package pinning around a sandboxed agent.
System Design
Its case studies on repository-scoped tokens and workload identity show how to give each agent and CI job a narrow credential.
System Design
Applies Little's law with headroom to size concurrency, the same arithmetic as peak concurrent agent runs.
Practice the interview question
Answer it in your own words before you look at the reference answer.