AI for Team and Tech Leads · 2 to 3 hours
Judge whether coding assistants helped your team
Use three quarters of team delivery data to separate what coding assistants changed from a holiday freeze and a migration, check whether code review became the bottleneck, and recommend which seats to renew and whether and how to try agent mode.
Business case
3 h
Used in interview rounds:
Leadership Scenarios (Medium)1
Step 1: The question
The CTO asks you, the product group's engineering manager, whether to renew all 120 coding-assistant seats for a year and add agent mode for every engineer. Using the scenario, the team tables, and the slide's claims, separate activity from outcome measures, work out what changed in each team once the freeze and the migration are accounted for, and compute review load per senior reviewer and cost per active seat. Then recommend which seats to renew, what to change in how the teams work, and whether and how to try agent mode, with the measures and stop rules you would fix in advance.
Scenario: the route-planning company's product group
All figures are fictional.
Suppose a fictional company sells route-planning and warehouse software to freight carriers. It has 120 engineers. You manage its product group: 32 engineers in four teams of 8, named Routing, Warehouse, Driver App (the mobile app drivers use), and Billing. In every team, each pull request (PR: a proposed code change that a teammate reviews before it is merged) needs approval from one of the team's two senior reviewers.
At the start of Quarter 1, the company bought coding-assistant seats for all 120 engineers from Vendor A, at a fictional price of $23.70 per seat per month, billed for every seat whether it is used or not. Quarter 2 has just ended, and the first term ends with it; seats released at renewal can be added back within a day. The CTO asks for your recommendation: renew all 120 seats for 12 months, and add Vendor A's agent mode for all 120 at a fictional $41.70 per seat per month. Agent mode lets the assistant plan a change, edit several files, and run tests on its own, then open a PR for a person to review.
Facts and rules (each quarter has 13 weeks):
- Quarter 0 (October to December) is the baseline, before the seats. The customers' holiday peak froze merging for 2 weeks in December, so Quarter 0 had 11 weeks open for merging; nothing was merged or reviewed during the freeze. Many engineers also take leave in late December.
- Quarters 1 (January to March) and 2 (April to June) had all 13 weeks open.
- From mid-January to the end of June, Driver App moved its screens to a new user-interface framework (the code library its screens are built on). A conversion script plus the assistant produced the migration PRs: 260 in Quarter 1 and 208 in Quarter 2, all counted in Driver App's PRs merged.
- At the start of Quarter 1, Warehouse adopted a 400-line limit per PR and a rule that authors list the tests they ran.
- Billing works under payment-compliance rules and chose to use the assistant little.
- The senior reviewers agreed they can each give about 6 hours a week to reviews before their own planned work slips.
- Reviewer minutes per PR are the senior reviewer's time on that PR, as an average.
- A failed deployment needed an immediate fix or rollback; failed deployments divided by deployments is the change fail rate.
- Escaped defects (bugs found in production) are traced back to the quarter of the change that caused them. Here, defects usually surface 4 to 10 weeks after the change ships. Counts are as of the end of Quarter 2.
- No one joined or left the four teams during the three quarters.
Vendor A's admin dashboard for Quarter 2 (whole company, fictional): 120 seats; 79 weekly active seats (the average number used in a week); 96 seats used at least once in the last 28 days; 11 seats never used since purchase; 118,000 suggestions accepted, an acceptance rate of 31%; "estimated hours saved" 2,950, which the vendor computes as accepted suggestions x 1.5 minutes. Only the product group has delivery data; for the other 88 engineers you have only this dashboard.
Three quarters of team data
All figures are fictional. Q0, Q1, and Q2 are Quarters 0, 1, and 2.
Delivery and review:
| Team, quarter | PRs merged (count) | PR size, lines (median) | Hours to first review (median) | Reviewer minutes per PR (average) | Lead time, commit to production, days (median) |
|---|---|---|---|---|---|
| Routing Q0 | 264 | 120 | 4 | 20 | 2.0 |
| Routing Q1 | 364 | 210 | 9 | 27 | 2.5 |
| Routing Q2 | 416 | 260 | 14 | 30 | 3.5 |
| Warehouse Q0 | 220 | 140 | 5 | 24 | 2.5 |
| Warehouse Q1 | 286 | 150 | 5 | 24 | 2.0 |
| Warehouse Q2 | 312 | 145 | 6 | 25 | 2.0 |
| Driver App Q0 | 176 | 150 | 6 | 30 | 3.0 |
| Driver App Q1 | 520 | 90 | 7 | 12 | 3.0 |
| Driver App Q2 | 468 | 100 | 8 | 15 | 3.0 |
| Billing Q0 | 176 | 100 | 8 | 30 | 4.0 |
| Billing Q1 | 234 | 105 | 8 | 30 | 4.0 |
| Billing Q2 | 234 | 100 | 8 | 30 | 4.0 |
Quality:
| Team, quarter | Reverts within 7 days | Failed deployments / deployments | Escaped defects traced to the quarter's changes (as of the end of Q2) |
|---|---|---|---|
| Routing Q0 | 4 | 4 / 50 | 3 |
| Routing Q1 | 7 | 5 / 55 | 7 |
| Routing Q2 | 13 | 9 / 60 | 2 |
| Warehouse Q0 | 3 | 4 / 50 | 3 |
| Warehouse Q1 | 4 | 4 / 52 | 3 |
| Warehouse Q2 | 4 | 4 / 55 | 1 |
| Driver App Q0 | 2 | 3 / 30 | 2 |
| Driver App Q1 | 9 | 5 / 40 | 5 |
| Driver App Q2 | 8 | 4 / 40 | 1 |
| Billing Q0 | 1 | 2 / 26 | 2 |
| Billing Q1 | 2 | 2 / 26 | 2 |
| Billing Q2 | 2 | 2 / 26 | 1 |
Adoption and survey (8 engineers per team). Quarter 0 has no seat or survey figures: the seats did not exist yet.
| Team | Weekly active seats, Q1 | Weekly active seats, Q2 | "I feel more productive with the assistant" (end of Q2) |
|---|---|---|---|
| Routing | 6 | 7 | 7 of 8 |
| Warehouse | 7 | 7 | 6 of 8 |
| Driver App | 7 | 8 | 8 of 8 |
| Billing | 3 | 2 | 2 of 8 |
Constraints
- Use only the scenario's figures; all are synthetic and both prices are fictional.
- Each quarter has 13 weeks; Quarter 0 had 11 weeks open for merging.
- Work at team level: do not rank or judge individual engineers.
- Keep the recommendation to one page the CTO can read in five minutes.
Deliverables
- Each figure sorted into tool activity, delivery activity, or outcome, with the measures you would report
- Each team's change in throughput, review hours per senior reviewer against capacity, lead time, and quality, with the lag stated
- The events besides the assistant that moved the figures, and how you adjusted for each
- The seat bill, cost per active seat, and the cost of each option, including agent mode
- A recommendation on seats, team norms, and agent mode, with trial measures, stop rules, an owner, and review dates
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Predict the outcome
Work the team figures
Using the scenario and the team tables, work out each figure from the inputs listed, then reveal the value and the working. Divide Quarter 0 by its 11 open weeks unless a row says otherwise.
| Figure | Inputs from the scenario | Value | Working |
|---|---|---|---|
| Product group: PRs per engineer per week in Q0 and Q2, the slide's way | All four teams' PRs merged; 32 engineers; 13 weeks in every quarter | ? | ? |
| Product group: the same, per week open for merging | Q0 had 11 open weeks | ? | ? |
| Product group: per open week without Driver App's migration PRs | 208 migration PRs in Q2, none in Q0 | ? | ? |
| Each team: PRs per engineer per open week, Q0 to Q2 | 8 engineers per team; 11 and 13 open weeks | ? | ? |
| High-adoption teams together against Billing | Routing, Warehouse, and Driver App without migration PRs | ? | ? |
| Review hours per senior reviewer per week, Q0 and Q2, against the 6-hour capacity | PRs per open week x average reviewer minutes per PR / 60 / 2 reviewers | ? | ? |
| Reverts within 7 days per 100 PRs, Q0 to Q2 | Reverts and PRs merged | ? | ? |
| Change fail rate, Q0 to Q2 | Failed deployments / deployments | ? | ? |
| Escaped defects: which quarters can be compared, and Routing's change | Defects surface 4 to 10 weeks after the change; counts as of the end of Q2 | ? | ? |
Predict the outcome
Work the seat and review cost
Using the fictional prices and Vendor A's dashboard, work out each figure, then reveal the value and the working.
| Figure | Inputs from the scenario | Value | Working |
|---|---|---|---|
| Seat bill today, monthly and yearly | 120 seats at a fictional $23.70 per seat per month | ? | ? |
| Cost per active seat under two definitions | 79 weekly active seats; 96 used in the last 28 days | ? | ? |
| Renew only the seats used in the last 28 days | 96 seats; 79 weekly active | ? | ? |
| Renew only the weekly active seats | 79 seats | ? | ? |
| Agent mode for all 120 | A fictional $41.70 per seat per month | ? | ? |
| Agent-mode trial for one 8-engineer team for one quarter | 8 seats; 3 months | ? | ? |
| Change in Routing's senior review time per quarter, Q2 against Q0 | Review hours per senior reviewer per week from the team figures; 2 reviewers; 13 weeks | ? | ? |
| The vendor's hours saved | 118,000 accepted suggestions x 1.5 minutes | ? | ? |
Diagnose
Test the slide's claims
A staff analyst and Vendor A's account team put these claims on a slide for the CTO before your analysis:
- PRs merged per engineer per week rose 71% since the seats arrived.
- Driver App more than doubled its output, so the assistant helps most on mobile work.
- Escaped defects fell in Quarter 2 in every team, so quality held.
- 72% of engineers feel more productive, so the seats are paying off.
- The assistant saved 2,950 hours last quarter, far more than the seats cost.
- Billing hardly used the assistant and still improved, so the tool made no difference.
- Agent mode for all 120 engineers will multiply these gains.
For each claim, say whether the scenario's figures support it, partly support it, or do not; name the figures that decide it; and say what evidence would settle it.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: Sorts every figure into tool activity, delivery activity, or outcome, explains why modeled or self-reported figures are not delivery evidence, and bases the judgment on outcome measures with their denominators.
What earns a 2: Computes review hours per reviewer against the stated capacity, connects change size and review wait to lead time and quality for each team, and checks which periods are old enough to judge escaped defects.
What earns a 2: Names each event besides the assistant that moved the figures, adjusts for it with working weeks, separated one-off work, or a comparison team, and states what the adjustment still cannot rule out.
What earns a 2: Computes the bill, cost per active seat under a stated definition, and the cost of each option including add-ons, and counts review time the bill does not show.
What earns a 2: Makes a scoped recommendation on seats, team norms, and agent mode, with trial measures and stop rules fixed in advance, an owner, a review date that allows for quality lag, and what would change the decision.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Activity versus outcome measures or Confounders and fair comparison or Renewal and trial decision
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can adjust a before-and-after comparison for the events that distort it, compute review load and cost per active seat from the tables, and recommend a renewal and a trial whose measures and stop rules are fixed before it starts.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Which figures on a coding-assistant dashboard are activity measures, and what makes a measure an outcome?
How do you turn team figures into review hours per reviewer per week?
When is a quarter's escaped-defect count ready to compare with earlier quarters?
How do you compare throughput fairly when one quarter had a code freeze?
Why does cost per active seat depend on how you define active?
7