Judge whether coding assistants helped your team

Figures about a coding assistant come in three kinds, and only one of them says whether the team got better.

Activity is easy to count and easy to inflate: smaller slices of the same work make more pull requests, and an accepted suggestion may be rewritten an hour later. Report activity as context; judge on outcomes.

The SPACE framework (Forsgren and colleagues, ACM Queue, 2021) holds that developer productivity is about more than activity and cannot be measured by a single metric. Its five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow.

DORA (DevOps Research and Assessment, the research program behind the best-known delivery metrics) names two pitfalls: setting a metric as a goal, such as a mandate to deploy more often, makes teams likely to game it, and the metrics are for improving a team over time, not for competing with other teams. Work at team level, in each team's context, and never rank engineers by pull-request counts. The AI Product & Experimentation Easy question covers outcomes and baselines for user-facing features.

Two rules apply to every figure. A rate needs a stated denominator: per 100 PRs, per deployment, or per engineer per week. And totals come from averages: a median time multiplied by a count does not give total time.

When writing code gets faster, the wait moves to review and testing. Reviewers are often the most senior people, so their own planned work slips first, and no tool dashboard shows it.

DORA's guidance on working in small batches (updated December 2025) says AI adoption often leads to increased software delivery instability, that large changes are difficult to review, test, and integrate safely, and that reviewing machine-generated code can take more effort per line than reviewing code a person wrote. It advises against generating massive pull requests. DORA's 2025 report calls AI an amplifier of an organization's existing strengths and weaknesses.

To see whether review is the bottleneck, turn the figures into hours:

Review hours per reviewer per week = PRs merged per week x average reviewer minutes per PR / 60 / number of reviewers.

Compare the result with the time each reviewer can give. For example, 25 PRs a week at 18 minutes is 450 minutes, or 7.5 hours: 2.5 hours each for three reviewers. If time to first review and lead time rise as review hours pass capacity, the queue is in review.

Norms that help: a size limit per PR, authors listing the tests they ran, spreading review across more people (train another reviewer or rotate review), and review time protected in each reviewer's plan.

Quality signals arrive at different times: reverts within days of merging, failed deployments at release, and escaped defects weeks later, when a customer or a monitor finds them.

When defects are traced back to the period of the change, the most recent period always looks best, because its defects have not surfaced yet. A falling count for the latest period is usually the lag, not an improvement.

A period is complete once its newest change is older than the longest usual time for defects to surface. Compare only complete periods, or periods counted at the same age, and put the follow-up date in the plan. For example, if defects usually surface 3 to 8 weeks after a change, a quarter that ended 2 weeks ago is not ready; use reverts and failed deployments meanwhile.

Normalize every count, per 100 PRs or per deployment, before comparing: a team that merges more also reverts more in absolute terms.

A before-and-after comparison measures everything that changed between the periods, not only the tool. List the other changes, then adjust for each.

• Seasons. A code freeze or holiday leave lowers a quarter's output. Divide by the weeks open for merging, or compare with the same quarter a year earlier. A baseline quarter with 10 open weeks divided by 13 makes the next quarter look 30% better with no change at all (13 / 10 = 1.3). Leave also lowers output inside open weeks, which no week count corrects.
• One-off projects. A migration or a batch of scripted changes adds many PRs: count them separately.
• Changes that started at the same time. A team that changed its review rules when the seats arrived shows the effect of both.
• People. Joiners, leavers, and reorganizations change output and review capacity.

A team that barely used the tool gives a rough comparison team: the change in the high-use teams minus the low-use team's change, a "difference of differences", suggests how much of the change would have happened anyway. If the low-use team chose low use, its work and people may differ, so treat the gap as rough. The method is taught in full in "Measure an AI pilot's quality, cost, and return", a prerequisite of this level.

Satisfaction is a real dimension of productivity in SPACE, so keep asking, and add specific questions such as "Where do you lose time?" and "What takes longer to review now?".

Perception can differ from measurement. In a randomized study published on July 10, 2025, METR followed 16 experienced open-source developers on 246 real issues in large repositories they had contributed to for multiple years; each issue was randomly assigned to allow or disallow AI tools. Issues with AI allowed took 19% longer, yet afterwards the developers still believed AI had sped them up by 20% (beforehand they had expected 24%). The authors did not claim that their setting represents most software work.

That page now says its results are out of date and, in METR's view, no longer reflect the current impact of AI models. A post on February 24, 2026 about a follow-up (57 developers, more than 800 tasks) called the new data an unreliable signal, because many developers withheld tasks they did not want to do without AI and others declined to take part. It adds that self-reported speedups "can be quite unreliable".

Use these studies as a reason to measure carefully, never as a figure to expect for your own team.

• Cost per active seat = monthly bill / active users. State the definition of active (used in a typical week, or at least once in the last 28 days), because the same bill gives different figures. If only half the seats are used in a typical week, each active seat costs twice the list price.
• Seats never used since purchase are the first to release.
• Renewal options come down to all seats; seats for active users, with a quick way to ask for one back; or none. The request path keeps a saving from blocking someone who needs a seat.
• Add-ons such as an agent mode (the assistant makes a multi-file change on its own and opens a PR for review) are a different tool with their own price and risks. Price them separately, and do not assume results from plain seats carry over.
• The full cost includes reviewer and rework time, training, and admin, none of which is on the bill.
• A vendor's "time saved" usually multiplies a count, such as accepted suggestions, by an assumed number of minutes each: a model, not a measurement. It counts suggestions later rewritten and leaves out review and rework.

When the evidence is mixed, or a new tool has none yet, try it on one team before buying it for everyone.

1. Change one thing, in one team, for a fixed period, and compare with that team's own recent quarters measured the same way.
2. Pick a team where the effect can be seen: steady work with no one-off project, spare review capacity, review norms, such as a size limit, in place before the trial starts, and people willing to take part.
3. Before starting, write down one or two outcome measures, the guardrails (such as reverts, failed deployments, review hours per reviewer, and change size), the cost, and the thresholds that mean expand, adjust, or stop.
4. Stop rules apply during the trial, not only at the end. Name an owner. Set one date to judge flow and cost, and a later one to judge quality once defects have had time to surface.
5. Tell leadership what is known, what is not, and what you will know by when.

Work each figure from the scenario and the tables before you reveal it.

Use the fictional prices and Vendor A's dashboard from the scenario.

A slide went to the CTO before your analysis. Check each claim against the scenario and the tables.