Measure an AI pilot's quality, cost, and return

A pilot is a short trial of a tool on real work. First fix the unit of work: for an order desk, a correctly submitted order, not a draft the tool produced. Measure the full time per order, from the document arriving to the order being submitted, including the time people spend reviewing AI output, and add the time spent fixing orders that turn out wrong.

A pilot answers three questions:

1. Time. Does the tool save time compared with what would have happened anyway?
2. Quality. Does quality hold where errors reach customers?
3. Return. Does it pay, and at what scope?

The options are expand, fix and re-test, or stop, and they can differ by type of work. Write down what "worth it" means before you open the results (for example, "quality no worse than the baseline"), so the numbers cannot be bent to fit a wish.

A baseline is the same measure taken before the change. A before-and-after on the pilot team alone mixes the tool's effect with everything else that changed in those weeks: the season, a new catalogue, people getting better at the job.

A comparison team doing the same work without the tool shows what would have happened anyway. Take each team's change from its own baseline, then subtract the comparison team's change from the pilot team's. This is a "difference of differences" (also called difference in differences). A neutral example:

Three checks keep it honest:

• Parallel trends. The method assumes both teams would have moved in parallel without the tool. Did their baseline weeks move together?
• What it cannot remove. It removes a fixed gap (one team was simply faster), not a difference in how teams respond. Volunteers may take to a tool faster than people who join later, so measure the next group instead of assuming.
• Small groups. With about a dozen people per team, count how many individuals improved, so that one or two people cannot drive the result.

List every cost line, not only the vendor's invoice:

One-time costs set the payback; monthly costs decide whether the tool pays at all.

A freed staff hour is money only when it removes a real cost: overtime, temporary staff, or a planned hire. Otherwise it is capacity, reported in hours, not dollars.

Money already spent (a sunk cost) does not argue for continuing: it is gone whatever you decide. Finance may still report payback from the program's start; that is a reporting view, not a reason to continue.

Two measures answer different questions. Errors in AI drafts before review, found by an audit that checks a sample of drafts field by field against the source, show what reviewers must catch. Work corrected after submission shows what got through to customers. Report both as rates per 100, by field and by type of input.

Weight an oversampled audit. Audits often oversample a rare or risky type on purpose. A pooled rate (all errors over all drafts checked) then overcounts that type, so weight each type's rate by its share of real work. Neutral example: 80% of items are type P and 20% type Q, and an audit of 100 of each finds 5 and 30 errors. Pooled: 35 / 200 = 17.5 per 100, too high. Weighted: 0.8 x 5 + 0.2 x 30 = 10 per 100.

Small counts move a lot. 2 errors in 200 is 1.0 per 100; one more makes it 1.5. Do not rank fields on counts that small; a gap of several times between types is worth acting on.

Averages hide segments. One type of input can lose time and add errors while the others gain, and the average still looks good.

Watch for over-trust. People tend to accept output that looks clean and confident (automation bias: excessive deference to automated systems). Review that catches errors on easy inputs can miss them on hard ones, so set draft errors beside corrections for each type.

Vendor dashboards often report "time saved": usually activity (drafts, documents, runs) multiplied by an assumed number of minutes per item. Ask three questions of any such figure:

1. What is counted? Items the tool touched, or finished work that reached the customer correctly?
2. What value per item is assumed, and where did it come from? A default setting, or a measurement on your own work?
3. Compared with what? Your own baseline, and a comparison group over the same weeks?

Report your measured figure with a short, line-by-line reconciliation from the vendor's figure, so finance can see why they differ. The activity count is still a useful usage signal, for example to watch spend.

Use the three scenario sections. Work each figure before you reveal it.

• Cost per order = total monthly cost / orders in scope. Fixed costs make a small scope expensive per order, and adding orders can lower cost per order even when the added orders lose money. Cost per order also leaves out time saved or lost, so compare scopes on net monthly benefit.
• Net monthly benefit = value of freed hours (within the cash rule) minus every monthly cost, including the cost of errors.
• Payback = one-time costs / net monthly benefit, in months.
• Break-even = the time saving at which freed hours just cover the monthly cost. Divided by the orders in scope, it gives a break-even saving per order to watch later.

Neutral example (fictional figures): a tool costs $3,150 a month to run and $14,400 to set up, and saves 150 hours a month at $37 an hour, so $5,550. Net: $5,550 - $3,150 = $2,400 a month. Payback: $14,400 / $2,400 = 6 months. Break-even: $3,150 / $37 = 85.1 hours a month.

Then run a pessimistic case, such as "the next group saves less", and check that the decision still holds.

Three ways to continue, plus one pessimistic case.

A recommendation a director can act on has more than a verdict:

• Scope per segment. Expand where the evidence is good, fix and re-test where a cause is known, stop where the tool does harm.
• Conditions and a stop rule, written before the next stage and tied to measured figures: a break-even saving and a quality floor such as "corrections no worse than baseline".
• A budget cap that falls back to manual work when it is reached, so it never stops orders, sized from forecast volume plus headroom.
• A spend alert on a ratio that should be stable, such as documents per order, and on the month-to-date pace, so a spike shows before the bill does.
• Staggered waves, so the group not yet using the tool is the comparison; judge each group after its learning weeks.
• An owner and a review date, and a word to staff on what changes for them: who reviews, and what freed time is for.