AI for Team and Tech Leads · 90 minutes
Choose which team workflows to hand to AI
Judge six workflows at a fictional home-services company by volume, error cost, checking effort, data, and the people affected, set an automation level and a human checkpoint for each, and choose the first pilot.
Scenario
1.5 h
Used in interview rounds:
Leadership Scenarios (Easy)Overview
Learn how a lead decides which of a team's recurring tasks to hand to AI, how far to automate each one, and where a person stays involved. You practice on six workflows at a fictional home-services company and finish by choosing the first pilot.
Study time: 90 minutes
Interview format: Hiring-manager scenario with one changed constraint
By the end you can
- Describe a team workflow by its volume, time per item, and what happens to its output
- Judge a workflow's fit for AI from error cost, reversibility, checking effort, data, and the people affected
- Estimate the weekly hours at stake and the time that checking AI output still costs
- Assign an automation level and place a human checkpoint that can actually catch errors
- Choose a first pilot and write its success measure, stop rule, and decision date before it starts
Prerequisites
Helpful background: No coding or AI background needed. The only arithmetic is multiplying weekly volume by minutes per item and dividing by 60.
This topic covers the decisions a lead makes about a team's use of AI: which work to automate and where people stay involved, staff data rules, building or buying tools, measuring quality, cost, and return, rolling out and training, and leading the response when AI output goes wrong. Responsible AI & Governance designs oversight and accountability for one system's effect on the people it serves, AI Product & Experimentation measures user-facing features with experiments, and AI Developer Tooling covers how engineers set up and secure coding assistants and agent harnesses.
Start from the work, not the tool
In this topic, a workflow is a recurring piece of business work: something arrives, a person produces an output, and someone uses that output. It is a business process, not an engineering pipeline.
Describe the work before you pick a tool:
- What arrives, and how often?
- What is produced, and who uses it?
- What happens if the output is late or wrong?
Then name the outcome that matters: hours saved, a faster answer for the customer, or fewer errors. Some tasks improve more with a template, a form, or a simple rule than with AI, so compare against that option too.
The UK government's AI Playbook makes the same point for public-sector teams: use cases should be led by business and user needs, not by what the technology can do, and sometimes AI is not the best solution.
Five questions for every workflow
Ask the same five questions of every workflow. They are the criteria for every decision in this guide.
| Question | Why it matters |
|---|---|
| How many items a week, and how many minutes each? | Sets how much time is at stake |
| What does a wrong output cost, and can it be undone before anyone is harmed? | Sets how close a person must stay to the work |
| How long does a person need to check one output, and can they tell a right output from a wrong one? | Decides whether checking saves time and whether the check is real |
| Which data does the work put into the tool, and is the tool approved for it? | Decides whether the workflow can start now |
| Who bears a wrong output: staff, customers, or a person a decision is about? | Raises the bar when customers or decisions about people are involved |
Volume tells you how much is at stake, not how far to automate. The level comes from the other four questions.
NIST's AI Risk Management Framework (voluntary guidance, not law) asks teams to document the expected benefits of an AI system and the costs of its errors (Map 3.1 and Map 3.2), and to define how people oversee it (Map 3.5).
Checking effort decides the saving
Two rules turn a list of workflows into hours:
- Weekly hours = weekly volume x minutes per item / 60.
- The most that checking each item can save, per item = minutes to do the task today minus minutes to check (and fix) the AI output.
Worked example from another setting (all figures synthetic). A school office writes 200 letters a week at 6 minutes each. Checking an AI draft takes 2 minutes.
| Measure | Arithmetic | Result |
|---|---|---|
| Today | 200 x 6 = 1,200 minutes | 20 hours |
| A person checks every draft | 200 x 2 = 400 minutes | about 6.7 hours |
| Most that can be saved | 1,200 - 400 = 800 minutes | about 13.3 hours, before edits and rework |
If checking takes as long as doing the work, checking every item saves nothing.
A check also counts only if the checker can catch the error: they know the facts, can read the output, and have the source in front of them. Automation bias is the tendency to accept what an automated system produces; NIST's Generative AI Profile describes it as excessive deference to automated systems. A check that takes a few seconds is often not a check at all.
Five levels of automation
An automation level says how much of the work the AI does and where a person stays involved. This guide uses five levels. The examples come from other settings.
| Level | What the AI does | Where the person is | Example |
|---|---|---|---|
| 1. No AI | Nothing | A person does all the work | Negotiating a contract renewal with a key client |
| 2. AI suggests, a person does the work | Gathers information or suggests options; produces nothing final | A person does the work and decides | Lists three possible meeting times; the organizer picks one and sends the invite |
| 3. AI drafts, a person approves each item | Produces a draft | A person checks, edits, and approves every item before it is used | Drafts meeting minutes; the chair corrects and sends them |
| 4. AI acts, a person audits a sample and handles exceptions | Produces final outputs | Rules send unusual items (exceptions) to a person, and a person checks a regular random sample | Routes incoming email to the right shared inbox; mail it cannot place goes to a person, and a lead checks 20 a week |
| 5. AI acts alone, with monitoring | Produces final outputs | A person watches overall measures, such as error reports and complaints, and can stop it | Removes duplicate contacts from a mailing list, with a weekly count someone reviews |
- One workflow can have steps at different levels. For example, a recruiting team might let the AI check that each job advert has the required sections at Level 4 while every hiring decision stays with people at Level 1.
- Levels are earned. Start lower, raise the level with evidence, and drop a level when errors rise or the work changes.
- The idea comes from automation research: Parasuraman, Sheridan, and Wickens (2000) show that each function of a system (gathering information, analyzing it, deciding, acting) can sit at its own level, from fully manual to fully automatic. These five levels are a simplified version for team decisions, not their scale.
Place the human checkpoint
A checkpoint is the point where a person checks the AI's work. Write it as three facts: who checks, what they look at, and when.
- When. Put it before the point of no return: before an output reaches a customer, before money moves or a commitment is made, and before a decision about a person takes effect.
- Exceptions. Simple rules send unusual items to a person: no matching record, low confidence, a large amount, or certain words.
- Random sample. A regular random sample of the items nobody checked one by one measures their error rate. Compare it with today's error rate, measured the same way.
- The checker. They need the knowledge to judge the output, the source on screen, the time, and the authority to change or stop it.
An approval rate shows what reviewers did, not whether the outputs were right. The UK AI Playbook's Principle 4 asks for meaningful human control at the right stages, with people validating high-risk decisions that AI influences.
This question decides where a checkpoint goes and who staffs it. Review screens and review capacity are covered in depth in the Responsible AI & Governance Easy question, "Make human oversight meaningful" (a prerequisite of this one).
Data and the people affected
List the data the work would put into the tool: customer names and addresses, messages, photos taken inside homes, payment details. Then check what the tool is approved for. If a data-use review is still pending, the workflow waits or uses only approved data. The rules themselves are covered in this topic's data-use question, "Set team rules for AI tools and company data".
Next, ask who bears a wrong output:
- Staff, who redo the work.
- Customers, who receive wrong information.
- A person a decision is about, such as someone whose request is refused. Decisions about a person call for a person deciding. The UK AI Playbook advises caution with fully automated significant decisions, and NIST's Map 5.1 asks teams to weigh how likely and how large each impact is.
The staff whose work changes are affected too. Involve them early.
Choose the first pilot
A pilot is a small, time-boxed trial that tells you whether to go further. A good first pilot lets you learn safely:
- Errors are cheap and can be undone.
- A person can check an output quickly and reliably.
- The tool is already approved for the data.
- A small team runs it, with enough volume to measure within a few weeks.
- A baseline exists: today's error rate or time, measured the same way the pilot will be measured.
Weigh the size of the saving against how far a mistake would reach: how many staff need training and how many customers would see the output.
Start one level below the target. Check every item first to measure the error rate, then move to sampling if the rate holds.
Before the pilot starts, write down four things: the success measure, the stop rule (the result that makes you pause or drop back a level), the owner, and the decision date. Measuring a pilot's full cost and return is this topic's Medium question, "Measure an AI pilot's quality, cost, and return".
Scenario: six workflows at a home-services company
Worked example
Suppose a regional home-services company does plumbing, heating, and electrical work. It has about 120 technicians and a 14-person office: 8 on the booking desk, 2 accounts clerks, 1 customer care lead, and 3 estimators. You are the operations lead. The company is trialing one business AI assistant, Option A. All figures are synthetic.
Table 1. Work and time
| ID | Workflow | Weekly volume | Minutes per item today | Done today by | Minutes to check one AI output |
|---|---|---|---|---|---|
| W1 | Turn technician visit notes into customer job reports | 1,500 | 8 | The technician, on a tablet after each visit | 2, by the technician who made the visit |
| W2 | Draft replies to booking and rescheduling emails | 720 | 5 | Booking desk (8 staff) | 1.5, with the booking calendar open |
| W3 | Code supplier invoices to a job and a cost category | 600 | 3 | 2 accounts clerks | 1, against the purchase order |
| W4 | Approve or deny warranty claims | 30 | 25 | The customer care lead | 20: checking a decision needs the same evidence as making it |
| W5 | Translate incoming customer messages into English | 60 | 12 | The one bilingual booking-desk member, or a phone interpreting service | Most staff cannot check. The bilingual member needs 2, but only for the 2 languages they read |
| W6 | Quote prices for new jobs from customer photos | 180 | 20 | 3 estimators | 20: the estimator must do the full assessment |
Table 2. Risk, data, and people
| ID | What a wrong output costs | Can it be undone? | Data involved | Who is affected |
|---|---|---|---|---|
| W1 | The customer gets a wrong record of work, parts, or safety advice. The report also backs invoices and warranty claims | Partly: a corrected report can be sent, but the customer may already have acted on it | Customer name and address, photos inside the home, appliance details | Customers; 120 technicians whose routine changes |
| W2 | A wrong date, time, or address means a missed visit and a wasted technician trip | Yes, if caught before the visit | Customer name, address, phone number, availability | Customers; booking desk |
| W3 | A wrong job or category misstates job profit in the monthly report | Yes: a correcting entry before the month-end close | Supplier names, amounts, job numbers | Accounts clerks; managers who read job profit reports |
| W4 | A wrong denial refuses a customer what they may be entitled to; a wrong approval costs a free return visit | A denial can be overturned on appeal, but trust suffers and complaints escalate | Customer details, job and payment history, photos of the home | The customer: it is a decision about them |
| W5 | Staff misread a request and answer the wrong question | Usually: staff can ask the customer to clarify | Customer name, address, message content | Customers who write in other languages |
| W6 | An accepted quote is a commitment: too low loses money, too high loses the job. Photos miss hidden conditions such as pipework behind walls or old wiring | No: changing an accepted price is hard and often loses the customer | Photos inside customers' homes (they can show people and documents), contact details | Customers; estimators; the company's margin |
Facts
- Option A: the IT lead has approved it for business documents such as supplier invoices and purchase orders (a purchase order is the company's record of what it ordered). Customer personal data (names, addresses, messages, photos of homes) may go into it only after a data-use review, expected to finish in about 6 weeks.
- About 1 in 10 supplier invoices (60 a week) match no purchase order and need a clerk to investigate. Payment approval is a separate step and is not changing.
- Today's month-end check finds about 3 miscoded invoices per 100. The month-end close is when the month's accounts are finalized; a correction is simple before it and harder after it.
- The 60 messages a week in other languages arrive in 5 languages. About half (30 a week) are in the 2 languages the bilingual booking-desk member reads.
- Emergencies (smell of gas, sparks, flooding) are handled by phone. The website and the email auto-reply tell customers to call the emergency line.
- The operations lead can run one pilot at a time this quarter.
Set a level and checkpoint per workflow
Use the two tables and the facts in the scenario above. Work every row before you open the hidden columns.
Predict the outcome
Set a level and checkpoint per workflow
Using the scenario tables and facts, fill in each row: the weekly hours today, the hours if a person checks every AI output, the automation level, the human checkpoint (who checks, what they look at, and when), and the criteria that decided it. Hours today = weekly volume x minutes per item / 60. Hours checking each = weekly volume x minutes to check one AI output / 60, or "not possible" when the person checking cannot judge the output.
| Workflow | Hours today | Hours checking each AI output | Level | Checkpoint | Deciding criteria |
|---|---|---|---|---|---|
| W1 Visit notes to job reports | ? | ? | ? | ? | ? |
| W2 Booking and rescheduling replies | ? | ? | ? | ? | ? |
| W3 Supplier invoice coding | ? | ? | ? | ? | ? |
| W4 Warranty claims | ? | ? | ? | ? | ? |
| W5 Translate incoming messages | ? | ? | ? | ? | ? |
| W6 Quotes from photos | ? | ? | ? | ? | ? |
When one fact changes
Interviewers often change one fact after your answer. Try each change against the levels you chose in the previous exercise.
Predict the outcome
When one fact changes
Each row changes one fact about a workflow in the scenario. Say whether the level you chose for that workflow moves, to what, and why. A change may split a workflow into steps at different levels.
| Change | Workflow | Level after the change | Why |
|---|---|---|---|
| C1. Finance asks for the coding step to also release payment for invoices below an amount it sets | W3 | ? | ? |
| C2. A faulty batch of parts pushes warranty claims from 30 to 300 a week for a month | W4 | ? | ? |
| C3. A new booking system lets Option A read the live calendar and put the confirmed slot into the reply itself | W2 | ? | ? |
| C4. The company wants Option A to translate outgoing replies into the customer's language | W5 | ? | ? |
| C5. The company introduces a fixed-price menu for 12 small jobs, such as replacing a tap or a socket. Checking that a photo matches a menu item takes an estimator 2 minutes | W6 | ? | ? |
Read a 98% approval rate
A later workflow at the same company reports a high approval rate, and the team wants to raise its level. Decide what the evidence supports.
Diagnose
Read a 98% approval rate
Suppose that later in the year, after the data-use review, the company adds a seventh workflow: Option A drafts replies to online customer reviews, and two booking-desk staff approve each draft before it is posted publicly (Level 3). After 3 weeks the dashboard shows:
| Measure | Value |
|---|---|
| Drafts | 150 |
| Approved without changes | 147 (98%) |
| Median (middle) time on the approval screen | 9 seconds |
| What the approval screen shows | The customer's review and the draft reply. The job record (visit date, arrival time, work done, amount charged) is in another system |
The team proposes moving review replies to Level 4 because 98% are approved unchanged. Before deciding, a team lead compared 50 randomly chosen approved replies with their job records. Three stated something the record contradicts: two said the technician arrived on time when the visit started late, and one offered a discount the company does not give.
- What does the approval rate show, and what does it not show, as evidence for a higher level?
- What does the audit imply for replies that would go out unchecked at Level 4?
- As the lead, what would you require before raising the level?
Completion checklist
- I can compute the weekly hours a task takes and what checking AI output would still cost.
- I can explain why a task whose check takes as long as the work gains nothing from checking every item.
- I can name who checks an AI output, what they look at, and when.
- I can say which data a task puts into a tool and whether the tool is approved for it.
- I can pick a first pilot and state its success measure, stop rule, owner, and decision date.
Sources
- A model for types and levels of human interaction with automation (Parasuraman, Sheridan, and Wickens, IEEE Transactions on Systems, Man, and Cybernetics, Part A, 2000) Describes automation applied to four classes of function (information acquisition, information analysis, decision and action selection, action implementation), each at levels from fully manual to fully automatic, and notes that one system can mix levels; automation reliability and the costs of decision and action consequences are among the criteria for choosing a level. The five-level scale in this guide is a simplified editorial version, not the paper's scale.
- NIST AI Risk Management Framework NIST's overview of AI RMF 1.0, a framework intended for voluntary use in managing AI risks. Cited as guidance, not as law.
- NIST AI RMF Core The Map function: potential benefits examined and documented (Map 3.1), potential costs of expected or realized AI errors, including non-monetary costs (Map 3.2), processes for human oversight defined, assessed, and documented (Map 3.5), and the likelihood and magnitude of each identified impact (Map 5.1).
- NIST AI 600-1: Generative AI Profile Describes human-AI configuration risks, including automation bias (excessive deference to automated systems) and over-reliance on generative AI output.
- AI Playbook for the UK Government (Government Digital Service, 2025) Says AI use cases should be led by business and user needs rather than by what the technology can do, that sometimes AI is not the best solution, and that fully automated significant decisions call for caution; Principle 4 asks for meaningful human control at the right stages, including people validating high-risk decisions influenced by AI. Written for UK civil servants and cited as public-sector guidance, not law.
The home-services company, its workflows, figures, Option A, the five-level scale, and the rubric are original synthetic teaching material about a fictional company. The scale is a simplified editorial version of the levels-of-automation idea. NIST and UK government material is cited as voluntary guidance, not law, and nothing here is legal advice. Study times and readiness thresholds are editorial estimates. Sources were checked as of November 2026.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
AI Engineering
Shows how to fix an outcome and a fair baseline, which the first pilot's success measure here depends on.
Behavioral Interview
Surface your assumptions and adjust out loud when the interviewer changes one, as the changed-fact exercise does.
Community article
A developer's account of tasks where checking plausible AI output costs more than it saves, the same test in another setting.
AI Engineering
For tech leads: the engineering version of fitting the design to the task, decided from pilot success and cost per task.
Practice the interview question
Answer it in your own words before you look at the reference answer.
