AI Engineering
/
AI for Team and Tech Leads
/

Choose which team workflows to hand to AI

AI for Team and Tech Leads · 90 minutes

Choose which team workflows to hand to AI

Judge six workflows at a fictional home-services company by volume, error cost, checking effort, data, and the people affected, set an automation level and a human checkpoint for each, and choose the first pilot.

Easy
Conceptual

Scenario

1.5 h

Used in interview rounds:

Leadership Scenarios (Easy)

Overview

Learn how a lead decides which of a team's recurring tasks to hand to AI, how far to automate each one, and where a person stays involved. You practice on six workflows at a fictional home-services company and finish by choosing the first pilot.

Study time: 90 minutes

Interview format: Hiring-manager scenario with one changed constraint

By the end you can

  • Describe a team workflow by its volume, time per item, and what happens to its output
  • Judge a workflow's fit for AI from error cost, reversibility, checking effort, data, and the people affected
  • Estimate the weekly hours at stake and the time that checking AI output still costs
  • Assign an automation level and place a human checkpoint that can actually catch errors
  • Choose a first pilot and write its success measure, stop rule, and decision date before it starts

Prerequisites

Helpful background: No coding or AI background needed. The only arithmetic is multiplying weekly volume by minutes per item and dividing by 60.

This topic covers the decisions a lead makes about a team's use of AI: which work to automate and where people stay involved, staff data rules, building or buying tools, measuring quality, cost, and return, rolling out and training, and leading the response when AI output goes wrong. Responsible AI & Governance designs oversight and accountability for one system's effect on the people it serves, AI Product & Experimentation measures user-facing features with experiments, and AI Developer Tooling covers how engineers set up and secure coding assistants and agent harnesses.

Start from the work, not the tool

In this topic, a workflow is a recurring piece of business work: something arrives, a person produces an output, and someone uses that output. It is a business process, not an engineering pipeline.

Describe the work before you pick a tool:

  • What arrives, and how often?
  • What is produced, and who uses it?
  • What happens if the output is late or wrong?

Then name the outcome that matters: hours saved, a faster answer for the customer, or fewer errors. Some tasks improve more with a template, a form, or a simple rule than with AI, so compare against that option too.

The UK government's AI Playbook makes the same point for public-sector teams: use cases should be led by business and user needs, not by what the technology can do, and sometimes AI is not the best solution.

Five questions for every workflow

Ask the same five questions of every workflow. They are the criteria for every decision in this guide.

QuestionWhy it matters
How many items a week, and how many minutes each?Sets how much time is at stake
What does a wrong output cost, and can it be undone before anyone is harmed?Sets how close a person must stay to the work
How long does a person need to check one output, and can they tell a right output from a wrong one?Decides whether checking saves time and whether the check is real
Which data does the work put into the tool, and is the tool approved for it?Decides whether the workflow can start now
Who bears a wrong output: staff, customers, or a person a decision is about?Raises the bar when customers or decisions about people are involved

Volume tells you how much is at stake, not how far to automate. The level comes from the other four questions.

NIST's AI Risk Management Framework (voluntary guidance, not law) asks teams to document the expected benefits of an AI system and the costs of its errors (Map 3.1 and Map 3.2), and to define how people oversee it (Map 3.5).

Checking effort decides the saving

Two rules turn a list of workflows into hours:

  • Weekly hours = weekly volume x minutes per item / 60.
  • The most that checking each item can save, per item = minutes to do the task today minus minutes to check (and fix) the AI output.

Worked example from another setting (all figures synthetic). A school office writes 200 letters a week at 6 minutes each. Checking an AI draft takes 2 minutes.

MeasureArithmeticResult
Today200 x 6 = 1,200 minutes20 hours
A person checks every draft200 x 2 = 400 minutesabout 6.7 hours
Most that can be saved1,200 - 400 = 800 minutesabout 13.3 hours, before edits and rework

If checking takes as long as doing the work, checking every item saves nothing.

A check also counts only if the checker can catch the error: they know the facts, can read the output, and have the source in front of them. Automation bias is the tendency to accept what an automated system produces; NIST's Generative AI Profile describes it as excessive deference to automated systems. A check that takes a few seconds is often not a check at all.

Five levels of automation

An automation level says how much of the work the AI does and where a person stays involved. This guide uses five levels. The examples come from other settings.

LevelWhat the AI doesWhere the person isExample
1. No AINothingA person does all the workNegotiating a contract renewal with a key client
2. AI suggests, a person does the workGathers information or suggests options; produces nothing finalA person does the work and decidesLists three possible meeting times; the organizer picks one and sends the invite
3. AI drafts, a person approves each itemProduces a draftA person checks, edits, and approves every item before it is usedDrafts meeting minutes; the chair corrects and sends them
4. AI acts, a person audits a sample and handles exceptionsProduces final outputsRules send unusual items (exceptions) to a person, and a person checks a regular random sampleRoutes incoming email to the right shared inbox; mail it cannot place goes to a person, and a lead checks 20 a week
5. AI acts alone, with monitoringProduces final outputsA person watches overall measures, such as error reports and complaints, and can stop itRemoves duplicate contacts from a mailing list, with a weekly count someone reviews
  • One workflow can have steps at different levels. For example, a recruiting team might let the AI check that each job advert has the required sections at Level 4 while every hiring decision stays with people at Level 1.
  • Levels are earned. Start lower, raise the level with evidence, and drop a level when errors rise or the work changes.
  • The idea comes from automation research: Parasuraman, Sheridan, and Wickens (2000) show that each function of a system (gathering information, analyzing it, deciding, acting) can sit at its own level, from fully manual to fully automatic. These five levels are a simplified version for team decisions, not their scale.

Place the human checkpoint

A checkpoint is the point where a person checks the AI's work. Write it as three facts: who checks, what they look at, and when.

  • When. Put it before the point of no return: before an output reaches a customer, before money moves or a commitment is made, and before a decision about a person takes effect.
  • Exceptions. Simple rules send unusual items to a person: no matching record, low confidence, a large amount, or certain words.
  • Random sample. A regular random sample of the items nobody checked one by one measures their error rate. Compare it with today's error rate, measured the same way.
  • The checker. They need the knowledge to judge the output, the source on screen, the time, and the authority to change or stop it.

An approval rate shows what reviewers did, not whether the outputs were right. The UK AI Playbook's Principle 4 asks for meaningful human control at the right stages, with people validating high-risk decisions that AI influences.

This question decides where a checkpoint goes and who staffs it. Review screens and review capacity are covered in depth in the Responsible AI & Governance Easy question, "Make human oversight meaningful" (a prerequisite of this one).

Data and the people affected

List the data the work would put into the tool: customer names and addresses, messages, photos taken inside homes, payment details. Then check what the tool is approved for. If a data-use review is still pending, the workflow waits or uses only approved data. The rules themselves are covered in this topic's data-use question, "Set team rules for AI tools and company data".

Next, ask who bears a wrong output:

  • Staff, who redo the work.
  • Customers, who receive wrong information.
  • A person a decision is about, such as someone whose request is refused. Decisions about a person call for a person deciding. The UK AI Playbook advises caution with fully automated significant decisions, and NIST's Map 5.1 asks teams to weigh how likely and how large each impact is.

The staff whose work changes are affected too. Involve them early.

Choose the first pilot

A pilot is a small, time-boxed trial that tells you whether to go further. A good first pilot lets you learn safely:

  • Errors are cheap and can be undone.
  • A person can check an output quickly and reliably.
  • The tool is already approved for the data.
  • A small team runs it, with enough volume to measure within a few weeks.
  • A baseline exists: today's error rate or time, measured the same way the pilot will be measured.

Weigh the size of the saving against how far a mistake would reach: how many staff need training and how many customers would see the output.

Start one level below the target. Check every item first to measure the error rate, then move to sampling if the rate holds.

Before the pilot starts, write down four things: the success measure, the stop rule (the result that makes you pause or drop back a level), the owner, and the decision date. Measuring a pilot's full cost and return is this topic's Medium question, "Measure an AI pilot's quality, cost, and return".

Scenario: six workflows at a home-services company

Worked example

Suppose a regional home-services company does plumbing, heating, and electrical work. It has about 120 technicians and a 14-person office: 8 on the booking desk, 2 accounts clerks, 1 customer care lead, and 3 estimators. You are the operations lead. The company is trialing one business AI assistant, Option A. All figures are synthetic.

Table 1. Work and time

IDWorkflowWeekly volumeMinutes per item todayDone today byMinutes to check one AI output
W1Turn technician visit notes into customer job reports1,5008The technician, on a tablet after each visit2, by the technician who made the visit
W2Draft replies to booking and rescheduling emails7205Booking desk (8 staff)1.5, with the booking calendar open
W3Code supplier invoices to a job and a cost category60032 accounts clerks1, against the purchase order
W4Approve or deny warranty claims3025The customer care lead20: checking a decision needs the same evidence as making it
W5Translate incoming customer messages into English6012The one bilingual booking-desk member, or a phone interpreting serviceMost staff cannot check. The bilingual member needs 2, but only for the 2 languages they read
W6Quote prices for new jobs from customer photos180203 estimators20: the estimator must do the full assessment

Table 2. Risk, data, and people

IDWhat a wrong output costsCan it be undone?Data involvedWho is affected
W1The customer gets a wrong record of work, parts, or safety advice. The report also backs invoices and warranty claimsPartly: a corrected report can be sent, but the customer may already have acted on itCustomer name and address, photos inside the home, appliance detailsCustomers; 120 technicians whose routine changes
W2A wrong date, time, or address means a missed visit and a wasted technician tripYes, if caught before the visitCustomer name, address, phone number, availabilityCustomers; booking desk
W3A wrong job or category misstates job profit in the monthly reportYes: a correcting entry before the month-end closeSupplier names, amounts, job numbersAccounts clerks; managers who read job profit reports
W4A wrong denial refuses a customer what they may be entitled to; a wrong approval costs a free return visitA denial can be overturned on appeal, but trust suffers and complaints escalateCustomer details, job and payment history, photos of the homeThe customer: it is a decision about them
W5Staff misread a request and answer the wrong questionUsually: staff can ask the customer to clarifyCustomer name, address, message contentCustomers who write in other languages
W6An accepted quote is a commitment: too low loses money, too high loses the job. Photos miss hidden conditions such as pipework behind walls or old wiringNo: changing an accepted price is hard and often loses the customerPhotos inside customers' homes (they can show people and documents), contact detailsCustomers; estimators; the company's margin

Facts

  • Option A: the IT lead has approved it for business documents such as supplier invoices and purchase orders (a purchase order is the company's record of what it ordered). Customer personal data (names, addresses, messages, photos of homes) may go into it only after a data-use review, expected to finish in about 6 weeks.
  • About 1 in 10 supplier invoices (60 a week) match no purchase order and need a clerk to investigate. Payment approval is a separate step and is not changing.
  • Today's month-end check finds about 3 miscoded invoices per 100. The month-end close is when the month's accounts are finalized; a correction is simple before it and harder after it.
  • The 60 messages a week in other languages arrive in 5 languages. About half (30 a week) are in the 2 languages the bilingual booking-desk member reads.
  • Emergencies (smell of gas, sparks, flooding) are handled by phone. The website and the email auto-reply tell customers to call the emergency line.
  • The operations lead can run one pilot at a time this quarter.

Set a level and checkpoint per workflow

Use the two tables and the facts in the scenario above. Work every row before you open the hidden columns.

Predict the outcome

Set a level and checkpoint per workflow

Using the scenario tables and facts, fill in each row: the weekly hours today, the hours if a person checks every AI output, the automation level, the human checkpoint (who checks, what they look at, and when), and the criteria that decided it. Hours today = weekly volume x minutes per item / 60. Hours checking each = weekly volume x minutes to check one AI output / 60, or "not possible" when the person checking cannot judge the output.

WorkflowHours todayHours checking each AI outputLevelCheckpointDeciding criteria
W1 Visit notes to job reports?????
W2 Booking and rescheduling replies?????
W3 Supplier invoice coding?????
W4 Warranty claims?????
W5 Translate incoming messages?????
W6 Quotes from photos?????

When one fact changes

Interviewers often change one fact after your answer. Try each change against the levels you chose in the previous exercise.

Predict the outcome

When one fact changes

Each row changes one fact about a workflow in the scenario. Say whether the level you chose for that workflow moves, to what, and why. A change may split a workflow into steps at different levels.

ChangeWorkflowLevel after the changeWhy
C1. Finance asks for the coding step to also release payment for invoices below an amount it setsW3??
C2. A faulty batch of parts pushes warranty claims from 30 to 300 a week for a monthW4??
C3. A new booking system lets Option A read the live calendar and put the confirmed slot into the reply itselfW2??
C4. The company wants Option A to translate outgoing replies into the customer's languageW5??
C5. The company introduces a fixed-price menu for 12 small jobs, such as replacing a tap or a socket. Checking that a photo matches a menu item takes an estimator 2 minutesW6??

Read a 98% approval rate

A later workflow at the same company reports a high approval rate, and the team wants to raise its level. Decide what the evidence supports.

Diagnose

Read a 98% approval rate

Suppose that later in the year, after the data-use review, the company adds a seventh workflow: Option A drafts replies to online customer reviews, and two booking-desk staff approve each draft before it is posted publicly (Level 3). After 3 weeks the dashboard shows:

MeasureValue
Drafts150
Approved without changes147 (98%)
Median (middle) time on the approval screen9 seconds
What the approval screen showsThe customer's review and the draft reply. The job record (visit date, arrival time, work done, amount charged) is in another system

The team proposes moving review replies to Level 4 because 98% are approved unchanged. Before deciding, a team lead compared 50 randomly chosen approved replies with their job records. Three stated something the record contradicts: two said the technician arrived on time when the visit started late, and one offered a discount the company does not give.

  1. What does the approval rate show, and what does it not show, as evidence for a higher level?
  2. What does the audit imply for replies that would go out unchecked at Level 4?
  3. As the lead, what would you require before raising the level?

Completion checklist

  • I can compute the weekly hours a task takes and what checking AI output would still cost.
  • I can explain why a task whose check takes as long as the work gains nothing from checking every item.
  • I can name who checks an AI output, what they look at, and when.
  • I can say which data a task puts into a tool and whether the tool is approved for it.
  • I can pick a first pilot and state its success measure, stop rule, owner, and decision date.

Sources

  • A model for types and levels of human interaction with automation (Parasuraman, Sheridan, and Wickens, IEEE Transactions on Systems, Man, and Cybernetics, Part A, 2000) Describes automation applied to four classes of function (information acquisition, information analysis, decision and action selection, action implementation), each at levels from fully manual to fully automatic, and notes that one system can mix levels; automation reliability and the costs of decision and action consequences are among the criteria for choosing a level. The five-level scale in this guide is a simplified editorial version, not the paper's scale.
  • NIST AI Risk Management Framework NIST's overview of AI RMF 1.0, a framework intended for voluntary use in managing AI risks. Cited as guidance, not as law.
  • NIST AI RMF Core The Map function: potential benefits examined and documented (Map 3.1), potential costs of expected or realized AI errors, including non-monetary costs (Map 3.2), processes for human oversight defined, assessed, and documented (Map 3.5), and the likelihood and magnitude of each identified impact (Map 5.1).
  • NIST AI 600-1: Generative AI Profile Describes human-AI configuration risks, including automation bias (excessive deference to automated systems) and over-reliance on generative AI output.
  • AI Playbook for the UK Government (Government Digital Service, 2025) Says AI use cases should be led by business and user needs rather than by what the technology can do, that sometimes AI is not the best solution, and that fully automated significant decisions call for caution; Principle 4 asks for meaningful human control at the right stages, including people validating high-risk decisions influenced by AI. Written for UK civil servants and cited as public-sector guidance, not law.

The home-services company, its workflows, figures, Option A, the five-level scale, and the rubric are original synthetic teaching material about a fictional company. The scale is a simplified editorial version of the levels-of-automation idea. NIST and UK government material is cited as voluntary guidance, not law, and nothing here is legal advice. Study times and readiness thresholds are editorial estimates. Sources were checked as of November 2026.

Practice the interview question

Answer it in your own words before you look at the reference answer.