AI Engineering
/
AI for Team and Tech Leads
/

Turn an executive's AI mandate into a scoped pilot

AI for Team and Tech Leads · 90 minutes

Turn an executive's AI mandate into a scoped pilot

Answer a COO who wants AI on every customer email and support costs down by a third: find the goal behind the request, check both targets against the team's figures and rules, and commit to a pilot you can measure.

Easy
Conceptual

Scenario

1.5 h

Used in interview rounds:

Leadership Scenarios (Easy)

Overview

Learn how a lead answers an executive's AI request: find the goal behind it, check the target against the team's own figures and rules, and commit to a pilot that can be measured. You practice on a fictional support team whose chief operating officer wants AI on every customer email.

Study time: 90 minutes

Interview format: Hiring-manager scenario with a written reply to an executive

By the end you can

  • Separate an AI request's method and number from the outcome behind it, and say what to confirm with the executive
  • Size a target from volume, handling minutes, and budget share, and test its best case
  • Name the rules, authority, data access, and review time that limit what AI can take on
  • Scope a first pilot with a human checkpoint, a comparison, success and stop measures, and a decision date
  • Write a reply that commits to a dated, measured step, and tell the team what changes and what does not

Prerequisites

Helpful background: No coding needed. Comfort with percentages and a few multiplications helps.

This topic covers the decisions a lead makes about a team's use of AI: which work to automate and where people stay involved, staff data rules, building or buying tools, measuring quality, cost, and return, rolling out and training, and leading the response when AI output goes wrong. Responsible AI & Governance designs oversight and accountability for one system's effect on the people it serves, AI Product & Experimentation measures user-facing features with experiments, and AI Developer Tooling covers how engineers set up and secure coding assistants and agent harnesses.

Why AI mandates arrive as slogans

Executives often state an AI request as a method plus a number: "AI on every customer email", "costs down by a third". The wording answers a pressure they face, such as a board question, a budget round, or a rival's announcement.

The method and the number stand in for an outcome. A lead's job is neither yes nor no: it is to turn the request into a commitment the team can keep and both of you can check. A yes to an unchecked target sets the team up to miss it or to cut corners. A flat no leaves the executive with nothing to report, and the decision gets made without you.

Find the goal behind the request

Separate the outcome (cost per customer, response time, room to grow without hiring, quality) from the method ("AI on every email"), which is one possible route to it. Confirm the outcome with a few questions:

  • Which outcome matters most if only one can move?
  • How will we know it improved: measured how, and by when?
  • What is fixed: the budget, hiring plans, quality rules?
  • What do you need to say, and to whom?

Then restate it: "I read the goal as X by Y. Is that right?" A confirmed goal lets you question the method without refusing the executive. Measures follow the outcome; the prerequisite question "Define a user outcome and baseline" covers outcomes, proxies (stand-in measures that are easier to count), and baselines.

Public guidance starts here too: the NIST AI Risk Management Framework, a voluntary US guide, asks organizations to understand and document their goals for an AI technology, and the UK government's service manual says to understand the problem before you commit to building a service.

Check a target against your own numbers

Size the work before you discuss the number.

  1. Hours. Volume times minutes per item gives hours; compare them with the team's total handling hours.
  2. Budget. Multiply the share of hours by the share of the budget that pays for people's time.
  3. Best case first. Suppose the tool did all of a type's work with no human minutes. If even that misses the target, the target needs to change, not the plan.

A generic example (not the scenario's figures): a queue uses 400 of a team's 5,000 handling hours (8%), so even removing all of it moves at most 8% of hours. A tool that halves its time frees 200 hours, 4% of handling hours. If people's time is 75% of the budget, that is 3% of the budget, before fees.

Then keep the estimate honest:

  • A vendor's claim is a hypothesis. Test it on your own work, counting review, edits, discarded drafts, and repeat contacts caused by wrong replies.
  • Time saved is not money saved. Freed hours lower spending only if hiring, overtime, contractors, or staffing change; otherwise they buy speed, quality, or room for growth. Say which.
  • Count the tool's cost: its fee and the setup time.

Limits a deadline cannot move

Some limits hold however urgent the request is. Check each type of work for:

  • Rules that require a person. A company rule, or sometimes a regulation, says a person must decide or respond. Legal questions go to counsel.
  • Authority over money or commitments. Credits, refunds, discounts, and contract changes need someone allowed to grant them; an AI draft must not grant or promise them.
  • Data the tool cannot see. If a correct reply needs a system the tool cannot read, the draft will guess.
  • The cost of a wrong reply. Repeat contacts, complaints, harm. Work with many repeat contacts already multiplies mistakes.
  • Review time that remains. When a person approves every draft, they still read every item. Fluent drafts can be wrong (NIST calls confidently stated false content "confabulation"), and reviewers can come to over-trust a tool that is usually right (automation bias).

The prerequisite question "Choose which team workflows to hand to AI" teaches how to pick an automation level. The five levels are:

  1. No AI
  2. AI suggests, a person does the work
  3. AI drafts, a person approves each item
  4. AI acts, a person audits a sample and handles exceptions
  5. AI acts alone, with monitoring

Scope a pilot you can judge

The prerequisite "Choose which team workflows to hand to AI" covers what to write down before any pilot: the baseline, the success measure, the stop rule, and the owner. A pilot that answers an executive's target also needs these fixed before the start, so nobody can argue about them afterwards:

ElementFix before the start
ComparisonThe same work in the same weeks without the tool, so a busy month does not pass for the tool's effect
SampleEnough reviewed items to tell rates apart: a few dozen cannot show a rate of 3 in 100
Decision dateAfter every measurement window has closed, including any follow-up period the measures count

Agree the three possible outcomes in advance too: expand, change and rerun, or stop.

The UK government's service manual describes the same habit for any new service: try out solutions with prototypes, do the minimum you need to test your riskiest assumptions, and treat stopping as an acceptable result. At the decision date, the NIST framework asks whether the system achieves its intended purposes and stated objectives, and whether deployment should proceed.

Commit to what you control

A commitment has three parts: what you will do, by when, and how it will be judged. Without all three, it is a hope.

  • Decline in the executive's terms: the rule, the figure, or the risk, one sentence each. "AI is not ready" gives the executive nothing to repeat; "custom orders need a site survey first" does.
  • Offer something usable now: a sentence for the board, and a date when evidence arrives.
  • Give estimates as ranges with the assumption named, such as "if the vendor's claim holds on our emails".
  • Agree check-ins and what result would change the plan.

Saying what you will not commit to is part of the commitment: it stops the original request from returning as the target you are judged on.

Worked example: a quoting mandate

Worked example

The company and its rules are fictional. A sales director at an office-equipment leasing company writes: "By March, AI writes every customer quote."

  • Goal behind it. Quotes take 3 working days and deals are lost to faster rivals. The director confirms that quote speed matters more than saving sales staff hours.
  • Limits. Discounts below the price list need a manager's approval (company rule). Custom fit-outs need a site survey. The quoting tool can read the price list but not stock levels.
  • Commitment. For 6 weeks in one region, AI drafts standard catalogue quotes and a salesperson approves each. The pilot is judged on time to quote, pricing errors in a weekly reviewed sample, and customer follow-up questions, against the other regions in the same weeks, with a decision date set in advance.
  • Not committed. Every quote by March. Discounted and custom quotes stay with people, and the director hears the reason in one sentence: the rule and the survey, not the technology.

Tell the team

The team will hear about an AI pilot from you or from rumor. Tell them early and plainly:

  • Why: the outcome you are working toward, not "management wants AI".
  • What changes in the pilot and for whom, and what does not.
  • Accountability: the person who approves a draft is accountable for what is sent, can reject or edit any draft, and knows how to flag a bad one.
  • Training: a short session on what to check, such as amounts, dates, and promises.
  • Their voice: their measures and comments decide the next step.

On jobs, say what you know, what you do not, who decides, and when they will hear more. Never promise beyond your authority. The rollout question in this topic covers this conversation in depth.

Scenario: an AI mandate for a support team

Worked example

The company, team, figures, rules, and prices are fictional.

You lead customer support at a regional internet service provider that sells home broadband. Your team has 45 agents who answer phone, chat, and email, plus 3 team leads and 2 QA specialists. This morning the chief operating officer (COO) wrote to you:

The board wants to hear our AI plan. By the end of next quarter I want AI answering every customer email, and support costs down by a third. Can you own this?

Next quarter ends in 20 weeks.

Team facts (a typical month)

  • Agents log 6,000 handling hours: phone 3,250, chat 820, and email 1,930. The help desk logs handling time (from opening an item to sending the reply) for every email.
  • The support budget is $307,500 a month. Agent time is $246,000 of it; the rest is the phone system, help desk software, training, and management.
  • Email volume grew 18% over the past year with the same headcount. The median wait for a first email reply is 26 hours.
  • Billing emails go to a pod of 10 agents, and the help desk hands them out in turn (round-robin).
  • QA reviews 100 sent email replies a week. About 3 in 100 contain an error: a wrong amount, a wrong date, or a wrong policy.

Email mix (a typical month)

Re-contact is the share of emails where the same customer writes again about the same issue within 7 days.

Email typeEmailsAverage handling minutesRe-contactWhat a correct reply needs
Billing question, no credit asked3,600510%Bill lines and payment dates from the account
Credit or refund request1,200916%A decision on a credit by staff allowed to grant it
Outage report2,400425%Live outage status for the customer's area
Moving home1,2001215%A service check at the new address and an install date
Cancellation900108%Account changes, a final bill, equipment return
Troubleshooting3,0001430%Line test results and router details; often a technician visit
Complaint6002022%The customer's case history

Rules and tools

  • Only staff may grant credits or refunds, each role up to a set limit. No automated system may grant or promise one (company rule).
  • A person must acknowledge every complaint within 2 business days (a company rule in this scenario).
  • Every cancellation goes to the retention group, which decides whether to make a retention offer; no one else may make one.
  • The help desk has macros (fixed reply templates an agent inserts), and a network status page lists known outages by area.
  • Option A is the help desk vendor's AI drafting add-on, at $0.047 per drafted reply (fictional price). It can read the customer's account summary (plan, bill lines, payments) and the knowledge base. It cannot read the status page, line tests, the service check for a new address, or past case history.
  • The vendor's material says agents using its drafts handle emails in half the time. Nobody has measured that on your emails.
  • IT's standard review of a new tool's data terms takes 2 weeks.

Size the target and test each email type

Use the scenario above.

Predict the outcome

Size the target and test each email type

For each email type, compute hours a month (emails x minutes / 60) and the share of the team's 6,000 handling hours, to one decimal. For the all-email row, also turn the hours into dollars and a share of the $307,500 budget. Then apply two tests: (1) does a company rule or authority require a person to make the reply's decision, and (2) can Option A read everything a correct reply needs? A type can join the first pilot only if no person is required and Option A can read everything. Say what limits AI on each type and its role in a first pilot, then reveal the answers.

Email typeEmails a monthAverage handling minutesHours a monthShare of 6,000 handling hoursWhat limits AI hereRole in a first pilot
Billing question, no credit asked3,6005????
Credit or refund request1,2009????
Outage report2,4004????
Moving home1,20012????
Cancellation90010????
Troubleshooting3,00014????
Complaint60020????
All email12,900mixed????

Reply to the COO

Use the scenario above and your sized table.

Design exercise

Reply to the COO

Use the support scenario. The COO wants your answer today, before the board pack is written, and will read it on a phone between meetings.

Requirements

  • State the outcome you think the COO wants and ask them to confirm it
  • Say what you will deliver and by which week
  • Say what you will not commit to, with the reason for each in plain figures or rules
  • Give the COO one sentence to use with the board now

Constraints

  • Under about 200 words
  • Only figures you can show from the scenario
  • No technical terms

Deliverables

  • The reply as you would send it

Completion checklist

  • I can name the outcome behind an AI request and the question that confirms it.
  • I can turn volume and minutes into hours and a budget share, and test a target's best case.
  • I can explain why time saved is not money saved, and say what freed hours will become.
  • I can scope a first pilot with a checkpoint, a comparison, measures, stop rules, and a decision date.
  • I can tell an executive and a team what I commit to, by when, and what is not yet decided.

Sources

  • NIST AI Risk Management Framework NIST's overview of AI RMF 1.0, a framework for voluntary use, organized into four functions: Govern, Map, Measure, and Manage. Cited as guidance, not as law.
  • NIST AI RMF Core Map 1.3 (the organization's mission and relevant goals for AI technology are understood and documented), Map 3.1 and 3.2 (potential benefits, and potential costs including those from AI errors, are examined and documented), Govern 2.1 (roles, responsibilities, and lines of communication are documented and clear), and Manage 1.1 (a determination is made whether the system achieves its intended purposes and stated objectives and whether its development or deployment should proceed).
  • NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024) Defines confabulation as the production of confidently stated but erroneous or false content, and lists human-AI configuration risks including automation bias and over-reliance.
  • GOV.UK Service Manual: How the discovery phase works Before committing to build a service, understand the problem that needs to be solved; stopping at the end of discovery is not a failure when research shows it is the best thing to do. General service-delivery guidance, not specific to AI.
  • GOV.UK Service Manual: How the alpha phase works Alpha tries out different solutions with prototypes and does the minimum needed to test the riskiest assumptions; a team that is not confident at the end can stop or repeat discovery or alpha. General service-delivery guidance, not specific to AI.

The internet service provider, its support team, figures, rules, vendor claim, prices, thresholds, and rubric are original synthetic teaching material about a fictional company, as is the leasing-company example. All prices are fictional and match no real list price. The complaint and credit rules are company rules in the scenario, not legal requirements. Study times and readiness thresholds are editorial estimates. Sources checked November 2026.

Practice the interview question

Answer it in your own words before you look at the reference answer.