Review a team-built AI automation before launch

A flow that staff built for their own work starts acting for the company once it launches to everyone. A go-live review asks four questions of every step:

1. What can it do on its own? Label, draft, send, book, or pay.
2. What does a wrong output cost? Money, a customer's trust, or staff time.
3. Can it be undone? A label can be changed; a sent email cannot.
4. Who would catch it today? Someone who reads every output, someone who sees a few, or nobody.

Each step gets a decision (Go, Go with conditions, or Not yet) and an automation level from "Choose which team workflows to hand to AI"; the scenario lists both.

Review the shared setup separately: the account the flow runs as, limits on repeats and spend, the owner, how long records last, and the off switch. A gap there blocks every step, however well the step itself performs.

Judge from a test on real work, and list the evidence still missing, such as outputs nobody checked.

A flow that runs on one person's login can do everything that person can do. Every email it sends looks like theirs, and it stops working when their password, sign-in method, or job changes.

• Service account: an account the company owns for one automation, with its sign-in details held by IT rather than by a person.
• Least privilege: give each step only the access it needs; a step that only drafts needs no permission to send, book, or pay. OWASP's guidance on excessive agency asks for the minimum access a task requires.
• Separate duties: the identity that adds a payee or a payment should not be the one that approves it.

List what each step needs next to what the current account can reach. The gap is what a mistake, or a message written to mislead the model, could use.

Draft or act. A draft reaches a person first; a send, a booking, or a payment reaches a customer or money directly. A sent email cannot be taken back, and a payment to the wrong account may never come back.

Put a person in front of costly or permanent actions. OWASP recommends human approval for high-impact actions; its own example is an email assistant where the user reviews and sends every drafted mail. Add rule checks that run outside the model, such as "every price matches the price sheet" or "the payee matches the supplier record", and route anything that fails to a person.

Prompt injection, in one line: text inside content the model reads, such as an email, that the model follows as if the company had said it. Telling the model to ignore such text helps but is not a control: OWASP notes it is unclear whether fool-proof prevention exists. For trust boundaries in depth, see "Contain an indirect prompt injection" (Security & Privacy).

Business email compromise, in one line: a message that impersonates someone the company pays, often a supplier, and asks for payment to new bank details. The FBI recommends verifying any request to change account information through a second channel, for example a call to the number already on file. An amount limit is easy to stay under, so on its own it is not a payment control.

Unchecked outputs. A label nobody checked has an unknown error rate; a sample check turns it into a number.

Work both cases from the lead's side before you reveal the reasoning.

Repeats after a timeout. When a service does not answer in time, the workflow tool may try again even though the first attempt worked, and an action that creates something then makes a second copy. Ask for a check before creating and again after a timeout, proven on the known cases. Engineers call this idempotency; "Design an agent with idempotent side effects" (Tool Calling & Agents) covers how it is built.

Mail loops. Two automatic responders can answer each other without end. The mail standard RFC 3834 asks automatic replies to carry a header (a hidden label in the email) called Auto-Submitted, asks responders not to answer messages that carry it, and asks personal responders such as out-of-office replies not to send the same reply to the same sender more than once within several days. Not every system follows it, so add your own limit on automatic emails per address.

Spend. Cost per run times the number of runs gives normal cost, the baseline for any cap. Test a proposed cap against the worst case you know, such as a loop: how much harm gets through before it fires? A cap on the whole flow pauses every step at once, so it is a backstop that alerts a named person, not the loop guard. Set the alert above the busiest normal day. OWASP lists rate limits, quotas, and monitoring against uncontrolled use and its cost.

Two kinds of cost. Put the running cost next to the money that wrong actions put at stake.

Use the fictional costs and planning rule from the scenario. Compute each value before you reveal it.

• Owners. Name a business owner (decides what the flow should do), a technical owner (keeps its accounts, limits, and records right), and a trained backup, in writing. The NIST AI Risk Management Framework asks for documented, clear roles and responsibilities.
• Run records. A run record shows what one run received, did, and sent, which you need to investigate a complaint, a payment, or an incident. Keep records at least as long as complaints and disputes take to arrive, in a place the company owns, and no longer than the records rule allows, because they hold customer emails. Staff data rules are covered in "Set team rules for AI tools and company data".
• Off switch and fallback. Give each step an off switch that the backup has practiced, and a one-page manual fallback so work continues while it is off. NIST asks for mechanisms, and people assigned, to disengage or deactivate a system that behaves outside its intended use.
• After launch. Plan the checks before go-live: a regular sample of outputs, a duplicate report, spend and loop alerts, and a review date (NIST calls these post-deployment monitoring plans).

The person who built the flow solved a real problem without any guidance on access, limits, or records. How you give the review decides whether staff bring the next automation to you or keep it quiet.

• Open with what works and what the test showed, not with blame.
• Pair on the fixes: the builder knows the process; internal tools knows accounts, limits, and records.
• Agree the roles in writing, and train the backup before the builder is away.
• Tell the wider team what changes at launch and how to report a wrong output.

A launch decision fits on one page:

1. Scope: which steps launch, at which level, and which wait.
2. Conditions: each with an owner and a done-by date, checked before go-live, and what happens if one is not done in time.
3. Measures: what you watch in the first weeks, and the review date.
4. Pause and widen: what would pause a step, and what evidence would let a step do more.

Tie each condition to a finding from the test, and each measure to a condition, so the review date has something to judge.

Use the decision words and automation levels from the scenario. Fill in every hidden column before you reveal it.