Tool Calling & Agents · 2 to 3 hours
Choose a fixed workflow or an agent loop
Compare a fixed workflow and an agent loop for a parcel-delivery assistant's rescheduling and stuck-parcel tasks using a pilot's success, cost, and latency figures, then recommend a design per task and say what would change it.
Conceptual comparison
3 h
Overview
Decide between a fixed workflow and an agent loop the way a technical screen expects: state the product's constraints, compare both designs on the same dimensions, and let pilot evidence settle the choice. You practice on a fictional parcel-delivery assistant with one predictable task and one open-ended task.
Study time: 2 to 3 hours
Interview format: Technical screen comparison with a pilot table and changed constraints
By the end you can
- State a product's goal and constraints, then compare a fixed workflow and an agent loop on the same dimensions
- Turn a pilot into cost per successful task and total daily cost, counting the work that failed runs create
- Use repeated trials and per-slice results to tell a missing path from run-to-run variance
- Keep validation, required steps, and approvals in code whichever design you choose
- Recommend a design per task, name the evidence still missing, and say what would change the decision
- Recognize when a workflow with one bounded agentic step is worth piloting against both pure designs
Prerequisites
Helpful background: Comfort with percentages and per-million-token prices helps.
This topic covers workflows versus agents, typed tools, approval boundaries, state, and termination. Specific frameworks and protocols appear as tags.
Who chooses the next step
In a fixed workflow, code decides the order of steps and every branch; the model fills bounded steps inside that path, such as extracting fields or writing a reply. In an agent loop, the model chooses the next tool call from the goal and the results so far, until it answers or a limit in code stops it.
The difference is who controls the next step, not how many tools or model calls run: a workflow can call a model five times, and an agent can finish in two calls. Most designs sit between the two. A router that picks one of several fixed paths is still a workflow, and a workflow can hold one bounded agentic step where its fixed path runs out.
Scenario: a parcel-delivery assistant
Worked example
Suppose a parcel-delivery company runs a chat assistant on its website. The company, its tools, and every figure in this question are fictional. For each of two tasks, the team built both a fixed workflow and an agent loop. Both designs use the same model, tools, and argument validation; only the control flow differs.
Task A: reschedule a delivery. A customer gives a tracking number and asks for another day. A parcel can be rescheduled only while it is at a depot (not out for delivery, delivered, or held at customs). A rebook is a write: a wrong one sends a driver on the wrong day.
| Tool | What it does |
|---|---|
verify_customer | Checks the tracking number against the postcode the customer gives |
get_shipment | Returns the parcel's state and depot |
list_slots | Returns open delivery slots for a day |
rebook_delivery | Moves the delivery to a slot (write) |
- Workflow A. The model extracts the tracking number, postcode, and preferred day into validated JSON (one repair attempt). Code verifies the customer, reads the shipment, checks that its state allows rescheduling, and lists slots. The customer picks and confirms a slot with buttons; code rebooks, and the model writes the confirmation from the result. Anything outside this path, such as a redirect to a neighbor, goes to a person.
- Agent A. The model receives the four tools and instructions that include "verify the customer first" and "confirm the slot with the customer before rebooking". The executor validates arguments and rejects
get_shipment,list_slots, andrebook_deliveryuntilverify_customerhas succeeded. Cap: 10 model calls, then a hand-off.
Task B: explain why a parcel is stuck. The customer asks why tracking has not moved. The reply must name the cause and the next step and cite the records it used. Every Task B tool is read-only.
| Tool | What it returns |
|---|---|
get_tracking_events | The public tracking history |
get_depot_scans | Internal scans at each depot |
get_carrier_notes | Free-text notes from drivers and partner carriers |
get_customs_status | Customs holds and missing documents |
get_partner_tracking | A partner carrier's history, given the partner reference found in a carrier note |
check_address | Whether the address failed validation at the depot |
- Workflow B. Code calls the first four tools in parallel, and one model call writes the explanation from their results (one repair attempt if the reply fails its format check). Code checks that every cause the reply cites comes from a fetched record; when that check fails or the model finds no record that explains the hold, the conversation goes to a person.
- Agent B. The model receives all six tools and chooses which to call and in what order. Code applies the same cited-record check to its reply. Cap: 12 model calls, then a hand-off.
Product constraints
| Constraint | Task A | Task B |
|---|---|---|
| Conversations per day | 6,000 | 2,000 |
| p95 processing time target (excluding customer wait) | 8 s | 20 s |
| Writes | rebook_delivery; no wrong rebooks allowed | None |
Model spend for both tasks together must stay at or below $100 per day. Prices: $1.00 per million input tokens and $3.00 per million output tokens. Each conversation a person has to finish costs $2.50 of staff time.
Pilot results
Worked example
The team labeled 200 recent conversations per task with the correct outcome. Each design ran every case three times (600 runs per design), and a reviewer graded every run's outcome against its label and read every trace.
- Successful: the labeled outcome without a person. Task A: the confirmed slot is booked, or the assistant correctly says the parcel cannot be rescheduled and why. Task B: the right cause and next step, with the records cited.
- Wrong outcome: the assistant finished, but the outcome was wrong.
- Handed off: the assistant passed the conversation to a person, including every run that hit the cap.
Every unsuccessful run, wrong or handed off, ends with a person finishing the conversation.
| Measure (600 runs per design) | Workflow A | Agent A | Workflow B | Agent B |
|---|---|---|---|---|
| Successful runs | 582 | 570 | 426 | 516 |
| Wrong outcomes | 0 | 3 | 54 | 30 |
| Handed off | 18 | 27 | 120 | 54 |
| Cases successful in all 3 runs (of 200) | 192 | 176 | 136 | 158 |
| Model calls per run, mean / p95 | 2.1 / 3 | 5.4 / 9 | 1.2 / 2 | 4.6 / 10 |
| Mean input tokens per run | 3,000 | 16,000 | 9,000 | 21,000 |
| Mean output tokens per run | 300 | 900 | 400 | 1,100 |
| p95 processing time | 4.1 s | 11.8 s | 5.2 s | 17.5 s |
| Runs that hit the cap | n/a | 6 | n/a | 12 |
| Runs with a wrong or unnecessary tool call | n/a (code picks every call) | 41 | n/a (code picks every call) | 48 |
Trace notes:
- Agent A: in 9 runs (included in the 41) the model called
rebook_deliverybeforeverify_customer; the executor rejected each call and the run continued. All 3 wrong outcomes booked a slot the customer had not confirmed. - Workflow A: the 18 hand-offs were requests outside the path (redirects, two parcels in one message) and extractions that still failed after the repair.
- Workflow B: each of its 54 wrong outcomes cited a real fetched record, but not the record that explained the hold.
- Agent B: the cap hits alternated between
get_depot_scansandget_carrier_noteswithout new information.
Task B by case type (successful runs; each case ran 3 times)
| Case type | Cases | Runs | Workflow B successful | Agent B successful |
|---|---|---|---|---|
| Depot backlog | 80 | 240 | 228 | 216 |
| Customs hold | 50 | 150 | 138 | 132 |
| Partner carrier handover | 40 | 120 | 36 | 96 |
| Address problem | 30 | 90 | 24 | 72 |
Compare both designs on the same dimensions
State the goal and constraints first: what a correct outcome is, whether the task ends in a write, the latency target, and the budget. Then compare both designs row by row in one table.
| Dimension | Fixed workflow | Agent loop |
|---|---|---|
| Next step chosen by | Code | The model, inside limits set in code |
| Testing | Unit tests per step, fixtures per branch | Outcome grading over repeated trials, plus transcript review |
| Cost and latency | A known number of model calls per path | Varies by case; input tokens grow because each call usually resends the conversation |
| Typical failure | A case no path covers: handed off, or answered from partial evidence | Loops, repeated or unnecessary calls, the wrong tool, stopping early, skipping a required step |
| Limits | Path length fixed in code | Step, time, and cost caps and a no-progress check, all in code |
| Validation and approvals | Enforced by code | Enforced by the executor, never by the prompt alone |
| Traces | Name the failing step | Need every call and a stop reason |
| Adding a case type | A new branch | A new tool, description, and test cases |
Cost per successful task
Cost per run is input tokens times the input price plus output tokens times the output price (prices per million tokens). It undercounts on its own, because failed runs spend tokens too.
- Cost per successful task: total model spend divided by successful runs.
- Cost of a failure: the staff time to finish the conversation. Count wrong outcomes separately, since they can also harm the customer.
- Daily cost: per-run cost times daily volume, plus unsuccessful conversations per day times the handling cost. Compare the model share with the budget.
- Tails: agent runs vary in length, so read p95 calls and p95 latency, not only the means.
For example, $0.010 per run at 80% success is $0.010 / 0.8 = $0.0125 per successful run, before any handling cost.
Work the pilot figures
Use the pilot results. Work each row for all four columns before you reveal it.
Predict the outcome
Work the pilot figures
Using the pilot results, compute each row for all four columns before you reveal the answers. Write each ratio as a fraction first.
| Measure | Workflow A | Agent A | Workflow B | Agent B |
|---|---|---|---|---|
| Success rate per run | ? | ? | ? | ? |
| Cases successful in all three runs | ? | ? | ? | ? |
| Successful runs among the cases that did not succeed in all three runs | ? | ? | ? | ? |
| Model cost per run | ? | ? | ? | ? |
| Model cost per 1,000 successful runs | ? | ? | ? | ? |
| Daily model spend | ? | ? | ? | ? |
| Conversations a person finishes per day, and their cost | ? | ? | ? | ? |
| Daily total, model plus handling | ? | ? | ? | ? |
| p95 processing time against the task's target | ? | ? | ? | ? |
Evaluate both designs on the same cases
Run both designs on the same labeled cases with the same model, tools, and validation, so only the control flow differs.
- Grade outcomes. Compare the final state or answer with the label, and review transcripts for blocked, repeated, or unnecessary calls and cap hits; an exact tool-sequence check fails valid paths.
- Repeat trials. Outputs vary between runs, so run each case several times.
pass^kis the chance that all k runs of a case succeed, averaged over cases; with exactly k runs per case, it is the share of cases that succeed every time. A workflow with model steps varies too, usually less. - Read where failures fall. The same cases failing on every run means a missing path: add a branch or a tool. Failures spread thinly across many cases mean run-to-run variance: tighten tools, instructions, or limits.
- Slice by case type. An overall rate can hide that one design wins the common cases and loses the rare ones.
Read Task B by case type
Use the Task B table by case type in the pilot results.
Predict the outcome
Read Task B by case type
For each Task B case type and the two groupings below it, compute both designs' success rates from the pilot table, then say what the comparison shows before you reveal the answers.
| Case type | Workflow B | Agent B | What it shows |
|---|---|---|---|
| Depot backlog (80 cases) | ? | ? | ? |
| Customs hold (50 cases) | ? | ? | ? |
| Partner carrier handover (40 cases) | ? | ? | ? |
| Address problem (30 cases) | ? | ? | ? |
| Depot backlog and customs hold together (130 cases) | ? | ? | ? |
| Partner handover and address problem together (70 cases) | ? | ? | ? |
Failure modes, limits, and traces
An agent loop fails by looping between tools without new information, repeating a call, choosing a tool the case does not need, stopping early, or skipping a step only the prompt required. A fixed workflow fails when a case falls outside every path: it hands off or, worse, answers from partial evidence.
Both designs need controls in code:
- A step cap, a deadline, a cost cap, and a no-progress check end an agent run with a stated reason and a hand-off (loop mechanics are in this topic's Medium adapter question).
- Required steps and approvals are preconditions the executor checks before a tool runs, in either design.
- A retryable write needs a stable idempotency key per logical operation (this topic's Hard question).
A workflow trace names the failing step. An agent trace needs a record per model call and tool call and a stop reason, so failed runs can be grouped by stop reason and tool sequence.
Hybrids and moving between designs
Common hybrids:
- A router that sends each request to a fixed path, with an agent only for requests no path covers.
- A fixed sequence first, then one bounded agentic step with a small tool set and its own cap, used only when the fixed steps leave the question open.
- An agent whose executor enforces a checklist of required steps.
Move a workflow toward an agent when hand-offs concentrate on cases no branch covers and case types keep changing. Move an agent toward a workflow when one tool sequence dominates its traces: promote that sequence to code. Either way, evaluate the new design on the same labeled cases with repeated trials, and run it in shadow mode before it takes traffic.
When a constraint changes
Start from your own recommendation for each task, then decide whether each change alters it.
Predict the outcome
When a constraint changes
Each row changes one fact of the scenario. For each task, decide whether your recommendation changes and why, using figures where they apply, before you reveal the answers.
| Change | Task A | Task B |
|---|---|---|
| Volume grows ten times, to 60,000 rescheduling and 20,000 stuck-parcel conversations a day, with the same prices and pilot rates | ? | ? |
| A voice channel needs a p95 processing time of 3 seconds for both tasks | ? | ? |
| A refund tool is added to Task B: a parcel stuck more than 5 days gets its delivery fee back | ? | ? |
| A new partner carrier integration arrives every month, each with its own tracking tool and note format | ? | ? |
| The Task B pilot had sampled only depot-backlog and customs-hold cases | ? | ? |
| After a round of fixes, Agent A reaches 582 successful runs with no wrong rebooks and the same token use per run, and its p95 drops to 7.9 seconds | ? | ? |
Completion checklist
- I can say who chooses the next step in a design and place it between a fixed workflow and an agent loop.
- I can turn pilot counts and token figures into cost per successful task and a daily total that includes hand-offs.
- I can use repeated trials and per-slice results to tell a missing path from run-to-run variance.
- I can name the controls that stay in code whichever design runs a task.
- I can name the changes that would reverse a workflow-or-agent decision and the evidence that would confirm them.
Sources
- Anthropic: Building effective agents (December 2024) Defines workflows as predefined code paths and agents as model-directed, recommends the simplest solution that works, and notes that agents trade latency and cost for task performance and need stopping conditions such as a maximum number of iterations.
- OpenAI: A practical guide to building agents (PDF, April 2025) Recommends an agent for complex decisions, hard-to-maintain rules, or unstructured data and says a deterministic solution may otherwise suffice; describes exit conditions such as a maximum number of turns and human intervention for failure thresholds and high-risk actions.
- Yao et al.: tau-bench, a benchmark for tool-agent-user interaction (arXiv 2406.12045, June 2024) Proposes pass^k, the chance that all k trials of a task succeed, to measure reliability, and grades customer-service agents by comparing the final database state with the expected one.
- Anthropic: Demystifying evals for AI agents (January 2026) Runs multiple trials because outputs vary, contrasts pass@k with pass^k, recommends grading outcomes rather than exact tool-call paths, and tracks latency, tokens, and cost per task.
- LangChain docs: Workflows and agents (LangGraph) An example framework's description of workflows with predetermined code paths versus agents that choose their own tool usage, with routing and other workflow patterns.
The parcel company, its tools, pilot cases, prices, handling cost, figures, and rubric are original synthetic teaching material about a fictional application; no real pilot or benchmark result is claimed. Study times and readiness thresholds are editorial estimates. Cited guides and frameworks are examples, not endorsements. Product, protocol, and model details are as of November 2026.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
AI Engineering
Runs the same two-design comparison for another choice, with cost per question, latency, and a sliced pilot.
AI Engineering
Scores labeled cases with named denominators, slices, and repeated trials, the habits a fair pilot depends on.
Practice problem
Practice computing success rate and cost per success for each arm, with the spend of failed attempts counted.
Community article
Works a per-call cost from input and output token prices up to a monthly total, and shows why resent history keeps growing.
Behavioral Interview
Frames a recommendation by naming the axes, the options, weighted criteria, and the cost you accepted with your choice.
Practice the interview question
Answer it in your own words before you look at the reference answer.