Tool Calling & Agents · 2 to 3 hours
Choose a fixed workflow or an agent loop
Compare a fixed workflow and an agent loop for a parcel-delivery assistant's rescheduling and stuck-parcel tasks using a pilot's success, cost, and latency figures, then recommend a design per task and say what would change it.
Conceptual comparison
3 h
1
Step 1: The question
The parcel company's team asks whether each assistant task should run as a fixed workflow or as an agent loop. Using the scenario and the pilot results, state each task's goal and constraints, compare both designs for each task on the same dimensions with the pilot's figures, and recommend a design per task. Then name the evidence you still lack and the changes that would reverse each recommendation.
Scenario: a parcel-delivery assistant
Suppose a parcel-delivery company runs a chat assistant on its website. The company, its tools, and every figure in this question are fictional. For each of two tasks, the team built both a fixed workflow and an agent loop. Both designs use the same model, tools, and argument validation; only the control flow differs.
Task A: reschedule a delivery. A customer gives a tracking number and asks for another day. A parcel can be rescheduled only while it is at a depot (not out for delivery, delivered, or held at customs). A rebook is a write: a wrong one sends a driver on the wrong day.
| Tool | What it does |
|---|---|
verify_customer | Checks the tracking number against the postcode the customer gives |
get_shipment | Returns the parcel's state and depot |
list_slots | Returns open delivery slots for a day |
rebook_delivery | Moves the delivery to a slot (write) |
- Workflow A. The model extracts the tracking number, postcode, and preferred day into validated JSON (one repair attempt). Code verifies the customer, reads the shipment, checks that its state allows rescheduling, and lists slots. The customer picks and confirms a slot with buttons; code rebooks, and the model writes the confirmation from the result. Anything outside this path, such as a redirect to a neighbor, goes to a person.
- Agent A. The model receives the four tools and instructions that include "verify the customer first" and "confirm the slot with the customer before rebooking". The executor validates arguments and rejects
get_shipment,list_slots, andrebook_deliveryuntilverify_customerhas succeeded. Cap: 10 model calls, then a hand-off.
Task B: explain why a parcel is stuck. The customer asks why tracking has not moved. The reply must name the cause and the next step and cite the records it used. Every Task B tool is read-only.
| Tool | What it returns |
|---|---|
get_tracking_events | The public tracking history |
get_depot_scans | Internal scans at each depot |
get_carrier_notes | Free-text notes from drivers and partner carriers |
get_customs_status | Customs holds and missing documents |
get_partner_tracking | A partner carrier's history, given the partner reference found in a carrier note |
check_address | Whether the address failed validation at the depot |
- Workflow B. Code calls the first four tools in parallel, and one model call writes the explanation from their results (one repair attempt if the reply fails its format check). Code checks that every cause the reply cites comes from a fetched record; when that check fails or the model finds no record that explains the hold, the conversation goes to a person.
- Agent B. The model receives all six tools and chooses which to call and in what order. Code applies the same cited-record check to its reply. Cap: 12 model calls, then a hand-off.
Product constraints
| Constraint | Task A | Task B |
|---|---|---|
| Conversations per day | 6,000 | 2,000 |
| p95 processing time target (excluding customer wait) | 8 s | 20 s |
| Writes | rebook_delivery; no wrong rebooks allowed | None |
Model spend for both tasks together must stay at or below $100 per day. Prices: $1.00 per million input tokens and $3.00 per million output tokens. Each conversation a person has to finish costs $2.50 of staff time.
Pilot results
The team labeled 200 recent conversations per task with the correct outcome. Each design ran every case three times (600 runs per design), and a reviewer graded every run's outcome against its label and read every trace.
- Successful: the labeled outcome without a person. Task A: the confirmed slot is booked, or the assistant correctly says the parcel cannot be rescheduled and why. Task B: the right cause and next step, with the records cited.
- Wrong outcome: the assistant finished, but the outcome was wrong.
- Handed off: the assistant passed the conversation to a person, including every run that hit the cap.
Every unsuccessful run, wrong or handed off, ends with a person finishing the conversation.
| Measure (600 runs per design) | Workflow A | Agent A | Workflow B | Agent B |
|---|---|---|---|---|
| Successful runs | 582 | 570 | 426 | 516 |
| Wrong outcomes | 0 | 3 | 54 | 30 |
| Handed off | 18 | 27 | 120 | 54 |
| Cases successful in all 3 runs (of 200) | 192 | 176 | 136 | 158 |
| Model calls per run, mean / p95 | 2.1 / 3 | 5.4 / 9 | 1.2 / 2 | 4.6 / 10 |
| Mean input tokens per run | 3,000 | 16,000 | 9,000 | 21,000 |
| Mean output tokens per run | 300 | 900 | 400 | 1,100 |
| p95 processing time | 4.1 s | 11.8 s | 5.2 s | 17.5 s |
| Runs that hit the cap | n/a | 6 | n/a | 12 |
| Runs with a wrong or unnecessary tool call | n/a (code picks every call) | 41 | n/a (code picks every call) | 48 |
Trace notes:
- Agent A: in 9 runs (included in the 41) the model called
rebook_deliverybeforeverify_customer; the executor rejected each call and the run continued. All 3 wrong outcomes booked a slot the customer had not confirmed. - Workflow A: the 18 hand-offs were requests outside the path (redirects, two parcels in one message) and extractions that still failed after the repair.
- Workflow B: each of its 54 wrong outcomes cited a real fetched record, but not the record that explained the hold.
- Agent B: the cap hits alternated between
get_depot_scansandget_carrier_noteswithout new information.
Task B by case type (successful runs; each case ran 3 times)
| Case type | Cases | Runs | Workflow B successful | Agent B successful |
|---|---|---|---|---|
| Depot backlog | 80 | 240 | 228 | 216 |
| Customs hold | 50 | 150 | 138 | 132 |
| Partner carrier handover | 40 | 120 | 36 | 96 |
| Address problem | 30 | 90 | 24 | 72 |
Constraints
- Use only the scenario, prices, and pilot figures shown; all of them are fictional
- Both designs share the model, tools, and argument validation; only the control flow differs
- Work the figures first, then present in about five minutes, quoting the figures that decide each task
Deliverables
- Each task's goal and binding constraints
- A comparison of both designs for each task on the same dimensions
- Cost per successful run, daily cost including conversations a person finishes, and the latency and budget checks
- A recommendation per task with the figures behind it and the evidence still missing
- What would change each recommendation
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: States each task's correct outcome, whether it ends in a write, its latency target, and the shared budget before comparing designs.
What earns a 2: Compares both designs for each task on the same rows (for example who picks the next step and its limits, cost, latency, failure modes, controls in code, and testing) using the pilot's figures.
What earns a 2: Shows cost per run, cost per successful run, daily model and handling cost, consistency across the three runs, and the per-slice rates from the stated prices and counts.
What earns a 2: Recommends a design per task tied to named figures, explains the cause of each failure class before weighing it, and names the evidence still missing.
What earns a 2: For each changed constraint, re-checks the affected figures and says whether each task's recommendation holds, and why.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Same-dimension comparison or Decision and evidence
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can turn a two-design pilot into cost per successful task and a daily total, compare the designs task by task on the same dimensions, and say which changes would reverse your choice.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
What makes a design an agent loop rather than a fixed workflow?
Why compare cost per successful task instead of cost per run?
Why run every pilot case more than once?
How do you grade an agent run without requiring one exact tool sequence?
Where do required steps and approvals live in a fixed workflow and in an agent loop?
7