LLM Foundations · 2 to 3 hours
Diagnose incomplete model output
Work out why a model's answer looks cut off or incomplete by reading termination metadata, token budgets, and stream evidence before changing any setting.
Hands-on debugging
3 h
Used in interview rounds:
Debugging & Evaluation (Medium)1
Step 1: The question
Analyze cases A to D. Produce a diagnosis table (likely cause, confirming check, minimal correction) and a regression checklist. Then check the planning budget below: does it fit the capacity, and what would you remove first?
Planning budget for a fictional model
| Part | Tokens |
|---|---|
| Context capacity | 16,000 |
| Instructions | 1,000 |
| History | 5,000 |
| Task data | 7,000 |
| Message overhead | 500 |
| Output reserve | 3,000 |
Constraints
- Use only the evidence in the fixture table and the budget
- Confirm each hypothesis before changing any configuration
Deliverables
- Diagnosis table for cases A to D
- Regression checklist
- Budget check with what you would remove first
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Diagnose
Four incomplete outputs
The fixture below uses normalized termination labels. It represents a fictional API with separate, adequate input capacity.
| Case | Output cap | Generated tokens | Termination | Observation |
|---|---|---|---|---|
| A | 120 | 120 | output_limit | JSON stops halfway through the second object |
| B | 1,000 | 88 | stop_sequence | Text ends at END, which occurs in the source |
| C | 1,000 | 410 | completed | Server response complete, browser displays the first 90 tokens |
| D | 1,000 | 70 | completed | Returns one summary sentence as requested |
For each case, identify the most likely cause, name one confirming check, and propose a minimal correction. Then explain why changing temperature is an insufficient diagnosis.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 8 · Scored 0 of 4 dimensions
Ready at 6 of 8, with no 0 in Evidence selection
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can diagnose all four cases from evidence, explain the difference between capacity, output cap, and visible output length, and record an unknown termination state without inventing a cause.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why is a larger output cap insufficient for every truncated response?
Why might a longer prompt produce worse results?
How do you make a comparison fair?
What does normal termination prove?
Can a seed guarantee repeatability?
7