LLM Foundations · 2 to 3 hours

Diagnose incomplete model output

Work out why a model's answer looks cut off or incomplete by reading termination metadata, token budgets, and stream evidence before changing any setting.

Medium

Hands-on

Hands-on debugging

3 h

Used in interview rounds:

Debugging & Evaluation (Medium)

Jump to

Overview

Debug incomplete, inconsistent, and unsupported model responses from evidence: token budgets, termination metadata, and stream logs. You can do every exercise offline with the fixtures here; no model API is needed.

Study time: 2 to 3 hours

Interview format: Debugging exercise with synthetic fixtures

By the end you can

Build a token budget that includes message overhead and the output reserve
Separate model termination (output limit, stop sequence, tool call) from transport and rendering failures
Name the evidence that confirms each hypothesis before changing configuration
Explain why changing temperature does not fix truncated output
Design a controlled comparison of generation settings

Prerequisites

Correct common LLM assumptions

Helpful background: Basic JSON, request logging, and arithmetic are enough.

Understand the request boundary

Capture the exact model identifier, request configuration, input composition, and termination metadata. Distinguish the prompt the developer intended from the payload actually sent. Chat formatting, repeated history, tool schemas, and application-added instructions can change size and meaning.

Store sensitive content only under appropriate access and retention rules. In many investigations, token counts, request identifiers, configuration versions, and sanitized fixtures provide enough evidence.

Budget input and output explicitly

Use instructions + history + task data + message overhead + output reserve as a planning breakdown. Check the actual provider's accounting before treating it as an executable formula. Output token allowances may include hidden reasoning in some interfaces, so visible text length alone may not explain usage.

When a plan does not fit, remove input before cutting the output reserve, and keep a margin because estimates are approximate. Prefer removing repeated or irrelevant content. Summarization can discard dates, exceptions, and constraints, so test that the facts the task requires survive it.

Interpret termination evidence

Observation	Possible cause	Next evidence to inspect
Output ends mid-object and reaches its limit	Generation budget exhausted	Termination reason, usage, requested output cap
Output stops at a repeated delimiter	An explicit stop sequence matched	Stop configuration and raw output
UI shows half an answer but server has a complete one	Streaming or rendering failure	Server completion, client stream events, cancellation
Request fails before producing output	Capacity violation or invalid parameters	API error and actual serialized request
Output is short and termination is normal	Task interpreted narrowly or natural completion	Prompt, examples, expected task length
Response requests a tool	Model handed off control	Tool-call metadata and application continuation

Provider-specific termination labels should be normalized into application categories while retaining the original metadata. A tool handoff is a distinct state in the interaction.

Sampling and reproducibility

Temperature changes the shape of the token distribution when the selected decoding method uses it. Top-p sampling limits choices to a set whose cumulative probability reaches a threshold. Greedy decoding selects a highest-probability token. Support and interactions differ across model interfaces.

Changing multiple settings simultaneously makes attribution difficult. Hold the model, prompt, input, output cap, and environment stable, then vary one supported parameter. Use repeated trials because one attractive answer can be an outlier. Even apparently deterministic configurations may vary across service revisions or execution conditions.

Evaluate completeness and factual support separately from diversity. A repetitive answer may be consistent and wrong. A varied answer may still satisfy the task.

Context quality and attention

Information can fit inside the allowed context without being used reliably. Repeated passages, conflicting versions, and irrelevant material can distract from the evidence needed. Test whether performance changes when relevant facts appear early, centrally, or late in a long input.

A summary is a lossy transformation. Keep provenance to the original material when exact claims matter. Use structured retained facts for critical requirements and compare answers against the original source during evaluation.

Diagnose hallucinations systematically

Start by asking whether the answer is supported by the supplied source. If the source is absent, the application may need retrieval or a verified tool. If the source is present but contradictory, define a precedence rule or request clarification. If the source is clear and the model contradicts it, inspect task wording and compare models or configurations on the same fixture.

Separate three measurements: whether relevant evidence was present, whether the response used it correctly, and whether the final output met the application's requirements. Retrieval implementation belongs in its dedicated topic, but this separation is essential to avoid blaming every failure on generation.

Debugging exercise: four incomplete outputs

Diagnose

Four incomplete outputs

The fixture below uses normalized termination labels. It represents a fictional API with separate, adequate input capacity.

Case	Output cap	Generated tokens	Termination	Observation
A	120	120	`output_limit`	JSON stops halfway through the second object
B	1,000	88	`stop_sequence`	Text ends at `END`, which occurs in the source
C	1,000	410	`completed`	Server response complete, browser displays the first 90 tokens
D	1,000	70	`completed`	Returns one summary sentence as requested

For each case, identify the most likely cause, name one confirming check, and propose a minimal correction. Then explain why changing temperature is an insufficient diagnosis.

Controlled experiment

Worked example

Use ten original summarization fixtures with short, long, contradictory, and insufficient-information inputs. Write expected facts and forbidden unsupported claims before generating outputs. Compare two supported sampling configurations with three trials per fixture. This creates 60 outputs.

Score factual support, required fact coverage, completeness, and output length. Report results per fixture group as well as overall. Retain an untouched set for confirming the chosen configuration. A small experiment can identify failure modes, but it cannot establish a precise rare-error rate.

The exercise can be performed offline using supplied response fixtures. Paid inference is optional.

Completion checklist

Explain the difference between capacity, output cap, and visible output length.
Diagnose all four fixture cases using evidence.
Identify an unknown termination state without inventing a cause.
Design an experiment that separates output variation from task quality.

Sources

Hugging Face text generation guide Generation settings and common pitfalls.
Hugging Face generation strategies Greedy decoding, sampling, temperature, and top-p.
Anthropic: effective context engineering for AI agents Context selection and compaction.

All numerical fixtures, normalized termination labels, and exercise outcomes are original synthetic teaching material. Study times and readiness thresholds are editorial estimates. Termination labels and token accounting differ between providers, so check the selected service's documentation.

Practice the interview question

Answer it in your own words before you look at the reference answer.

2 of 4

Previous: Correct common LLM assumptions Back to LLM Foundations

Next: Choose long context or retrieval