Handle an uncertain screenshot

A modality is a form of information, such as text, pixels, or sound. A multimodal application processes more than one form. Its underlying model may accept multiple modalities directly, or the application may combine specialized components.

Suppose a coding-practice platform lets learners upload a screenshot of code, a PDF exercise, or a recorded explanation. A useful response depends on correctly interpreting the input and connecting the answer to the relevant evidence.

Input support and output support are separate. A model that accepts images does not necessarily generate images. Check the selected system's actual interfaces and limits rather than assuming that "multimodal" means every combination is available.

A document contains content and structure. Headings, columns, tables, captions, footnotes, checkboxes, and diagrams influence meaning. Flattening everything into a string can separate a value from its label or combine unrelated columns.

A PDF may contain selectable text, scanned page images, or a mixture. Some scanned pages include an OCR text layer that may be inaccurate, and a text-extraction library reads that layer without checking it against the image. Successfully extracting text does not prove that the extracted order or wording matches the visible page.

Keep the original file so users and reviewers can inspect the source. A plain-text derivative is useful for search but is not a complete replacement for the document.

Different pages in one file may need different routes. Treat routing as a quality decision, not simply a filename-extension check.

Optical character recognition converts visible writing into machine-readable text. It does not automatically explain the document or determine whether a statement is true. Visual interpretation may infer relationships, but it can also invent missing details.

In code, one character can change the meaning. A blurry items[l] (the letter l) might be read as items[1] (the digit one). One indexes with a variable, the other always reads the second element, and each points to a different bug. When exact characters matter, an OCR result is a candidate transcription, not the code itself.

Image preprocessing can help OCR. Rotation correction, deskewing, and appropriate contrast may improve readability, but aggressive filtering can remove punctuation or faint marks. Inspect changes against the original. Tesseract's documentation discusses how resolution, noise, rotation, and skew affect recognition.

A useful document citation identifies the exact file revision and page, and ideally the region supporting the claim. Audio and video evidence use time ranges. A link to the entire file is often too broad for verification.

Keep printed page labels separate from physical page positions. "Page 3" printed in a report might be the fifth page in the PDF because of cover pages. State the convention clearly.

A region reference establishes where content came from. It does not prove that the content supports the generated answer. Check the relationship between the claim and the cited evidence.

Extraction maps source content into a defined schema. For an exercise handout, fields might include title, language, difficulty, constraints, and examples. Distinguish missing information from a failed extraction and from a field that does not apply.

{
  "field": "language",
  "value": "JavaScript",
  "status": "extracted",
  "evidence": {
    "document_revision": "handout-v2",
    "page_index": 0,
    "quote": "Language: JavaScript"
  }
}

This is an illustrative application schema, not a provider API. A field should not be invented just because the schema requires an answer. Use an explicit unresolved state and route it for review.

Tables require row and column relationships. A value without its header, unit, and relevant footnote can be misleading. Preserve merged cells and repeated headers when they affect meaning. Layout systems can expose cell coordinates and spatial regions; their exact schemas vary.

Charts also require axes, scales, legends, and units. A bar's height is not a reliable exact value unless the scale and rendering permit that precision. Prefer original tabular data for exact calculations when available.

Diagrams encode relations through arrows, position, labels, and grouping. Explain which connections are visible and which are inferred. Do not assume nearby labels refer to the same object.

Speech recognition creates a transcript. Diarization estimates who spoke when; it is different from identifying a person's real identity. Translation adds another transformation and can lose technical nuance.

Preserve timestamps so a reviewer can listen to uncertain words. For code tutorials, distinguish spoken punctuation from punctuation visible on screen. A transcript may omit a command that appears only visually.

Video analysis usually works under a sampling or processing budget. If frames are sampled every few seconds, the system cannot establish that nothing happened between them. Report observation limits when an answer depends on short events or precise timing.

A model's confident wording is not a calibrated probability. An OCR confidence score may refer to one word or region and may not predict whether a final answer is correct. Use scores as signals whose usefulness must be measured on representative data.

Review should focus on consequential ambiguity: unreadable operators, mismatched table headers, missing pages, conflicting sources, or unsupported conclusions. "Unable to determine from this input" is a valid result.

Decide in advance what each kind of ambiguity leads to: proceed with a stated caveat, ask the user for a better source, or send the item to a person. Base the choice on what a wrong reading would cost, not on how fluent the output sounds.

Uploaded documents and media are untrusted input. Text inside an image or PDF can be written as instructions to the assistant; it is source content with no authority to change application rules or invoke tools. Security & Privacy covers prompt injection in depth.

Enforce file access in the backend and apply the same permissions to extracted text, thumbnails, transcripts, and citations. Derived content can contain the same sensitive information as the original.

Provide accessible text descriptions, transcripts, and keyboard-accessible source navigation. Preserve uncertainty in those alternatives; a misleading transcript is not an adequate substitute for evidence.

This exercise applies the route table to a whole document, where different pages need different routes.