Document AI: How Models Read PDFs, Tables, and Receipts

Document AI systems turn semi-structured documents into machine-readable structured data by chaining OCR engines, layout detectors, and multimodal transformers. These architectures jointly model text, visual appearance, and spatial coordinates to handle forms, tables, and receipts at scale. At the core, models like LayoutLMv3 and Donut fuse text tokens with image patches and 2D positional embeddings inside a transformer encoder, then route through task-specific heads or generative decoders to produce JSON-like outputs, table grids, or classification labels.

Here is the uncomfortable truth: the best Document AI models do not read documents the way you do. They discard reading order entirely. They flatten a page into an unordered bag of tokens with coordinates, then rely on self-attention to rediscover which words belong together. A receipt line item with its price three centimeters to the right is not "read" left-to-right. It is connected geometrically, through learned spatial relationships encoded in the attention weights.

Where We Left Off

In Part 4 we examined how video models struggle with temporal consistency because every frame must relate plausibly to every other frame across time. Document AI faces a parallel challenge but in the spatial domain. A table cell must relate to its column header above it and its row label to the left. A form field must connect to its label, which might sit anywhere on the page. The spatial layout is the temporal dimension of documents, and the models we will examine here learn it without ever reading in order.

How Does a Document Become Model Input?

The first transformation is destructive, and that destruction is the key insight.

A scanned receipt, a PDF invoice, a multi-page claim form: every one of these starts as pixels. The pipeline converts these pixels into a sequence of tokens where each token carries three things: a piece of text, the coordinates where it sits on the page, and optionally a crop of the pixels around it. Reading order is deliberately thrown away. The tokens are serialized in whatever order the OCR engine produced them, which is often top-left to bottom-right but is not guaranteed to be meaningful. The model will rediscover reading order from spatial coordinates if it needs them. Often it does not need them at all.

This is LayoutLM's foundational idea. Each input token gets three embeddings summed together: a word embedding from a standard vocabulary, a 1D sequence position embedding, and a 2D spatial embedding encoding the token's bounding box coordinates, typically normalized to [0, 1000] in both axes LayoutLM paper. The transformer then sees a flat list of tokens floating in 2D space and learns to pull related tokens together through attention. Two tokens near each other on the page and semantically related will develop similar attention patterns, even if separated by dozens of tokens in the serialized sequence.

The 2D bounding box is the bridge between vision and language. Every OCR token produced by engines like Google's Enterprise Document OCR or AWS Textract carries coordinates (x_min, y_min, x_max, y_max) along with the recognized text and a confidence score Google Cloud Document AI overview. The model embeds these four numbers into a higher-dimensional vector, typically through a learned linear projection that maps four floats to the model's hidden dimension. That vector gets added to the token embedding.

The image itself enters through a parallel path. For LayoutLMv3, the document page image is split into fixed-size patches (16×16 pixels is standard), each patch flattened and projected into an embedding vector, exactly as Vision Transformers do LayoutLMv3 paper. These image patches live alongside the text tokens in the same transformer. A patch covering the corner of a table cell and the text token for the cell's contents attend to each other, binding visual and textual evidence. The architecture eliminates the separate CNN backbone that LayoutLMv2 required, making the model simpler and faster.

Why Do Multimodal Document Transformers Need Special Pre-training?

Standard masked language modeling is not enough. Masking a word and predicting it from context teaches the model about language but nothing about layout. Document transformers need objectives that force them to use spatial and visual signals.

LayoutLMv3 uses three pre-training objectives. Masked language modeling with layout context: the model predicts masked text tokens, but now it has access to the spatial positions and image patches of all tokens, so it learns to use "this token is in the top center of the page" as a feature. Masked image modeling: random image patches are masked, and the model predicts the visual features of the masked regions from surrounding patches and nearby text LayoutLMv3 GitHub. This is analogous to the pixel reconstruction objective in ViT pre-training but conditioned on text. A word-patch alignment objective: the model learns whether a given text token corresponds to a given image patch, forcing explicit alignment between visual regions and the words that describe them.

These objectives together create a representation where "total_amount" is tightly bound to the bottom-right region of a receipt image, even if that word never appears near "total_amount" in any training sequence. The model has seen thousands of receipts where large numbers appear near the bottom-right and has learned that spatial location predicts semantic category.

DocFormer takes a different architectural choice. Instead of summing modalities into one embedding, it keeps text, vision, and spatial features as separate streams that interact through a dedicated multi-modal self-attention layer DocFormer paper. Each modality is encoded independently first, then cross-attention lets text tokens query visual tokens and vice versa. The spatial embeddings are shared across modalities, which means the same coordinate system encodes both where a word sits and where an image patch sits. This shared spatial grounding is what lets a visual patch representing a table line bind to the text tokens that represent table cell contents.

DiT (Document Image Transformer) takes an even more radical step: it ignores OCR text during pre-training entirely. DiT is pre-trained on 42 million document images from the IIT-CDIP dataset using only masked visual token prediction DiT paper. The idea is that a pure vision model trained on enough documents will learn visual patterns that generalize to text and layout tasks. Fine-tuning then adds task-specific heads for document classification, layout analysis, and table detection. DiT effectively says: the visual patterns of documents are so rich and structured that understanding text as text can be deferred to downstream tasks.

Each of these architectures makes a different bet on how much language matters for document understanding. LayoutLMv3 bets on joint modeling from the start. DocFormer bets on keeping modalities separate with explicit cross-attention. DiT bets on vision-only pre-training at scale. The right choice depends on the task. For form understanding where field labels matter, joint modeling wins. For layout analysis where region detection dominates, DiT's approach is competitive with fewer parameters.

How Do Tables Get Parsed into Cells?

Table detection and structure recognition are genuinely distinct problems that require different solutions.

Table detection finds the bounding box of a table on a page. This is a standard object detection problem. Models like Table Transformer use DETR, a transformer-based object detector, trained on PubTables1M to locate table regions Table Transformer paper. DETR predicts a fixed set of bounding boxes and labels using bipartite matching during training, which naturally handles the variable number of tables per page.

Table structure recognition is harder. Once a table region is cropped, the model must identify rows, columns, headers, and spanning cells. Table Transformer handles this by treating row separators and column separators as objects with bounding boxes. The output is two sets of boxes: horizontal boxes representing rows and vertical boxes representing columns. Their intersections define cells. A cell's box is the intersection of a row bounding box and a column bounding box Table Transformer HuggingFace demo.

This intersection approach is elegant but brittle. Merged cells break it. A cell spanning two columns creates an ambiguous intersection. DeepTabStR addresses this with deformable convolutional networks that can warp their receptive fields to fit irregular table geometries DeepTabStR paper. The model learns to output a grid representation directly, with explicit handling for spanning cells coded as special tokens in the output sequence.

The practical pipeline for table extraction in production systems like AWS Textract works in stages. Detect the table region with a detector model. Crop the table image. Run structure recognition to identify rows and columns. For each cell implied by row-column intersection, run OCR on the cell region to extract text. Assemble the results into a CSV or JSON structure AWS Textract docs. This staged approach is modular and debuggable but introduces latency from multiple model calls.

An emerging alternative is end-to-end table extraction where a single model takes a document image and outputs a structured representation directly. Google's Document AI processors for form parsing effectively do this for key-value pairs, outputting the parsed fields without explicit table structure recognition Google Cloud Document AI processors. The model internally handles both detection and extraction, but the mechanics are less transparent than the staged approach.

How Do OCR-Free Models Skip Text Extraction Altogether?

OCR is slow, error-prone, and language-dependent. Donut asks: what if the model reads text directly from pixels without ever producing an intermediate text representation?

Donut uses a Swin Transformer encoder and a BART decoder to generate structured outputs directly from document images Donut paper. The input is a raw image. The output is a JSON string. For a receipt, the output is {"total": "42.00", "date": "2024-01-15", "items": [...]}. No OCR step. No text bounding boxes. The Swin Transformer encoder processes the image into a sequence of patch embeddings through its hierarchical architecture. The BART decoder attends to these embeddings and generates tokens autoregressively.

The training signal comes from SynthDoG, a synthetic document generator that creates training images by rendering text onto document backgrounds with realistic variations in font, layout, noise, and distortion SynthDoG GitHub. Because SynthDoG produces the ground truth text during rendering, the model learns to map pixel patterns directly to character sequences. Fine-tuning on real documents then adapts this synthetic pre-training to the target domain.

Donut's limitation is that it operates on a fixed vocabulary. The decoder generates text tokens from a standard language model vocabulary, so it can only read text it knows how to write. For receipts and forms with constrained vocabularies (numbers, dates, common words), this works well. For documents with rare technical terms or arbitrary names, OCR-free models still underperform traditional OCR-based pipelines.

The architecture reveals something fundamental about document reading. When you or I read a receipt, we do not consciously OCR each character. We perceive words as visual gestalts. Donut's encoder learns these visual gestalts directly. A word like "Total" rendered in bold on a receipt background becomes a visual pattern that the encoder maps to a semantic representation, bypassing explicit character recognition. This is how human expert readers process familiar document types, and it is why OCR-free models can be faster and more robust for domain-specific tasks.

Quick Reference

Property	Value
LayoutLMv3 backbone	Unified transformer (text + image patches + 2D positions)
Pre-training objectives (LayoutLMv3)	MLM, MIM, word-patch alignment
Table detection model	Table Transformer (DETR on PubTables1M)
Table structure output	Row/column bounding boxes → cell intersections
OCR-free model	Donut (Swin encoder + BART decoder + SynthDoG pre-training)
Key datasets	PubLayNet (layout), FUNSD (forms), CORD (receipts), PubTables1M (tables)
Typical page coordinate normalization	[0, 1000] or [0, 1] across both axes
Image patch size	16×16 pixels

Frequently Asked Questions

Q: Why do document transformers use 2D positional embeddings instead of 1D sequence positions?

2D embeddings encode spatial layout directly. A 1D sequence position only tells the model the token's order in the linearized text, which for a multi-column PDF or a form is arbitrary. The 2D bounding box tells the model "this token is in the top-left" or "this token is directly below the header," which is what matters for layout understanding. The model learns that proximity in the 2D coordinate space correlates with semantic relationship, regardless of sequence position.

Q: Can Document AI handle handwritten text in scanned forms?

Yes, with caveats. TrOCR was explicitly designed to handle printed, handwritten, and scene text within a single transformer architecture TrOCR paper. It is pre-trained on synthetic data that includes varied handwriting styles. Production systems like AWS Textract also support handwriting. The limitation is accuracy: unstructured handwriting with heavy cursiveness still degrades recognition, and downstream layout models may struggle with irregular spacing in handwritten forms.

Q: How does Document AI handle multi-page PDFs with hundreds of pages?

Pages are processed independently, then results are merged. Each page goes through OCR and layout analysis separately. For models with token limits like LayoutLMv3 (512 tokens), long pages or dense tables require sliding-window approaches where the page is processed in overlapping chunks. Multi-page semantics, like a table that spans two pages, are typically handled by post-processing heuristics rather than being modeled by the transformer directly.

Q: What is the difference between LayoutLMv3 and Donut's approach to the image modality?

LayoutLMv3 treats image patches as additional tokens that coexist with text tokens in the transformer. It requires OCR text at inference time. Donut skips text entirely: the encoder produces visual representations, and the decoder generates text output directly from those visual representations. LayoutLMv3 excels at tasks where OCR is available and the goal is semantic enrichment. Donut excels at end-to-end tasks where OCR latency or error is the bottleneck.

Q: How are these models evaluated on real-world document understanding tasks?

The standard benchmarks are FUNSD for noisy scanned forms (token-level semantic labeling into "question," "answer," "header," and "other") FUNSD dataset, CORD for receipts (entity extraction for prices, items, totals), and PubLayNet for layout analysis (region detection and classification). Table extraction is evaluated on ICDAR-13 and PubTables1M using mean Average Precision for detection and F-measure for structure recognition.

Test Yourself

Your team is building an expense report system that ingests photographed receipts from a mobile app. The photos vary wildly: some are well-lit and flat, others are crumpled, skewed, or have coffee stains. You choose LayoutLMv3 with a pre-trained OCR engine. At inference, you notice the field extraction accuracy drops 30% for photos taken at an angle. What is the most likely cause, and what change would you make first?

Answer: The OCR engine is producing bounding boxes in the original skewed coordinate space. LayoutLMv3's 2D positional embeddings encode these skewed coordinates directly, so tokens that should align horizontally (like a line item and its price) appear at different y-coordinates. The model's spatial attention patterns, which were trained on predominantly aligned documents, fail to connect them. The first fix is adding a geometric rectification step before OCR: detect the document corners, apply a homography transform to flatten the perspective, and crop to a top-down view. This normalization step is standard in production Document AI pipelines like Google Cloud Document AI's preprocessing. If rectification is not possible at the image level, a lighter alternative is training the model with augmented synthetic data that includes perspective distortions, teaching the spatial embeddings to become more robust to skew, but rectification is the higher-leverage change.

What's Next

In the final episode, we will examine how vision models handle 3D data: point clouds, depth maps, and the spatial reasoning tasks that flat images cannot solve. The same transformer architectures we have traced through CLIP, LLaVA, diffusion, video, and documents will face their hardest test yet: understanding the world in three dimensions.

If these articles reveal how the systems you rely on actually work, subscribe to Internals Decoded at internalsdecoded.com. Next week we break 3D vision.

Sources

• LayoutLM: Pre-training of Text and Layout for Document Image Understanding
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
• LayoutLM GitHub (Microsoft Unilm)
• DocFormer: End-to-End Transformer for Document Understanding
• DiT: Self-supervised Pre-training for Document Image Transformer
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• Donut: Document Understanding Transformer without OCR
• SynthDoG (Donut synthetic data)
• Table Transformer (TATR) and PubTables1M
• Table Transformer HuggingFace Documentation
• DeepTabStR: Deep Learning for Table Structure Recognition
• PubLayNet: Largest Dataset Ever for Document Layout Analysis
• FUNSD: Form Understanding in Noisy Scanned Documents
• Google Cloud Document AI Overview
• Google Cloud Document AI Processors
• AWS Textract How It Works