Document AI: How Models Read PDFs, Tables, and Receipts

Document AI turns semi-structured documents (forms, invoices, receipts, tables) into structured data a program can use. A classic system chains an OCR (optical character recognition) engine, a layout detector and a multimodal transformer. LayoutLMv3 fuses OCR text tokens, image patches and 2D positions inside one transformer, then routes the result to task-specific heads; Donut skips OCR, reads straight from pixels and writes JSON. Since 2024, vision-language models that read the page image directly have become a strong alternative, and this article covers both.

The key idea of layout-aware models is that they do not rely on reading order alone. Each word carries both its place in the OCR sequence and its box on the page, so attention can link a price to its item even when the OCR order is wrong. A receipt line item with its price three centimeters to the right is connected geometrically, through spatial relationships the model learned in training.

Where we left off

In Part 4 we saw that video models must keep each frame consistent with the frames around it in time. Document AI faces a parallel challenge in space. A table cell must relate to the column header above it and the row label to its left, and a form field must connect to its label, which might sit anywhere on the page. Layout plays the role for documents that time plays for video. Our photo app meets this problem when it starts reading receipts from the camera roll.

How does a document become model input?

Start with a practical question: does the file need OCR at all? Many PDFs are not pixels. If a file was exported from a word processor or a website, the text is stored inside it and a library can read it directly, as How ChatGPT Reads a PDF explains. OCR is needed for scans and photos, such as the receipts in your camera roll.

For those, the pipeline turns pixels into a sequence of tokens, and each token carries three things: a piece of text, the coordinates of its box on the page, and optionally the pixels around it. The OCR engine emits words in an order that is usually top left to bottom right, but often wrong for columns, forms and tables. Layout-aware models keep that order as one signal and add the box positions as another, so they can recover the right relationships even when the order misleads.

This is LayoutLM's foundational idea. Each input token gets embeddings summed together: a word embedding from a standard vocabulary, a 1D position embedding for its place in the OCR sequence, and a 2D position embedding for its box (Xu et al. 2019, LayoutLM: Pre-training of Text and Layout for Document Image Understanding). The box coordinates are scaled to the page size (to a range of 0 to 1,000 in the Hugging Face LayoutLM documentation), and each coordinate is looked up in a learned embedding table, one table for x positions and one for y. The transformer then sees a list of tokens placed on a 2D page and learns to relate them through attention. Two tokens that sit near each other on the page can attend to each other strongly, even when dozens of tokens separate them in the OCR sequence.

Diagram loading

The 2D box is the bridge between vision and language, and OCR services return one with every word. Amazon Textract, for example, returns each detected item as a block with its geometry (a bounding box and a polygon) and a confidence score, as its documentation on Textract response objects shows. LayoutLM does not push those four numbers through a single linear layer: it looks each integer coordinate up in its embedding tables and sums the results. LayoutLMv2 and v3 also embed the box's width and height.

The image itself enters through a parallel path. LayoutLMv3 resizes the page image to 224 × 224 pixels, splits it into 16 × 16 pixel patches, and linearly projects each patch into a vector, the same patch embedding that Vision Transformers use, which you met in Part 1 (Huang et al. 2022, LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking). These image patches sit alongside the text tokens in the same transformer, so a patch covering the corner of a table cell and the text token for the cell's contents can attend to each other. LayoutLMv3 was the first multimodal Document AI model to need no CNN or object detector for its image features, which made it simpler than LayoutLMv2.

Why do multimodal document transformers need special pre-training?

Standard masked language modeling (hide some words and train the model to predict them from context) is not enough. It teaches the model about language but nothing about layout. Document transformers need objectives that force them to use spatial and visual signals.

LayoutLMv3 uses three pre-training objectives. Masked language modeling with layout context: the model predicts masked text tokens, but it can see the positions and image patches of all tokens, so it learns to use "this token is in the top center of the page" as a feature. Masked image modeling: about 40% of the image patches are masked, and the model predicts discrete image tokens for the hidden patches, an idea borrowed from BEiT-style image pre-training, with the surrounding text and patches as context (LayoutLMv3 code). Word-patch alignment: the model predicts whether the image patch under a text word has been masked, which forces explicit alignment between visual regions and the words they hold.

Pre-training used about 11 million scanned business documents from the IIT-CDIP collection, not receipts. The receipt knowledge comes from fine-tuning: after training on thousands of labeled receipts (a dataset such as CORD), the model learns that a large number near the bottom of a receipt, next to a word like TOTAL, is probably the total.

DocFormer makes a different architectural choice. It takes visual features from a plain ResNet50 CNN, keeps text and visual features as separate streams, and combines them with a new multi-modal self-attention layer that shares learned spatial embeddings across the two, which makes it easy to correlate text with visual tokens and back (Appalaraju et al. 2021, DocFormer: End-to-End Transformer for Document Understanding). Because both streams use the same coordinate system, a visual patch showing a table line can bind to the text tokens of the cells it encloses.

DiT (Document Image Transformer) takes an even more radical step. Note the name: it is not the Diffusion Transformer, also called DiT, behind the image and video generators in Parts 3 and 4. This DiT uses no text at all during pre-training: it learns from 42 million document images from IIT-CDIP with masked image modeling only (Li et al. 2022, DiT: Self-supervised Pre-training for Document Image Transformer). The bet is that a pure vision model trained on enough documents learns visual patterns that carry over to layout tasks. Fine-tuning then adds task heads for document classification, layout analysis, table detection and text detection.

Each of these architectures makes a different bet on how much language matters. LayoutLMv3 models text and image jointly from the start. DocFormer keeps the modalities separate and links them through shared spatial attention. DiT bets on vision-only pre-training at scale. Joint text and layout modeling tends to help where field labels matter, as in forms; a vision-only backbone is a natural fit where finding regions dominates, as in layout analysis.

How do tables get parsed into cells?

Table detection and table structure recognition are distinct problems that need different solutions.

Table detection finds the box of a table on a page, a standard object detection problem. Table Transformer (TATR) uses DETR, an object detector that predicts a fixed-size set of boxes and labels and matches them one to one with the true boxes during training, so it handles any number of tables per page. It was trained on PubTables-1M, nearly one million tables from scientific articles (Smock et al. 2021, PubTables-1M: Towards comprehensive table extraction from unstructured documents).

Table structure recognition is harder. Once a table region is cropped, the model must find the rows, columns, headers and merged cells. TATR treats each of these as an object class with its own boxes: table, table column, table row, table column header, table projected row header and table spanning cell. Each pair of a column box and a row box intersects to form a grid cell (Table Transformer documentation).

Merged cells are the hard case, and TATR gives them a class of their own, the spanning cell. Earlier work attacked irregular tables with deformable convolutions, whose filters can bend to fit irregular rows and columns: DeepTabStR detects rows and columns this way (Siddiqui et al. 2019, DeepTabStR: Deep Learning based Table Structure Recognition).

A typical staged pipeline looks like the diagram below. Commercial services such as Amazon Textract do not publish their internals, but they return results in this shape: tables made of cells, each cell with its words, boxes and confidence scores. Detect the table region with a detector model, crop the table image, recognize the rows and columns, read the text in each cell region (or assign the page's OCR words to cells), and assemble the result into CSV or JSON. The staged approach is modular and easy to debug, but each stage adds latency and can pass its errors to the next.

Diagram loading

The alternative is end-to-end extraction, where one model takes a document image and outputs a structured result directly. Cloud services sell this as ready-made processors, such as Google's Form Parser, Layout Parser and custom extractors (see the Document AI overview), but they do not publish how those work inside.

How do OCR-free models skip text extraction altogether?

An OCR step adds cost, can fail on unusual languages or document types, and passes its errors downstream. Donut asks: what if the model reads text directly from pixels, without ever producing an intermediate list of words and boxes?

Diagram loading

Donut uses a Swin Transformer encoder (a vision transformer that looks at small windows of the image first, then larger ones) and a BART decoder (a text generator; Donut uses the first four layers of a multilingual BART) to produce structured output directly from document images (Kim et al. 2021, OCR-free Document Understanding Transformer). The input is a raw image. The output is a token sequence that converts to JSON, for a receipt something like {"total": "42.00", "date": "2024-01-15", "items": [...]}. There is no OCR step and there are no word boxes.

Pre-training teaches the model to read. It must output all the text in an image in reading order, top left to bottom right, a task the authors call pseudo-OCR. The data is 11 million scanned documents from IIT-CDIP, whose text labels came from a commercial OCR engine, plus 2 million synthetic pages from SynthDoG, a generator that renders text onto document backgrounds with varied fonts, layouts and noise (SynthDoG). Fine-tuning on real documents then teaches the output format of each task.

Donut's subword vocabulary can spell any string, so its limits lie elsewhere. Its input resolution is fixed (2,560 × 1,920 in the paper), so very small print can be lost. Its decoder can write plausible text that is not on the page. It gives no word boxes or confidence scores to audit an answer against. And it writes token by token, which is slow on dense pages. OCR-free models trade transparency for simplicity: one model and no OCR errors to inherit, but nothing to check each answer against.

How do vision-language models read documents now?

The biggest change since LayoutLMv3 and Donut is that vision-language models, the LLaVA-style systems from Part 2, now read page images well. You give the model the page as an image and ask for its content as Markdown or JSON, in reading order, with tables kept as tables.

Two kinds of systems do this. General models answer questions about any image, documents included. Specialized parsers are trained only to convert pages. olmOCR fine-tunes a 7B vision-language model on 260,000 pages and converts a million PDF pages for about 176 US dollars, against more than 6,240 dollars for the same job with GPT-4o (Poznanski et al. 2025, olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models). PaddleOCR-VL joins a dynamic resolution vision encoder to a 0.3B language model, 0.9B parameters in all, and supports 109 languages (Cui et al. 2025, PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model). Earlier end-to-end page readers include Nougat for scientific papers (Blecher et al. 2023, Nougat: Neural Optical Understanding for Academic Documents) and GOT, a 580M parameter model that also reads formulas, tables and charts (Wei et al. 2024, General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model).

Pipeline toolkits wrap the classic stages instead. Docling, an MIT-licensed package, combines a layout model and a table structure model and runs on ordinary hardware (Auer et al. 2024, Docling Technical Report), and Tesseract is the long-running open source OCR engine underneath many simple pipelines. To compare all of these, OmniDocBench tests pipelines and end-to-end models across nine kinds of documents, from academic papers to handwritten notes and dense newspapers (Ouyang et al. 2024, OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations).

These models are strong where the classic pipeline struggles: multi-column pages, complex tables, formulas and handwriting. Their failure modes come from being text generators. The olmOCR authors found that prompting with only the page image led models to complete unfinished sentences, or to invent larger passages when the image was ambiguous, and that their own model's most common failure was output that degenerated into endless repetition. A generated parse also has no word boxes or confidence scores to audit, and every page costs a full model call. So a classic OCR and layout pipeline is still the better choice when reviewers need boxes and confidences, when every digit must be traceable (the amounts on an invoice), or when cost per page matters at large volume. A vision-language parser is the better choice for messy layouts and mixed content, ideally with a check on the fields that matter most.

Quick Reference

Property	Value
LayoutLMv3 backbone	One transformer over text tokens, image patches (16×16, no CNN) and 2D positions
LayoutLMv3 pre-training objectives	Masked language modeling, masked image modeling, word-patch alignment
LayoutLMv3 pre-training data	About 11 million IIT-CDIP document images; up to 512 text tokens
Table model	Table Transformer: DETR trained on PubTables-1M, six object classes including spanning cells
Table structure output	Row and column boxes whose intersections form cells, plus header and spanning cell boxes
OCR-free model	Donut: Swin encoder, BART decoder, pseudo-OCR pre-training with SynthDoG
Vision-language parsers	olmOCR (7B), PaddleOCR-VL (0.9B); toolkits such as Docling
Key datasets and benchmarks	PubLayNet (layout), FUNSD (forms), CORD (receipts), PubTables-1M (tables), OmniDocBench (page parsing)
Coordinate normalization	Boxes scaled to the page size; 0 to 1,000 for LayoutLM

Frequently Asked Questions

Q: Why do document transformers add 2D position embeddings to the 1D sequence positions?

Because the OCR order alone can mislead. A 1D position only tells the model where a token sits in the OCR output, which for a multi-column PDF or a form is often not the true reading order. The 2D box tells it "this token is in the top left" or "this token is directly below the header", which is what layout understanding needs. Models use both signals. When the true reading order matters in its own right, there are dedicated models: LayoutReader predicts reading order from text and layout, trained on ReadingBank, 500,000 document images whose order was taken from Word files' metadata (Wang et al. 2021, LayoutReader: Pre-training of Text and Layout for Reading Order Detection).

Q: Can Document AI handle handwritten text in scanned forms?

Yes, with caveats. TrOCR uses a pretrained image transformer and a text transformer, is pre-trained on large-scale synthetic data, and beat earlier models on printed, handwritten and scene text recognition (Li et al. 2021, TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models). Amazon Textract marks each detected line or word as HANDWRITING or PRINTED, and olmOCR's training pages include handwritten text. Heavy cursive still degrades recognition, and layout models may struggle with the irregular spacing of handwritten forms.

Q: How does Document AI handle multi-page PDFs with hundreds of pages?

Pages are usually processed one at a time, then the results are merged. Each page goes through OCR and layout analysis separately. LayoutLMv3 reads at most 512 text tokens at a time, so a dense page needs a sliding window over overlapping chunks. Relationships across pages, such as a table that continues on the next page, are typically handled by post-processing rules rather than by the transformer itself.

Q: What is the difference between LayoutLMv3 and Donut's approach to the image?

LayoutLMv3 treats image patches as extra tokens that sit beside the text tokens in the transformer, and it needs OCR text at inference time. Donut skips the text input entirely: its encoder produces visual representations, and its decoder writes the text output directly from them. LayoutLMv3 suits tasks where OCR is available and you want to enrich it with layout and appearance. Donut suits end-to-end tasks where the cost or errors of OCR are the bottleneck.

Q: How are these models evaluated?

The standard benchmarks are FUNSD for noisy scanned forms, where each word is labeled question, answer, header or other (FUNSD dataset); CORD for receipts (extracting prices, items and totals); and PubLayNet for layout analysis (finding and classifying regions on a page). Table models are scored with detection metrics such as average precision for finding tables, and with structure metrics such as F-measure, used by DeepTabStR on ICDAR-13, or GriTS, introduced with PubTables-1M. Whole-page parsers, including vision-language models, are compared on OmniDocBench.

Test yourself

Your photo app now scans receipts from the camera roll to build expense reports. The photos vary wildly: some are flat and well lit, others are crumpled, skewed or stained. You use an OCR engine and LayoutLMv3 fine-tuned on receipts. Field accuracy drops sharply for photos taken at an angle. What is the most likely cause, and what would you change first?

Answer: The OCR engine reports word boxes in the skewed photo's own coordinates. LayoutLMv3's 2D position embeddings take those coordinates at face value, so a line item and its price that should sit on one horizontal line arrive at different heights, and the spatial patterns the model learned from mostly straight scans no longer connect them. The first fix is to straighten the image before OCR: find the receipt's four corners, apply a homography (the math that maps a tilted photo of a flat page back to a straight-on view), and crop. Some cloud OCR services offer automatic deskewing; Google's Enterprise Document OCR lists it as an add-on. If you cannot straighten every image, fine-tune on augmented data with perspective distortions so the model learns to cope with tilt, but straightening is the bigger win. Badly crumpled receipts may defeat both, so test a vision-language parser on those.

Next: Multimodal Agents: When Vision Meets Action. It takes the models from this series off the page and onto the screen, where they click, type and act.

If these articles help you understand how the systems you rely on actually work, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• LayoutLM: Pre-training of Text and Layout for Document Image Understanding (Xu et al. 2019)
• Hugging Face LayoutLM documentation
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (Huang et al. 2022)
• LayoutLMv3 code (Microsoft)
• DocFormer: End-to-End Transformer for Document Understanding (Appalaraju et al. 2021)
• DiT: Self-supervised Pre-training for Document Image Transformer (Li et al. 2022)
• PubTables-1M: Towards comprehensive table extraction from unstructured documents (Smock et al. 2021, Table Transformer)
• Table Transformer documentation (Hugging Face)
• DeepTabStR: Deep Learning based Table Structure Recognition (Siddiqui et al. 2019)
• OCR-free Document Understanding Transformer (Kim et al. 2021, Donut)
• SynthDoG (Donut's synthetic data generator)
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models (Li et al. 2021)
• LayoutReader: Pre-training of Text and Layout for Reading Order Detection (Wang et al. 2021)
• olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models (Poznanski et al. 2025)
• PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model (Cui et al. 2025)
• Nougat: Neural Optical Understanding for Academic Documents (Blecher et al. 2023)
• General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model (Wei et al. 2024)
• Docling Technical Report (Auer et al. 2024)
• Tesseract (open source OCR engine)
• OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (Ouyang et al. 2024)
• PubLayNet: largest dataset ever for document layout analysis (Zhong et al. 2019)
• FUNSD: Form Understanding in Noisy Scanned Documents
• Textract response objects (Amazon Web Services)
• Document AI overview (Google Cloud)