Document AI Now: From OCR to Vision-Language Models

Document processing used to mean OCR plus rules. In September 2026, nearly every serious parser puts a vision-language model somewhere in the loop, and that changes what you get back, what can go wrong, and what you should pay for.

This is part 1 of the Document AI series. It explains the shift from the documentation of the tools builders actually choose between. It does not run a benchmark. Every product claim below links to the source it came from, read on September 26, 2026.

The old stack: characters first, meaning later

A classic pipeline worked bottom-up. It pulled the embedded text layer out of a born-digital PDF, or ran an OCR engine over a scan to get lines of characters with bounding boxes. Everything after that was your problem: which line belongs to which column, where a table starts, which number is the total.

Teams solved that with templates, zones, and rules per document layout. A new supplier meant a new template. A table that wrapped onto a second page, a checkbox, or a handwritten correction broke the rules quietly, because the OCR step had no idea what the page meant. It only knew what the characters were.

That design had one virtue worth keeping. It could not invent text. An OCR engine may misread a character, but it does not write a sentence that isn't on the page.

What a vision-language model changes

A vision-language model reads the page image the way a person does and writes out structure directly: headings, paragraphs in reading order, tables as HTML, equations as LaTeX, form fields as label and value pairs.

Datalab's Chandra is a clean example. Its README describes an OCR model that turns images and PDFs into HTML, Markdown, or JSON while preserving layout, with handwriting, forms with checkboxes, tables, and math in scope. Chandra 1 shipped in October 2025 and Chandra 2 in March 2026, per the same README. Allen AI's olmOCR took the same approach in the open, using a 7-billion-parameter VLM to turn document images into markdown, first released in February 2025.

The gain is layout understanding. Reading order across columns, merged table cells, and a checkbox next to its label stop being post-processing puzzles. The model resolves them while it reads.

The cost is that the output is generated. A model that writes text can write text that is plausible and wrong, and a clean markdown table looks equally confident either way. That single fact drives most of the design choices in the rest of this series: citations back to the page, confidence scores, validation rules, and a review queue.

The practical default is hybrid

Almost nobody runs a large VLM over every page of every document. The products worth a look decide per page, or per block, when the expensive model is needed.

Marker documents this in detail. It extracts the embedded text with pdftext, detects layout, and decides page by page whether that text is usable. In fast mode it calls the Surya VLM only for equations, garbled blocks, and scanned pages, and a clean digital document without equations never starts the VLM server. In balanced mode it uses the VLM for layout and re-OCRs a whole page whenever its embedded text is bad. Tables are rebuilt from the text layer, with low-confidence reconstructions falling back to the model.

Extend makes the same trade explicit in its engines. Its parse configuration offers parse_light for high-volume ingestion, parse_performance for hard tables, handwriting, and checkboxes, and parse_auto, which routes each page by difficulty. Optional agentic corrections use a VLM to review low-confidence text and tables, and Extend's credit docs say that surcharge applies only on pages where the correction actually triggered.

The cloud suites have moved too. Microsoft's Document Intelligence overview (updated September 8, 2026) positions the service for deterministic extraction from structured documents, while Azure Content Understanding offers LLM-powered analyzers for unstructured content. Its layout model in the v4.0 API from 2024-11-30 can return Markdown, with tables as HTML. Google's Document AI overview (updated September 24, 2026) says the product is built on Vertex AI with generative AI, lists a custom extractor that uses it, and describes a Layout Parser that returns context-aware chunks.

Mistral joined as a pure API. Its OCR 4 announcement from June 23, 2026 lists bounding boxes, block types, and per-page and per-word confidence scores alongside markdown text.

What the benchmark numbers tell you

The number you will see most often in 2026 is an olmOCR-Bench score. The olmOCR repository describes the benchmark as roughly 1,400 documents with more than 7,000 pass-or-fail tests covering old scans, math, tables, headers and footers, multi-column layouts, and long tiny text.

Vendors quote it constantly, on their own runs. The Chandra README lists 85.8 overall for Chandra 2 and 86.7 for the hosted Datalab API. The Marker README lists 76.0 for Marker's balanced mode. Mistral's announcement gives 85.20 for OCR 4. Each is self-reported, each is from the vendor's own harness, and the runs are not guaranteed to use the same settings or date.

Two things matter more than the ranking. First, the benchmark tests whether text and structure come out of a PDF correctly. It does not test whether your extractor put the right total into the right field of an invoice, which is the error that costs money. Second, the categories vary enormously. Marker's own per-category table shows 99.7 on the baseline category and 43.2 on old scans in balanced mode, so an overall score hides where a parser fails.

Use these numbers to decide what to try, not what to buy. The only score that settles a decision is one you compute on your own documents, which is the subject of part 7 of this series.

What changed for the person building the pipeline

Four consequences follow from the shift, and they shape the rest of the series.

1. Parsing is now the cheap, commodity step. Datalab's pricing page lists fast conversion at $4 per 1,000 pages, and Mistral lists OCR at the same rate. The work that decides whether a pipeline is useful sits after it: extraction into a schema, validation, and review.
2. Grounding matters more than it used to. Because a generative model can write text that isn't on the page, you want every extracted value tied to a location. Datalab returns per-field citations to block IDs in its extraction API, and Extend returns page and bounding-box citations when citationsEnabled is on in its extraction config.
3. Cost is per page, per operation, and additive. Parse, extract, split, and classify are usually billed separately, so the price of a pipeline is the sum of its steps. Datalab's billing page says processors are additive, and Extend's credit docs show parse running automatically under extract. The document processing cost calculator puts those per-page rates next to the cost of human review.
4. Model versions move under you. A VLM-based parser improves by shipping new model versions, which can change your outputs. Extend lets you pin engineVersion on parse and publish immutable processor versions, and Datalab's pipeline docs describe immutable published pipeline versions. Pin what goes to production and re-run your evaluation set before you move the pin.

Where to go next

If you are choosing a vendor, the series began with a shortlist of enterprise document processing tools and five questions to take to every sales call. Read it for the buying questions, then come back for the mechanics.

Next in the series: parse vs extract vs classify vs split, which walks one synthetic invoice through each operation so you can see which one your problem actually needs.

Previous: Enterprise document processing: where Datalab fits. Next: Parse, extract, classify, split.