Document AI Now: From OCR to Vision-Language Models

Picture a scanned invoice with two columns, a table that runs onto page two, and a handwritten note in the margin. For years, getting that into a spreadsheet meant reading the characters first and writing rules to figure out what they meant.

That's changed. As of September 2026, nearly every serious document tool puts an AI model that can look at the whole page somewhere in the process. That changes what you get back, what can go wrong, and what's worth paying for.

This is part 1 of the Document AI series. It explains the change using the docs of the tools builders actually choose between. It doesn't run any tests of its own. Every product claim links to its source, read on September 26, 2026.

The old way: read the characters, then guess the meaning

The classic approach worked from the bottom up. If the PDF was created on a computer, you pulled out the text already stored inside it. If it was a scan, you ran OCR (optical character recognition), which turns a picture of text into characters, each with a box marking where it sat on the page.

Everything after that was your problem. Which line belongs to which column? Where does the table start? Which number is the total?

Teams handled that with templates and rules for each layout. A new supplier meant a new template. A table that spilled onto a second page, a checkbox, or a handwritten correction broke the rules without anyone noticing. The OCR step knew what the characters were. It had no idea what the page meant.

That approach had one thing going for it. It couldn't make things up. OCR might misread a 5 as an S, but it won't write a sentence that isn't on the page.

What the new models change

A vision-language model, or VLM, is an AI model that takes in an image and writes text about it. Point one at a page and it reads it more like a person does. It writes out the structure directly: headings, paragraphs in the right order, tables as HTML, equations as LaTeX, and form fields as label and value pairs.

Datalab's Chandra is a clear example. Its README describes an OCR model that turns images and PDFs into HTML, Markdown or JSON while keeping the layout, and it handles handwriting, forms with checkboxes, tables and math. Chandra 1 shipped in October 2025 and Chandra 2 in March 2026, per the same README. Allen AI's olmOCR took the same approach in the open, using a 7-billion-parameter VLM to turn document images into markdown. It was first released in February 2025.

What you gain is a model that understands layout. Reading across columns in the right order, table cells that span several columns, a checkbox next to its label: those stop being puzzles you solve afterward. The model sorts them out while it reads.

What you give up is the guarantee. The model writes its output, and a model that writes can write something that looks right and isn't. A clean markdown table looks just as confident either way.

That one fact drives most of the rest of this series: links back to the spot on the page, confidence scores, checking rules, and a person reviewing the doubtful cases.

Most tools now mix the two

Running a big VLM over every page of every document is slow and expensive, so almost nobody does it. The tools worth a look decide page by page, or even block by block, when the expensive model is actually needed.

Marker explains how it does this in detail. It pulls out the text stored in the PDF with a tool called pdftext, works out the layout, and checks page by page whether that stored text is usable.

• In fast mode, it only calls its Surya VLM for equations, garbled blocks and scanned pages. A clean digital document with no equations never starts the VLM at all.
• In balanced mode, it uses the VLM for layout, and re-reads a whole page with OCR whenever the stored text is bad.
• Tables are rebuilt from the stored text, and only the ones it isn't sure about get sent to the model.

Extend makes the same trade with a setting. Its parse configuration offers parse_light for high volume, parse_performance for hard tables, handwriting and checkboxes, and parse_auto, which picks for each page based on how hard it looks. There's also an optional step where a VLM double-checks text and tables the system isn't sure about. Extend's credit docs say you only pay extra for that on pages where it actually ran.

The big cloud providers have moved too. Microsoft's Document Intelligence overview (updated September 8, 2026) pitches the service for predictable extraction from structured documents, and points to Azure Content Understanding for messier content handled by LLMs. Its layout model, in the v4.0 API from 2024-11-30, can return Markdown with tables as HTML.

Google's Document AI overview (updated September 24, 2026) says the product is built on Vertex AI with generative AI. It lists a custom extractor that uses it, and a Layout Parser that splits documents into chunks that keep their context.

Mistral came in as a plain API. Its OCR 4 announcement from June 23, 2026 lists location boxes, block types, and confidence scores for each page and each word, along with the markdown text.

What the benchmark scores tell you

The score you'll see quoted most in 2026 is from olmOCR-Bench. The olmOCR repository describes it as roughly 1,400 documents with more than 7,000 pass-or-fail tests. They cover old scans, math, tables, headers and footers, multi-column layouts, and long stretches of tiny text.

Vendors quote it all the time, on their own runs. The Chandra README lists 85.8 overall for Chandra 2 and 86.7 for the hosted Datalab API. The Marker README lists 76.0 for Marker's balanced mode. Mistral's announcement gives 85.20 for OCR 4.

Every one of those is self-reported, from the vendor's own setup. Nothing guarantees they used the same settings or ran on the same date.

Two things matter more than who's on top.

First, the benchmark checks whether text and structure come out of a PDF correctly. It doesn't check whether the right total ended up in the right field of an invoice. That's the mistake that costs money.

Second, scores swing a lot by category. Marker's own table shows 99.7 on the baseline category and 43.2 on old scans in balanced mode. One overall number hides where a tool falls apart.

Use these scores to decide what to try, not what to buy. The only score that settles it is one you work out on your own documents. Part 7 of this series covers how.

What this means if you're building the pipeline

Four things follow from the change, and they shape the rest of the series.

1. Turning a page into text is now the cheap part. Datalab's pricing page lists fast conversion at $4 per 1,000 pages, and Mistral lists OCR at the same rate. Whether a pipeline is useful gets decided after that step: pulling out the fields you need, checking them, and reviewing the doubtful ones.
2. You need to know where each answer came from. Because the model can write text that isn't on the page, you want every value tied to a spot on the page. Datalab returns a citation for each field, pointing to a block on the page, in its extraction API. Extend returns the page and the box the value came from when citationsEnabled is on in its extraction config.
3. Every step costs money, and the costs add up. Parsing, extracting, splitting and classifying are usually billed separately, so the pipeline costs the sum of its steps. Datalab's billing page says its processors are billed separately and added together, and Extend's credit docs show a parse running automatically under each extraction. The document processing cost calculator puts those per-page prices next to the cost of people reviewing the results.
4. The model can change under you. These tools get better by shipping new model versions, and a new version can change your results. Extend lets you lock engineVersion on parse and publish versions that never change. Datalab's pipeline docs describe published pipeline versions that are locked the same way. Lock the version you run in production, and rerun your test documents before you move to a new one.

Where to go next

Next in the series: parse vs extract vs classify vs split. It walks one made-up invoice through each step so you can see which one your problem actually needs.

If you're picking a vendor, the series started with a shortlist of enterprise document processing tools and five questions to ask on every sales call. Read it for the buying questions, then come back for how things work.