Designing the Document Review Queue

An invoice comes in. The AI reads it and pulls out the vendor, the date and the total. Now someone has to decide: does a person check that total before it gets paid, or does it go straight through?

That decision, made thousands of times a month, is the review queue. Send too much to it and you've rebuilt manual data entry with extra steps. Send too little and a wrong total lands in the books with a confident-looking score next to it.

This guide walks through five choices: what sends a document to a person, how many of the auto-approved ones you spot-check anyway, what the reviewer needs on screen, how many hours of review you need, and how corrections make the system better. It's built from the review features Extend, Rossum, Microsoft and AWS describe in their docs, read on September 26, 2026, and from the per-field scoring in the previous post in this series. Nothing here was measured on a live queue. The numbers are worked examples on stated inputs.

Why every vendor ships a queue

The vendors' own docs tell you not to trust the score on its own.

Microsoft's accuracy and confidence page for Document Intelligence describes a field's confidence as an estimate of the chance the answer is right. It says you can use that number to decide whether to accept the answer automatically or send it to a person. Extend's confidence score docs say a high score does not guarantee accuracy.

Put those together and the score is a hint for routing, not a verdict.

The queue doesn't always come with the product, either. Google's Document AI deprecations page lists its Human-in-the-Loop feature as deprecated on January 16, 2024. AWS's Augmented AI docs say A2I is no longer open to new customers. If your extractor comes from either, you build the queue yourself, like the n8n pipeline earlier in this series.

Choice 1: what sends a document to review

There are three kinds of reasons to send a document to a person. Keep them separate, so the reviewer can see which one fired.

Business rules. These are plain checks that don't care what the AI thinks. Is every required field there? Is the date a real date? Is the currency one you accept? Do the line items add up to the subtotal? Do you know this vendor? Has this invoice number already been paid? These catch the mistakes that cost the most, and they never drift. Rossum's automation guide lists built-in data checks and history-based checks next to confidence as ways to validate a field.

Low confidence. The AI gives each field a score, and anything under your cutoff goes to a person. The catch is that every vendor scores differently, so the first job is to put them on the same scale:

Sources, all read 2026-09-26: Extend confidence scores, Extend Review Agent, Rossum queue API and automation guide, Azure transparency note, Datalab confidence scoring.

Look at the Rossum row again. Its API reference and its help docs give different defaults for the same setting. That's a small sign of a bigger point: no vendor's default is your cutoff. Microsoft's transparency note says so directly. It tells customers to run their own evaluation and a pilot to figure out which results can skip review.

Missing information. The extraction failed, the score never arrived, or the system wasn't sure what kind of document it was looking at. Datalab's docs warn that its confidence scoring can fail even when extraction works, and that a missing average means "unavailable," not zero. Send these to review with their own label, so nobody mistakes them for low confidence.

Setting your first cutoff

Run the extractor on a batch of documents where you already know the right answers. Save each field's score next to the field-level scoring results.

Then try different cutoffs. For each one, work out two numbers. First, what share of fields scored above it. That's how much work skips review. Second, how often those skipped fields were actually right. That's your precision.

Set the cutoff field by field, based on what a mistake would cost. A typo in a memo field can slip through. A wrong total that becomes a payment can't. Rossum and Extend both let you set this per field: Rossum through a score threshold column in its automation settings, Extend through per-field variables in its workflow conditions.

Choice 2: spot-check what the machine approved

Everything above your cutoff goes straight through. Your reviewers never see those mistakes.

So you won't know your real error rate unless you go looking.

AWS A2I has this built in. Its Sampling setting sends a fixed percentage of documents to people no matter what the score says. You can combine it with confidence rules, for example always reviewing low scores while also checking a small share of high ones. You can copy that anywhere. Every week, pull a random handful of auto-approved documents and review them exactly like the flagged ones.

Spot-checks also keep the system learning. Rossum's automation guide says it learns from human corrections, not from its own past answers. It warns that if you automate almost everything, the AI has very little left to learn from. A steady spot-check keeps human-checked examples coming in, including from the documents the AI usually handles well.

How many should you check? To estimate an error rate within a given margin, the standard formula from Penn State's STAT 200 notes is: z squared, divided by the margin squared, times p times one minus p. Here p is the error rate you expect. At 95% confidence, z is 1.96.

Those are totals for the whole period you report on, not per day. The formula gets shaky when error rates are very small, so treat these as a minimum. A few hundred documents a month is usually enough to tell whether your auto-approved error rate is closer to 1% or 5%, and that's the question you actually need answered.

Choice 3: what the reviewer needs on screen

Most of a reviewer's time goes into finding the right spot on the page, not typing the fix. Build the screen around that.

1. The document next to the fields. Extend's review screen puts the document on the left and the extracted values on the right. Without that, the reviewer is scrolling through a PDF hunting for a number.
2. A link from each value to where it came from. Datalab returns citation block IDs for each field, and Azure scores individual words. Use them to highlight the spot on the page. If the reviewer can see the number on the page, they can confirm it in seconds.
3. Why it's here. Show the rule or score that flagged it. A label like subtotal_plus_tax_not_total tells the reviewer where to look first.
4. Both the original and the corrected value. Extend's Human Review step records initialOutput and reviewedOutput with reviewed and edited flags, so later systems can tell machine output from corrected output. Keep both in your own records too.
5. A one-line note on every correction. Extend asks reviewers to explain their corrections when they approve, and says it uses those notes to improve accuracy. Something like "total on page 2, not page 1" is enough to turn into a fix later.
6. Approve, correct or reject. A rejection needs a reason and has to stop the document from going any further. Extend's docs say rejected runs don't trigger later steps.
7. One person per item. Two reviewers correcting the same invoice gives you two versions of the truth. Have reviewers claim an item before they edit it.

The same idea shows up when people approve what an AI agent wants to do: save the exact decision the person made, not a summary of it. Human approval for agents builds that in code, and it carries straight over to document review.

Choice 4: how many reviewer hours

This part is just arithmetic. Start with the same inputs as the document processing cost calculator: 2,000 documents a month, 8% of them flagged, and 6 minutes to review each flagged one. Those defaults are made up for illustration, so swap in your own pilot numbers.

• Flagged documents: 2,000 times 8% is 160 a month.
• Reviewing them: 160 times 6 minutes is 960 minutes, or 16 hours a month.
• Spot-checks: 189 documents a month at the same 6 minutes adds about 19 hours.

Yes, the spot-checks can take more time than the flagged documents. That's normal at low volume. It gets smaller as a share of the work as volume grows, because the number you need to check depends on how precise you want the estimate to be, not on how many documents you process.

Plan for the busy days. Your busiest day sets how many reviewers you need, not your average day. Over 20 business days, 160 flagged documents is 8 a day, or 48 minutes of review. If invoices pile up at month end and one day brings three times the usual load, that day needs 144 minutes. If your promise is that every flagged document is cleared within one business day, staff for that day.

Little's law ties the queue's numbers together: the average number of items waiting equals how many arrive per day times how long each one waits on average. At 8 a day and a one-day wait, expect about 8 in the queue. If it's sitting at 40, either more are coming in or items are now waiting five days. Both are worth an alert.

Track three numbers from day one: how many items are waiting, how old the oldest one is, and what share missed your deadline. The oldest-item number is the one that catches the invoice nobody wants to touch.

Choice 5: use corrections to fix the system

Every correction a reviewer makes is a free example of the right answer. Throwing them away is the most common reason a review queue never gets smaller.

• Turn reviewed work into a test set. Extend's evaluation docs describe turning a reviewed batch into a test set, then scoring a new version against it field by field. Whatever product you use, keep a fixed set of reviewed documents and rerun it before every change to your schema (the list of fields you ask for) or your prompt.
• Group corrections by reason. If one field gets most of the corrections, the fix is usually in the schema: a clearer description, a stricter type, or an instruction to return nothing when the value isn't there.
• Retrain where the product allows it. Microsoft's accuracy page suggests adding labeled examples, at least five of each document type, when a custom model shows low confidence on a layout it hasn't seen.
• Move cutoffs only on evidence. When spot-checks show a field's auto-approved answers staying accurate, lower its cutoff one step and watch the next round. When accuracy drops, raise it.

You want a queue that shrinks because the extractor got better, not because someone lowered a cutoff to hit a target.

Where to start

For your first month in production, before you have your own numbers:

1. Send every business-rule failure and every missing signal to review, whatever the score says.
2. Start with high cutoffs on money and identity fields, lower ones on descriptive fields.
3. Spot-check a random sample of auto-approved documents, big enough for the margin you need.
4. Record the original value, corrected value, reason and note for every change.
5. Look at your cutoffs once a month against the spot-checks and the test set.

Next in the series: document AI cost at scale, where review time turns out to be most of the bill. If you missed it, the previous post covers measuring extraction accuracy. Unfamiliar terms are in the Document AI glossary. For the vendor shortlist behind these docs, see how to choose an enterprise document tool.