# Designing the Document Review Queue

> How to set confidence thresholds, sample auto-accepted documents, size reviewer time and feed corrections back, from Extend, Rossum and Azure docs.

- Source: https://www.zarifautomates.com/blog/designing-the-document-review-queue
- Published: 2026-09-26
- Updated: 2026-09-26
- Pillar: Agents & AI Engineering
- Tags: document ai, ai document review, human in the loop, confidence thresholds, intelligent document processing
- Author: Zarif

---

The review queue is where a document AI system either earns trust or quietly loses it. Send too much to review and you have rebuilt manual data entry with extra steps. Send too little and wrong totals reach the ledger with a confident-looking score attached.

This guide covers the five decisions that shape the queue: what triggers review, how much of the auto-accepted work you audit anyway, what the reviewer's screen must show, how many reviewer hours the queue needs, and how corrections flow back into the system. It draws on the review features Extend, Rossum, Microsoft and AWS describe in their docs, read on September 26, 2026, and on the per-field scoring from [the previous post in this series](/blog/measuring-document-extraction-accuracy). Nothing here was measured on a live queue. The numbers are worked examples on stated inputs.

## Why the vendors all ship a queue

Every serious extraction product treats human review as part of the pipeline, not an afterthought, and the reason is in their own docs.

Microsoft's [accuracy and confidence page for Document Intelligence](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/accuracy-confidence?view=doc-intel-4.0.0) defines field confidence as an estimated probability that the prediction is correct, and says confidence can decide whether to accept a prediction automatically or flag it for human review. Extend's [confidence score docs](https://docs.extend.ai/extraction/confidence-scores) say a high score does not guarantee accuracy. Both statements point the same way: a score is an input to a routing decision, not a verdict.

The queue also does not always come with the extractor. Google's [Document AI deprecations page](https://docs.cloud.google.com/document-ai/docs/deprecation) lists its Human-in-the-Loop feature as deprecated on January 16, 2024. AWS's [Augmented AI docs](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-json-humantaskactivationconditions-textract-example.html) state that A2I is no longer open to new customers. If your extractor comes from either, the queue is yours to build, as in [the n8n pipeline](/blog/invoice-pipeline-in-n8n) earlier in this series.

## Decision 1: what sends a document to review

Use three kinds of trigger, and keep them separate so a reviewer can see which one fired.

**Business rules.** Deterministic checks that do not care what the model thinks: required fields present, dates valid, currency on the allowed list, line items summing to the subtotal, vendor known, invoice number not already paid. These catch the errors that matter most, and they never drift. Rossum's [automation guide](https://knowledge-base.rossum.ai/docs/guide-to-automation-process-in-rossum) lists built-in data integrity checks and history-based checks alongside confidence as sources that can validate a field.

**Confidence.** A per-field score below a threshold. The vendors expose this in different units, which is the first thing to normalize:

| Product | Score | Routing example or default from the docs |
| --- | --- | --- |
| Extend confidence | 0 to 1 per field | Route to review when the minimum field confidence is below 0.9 |
| Extend Review Agent | 1 to 5 per field | A score of 3 or below "usually warrants a review" |
| Rossum | 0 to 1 per field | Queue API default threshold 0.8, while the automation guide says 97.5% |
| Azure Document Intelligence | 0 to 1 per field | Transparency note example: accept at 0.80 or above, review below |
| Datalab fast-mode scoring | 1 to 5 per field | Docs suggest auto-accepting high scores and routing low ones |

Sources, all read 2026-09-26: [Extend confidence scores](https://docs.extend.ai/extraction/confidence-scores), [Extend Review Agent](https://docs.extend.ai/extraction/review-agent), [Rossum queue API](https://rossum.app/api/docs/openapi/api/queue/) and [automation guide](https://knowledge-base.rossum.ai/docs/guide-to-automation-process-in-rossum), [Azure transparency note](https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/document-intelligence/transparency-note), [Datalab confidence scoring](https://documentation.datalab.to/docs/recipes/structured-extraction/confidence-scoring).

The Rossum row is worth a second look. Its API reference and its knowledge base give different defaults for the same idea. That is a small sign of a larger point: no vendor default is your threshold. Microsoft's transparency note says it plainly, telling customers to run their own evaluation and a pilot to estimate thresholds for straight-through processing versus review.

**Missing signals.** A failed extraction, a score that never arrived, a document type the classifier was unsure about. Datalab's docs warn that its confidence scoring can fail independently of extraction and that a missing average means unavailable, not zero. Route these to review with their own reason code, so they are not confused with low confidence.

### Setting the first threshold

Run the extractor on a labeled sample, store each field's score next to the [field-level scoring results](/blog/measuring-document-extraction-accuracy), and group fields by score. For each candidate threshold, compute two numbers: the share of fields above it, which is your automation rate, and the precision of those fields, which is how often an auto-accepted value is right.

Then set the threshold per field, by consequence. A vendor name typo in a memo field can tolerate lower precision than a total that becomes a payment. Rossum and Extend both support per-field control, through a score threshold column in Rossum's automation settings and per-field variables in Extend's workflow conditions.

## Decision 2: audit what the machine accepted

Everything above the threshold goes straight through, so your queue never sees those errors. You will not learn your real error rate unless you look.

AWS A2I built this in: its `Sampling` condition sends a set percentage of documents to people regardless of confidence, and can combine with confidence rules, for example always reviewing low scores while sampling a small share of high ones. The same pattern works anywhere. Pick a random sample of auto-accepted documents every week and review them exactly like exceptions.

Sampling also protects the learning loop. Rossum's automation guide says it trains on human corrections, not on its own previous predictions, and warns that automating almost every document leaves very few for the AI to learn from. An audit sample keeps a steady stream of human-checked documents from the part of the distribution the model handles well.

How big a sample? To estimate an error rate within a margin of error, the standard formula from [Penn State's STAT 200 notes](https://online.stat.psu.edu/stat200/lesson/8/8.1/8.1.1/8.1.1.3) is the square of z divided by the margin, times p times one minus p. At 95% confidence, z is 1.96.

| Expected error rate | Margin you want | Documents to audit |
| --- | --- | --- |
| 2% | plus or minus 2 points | 189 |
| 2% | plus or minus 1 point | 753 |
| 5% | plus or minus 2 points | 457 |

Those counts are totals over the period you report on, not per day. For small error rates the normal approximation behind this formula gets rough, so treat the numbers as a floor. A monthly audit of a few hundred documents is usually enough to tell whether the auto-accepted error rate is closer to 1% or 5%, which is the decision you need.

## Decision 3: what the reviewer's screen needs

A reviewer's time goes into finding the evidence, not typing the fix. Design the screen around that.

1. **The document beside the fields.** Extend's [review screen](https://docs.extend.ai/workflows/reviewing-workflow-run) shows the document on the left and the extracted results on the right. Anything less sends the reviewer hunting through a PDF.
2. **A link from each value to where it came from.** Datalab returns citation block IDs per field, and Azure scores individual words. Use them to highlight the source region. A reviewer who can see the number on the page confirms it in seconds.
3. **The reason it is here.** Show the rule or score that sent it: `subtotal_plus_tax_not_total` tells the reviewer where to look first.
4. **Original and corrected values, both kept.** Extend's [Human Review step](https://docs.extend.ai/workflows/human-review) records `initialOutput` and `reviewedOutput` with `reviewed` and `edited` flags, so downstream systems can tell machine output from corrected output. Keep both in your own store too.
5. **A short explanation on every correction.** Extend asks reviewers to explain their corrections on approval and says the explanations are used to improve extraction accuracy. One line, "total on page 2, not page 1", is enough to become a schema fix later.
6. **Approve, correct or reject.** Rejection needs a reason and must stop the downstream write. Extend documents that rejected runs do not trigger later steps.
7. **One owner per item.** Two reviewers correcting the same invoice produces two versions of the truth. Claim items before editing.

The same principles show up in agent approvals: store and check the exact decision a person made, not a summary of it. [Human approval for agents](/blog/how-to-build-ai-agent-human-in-loop-approval) works through that pattern in code, and it transfers directly to document review.

## Decision 4: how many reviewer hours

The workload is arithmetic. Use the same inputs as the [document processing cost calculator](/lab/document-processing-cost-calculator), whose defaults are 2,000 documents a month, an 8% exception rate and 6 minutes to review one exception. Those defaults are illustrative, not measurements, so replace them with your pilot numbers.

- Exceptions: 2,000 times 8% is 160 documents a month.
- Exception review: 160 times 6 minutes is 960 minutes, or 16 hours a month.
- Audit sample: 189 documents a month at the same 6 minutes adds about 19 hours.

The audit can cost more reviewer time than the exceptions. That is expected at low volume and it shrinks as a share when volume grows, because the sample size depends on the precision you want, not on how many documents you process.

**Service level.** Peaks set the staffing, not averages. Across 20 business days, 160 exceptions is 8 a day, or 48 minutes of review. If invoices bunch at month end and a single day brings three times the usual volume, that day needs 144 minutes. If your service level says every exception is cleared within one business day, staff for that day.

[Little's law](https://en.wikipedia.org/wiki/Little%27s_law) links the queue's three numbers: the average number of items waiting equals the arrival rate times the average wait. At 8 exceptions a day and a one-day average wait, expect about 8 items in the queue. If the queue holds 40, either arrivals rose or items now wait five days, and both are worth an alert.

Track three numbers from day one: items waiting, age of the oldest item, and the share of items breached against the service level. Age of the oldest item catches the invoice nobody wants to touch.

## Decision 5: close the loop back to the schema

Every correction is a labeled example. Wasting them is the most common way a review queue stays the same size forever.

- **Turn reviewed work into an evaluation set.** Extend's [evaluation docs](https://docs.extend.ai/evaluation/overview) describe building a set by converting a reviewed batch run, then scoring a processor version against it per field. Whatever product you use, keep a frozen set of reviewed documents and rerun it before every schema or prompt change.
- **Group corrections by reason.** If one field accounts for most corrections, the fix is usually in the schema: a clearer field description, a stricter type, or an instruction to return nothing when the value is absent.
- **Retrain where the product supports it.** Microsoft's accuracy page suggests adding labeled samples, at least five of each document type, when a custom model shows low confidence on a variation it has not seen.
- **Move thresholds on evidence.** When the audit sample shows auto-accepted precision holding for a field, lower its threshold a step and watch the next sample. When it drops, raise it.

The goal is a queue that shrinks because the extractor got better, not because someone lowered a threshold to hit a target.

## A starting configuration

For a first production month, before you have your own numbers:

1. Route every business-rule failure and every missing signal to review, whatever the confidence.
2. Start confidence thresholds high on money and identity fields and lower on descriptive ones.
3. Audit a random sample of auto-accepted documents large enough for the margin you need.
4. Record original value, corrected value, reason and explanation for every change.
5. Review threshold choices monthly against the audit and the evaluation set.

Previous in the series: [measuring extraction accuracy](/blog/measuring-document-extraction-accuracy). Next: [document AI cost at scale](/blog/document-ai-cost-at-scale), where review time turns out to be most of the bill. Terms are defined in the [Document AI glossary](/blog/document-ai-glossary). For the vendor shortlist behind these docs, see [where Datalab fits among enterprise document tools](/blog/best-enterprise-ai-document-processing-tools).