# Measure Document Extraction Accuracy Field by Field > Score extracted invoice fields against a small ground-truth set with a tested Python script: precision, recall, exact vs normalized match. - Source: https://www.zarifautomates.com/blog/measuring-document-extraction-accuracy - Published: 2026-09-26 - Updated: 2026-09-26 - Pillar: Agents & AI Engineering - Tags: document ai, document-extraction, extraction accuracy, evaluation, precision and recall - Author: Zarif --- "The extractor is 90% accurate" is a sentence that hides almost everything you need to know. Accurate on which fields, against what answer key, counting blanks how, and after how much cleanup of dates and dollar signs? This lesson builds the answer key and the scorer. You get a five-invoice ground-truth file, a set of extracted outputs with deliberate mistakes, and a standard-library Python script that prints precision and recall for every field, twice: once with exact matching and once after normalization. The recorded run below shows the same outputs scoring 0.76 or 0.90 depending on that one choice, with only one of five invoices fully right either way. The data is synthetic. It exists to exercise the scoring rules, not to rate any vendor. This is part 7 of the Document AI series. It picks up where [the n8n invoice pipeline](/blog/invoice-pipeline-in-n8n) leaves off: that workflow decides what goes to review, and this script tells you whether those decisions deserve trust. ## What a field-level score counts Treat each field on each document as one comparison between the value on the page and the value the extractor returned. Four outcomes are possible: | Truth on the page | Extractor returned | Outcome | Counts as | | --- | --- | --- | --- | | A value | The same value | Correct | True positive | | A value | Nothing | Missed | False negative | | A value | A different value | Wrong | False positive and false negative | | Nothing | A value | Invented | False positive | | Nothing | Nothing | Correctly blank | Not counted | Precision is true positives divided by true positives plus false positives: of the values the extractor gave you, how many were right. Recall is true positives divided by true positives plus false negatives: of the values on the page, how many came back correctly. Those are the standard definitions in [scikit-learn's evaluation guide](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures) and in [Google Document AI's evaluation docs](https://docs.cloud.google.com/document-ai/docs/evaluate), both read on 2026-09-26. The row that trips people up is the wrong value. It counts against both numbers, because the extractor asserted something false and also failed to deliver the truth. If you count a wrong value only as a miss, precision looks better than it is, and precision is the number that tells you how often a confident-looking value will mislead a downstream system. The invented row matters just as much for invoices. A due date the model filled in from payment terms that weren't on the page is a false positive. It is also exactly the kind of value an accounts-payable system will act on without question. ## Build a small ground-truth set Ground truth is the answer key: what a careful person reads off each document. Five documents are enough to learn the mechanics. For a real decision you want enough documents that each important field, layout and failure type shows up several times, drawn from the documents you actually receive. Save this as `truth.json`. A `null` means the field is not on that document. ```json [ {"doc_id": "inv-001", "invoice_number": "INV-1001", "vendor_name": "Acme Supply Co.", "invoice_date": "2026-03-14", "due_date": "2026-04-13", "currency": "USD", "total_amount": 1080.00}, {"doc_id": "inv-002", "invoice_number": "INV-1002", "vendor_name": "Northwind Traders", "invoice_date": "2026-03-15", "due_date": null, "currency": "USD", "total_amount": 249.50}, {"doc_id": "inv-003", "invoice_number": "7731-A", "vendor_name": "Globex GmbH", "invoice_date": "2026-03-02", "due_date": "2026-04-01", "currency": "EUR", "total_amount": 3120.00}, {"doc_id": "inv-004", "invoice_number": "INV-1004", "vendor_name": "Acme Supply Co.", "invoice_date": "2026-03-20", "due_date": "2026-04-19", "currency": "USD", "total_amount": 64.99}, {"doc_id": "inv-005", "invoice_number": "SO-88213", "vendor_name": "Initech LLC", "invoice_date": "2026-03-21", "due_date": "2026-03-31", "currency": "USD", "total_amount": 15000.00} ] ``` Three rules keep an answer key honest: 1. Write down what the document says, in one canonical form per type: ISO dates, numbers without symbols, currency as a three-letter code. 2. Mark genuinely absent fields as `null`. Do not leave them out, and do not fill them from outside knowledge. 3. Have a second person check a sample of the key, and settle disagreements in a written rule. If two people disagree on what the invoice date is, the extractor cannot be scored on it yet. Now the extractor output. Save this as `predicted.json`. It mimics the kind of JSON a document AI API returns after you parse its extraction field, with mistakes planted on purpose. ```json [ {"doc_id": "inv-001", "invoice_number": "INV-1001", "vendor_name": "ACME SUPPLY CO", "invoice_date": "03/14/2026", "due_date": "2026-04-13", "currency": "USD", "total_amount": "$1,080.00"}, {"doc_id": "inv-002", "invoice_number": "INV-1002", "vendor_name": "Northwind Traders", "invoice_date": "2026-03-15", "due_date": "2026-04-14", "currency": "USD", "total_amount": 249.5}, {"doc_id": "inv-003", "invoice_number": "7731-A", "vendor_name": "Globex GmbH", "invoice_date": "2026-03-02", "due_date": "2026-04-01", "currency": "USD", "total_amount": 3120}, {"doc_id": "inv-004", "invoice_number": null, "vendor_name": "Acme Supply Co.", "invoice_date": "March 20, 2026", "due_date": "2026-04-19", "currency": "USD", "total_amount": 64.99}, {"doc_id": "inv-005", "invoice_number": "SO-88213", "vendor_name": "Initech LLC", "invoice_date": "2026-03-21", "due_date": "2026-03-31", "currency": "USD", "total_amount": 1500.00} ] ``` The planted problems: a date in US format, an amount with a dollar sign and comma, a vendor name in capitals, a due date invented where none exists, the wrong currency, a missing invoice number, and a total that dropped a zero. ## The scorer Save this as `score_extraction.py`. It uses only the standard library and was run with Python 3.9.6 and 3.11.15 on 2026-09-26. ```python """Score extracted invoice fields against a ground-truth file. Usage: python3 score_extraction.py truth.json predicted.json Both files hold a list of objects keyed by doc_id. A null value means "not on the document" in truth and "not extracted" in predictions. """ import json import re import sys from datetime import datetime from decimal import Decimal, InvalidOperation FIELD_TYPES = { "invoice_number": "id", "vendor_name": "text", "invoice_date": "date", "due_date": "date", "currency": "code", "total_amount": "amount", } DATE_FORMATS = ("%Y-%m-%d", "%m/%d/%Y", "%B %d, %Y", "%d %B %Y", "%b %d, %Y") def normalize(value, kind): if value is None: return None text = str(value).strip() if kind == "amount": cleaned = re.sub(r"[^0-9.\-]", "", text) try: return Decimal(cleaned).quantize(Decimal("0.01")) except InvalidOperation: return text if kind == "date": for fmt in DATE_FORMATS: try: return datetime.strptime(text, fmt).date().isoformat() except ValueError: continue return text if kind in ("id", "code"): return re.sub(r"\s+", "", text).lstrip("#").upper() return re.sub(r"[^\w\s]", "", " ".join(text.split())).casefold() def compare(truth, pred, kind, mode): if mode == "normalized": truth, pred = normalize(truth, kind), normalize(pred, kind) if truth is None and pred is None: return "tn" if truth is None: return "fp" if pred is None: return "fn" return "tp" if truth == pred else "wrong" def score(truth_docs, pred_docs, mode): preds = {doc["doc_id"]: doc for doc in pred_docs} counts = {field: {"tp": 0, "fp": 0, "fn": 0} for field in FIELD_TYPES} perfect_docs = 0 errors = [] for doc in truth_docs: pred = preds.get(doc["doc_id"], {}) doc_ok = True for field, kind in FIELD_TYPES.items(): outcome = compare(doc.get(field), pred.get(field), kind, mode) if outcome == "tp": counts[field]["tp"] += 1 elif outcome == "wrong": counts[field]["fp"] += 1 counts[field]["fn"] += 1 elif outcome in ("fp", "fn"): counts[field][outcome] += 1 if outcome not in ("tp", "tn"): doc_ok = False errors.append((doc["doc_id"], field, outcome, doc.get(field), pred.get(field))) perfect_docs += doc_ok return counts, perfect_docs, errors def ratio(numerator, denominator): return f"{numerator / denominator:.2f}" if denominator else "n/a" def report(truth_docs, pred_docs): for mode in ("exact", "normalized"): counts, perfect, errors = score(truth_docs, pred_docs, mode) print(f"\n{mode.upper()} MATCH") print(f"{'field':<16}{'tp':>4}{'fp':>4}{'fn':>4}{'precision':>11}{'recall':>8}") total = {"tp": 0, "fp": 0, "fn": 0} for field, c in counts.items(): for key in total: total[key] += c[key] print(f"{field:<16}{c['tp']:>4}{c['fp']:>4}{c['fn']:>4}" f"{ratio(c['tp'], c['tp'] + c['fp']):>11}{ratio(c['tp'], c['tp'] + c['fn']):>8}") print(f"{'ALL FIELDS':<16}{total['tp']:>4}{total['fp']:>4}{total['fn']:>4}" f"{ratio(total['tp'], total['tp'] + total['fp']):>11}" f"{ratio(total['tp'], total['tp'] + total['fn']):>8}") print(f"documents with every field right: {perfect} of {len(truth_docs)}") if mode == "normalized": print("\nremaining errors:") for doc_id, field, outcome, want, got in errors: print(f" {doc_id} {field}: {outcome} (truth={want!r}, predicted={got!r})") if __name__ == "__main__": if len(sys.argv) != 3: sys.exit("usage: python3 score_extraction.py truth.json predicted.json") with open(sys.argv[1]) as t, open(sys.argv[2]) as p: report(json.load(t), json.load(p)) ``` `FIELD_TYPES` is the part you edit. Each field gets a type, and the type decides how normalization treats it. Exact mode compares the decoded JSON values directly, so `249.5` equals `249.50` because JSON numbers decode to the same float, but `"$1,080.00"` does not equal `1080.0`. Normalized mode turns amounts into two-decimal numbers, parses a few date formats into ISO dates, strips spaces and a leading `#` from identifiers, and lowercases text with punctuation removed. ## Run it ```bash python3 score_extraction.py truth.json predicted.json ``` The recorded output: ```text EXACT MATCH field tp fp fn precision recall invoice_number 4 0 1 1.00 0.80 vendor_name 4 1 1 0.80 0.80 invoice_date 3 2 2 0.60 0.60 due_date 4 1 0 0.80 1.00 currency 4 1 1 0.80 0.80 total_amount 3 2 2 0.60 0.60 ALL FIELDS 22 7 7 0.76 0.76 documents with every field right: 0 of 5 NORMALIZED MATCH field tp fp fn precision recall invoice_number 4 0 1 1.00 0.80 vendor_name 5 0 0 1.00 1.00 invoice_date 5 0 0 1.00 1.00 due_date 4 1 0 0.80 1.00 currency 4 1 1 0.80 0.80 total_amount 4 1 1 0.80 0.80 ALL FIELDS 26 3 3 0.90 0.90 documents with every field right: 1 of 5 remaining errors: inv-002 due_date: fp (truth=None, predicted='2026-04-14') inv-003 currency: wrong (truth='EUR', predicted='USD') inv-004 invoice_number: fn (truth='INV-1004', predicted=None) inv-005 total_amount: wrong (truth=15000.0, predicted=1500.0) ``` ## How to read the numbers **Normalization is a policy, not a cleanup step.** The overall score moved from 0.76 to 0.90, and every point of that gap came from formatting: a date written as `03/14/2026`, an amount written as `$1,080.00`, a vendor name in capitals. Whether those count as correct depends on what happens next. If your code parses dates before writing them, the normalized number is the one that describes your system. If the raw string goes straight into a column that expects ISO dates, the exact number is the honest one. Pick the mode that matches the downstream system and state it next to every score you report. Google's fuzzy matching makes a different choice from this script. Its docs say fuzzy matching lowercases text, trims punctuation and strips currency symbols, but that `1` and `1.00` never match. This script treats them as equal amounts. Neither is wrong, but two scores computed under different rules are not comparable. **The averages hide the costly errors.** After normalization, `total_amount` shows 0.80 precision and recall. The one miss is 1,500 returned for 15,000. The currency miss labels a euro invoice as dollars. Both are single errors in a 30-comparison set, and both would pay the wrong amount. Weight your attention by consequence, and read the remaining-errors list before you read the table. **Document-level results are much lower than field-level ones.** Field-level accuracy was 0.90, yet only one of five invoices had every field right. An invoice that is 90% right still needs a person, so the document-level number is closer to your real review load. If six fields each come back right nine times in ten independently, all six are right together only about 53% of the time, since 0.9 to the sixth power is roughly 0.53. **Invented values need their own line.** `due_date` has perfect recall and 0.80 precision. That pattern means the extractor finds real due dates and also makes some up. A prompt or schema change that tells the model to return `null` when the field is absent is the usual fix, and you should expect precision to move while recall holds. **Small sets give wide error bars.** Each field here has five comparisons, so one mistake moves a score by 0.20. Treat a five-document score as a smoke test. When you compare two extractors or two schema versions, run both on the same documents and look at which specific errors changed, not only at the totals. ## Extend it to your own documents - **Add fields** by adding entries to `FIELD_TYPES`. Add a type and a branch in `normalize` when a field needs its own rule, such as a tax ID that should ignore dashes. - **Keep confidence alongside values.** If your API returns a per-field score, store it next to each prediction and compute precision and recall for values above a threshold. That tells you where the review cutoff belongs. The [review queue post](/blog/designing-the-document-review-queue) covers that decision. - **Line items need matching first.** Tables of line items cannot be scored field by field until each predicted row is paired with a true row. Match on a stable key such as a SKU or description before comparing quantities and prices, and count unmatched rows as misses or inventions. - **Freeze the set.** Keep the ground truth under version control and never tune prompts or schemas on the same documents you use for the final score. A held-out set is what stops a score from measuring how well you memorized the test. - **Log every error.** The remaining-errors list is the most useful part of the output. Keep it per run so you can see whether a change fixed one class of mistake and created another. The same thinking applies beyond documents. [Evaluate a ticket agent](/blog/evaluate-ai-agent-performance-reliability) uses a dataset, scoring code and a failure report for an agent, and [prompt contracts with evaluation fixtures](/blog/prompt-contracts-with-evaluation-fixtures) shows how to pin expected outputs before you change a prompt. ## Where this sits in the series Previous: [An invoice pipeline in n8n](/blog/invoice-pipeline-in-n8n). Next: [Designing the document review queue](/blog/designing-the-document-review-queue), which uses these per-field numbers to set confidence thresholds and sampling rates. For the terms used here, see the [Document AI glossary](/blog/document-ai-glossary). If you are still choosing a parser, [Extend vs Datalab vs other parsers](/blog/extend-vs-datalab-vs-other-document-parsers) is the comparison to run this scorer against.