Measure Document Extraction Accuracy Field by Field

"The extractor is 90% accurate" hides almost everything you need to know. Accurate on which fields? Checked against what answer key? Counting blanks how? And after how much cleanup of dates and dollar signs?

This lesson builds the answer key and the scorer. You get an answer key for five invoices, a set of extracted results with mistakes planted on purpose, and a Python script that needs nothing installed. The script scores every field twice: once requiring an exact match, and once after tidying up formatting like dates and dollar signs.

That one choice matters a lot. In the recorded run below, the same results score 0.76 or 0.90 depending on it. Either way, only one of the five invoices came back fully right.

The data is made up. It's there to exercise the scoring rules, not to rate any vendor. This is part 7 of the Document AI series. It picks up where the n8n invoice pipeline leaves off: that workflow decides which invoices go to a person for review, and this script tells you whether those decisions deserve trust.

What a field-level score counts

Take one field on one invoice, say the total. Compare the value printed on the page with the value the extractor returned. That's one comparison. Every field on every document gets one, and each lands in one of these rows:

From those counts you get two scores.

Precision answers: of the values the extractor gave you, how many were right? It's true positives divided by true positives plus false positives.

Recall answers: of the values on the page, how many came back correctly? It's true positives divided by true positives plus false negatives.

Those are the standard definitions in scikit-learn's evaluation guide and in Google Document AI's evaluation docs, both read on 2026-09-26.

The row that trips people up is the wrong value. It counts against both scores, because the extractor said something false and also failed to give you the real answer. Count it only as a miss and precision looks better than it is. Precision is the number that tells you how often a confident-looking value will mislead whatever system reads it next.

The invented row matters just as much for invoices. Picture a due date the model worked out from payment terms that weren't on the page. That's a false positive, and it's exactly the kind of value an accounts-payable system will act on without question.

Build a small ground-truth set

Ground truth is the answer key: what a careful person reads off each document. Five documents are enough to learn how scoring works. For a real decision, you want enough of your own real documents that each important field, layout and kind of mistake shows up several times.

Save this as truth.json. A null means the field is not on that document.

[
  {"doc_id": "inv-001", "invoice_number": "INV-1001", "vendor_name": "Acme Supply Co.", "invoice_date": "2026-03-14", "due_date": "2026-04-13", "currency": "USD", "total_amount": 1080.00},
  {"doc_id": "inv-002", "invoice_number": "INV-1002", "vendor_name": "Northwind Traders", "invoice_date": "2026-03-15", "due_date": null, "currency": "USD", "total_amount": 249.50},
  {"doc_id": "inv-003", "invoice_number": "7731-A", "vendor_name": "Globex GmbH", "invoice_date": "2026-03-02", "due_date": "2026-04-01", "currency": "EUR", "total_amount": 3120.00},
  {"doc_id": "inv-004", "invoice_number": "INV-1004", "vendor_name": "Acme Supply Co.", "invoice_date": "2026-03-20", "due_date": "2026-04-19", "currency": "USD", "total_amount": 64.99},
  {"doc_id": "inv-005", "invoice_number": "SO-88213", "vendor_name": "Initech LLC", "invoice_date": "2026-03-21", "due_date": "2026-03-31", "currency": "USD", "total_amount": 15000.00}
]

Three rules keep an answer key honest:

1. Write down what the document says, in one standard format per type: dates as year-month-day, numbers without symbols, currency as a three-letter code.
2. Mark fields that really aren't there as null. Don't leave them out, and don't fill them in from something you know that isn't on the page.
3. Have a second person check some of the key, and turn every disagreement into a written rule. If two people can't agree on the invoice date, you can't score the extractor on it yet.

Now the extractor's results. Save this as predicted.json. It looks like the JSON a document AI service hands back once you pull out its extracted fields, with mistakes planted on purpose.

[
  {"doc_id": "inv-001", "invoice_number": "INV-1001", "vendor_name": "ACME SUPPLY CO", "invoice_date": "03/14/2026", "due_date": "2026-04-13", "currency": "USD", "total_amount": "$1,080.00"},
  {"doc_id": "inv-002", "invoice_number": "INV-1002", "vendor_name": "Northwind Traders", "invoice_date": "2026-03-15", "due_date": "2026-04-14", "currency": "USD", "total_amount": 249.5},
  {"doc_id": "inv-003", "invoice_number": "7731-A", "vendor_name": "Globex GmbH", "invoice_date": "2026-03-02", "due_date": "2026-04-01", "currency": "USD", "total_amount": 3120},
  {"doc_id": "inv-004", "invoice_number": null, "vendor_name": "Acme Supply Co.", "invoice_date": "March 20, 2026", "due_date": "2026-04-19", "currency": "USD", "total_amount": 64.99},
  {"doc_id": "inv-005", "invoice_number": "SO-88213", "vendor_name": "Initech LLC", "invoice_date": "2026-03-21", "due_date": "2026-03-31", "currency": "USD", "total_amount": 1500.00}
]

The planted problems: a date in US format, an amount with a dollar sign and comma, a vendor name in capitals, a due date invented where none exists, the wrong currency, a missing invoice number, and a total that dropped a zero.

The scorer

Save this as score_extraction.py. It uses only the standard library and was run with Python 3.9.6 and 3.11.15 on 2026-09-26.

"""Score extracted invoice fields against a ground-truth file.

Usage: python3 score_extraction.py truth.json predicted.json
Both files hold a list of objects keyed by doc_id. A null value means
"not on the document" in truth and "not extracted" in predictions.
"""
import json
import re
import sys
from datetime import datetime
from decimal import Decimal, InvalidOperation

FIELD_TYPES = {
    "invoice_number": "id",
    "vendor_name": "text",
    "invoice_date": "date",
    "due_date": "date",
    "currency": "code",
    "total_amount": "amount",
}
DATE_FORMATS = ("%Y-%m-%d", "%m/%d/%Y", "%B %d, %Y", "%d %B %Y", "%b %d, %Y")


def normalize(value, kind):
    if value is None:
        return None
    text = str(value).strip()
    if kind == "amount":
        cleaned = re.sub(r"[^0-9.\-]", "", text)
        try:
            return Decimal(cleaned).quantize(Decimal("0.01"))
        except InvalidOperation:
            return text
    if kind == "date":
        for fmt in DATE_FORMATS:
            try:
                return datetime.strptime(text, fmt).date().isoformat()
            except ValueError:
                continue
        return text
    if kind in ("id", "code"):
        return re.sub(r"\s+", "", text).lstrip("#").upper()
    return re.sub(r"[^\w\s]", "", " ".join(text.split())).casefold()


def compare(truth, pred, kind, mode):
    if mode == "normalized":
        truth, pred = normalize(truth, kind), normalize(pred, kind)
    if truth is None and pred is None:
        return "tn"
    if truth is None:
        return "fp"
    if pred is None:
        return "fn"
    return "tp" if truth == pred else "wrong"


def score(truth_docs, pred_docs, mode):
    preds = {doc["doc_id"]: doc for doc in pred_docs}
    counts = {field: {"tp": 0, "fp": 0, "fn": 0} for field in FIELD_TYPES}
    perfect_docs = 0
    errors = []
    for doc in truth_docs:
        pred = preds.get(doc["doc_id"], {})
        doc_ok = True
        for field, kind in FIELD_TYPES.items():
            outcome = compare(doc.get(field), pred.get(field), kind, mode)
            if outcome == "tp":
                counts[field]["tp"] += 1
            elif outcome == "wrong":
                counts[field]["fp"] += 1
                counts[field]["fn"] += 1
            elif outcome in ("fp", "fn"):
                counts[field][outcome] += 1
            if outcome not in ("tp", "tn"):
                doc_ok = False
                errors.append((doc["doc_id"], field, outcome, doc.get(field), pred.get(field)))
        perfect_docs += doc_ok
    return counts, perfect_docs, errors


def ratio(numerator, denominator):
    return f"{numerator / denominator:.2f}" if denominator else "n/a"


def report(truth_docs, pred_docs):
    for mode in ("exact", "normalized"):
        counts, perfect, errors = score(truth_docs, pred_docs, mode)
        print(f"\n{mode.upper()} MATCH")
        print(f"{'field':<16}{'tp':>4}{'fp':>4}{'fn':>4}{'precision':>11}{'recall':>8}")
        total = {"tp": 0, "fp": 0, "fn": 0}
        for field, c in counts.items():
            for key in total:
                total[key] += c[key]
            print(f"{field:<16}{c['tp']:>4}{c['fp']:>4}{c['fn']:>4}"
                  f"{ratio(c['tp'], c['tp'] + c['fp']):>11}{ratio(c['tp'], c['tp'] + c['fn']):>8}")
        print(f"{'ALL FIELDS':<16}{total['tp']:>4}{total['fp']:>4}{total['fn']:>4}"
              f"{ratio(total['tp'], total['tp'] + total['fp']):>11}"
              f"{ratio(total['tp'], total['tp'] + total['fn']):>8}")
        print(f"documents with every field right: {perfect} of {len(truth_docs)}")
        if mode == "normalized":
            print("\nremaining errors:")
            for doc_id, field, outcome, want, got in errors:
                print(f"  {doc_id} {field}: {outcome} (truth={want!r}, predicted={got!r})")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit("usage: python3 score_extraction.py truth.json predicted.json")
    with open(sys.argv[1]) as t, open(sys.argv[2]) as p:
        report(json.load(t), json.load(p))

FIELD_TYPES is the part you edit. Each field gets a type, and the type decides how it gets tidied up before comparing. That tidying is called normalization.

Exact mode compares the values as they come out of the JSON. So 249.5 equals 249.50, because both turn into the same number, but "$1,080.00" doesn't equal 1080.0, because one is text.

Normalized mode turns amounts into numbers with two decimals, reads a few date formats and rewrites them as year-month-day, strips spaces and a leading # from IDs, and lowercases text with punctuation removed.

Run it

python3 score_extraction.py truth.json predicted.json

The recorded output:

EXACT MATCH
field             tp  fp  fn  precision  recall
invoice_number     4   0   1       1.00    0.80
vendor_name        4   1   1       0.80    0.80
invoice_date       3   2   2       0.60    0.60
due_date           4   1   0       0.80    1.00
currency           4   1   1       0.80    0.80
total_amount       3   2   2       0.60    0.60
ALL FIELDS        22   7   7       0.76    0.76
documents with every field right: 0 of 5

NORMALIZED MATCH
field             tp  fp  fn  precision  recall
invoice_number     4   0   1       1.00    0.80
vendor_name        5   0   0       1.00    1.00
invoice_date       5   0   0       1.00    1.00
due_date           4   1   0       0.80    1.00
currency           4   1   1       0.80    0.80
total_amount       4   1   1       0.80    0.80
ALL FIELDS        26   3   3       0.90    0.90
documents with every field right: 1 of 5

remaining errors:
  inv-002 due_date: fp (truth=None, predicted='2026-04-14')
  inv-003 currency: wrong (truth='EUR', predicted='USD')
  inv-004 invoice_number: fn (truth='INV-1004', predicted=None)
  inv-005 total_amount: wrong (truth=15000.0, predicted=1500.0)

How to read the numbers

Normalization is a decision, not a cleanup step. The overall score moved from 0.76 to 0.90, and every point of that gap came from formatting: a date written as 03/14/2026, an amount written as $1,080.00, a vendor name in capitals.

Whether those count as correct depends on what happens next. If your code converts dates before saving them, the normalized number describes your system. If the raw text goes straight into a column that expects year-month-day dates, the exact number is the honest one. Pick the mode that matches where the data goes next, and write it next to every score you report.

Google's fuzzy matching makes a different choice from this script. Its docs say fuzzy matching lowercases text, trims punctuation and strips currency symbols, but that 1 and 1.00 never match. This script treats them as the same amount. Neither is wrong, but you can't compare two scores worked out under different rules.

The averages hide the expensive mistakes. After normalization, total_amount shows 0.80 precision and recall. The one miss is 1,500 returned for 15,000. The currency miss labels a euro invoice as dollars. Each is one error out of 30 comparisons, and each would pay the wrong amount. Read the remaining-errors list before you read the table, and pay most attention to the mistakes that cost the most.

Whole invoices score much lower than single fields. Fields were right 0.90 of the time, yet only one of five invoices had every field right. An invoice that's 90% right still needs a person to look at it, so the whole-invoice number is closer to how much review work you'll really have. The math is unforgiving: if six fields are each right nine times in ten, independently, all six are right together only about 53% of the time, since 0.9 to the sixth power is roughly 0.53.

Made-up values need their own line. due_date has perfect recall and 0.80 precision. That pattern means the extractor finds the real due dates and also invents some. The usual fix is to change the prompt or schema so the model returns null when a field isn't on the page. Expect precision to go up while recall stays put.

Small sets give shaky scores. Each field here has five comparisons, so one mistake moves a score by 0.20. Treat a five-document score as a quick sanity check. When you compare two extractors or two schema versions, run both on the same documents and look at which specific mistakes changed, not just the totals.

Extend it to your own documents

• Add fields by adding entries to FIELD_TYPES. Add a type and a branch in normalize when a field needs its own rule, such as a tax ID that should ignore dashes.
• Keep confidence alongside values. If your service returns a confidence score for each field, store it next to each value. Then work out precision and recall for only the values above a cutoff. That tells you where to draw the line for sending an invoice to review. The review queue post covers that decision.
• Line items need matching first. You can't score a table of line items field by field until each extracted row is paired with the right row in the answer key. Pair them on something that doesn't change, such as a SKU or description, before comparing quantities and prices. Count unpaired rows as misses or inventions.
• Freeze the set. Keep the answer key in version control, and never tune prompts or schemas on the same documents you use for the final score. Keep a separate set aside for that. Otherwise the score only measures how well you memorized the test.
• Save every error. The remaining-errors list is the most useful part of the output. Keep it for every run so you can see whether a change fixed one kind of mistake and created another.

The same approach works beyond documents. Evaluate a ticket agent uses a dataset, scoring code and a failure report for an agent, and prompt contracts with evaluation fixtures shows how to pin expected outputs before you change a prompt.

Where this sits in the series

Previous: An invoice pipeline in n8n. Next: Designing the document review queue, which uses these per-field numbers to set confidence thresholds and sampling rates. For the terms used here, see the Document AI glossary. If you are still choosing a parser, Extend vs Datalab vs other parsers is the comparison to run this scorer against.