How document extraction accuracy is actually measured

"Accuracy" is the word every document-extraction vendor puts on a slide, and it is also the word that carries the least information, because almost nobody states what it is measuring. Ninety-five percent accurate could mean ninety-five percent of individual fields correct, ninety-five percent of whole documents fully correct, or ninety-five percent of a sample the vendor selected themselves. Those three numbers can differ by a wide margin on the same document set, and a buyer who does not ask which one is being reported is comparing nothing to nothing, which is worse than comparing nothing at all, because it feels like due diligence without doing any.

Field-level accuracy counts individual data points: a policy limit, a deductible, an effective date, a vendor name on an invoice. Document-level accuracy counts whole documents, and a document only passes if every field on it is correct. A pipeline can hit ninety-eight percent field-level accuracy and still fail document-level accuracy on a third of documents, because ninety-eight percent of fields correct across a twenty-field policy still leaves an average of one wrong value somewhere on most pages. For insurance, legal, or financial documents where a single wrong number changes a decision, document-level accuracy is the number that actually matters to the person relying on the output, and it is almost never the number vendors lead with, because it is the harder one to hit and the more honest one to report.

The instrument for measuring either number is a golden set: a fixed batch of real documents where every field has been hand-verified against the source by a person who knows the domain, and then held constant release over release. Without a golden set, "accuracy improved" is an unfalsifiable claim, because the test set moved along with the model that was supposedly being measured against it. Building one is slow, unglamorous work that cannot be shortcut: someone has to open each source document, confirm the correct value against the actual page, and record it as ground truth, field by field, document by document. The payoff is that every future release gets scored against the same fixed answer key in minutes rather than weeks, and a regression shows up immediately instead of surfacing three months later in a client's inbox as a wrong number on a document nobody double-checked.

Building a golden set well means resisting the urge to hand-pick the easy documents. A set built only from clean, well-formatted policies will always score high and will never tell you what happens on the scanned, handwritten, or oddly formatted ten percent that shows up in every real book of business. A useful golden set is deliberately weighted toward the hard cases: the carrier that formats things differently from everyone else, the multi-endorsement policy where the correct value depends on reading three amendments in order, the scanned document where OCR itself introduces error before extraction even begins. It also has to grow over time. Every time a new failure mode appears in production, that document becomes a candidate for the golden set, so the measurement gets stricter as the pipeline gets better, rather than staying easy forever while the marketing claim stays flattering.

A diff between the extracted value and the golden value catches wrong answers. It does not catch invented ones, because a diff only compares two values that already exist side by side; it says nothing about whether the extracted value was supposed to exist at all. A model can hallucinate a coverage limit that reads as entirely plausible, land close enough to a true value to pass a loose comparison, and still be a number nobody wrote down anywhere in the source document. A forbidden-value check runs the opposite direction: it holds a list of values a given field must never produce, built from what the corpus actually contains, and flags any output that matches one of them. Run alongside the golden-set diff, it catches the failure mode a diff structurally cannot see on its own: not "wrong," but "invented, and invented in a way that looks entirely correct until someone checks the source." Take a policy limit as the concrete case: a diff only fires if the extracted number does not match the golden number. If a model invents a plausible-sounding limit for a coverage that the policy never actually mentions, and that invented number happens to be close to a typical limit for that coverage type, a loose diff can miss it entirely, because there is no golden value to compare against on a field that should have come back empty. The forbidden-value check exists precisely for that gap: it does not ask whether the number is close to correct, it asks whether the field should have produced a value at all, and flags the case where the model answered a question the document never actually answered.

PolicyIQ, an insurance policy-comparison system Signalward built for an independent agency, was scored this way from the first release rather than retrofitted with measurement once a client asked hard questions. The extraction layer runs two frontier models against every coverage question independently, and the two must agree before a value is accepted into the ontology; where they diverge, or where a deterministic accuracy guard flags something as ambiguous, the value is held for a human reviewer instead of the system guessing at a middle ground. Nothing displays on screen that the source document does not literally say, and every displayed value carries a citation back to the exact passage it came from, so a broker checking a number is never taking the system's word for it on faith.

Independent audits against real client policies, run across sixteen lines of business, land the displayed-value accuracy at ninety-six to one hundred percent, with zero invented values on the golden scorer, backed by more than two thousand automated tests that re-run on every release. The range exists because insurance policies vary in how cleanly they are written, not because the pipeline itself is inconsistent: a clean, well-structured policy scores at the top of the range, a dense or unusually formatted one sits lower, and the honest move is reporting that range rather than one blended number that quietly hides the spread.

The practical version of this for anyone evaluating a document-extraction vendor, in insurance or anywhere else: ask for the field-level number and the document-level number separately, ask whether the test set is fixed or moves along with each new release, and ask what happens to a value the system is not confident about. If the honest answer to that last question is "it guesses," the accuracy number on the slide is measuring something other than what you actually need it to measure, which is whether you can hand the output to a client without checking it by hand first.

This question comes up most in Document & Data Processing work.