Extraction Accuracy Benchmarking: How to Run a Vendor Bake-Off

Write the decision rule down before the first document is scored: one primary metric, one hard gate, one tie-break order. Then build a stratified set of 300 to 500 of your own files, establish ground truth independently, run every vendor on the identical set as shipped, and publish per stratum. A bake-off scored after the fact is a rationalisation.

Key facts

• YuSight publishes 95.2% extraction accuracy against a manual benchmark. It is a field-level number on our document mix, and the entire point of this page is that it should not decide your shortlist — your own scoring sheet should. What each metric actually measures is in extraction accuracy vs straight-through rate.
• A bake-off is regulated due diligence, not a science project. The interagency third-party guidance states that "due diligence includes assessing the third party's ability to: perform the activity as expected, adhere to a banking organization's policies related to the activity, comply with all applicable laws and regulations, and conduct the activity in a safe and sound manner" (88 FR 37920, 9 June 2023).
• The winning number belongs in the contract. The same guidance: "Clearly defined performance measures can assist a banking organization in evaluating the performance of a third party. In particular, a service-level agreement between the banking organization and the third party can help specify the measures surrounding the expectations and responsibilities for both parties."
• 400 documents buys you a ±4.9 point margin, and no more. The worst-case 95% margin of error on a document-level proportion is 1.96 × √(0.25 ÷ n): ±9.8 points at n=100, ±6.9 at n=200, ±5.7 at n=300, ±4.9 at n=400, ±3.1 at n=1,000. Most POCs run on 50 files and announce a two-point winner.
• The thing you are testing is a control, not a convenience. BCBS 239 Principle 3 requires risk data to be "aggregated on a largely automated basis so as to minimise the probability of errors," with "effective mitigants in place" where a bank relies on spreadsheets and desktop applications (BIS, January 2013). That is the standard the incumbent manual process is also held to, which is why it belongs in the bake-off as a competitor.
• Never blend strata. OmniDocBench exists in stratified form — "1651 PDF pages... 10 document types, 5 layout types, and 5 language types," with attribute tags so scores read per stratum (OmniDocBench, OpenDataLab). A blended score is an artefact of your sample's mix.

Step 1 — Pre-register the decision rule

Before anyone sees a result, get four things signed by the person who will actually sign the contract.

• The primary metric. One. For a credit operation this is almost always straight-through rate on complete borrower files.
• The hard gate. A ceiling the primary metric is subject to — typically escaped-error rate, the proportion of documents that went through untouched and were wrong. Set the number now. Five per cent is a common starting point.
• The tie-break order, in writing, ranked.
• What disqualifies a run — vendor tuning against the test set, a vendor-side human in the loop, thresholds changed mid-run.

Pre-registration is the single control that separates a bake-off from a beauty parade. Without it, whoever demos best gets the metric that flatters them chosen afterwards.

Step 2 — Build the test set

Size. 300 to 500 documents, from your own book. Use the margin-of-error table above to argue the number to procurement rather than asserting it. Below 200 you cannot distinguish a real five-point gap from noise, and five points is roughly the size of gap that exists between serious vendors.

Strata. Design the mix from your actual intake, not from what is easy to collect. A workable frame:

The worst files are mandatory. Pull the five per cent of files your team complains about: the rotated scan, the 180-page consolidated set, the statement with a bank's redesigned template, the accounts in a second language, the one where the borrower photographed a screen. If a bake-off contains only files that work, it measures nothing you did not already know.

Freeze it. Hash the set, store it read-only, and keep a hold-out of 15% that no vendor sees until the final run. The hold-out is what catches tuning.

Step 3 — Establish ground truth, and name its owner

Ground truth is the largest single cost in a bake-off and the thing most often skipped.

1. Two experienced analysts independently prepare 30 documents. Where they disagree is the human ceiling. If they agree on 96.8% of fields, no vendor result above 96.8% is measuring vendor quality — it is measuring your ground truth.
2. A third, senior person adjudicates the disagreements and their adjudication becomes truth. Name this person in the protocol. Ground truth without an owner becomes negotiable the moment a vendor challenges a score.
3. Write the normalisation rules before scoring: sign conventions, scale (thousands, lakhs, millions), currency, period labelling, entity attribution, chart-of-accounts mapping, and what counts as one field. Every scoring dispute traces back to one of these.
4. Score judgement fields separately from lookup fields. "Cash at bank, 31 March, page 28" is a lookup. "Is this director's loan debt?" is a judgement, and disagreement there is not an extraction defect.

The benchmark-construction question in more depth — corpora, human ceilings, published datasets — is covered in how accurate is AI at extracting data from financial statements.

Step 4 — Fix the rules of engagement

Put these in the POC agreement, not the kick-off deck:

• As shipped. Confidence gating on, validation rules on, default thresholds. You are buying the product, not a demo configuration.
• No tuning against the test set, and the hold-out to prove it.
• Same set, same week, no re-runs. A re-run after seeing a score is a different experiment.
• No vendor-side human in the loop, and a written statement to that effect.
• Data handling: what leaves your premises, retention, deletion at exit, and whether your documents may be used for training. Get the answer in the contract, not the sales call.
• You do the scoring. Vendors may see their own results, not each other's, and not before the run ends.

Step 5 — Score, with the arithmetic shown

A worked sheet. Test set 400 documents, 120 fields each, so 48,000 fields. Three vendors, identical set, as shipped. Figures illustrative.

The arithmetic, line by line:

• Field accuracy A = 46,272 ÷ 48,000 = 0.964 → 96.4%. B = 46,752 ÷ 48,000 = 97.4%. C = 45,600 ÷ 48,000 = 95.0%.
• Document accuracy A = 268 ÷ 400 = 67.0%. B = 212 ÷ 400 = 53.0%.
• B reads better and delivers worse documents, because its errors are spread thin. A makes 48,000 − 46,272 = 1,728 errors across 400 − 268 = 132 imperfect documents, or 13.1 per bad document. B makes 1,248 errors across 188 imperfect documents, or 6.6 each. A's errors cluster in a few files you can route away; B's contaminate half the set.
• Escaped-error A = 9 ÷ 249 = 3.6%. B = 41 ÷ 302 = 13.6%. C = 7 ÷ 228 = 3.1%.

Apply the pre-registered rule: highest STP subject to escaped error ≤ 5.0%. B posts the best field accuracy and the best STP and fails the gate. It is out. A wins on 62.3% against C's 57.0%.

What the gate is worth, annualised at 60,000 documents a year (factor 150):

• A: 9 ÷ 400 = 2.25% → 1,350 wrong documents a year, unreviewed.
• C: 7 ÷ 400 = 1.75% → 1,050.
• B: 41 ÷ 400 = 10.25% → 6,150.

Buying B on field accuracy would have put roughly 4,800 more wrong documents a year into files nobody looks at again. That is the sentence the credit committee needs, and it is arithmetic, not opinion.

Step 6 — Score the hard 10% on its own

Report the deliberate-worst stratum as a separate result and never fold it into the headline. Three things to record on it:

• Failure mode, not just failure rate. Did the system return a wrong number confidently, or decline the field? The second is a working control; the first is the expensive one.
• Flag precision. Of the documents it routed to review, what proportion actually needed it? A system that flags everything has a perfect catch rate and no value.
• Recovery cost. Minutes to fix a hard file, measured, not estimated.

The reading-technology differences that drive this stratum are set out in OCR vs IDP vs LLM extraction.

Step 7 — Break ties honestly

A and C are 5.3 points apart on STP. Is that real?

Because both ran the identical documents, use the discordant pairs. Suppose A went straight through and C did not on 71 documents, and C did and A did not on 50 — a net difference of 21 documents, which is the 249 − 228 gap. The paired test statistic is (71 − 50) ÷ √(71 + 50) = 21 ÷ 11 = 1.91, roughly a two-sided p of 0.06.

So: suggestive, not decisive. When the primary metric does not separate the finalists, work down the pre-registered tie-break order. A defensible order:

1. Performance on the hard 10%
2. Escaped-error rate at the operating threshold
3. Review-flag precision — does it point at the right field, not just the right document?
4. Mean minutes to correct
5. Configuration burden: how many hours to reach the tested state on a new document type
6. Exit cost and data portability

Price is not on that list until the gate is passed and the tie-break is exhausted. The wider capability scorecard for spreading platforms specifically is in the 22-point scorecard.

Step 8 — Write it up so procurement can act

A decision memo procurement can sign has eight sections and fits on three pages:

1. The decision rule as pre-registered, with the date it was fixed and by whom.
2. The test set: size, strata, source, hash, hold-out share.
3. Ground truth: who prepared it, the human ceiling, who adjudicated.
4. Rules of engagement and any breach of them.
5. The scoring table, per stratum, never blended.
6. The decision, stated as the rule applied to the table — not as a preference.
7. The SLA lines to write into the contract: the primary metric, the gate, the measurement method, the re-measurement cadence and the remedy if it drifts.
8. What the test did not cover, listed honestly — document types absent from the set, volumes not stress-tested, integrations not exercised.

Section 7 is the one that pays for the whole exercise. A bake-off whose winning number never reaches the contract has told you something interesting and bought you nothing.

How long should it run?

Six weeks with roughly two people, of which weeks 3 and 6 are the ones that get compressed and should not be. Anything shorter than four weeks is a demo you scored. Anything longer than eight and the models have changed underneath you — which is also the argument for re-running annually against the frozen set.

FAQ

How do you benchmark document extraction vendors?

Fix the decision rule first — one primary metric, one hard gate, one ranked tie-break order — then build 300 to 500 of your own documents into a stratified, frozen set, establish ground truth independently, run every vendor on the identical set as shipped, and report per stratum. Scoring first and choosing the metric afterwards is how vendors win bake-offs they should lose.

How many documents make a valid test set?

Enough that the gap you care about is bigger than your margin of error. At 400 documents the worst-case 95% margin on a document-level rate is about ±4.9 points; at 100 documents it is ±9.8, which is wider than the difference between most serious vendors. Three hundred is the practical floor and 400 to 500 is comfortable.

Should accuracy be measured per field or per document?

Both, and report them separately. Field accuracy tells you whether the reading layer works; document accuracy tells you what your analysts will actually pick up. The worked sheet above has one vendor winning on fields and losing badly on documents, which is the normal case rather than a curiosity.

Do we have to include our worst files?

Yes, and around five per cent of the set is the right dose. Your worst files are where the cost is, and a test set of clean native PDFs will rank vendors identically to a coin toss because they all handle those. Score the hard stratum separately so it does not drown the headline either.

Who should establish ground truth?

Two of your own experienced analysts working independently, with a named senior adjudicator settling disagreements. Never the vendor, and never a single junior — a one-person ground truth is one person's opinion, and the first score dispute will show that.

Can a vendor tune their model to our test set?

Not during the bake-off, and you need a hold-out to prove it rather than a promise. Tuning is legitimate after selection — that is implementation — but a tuned score compared against an untuned one is not a comparison.

What if two vendors finish within a point of each other?

Then the primary metric has not decided it and you go to the pre-registered tie-break order: hard-stratum performance, escaped errors, flag precision, correction time, configuration burden, exit cost. Deciding on price at that point is defensible. Deciding on price before the gate is not.

Should we benchmark our current process too?

Yes, and almost nobody does. Manual preparation has an error rate and a cost per file, and until both are measured you are comparing a vendor's measured number against an assumption that your team is perfect. Run the same 30-document ceiling study on your own team and use it as the floor.

How often should we re-run it?

Annually against the frozen set, plus whenever a vendor ships a major model change or your intake mix moves — a jump in mobile-captured documents will move every number on the sheet. Write the re-measurement cadence into the SLA so it happens without a new business case.

Conclusion

Three things to take away:

1. The rule comes before the result. Pre-register the primary metric, the hard gate and the tie-break order, and the bake-off produces a decision instead of an argument.
2. The gate is where the money is. In the worked sheet the best-reading vendor would have shipped roughly 4,800 more wrong documents a year into files nobody re-opens. Field accuracy alone would have bought that.
3. Finish in the contract. The winning metric, the gate, the measurement method and the re-measurement cadence belong in the SLA, or the six weeks bought you a slide.

YuSight's Document Intelligence classifies and validates tax returns, bank statements, financial statements and bureau reports, maps each to the right borrower entity, and hands cited figures to spreading — measured at 95.2% extraction accuracy against a manual benchmark, and we would rather be scored on your worst 5% than on that number.

Upload a messy document set and see it classified — book a live demo.