Extraction Accuracy vs Straight-Through Rate: Which Metric Should Lenders Buy On?
Buy on straight-through rate, at a stated escaped-error ceiling. Field-level extraction accuracy measures how often the model reads a number correctly; straight-through rate measures how many documents get through with nobody touching them. Only the second one predicts your headcount, and a vendor can be excellent at the first and useless at it.
Key facts
- YuSight measures 95.2% extraction accuracy against a manual benchmark — a field-level figure. Read the honesty section below before you compare it to anyone else's number, including where it does and does not hold.
- A 99% field-level number is not a 99% document. Across a 120-field spread, 0.99¹²⁰ = 29.9% of documents perfect under naive independence, and an expected 1.2 wrong fields per document. The arithmetic is worked line by line further down.
- Benchmarks are stratified for a reason. OmniDocBench covers "1651 PDF pages... 10 document types, 5 layout types, and 5 language types," with page, text and table attribute tags so scores can be read per stratum rather than as one blended number (OmniDocBench, OpenDataLab).
- Supervisors expect automation and reconciliation, in that order. BCBS 239 Principle 3 requires risk data to be "aggregated on a largely automated basis so as to minimise the probability of errors," and reconciled to source (BIS, January 2013).
- An extraction pipeline that feeds a risk grade is in scope for validation. SR 11-7 requires that "all model components—inputs, processing, outputs, and reports—should be subject to validation," including vendor-supplied ones (Federal Reserve, SR 11-7, 4 April 2011). Note: SR 11-7 was superseded on 17 April 2026 by SR 26-2 and OCC Bulletin 2026-13, which narrow the definition of a model and place generative and agentic AI outside scope. Whether this requirement still reaches your tool is the live question — see what SR 26-2 changed.
What does field-level extraction accuracy actually measure?
It measures agreement between a system's output and a ground truth, counted one field at a time.
That sentence hides four choices, and vendors rarely state any of them:
- What counts as a field. Is "Turnover, FY24, Entity A, restated" one field or four? A 40-field spread and a 120-field spread of the same accounts produce different percentages from identical software.
- What counts as correct. Right digits is the easy case. Sign convention, scale (thousands vs lakhs vs millions), period label, entity and chart-of-accounts mapping are all separately failable. A system that reads 4,52,000 perfectly and maps it to the wrong line has produced a correct extraction and a wrong spread.
- What happens to abstentions. If the model declines a field it is unsure of, is that an error, a correct exception, or excluded from the denominator? Excluding abstentions can move a headline number by several points on its own.
- Whose ground truth. Two experienced analysts spreading the same audited accounts will disagree on judgement fields — what belongs in other income, whether a director's loan is debt. That disagreement rate is the ceiling on any accuracy score.
Field-level accuracy is a real engineering metric: it tells you whether the reading layer works. It does not tell you what your team will do on Monday.
What does straight-through rate actually measure?
Straight-through rate (STP) is the proportion of documents — or of complete borrower files, the more honest unit — that pass from intake to an accepted, usable output with zero human touches, measured with the vendor's confidence gating and validation rules switched on, as shipped.
Its complement is the touch rate. Its shadow metric, which almost nobody quotes, is the escaped-error rate: the proportion of documents that went straight through and were wrong.
STP without an escaped-error ceiling is meaningless, because STP is trivially maximisable. Turn off every threshold and every validation check and STP goes to 100%. The pair is the metric:
STP of X% at an escaped-error rate no higher than Y%, on document mix Z.
Anything less than that triple is a marketing number.
| Field-level extraction accuracy | Straight-through rate |
|---|---|---|
Unit counted | A field | A document or a borrower file |
What it describes | The reading layer | The whole pipeline, including exception routing |
Who controls it | Mostly the vendor | You and the vendor jointly — thresholds are your policy |
Sensitive to | Scan quality, layout variance, language | All of the above plus your document mix and your risk appetite |
Predicts | Whether the model can read | Your headcount |
Gameable by | Choosing a favourable field list | Turning off confidence gating |
Must be quoted with | Test set, document mix, field list, abstention rule | Escaped-error rate and document mix |
The compounding arithmetic, worked
Take a full spreading job: three years of financials, balance sheet, P&L and cash flow, plus the notes you actually use. Call it 120 fields.
Under naive independence — every field an independent draw at the stated accuracy — the probability that a document comes out with zero errors is accuracy¹²⁰:
Field-level accuracy | Expected wrong fields per document (120 × (1−p)) | Documents with zero errors (p¹²⁰) |
|---|---|---|
99.9% | 0.12 | 0.999¹²⁰ = 88.7% |
99.5% | 0.60 | 0.995¹²⁰ = 54.8% |
99.0% | 1.20 | 0.99¹²⁰ = 29.9% |
98.0% | 2.40 | 0.98¹²⁰ = 8.9% |
95.2% | 5.76 | 0.952¹²⁰ = 0.27% |
The working for the 99% row, since this is the one vendors quote: ln(0.99) = −0.010050; × 120 = −1.206; e^(−1.206) = 0.2994. So 29.9%.
Two things follow.
First, the headline number is doing far less work than it looks. The gap between 99.0% and 99.5% is half a percentage point at field level and 25 points at document level. Buyers routinely treat 99.0% and 99.5% as equivalent. They are not remotely equivalent.
Second, independence is the wrong assumption, and it flatters the pessimism. Extraction errors cluster: one badly scanned page produces eight errors in one document while thirty others come out clean. Real document-level accuracy is therefore materially higher than p¹²⁰, and no formula gets you from one to the other. The clustering factor is a property of your files, not of the vendor's model.
So the table is an argument for measuring, not a forecast. Use it to reject a single field-level percentage as a purchase criterion, then measure the document-level number and the STP on your own book. OCR vs IDP vs LLM extraction covers which technique fails where, and why scan quality moves the number more than model choice does.
Why the two metrics move independently
Four cases, all of which exist in the market.
| High field accuracy | Low field accuracy |
|---|---|---|
High STP | The genuine article — reads well and knows when it did not | Dangerous: high throughput because nothing is being checked. Escaped errors land in the credit file |
Low STP | Common and often correct: a cautious pipeline that flags aggressively. Safe, expensive, and improvable by tuning | Straightforwardly not fit for purpose |
The top-right cell costs money quietly. A system that returns wrong numbers with high confidence produces beautiful throughput statistics and a spread nobody caught. A pipeline that fails loudly is safer than one that is right more often and fails silently — which is why every accepted figure should be citable back to a page; see how to stop an AI credit memo tool from hallucinating numbers.
STP is a policy choice as much as a capability
The confidence threshold is a dial you own. Here is what turning it does, on 1,000 documents of 120 fields each.
Assume field accuracy of 99.0% → 120,000 fields, 1,200 wrong fields. Because errors cluster, suppose those 1,200 errors land in 180 documents (an average of 6.7 errors each). So 82% of documents are already perfect — far better than the 29.9% naive figure, and a useful demonstration of why clustering matters.
Threshold A — conservative. Flags any document containing a low-confidence field.
- Documents flagged for review: 400 (40%)
- Bad documents caught: 171 of 180 = 95%
- Bad documents escaping: 180 − 171 = 9 → escaped-error rate 0.9%
- STP = 1,000 − 400 = 600 → 60%
- Review precision: 171 ÷ 400 = 42.8% — most of what your analysts open was fine
Threshold B — aggressive. Flags only on hard validation failures.
- Documents flagged: 150 (15%)
- Bad documents caught: 112 of 180 = 62%
- Bad documents escaping: 68 → escaped-error rate 6.8%
- STP = 85%
- Review precision: 112 ÷ 150 = 74.7%
Same software. Same field accuracy. STP moved 25 points, and the escaped-error rate moved from 0.9% to 6.8%. Neither setting is wrong; they are different appetites. But note what a vendor quoting "85% straight-through" without the second number has done.
Ask every vendor for the curve, not the point: STP at three threshold settings, with the escaped-error rate at each.
Why STP depends on your document mix, not the vendor's
A vendor's published STP describes their reference mix. Yours will differ, and not by a rounding error. The variables that move it most:
- Native PDF vs scan vs phone photograph. A native PDF is a parsing problem; a 200 dpi photograph shot at an angle is a recognition problem. Never let one STP number span both.
- Number of distinct layouts. Twelve bank formats behaves nothing like two hundred audit-firm house styles.
- Multi-entity structures. A group borrower needs each document assigned to the right entity before extraction, and a mis-assignment is a document-level failure no field-level metric will show. See multi-entity document mapping.
- Language, script and separators.
1.234.567read as1.234is a 1,000x error that foots perfectly. - Your own acceptance rules. If policy requires an analyst to eyeball every audited financial statement regardless of confidence, your STP ceiling for that type is zero and no vendor can raise it.
So STP is both the metric to buy on and the one you cannot take from a vendor's website. It has to be measured on your files.
What STP predicts that accuracy does not: headcount
A mid-size lender: 1,000 credit files per month, averaging 12 documents each = 12,000 documents per month. Average analyst time to open, check and correct one flagged document: 6 minutes.
At 60% STP:
- Documents touched: 12,000 × 40% = 4,800
- Time: 4,800 × 6 = 28,800 minutes = 480 hours
- At 160 productive hours per analyst-month: 480 ÷ 160 = 3.0 FTE
At 85% STP:
- Documents touched: 12,000 × 15% = 1,800
- Time: 1,800 × 6 = 10,800 minutes = 180 hours
- 180 ÷ 160 = 1.125 FTE
Difference: 1.9 FTE, from a 25-point STP difference at identical field-level accuracy.
Now price the other side. At threshold B the escaped-error rate is 6.8% against 0.9% — on 12,000 documents a month, 816 wrong documents going through untouched instead of 108. Whether 1.9 FTE is worth 708 extra unexamined documents is a credit policy decision, not a procurement one, and it is the real trade behind any comparison of two vendors' STP figures. For the time-saving side, see how much analyst time automated spreading saves per file.
A protocol for measuring both on your own files
Eight steps, roughly a week for two people, and the only thing here that yields a number you can trust.
- Freeze a sample from your own book. 300–500 documents, stratified by document type, source quality (native PDF / clean scan / photograph), period and entity complexity. Record the strata.
- Measure your ground-truth ceiling first. Two analysts spread 30 documents independently. The fields they disagree on define the maximum achievable accuracy. If that is 97%, no vendor scoring 99% against one junior's spread is telling you anything.
- Publish the field list and normalisation rules before the test — sign conventions, scale, period labelling, chart-of-accounts mapping — or every scoring dispute becomes a definitional argument after the fact.
- Run every vendor on the identical set, as shipped. Confidence gating on, validation on, no tuning against your test set, no vendor-side human in the loop.
- Record four things per document: fields wrong, whether the system flagged it, whether the flag was on the right field, and analyst minutes to fix.
- Compute the full set: field-level accuracy, document-level accuracy, STP, escaped-error rate, review precision, mean touch time.
- Re-run at two more threshold settings for the STP / escaped-error curve. A single point is not a result.
- Report per stratum and never blend. A blended number is an artefact of your sample's mix; if phone photographs are 8% of intake today and 20% next year, it was always going to move.
Then run the same protocol on your incumbent manual process. It has an error rate too, and it is rarely measured.
Being honest about our own 95.2%
YuSight states 95.2% extraction accuracy, validated against a manual benchmark. Here is what that does and does not mean.
It is field-level, not document-level. As the table above shows, 0.952¹²⁰ under naive independence is 0.27% — a number that is obviously wrong as a description of real output, which is precisely the point: independence does not hold, errors cluster, and neither the naive product nor the headline percentage describes what lands on an analyst's desk.
The denominator is a manual benchmark — figures spread by analysts, compared field by field against the platform's output. It is a real comparison against human work rather than a synthetic dataset.
What we are not publishing alongside it, and you should ask for: the exact field list and count, the document mix by type and by scan quality, the market split, the abstention rule, and the corresponding straight-through and escaped-error rates.
So the honest position, which applies to us and to every competitor: any single accuracy number without a stated test set, field list and document mix should be treated as marketing. Ours included. The number that should decide your purchase is the one you generate yourself, on your own files, using the protocol above — and the vendor comparison worth doing is the one in Ocrolus vs Docsumo vs YuSight, run on your documents rather than anyone's brochure.
FAQ
How accurate is AI at extracting data from financial statements?
Accurate enough that reading is no longer the bottleneck, and not accurate enough to leave unchecked. Published field-level figures tend to cluster in the mid-90s to low-99s, and they are not comparable to each other because the field lists and document mixes differ.
How is extraction accuracy measured?
By comparing each extracted field against a ground truth and counting agreement. The result depends entirely on what counts as a field, what counts as correct, whether abstentions are excluded from the denominator, and who produced the ground truth. Ask all four questions before you accept a percentage.
What accuracy is good enough for credit decisions?
There is no threshold, because accuracy is not the safety property. What makes a spread safe is that errors are caught: deterministic footing and balance checks, cross-statement ties, and every figure citable to its source page. A 97% system that flags its own doubts is safer than a 99% system that does not.
What is the difference between straight-through rate and automation rate?
Automation rate usually means the share of steps a system performs. Straight-through rate means the share of documents that needed no human at all. A pipeline can automate 90% of the steps on every document and have a straight-through rate of zero.
Why is my straight-through rate lower than the vendor's published figure?
Almost always document mix and policy. Your scan quality, your layout variety, your multi-entity structures and your own rules about which documents must be eyeballed regardless. The vendor's number describes their reference sample, not your intake.
Can a vendor have high accuracy and a low straight-through rate?
Yes, and it is often the right design. A cautious pipeline flags aggressively, so a lot of correct extractions get reviewed anyway. That is a tuning conversation, not a defect — unlike the reverse case, where high throughput is bought by not checking.
What is escaped-error rate and why does nobody quote it?
It is the share of documents that went straight through and were wrong. Nobody quotes it because it is the number that makes a high straight-through rate look expensive. Ask for it, and insist that any STP figure comes paired with it.
How large should a proof-of-concept sample be?
300–500 documents, stratified, is enough to separate vendors on document-level accuracy and STP with sensible confidence. What matters far more than raw size is that the strata mirror your real intake, including the scan quality you actually receive rather than the files you wish you received.
Should I test my current manual process too?
Yes, and it is the step most banks skip. Manual spreading has a measurable error rate and a measurable cost per file, and neither is usually known. Without that baseline you are comparing a vendor's measured number against an unmeasured assumption that your team is perfect.
Conclusion
Three things to take away:
- Field-level accuracy tells you whether the model reads. Straight-through rate tells you what your team will do on Monday. Both are real; only one predicts headcount.
- Never accept an STP figure without an escaped-error rate. STP alone is maximisable by switching off the checks, which is the one change that makes a pipeline worse and its metrics better.
- Every number in this market, ours included, is a description of somebody's document mix. Run the eight-step protocol on 300–500 of your own files and the argument ends.
YuSight's Document Intelligence classifies and validates tax returns, bank statements, financial statements and bureau reports, maps each to the right borrower entity, and hands cited figures to financial spreading, where ratios are computed in code from traceable inputs. Extraction accuracy is measured at 95.2% against a manual benchmark, at field level — and we would rather you tested that on your own documents than took it from this page.
Upload a messy document set and see it classified — book a live demo.
