OCR vs IDP vs LLM Extraction: Which Handles a Credit File Best

None of them, alone. A credit file is five or six document types with different physics. Classical OCR wins on fixed forms, IDP on high-volume repeat layouts, LLM and VLM extraction on layouts nobody trained for. The right architecture is a router that sends each document to the technique that suits it, then checks the arithmetic in code.

That is the whole argument, and the rest of this post is the evidence for it. YuSight's Document Intelligence is built that way and measures 95.2% extraction accuracy against a manual benchmark. If you are choosing an approach for a commercial lending book, the question "which one is most accurate" has no answer until you say on what.

Key facts

• On OmniDocBench — 1,651 annotated PDF pages across 10 document types including financial reports and handwritten notes — the leading models now score above 94%, with GLM-OCR at 94.6% and Gemini 3 Pro at 90.3% (OmniDocBench; LlamaIndex). The same analysis notes the benchmark under-represents "complex financial presentations" — the documents you care about.
• A staged pipeline — image pre-processing, multilingual OCR, section retrieval, then a compact vision-language model — achieved 8.8x higher field-level accuracy than passing whole documents to a large VLM, at 0.7% of the GPU cost and 92.6% lower end-to-end latency (Jin et al., 2025). Architecture beats model size on this task.
• Table-extraction scores are worse than they look. TEDS correlates with human judgement at only r = 0.68, because it "over-penalises representational variation while under-penalising genuine content errors" — a dropped decimal point barely moves the score (Horn & Keuper, 2026).
• Cost separates the approaches by an order of magnitude before any model is involved. Amazon Textract lists $1.50 per 1,000 pages for plain text detection, $15 for tables, $50 for forms and $70 for its lending package, US West (Oregon), first million pages (AWS).
• A trained IDP extractor needs labelled data. Azure AI Document Intelligence requires a minimum of five labelled examples per document type, and its template models need a separate model for every layout variation (Microsoft Learn).

What actually separates the three approaches?

Strip away vendor language and there are three distinct things happening.

Classical OCR converts pixels to characters, and a template or zonal layer says which characters mean what. The rule is positional: "the value in the rectangle 340px from the left, 812px from the top, on page 2, is Gross Salary." It is deterministic. Run it twice and you get the same answer twice. It has no concept of a table, a column header or a fiscal year — only coordinates and characters.

IDP — intelligent document processing — is OCR plus two learned components: a layout model that recognises structure (this region is a table, these cells form a row, this is a header), and a trained extractor that maps regions to named fields. It generalises across layout variation within the distribution it was trained on. Show it a bank statement format it has seen a thousand variants of and it will handle the thousand-and-first. Show it a format from a different banking system and it degrades quietly.

LLM / VLM extraction hands the page — as text, as an image, or both — to a general model and asks for structured output. It has never been trained on your specific document and does not need to be. It reads a column header that says "FY24 (Restated)" and understands what that means, because it understands language. It also has no arithmetic guarantee, no positional certainty, and a well-documented tendency to produce a plausible figure where it cannot read one.

The distinction that matters commercially is not accuracy. It is what each does when it is wrong, which we come to below.

Why does classical OCR fail on a financial statement?

Because a financial statement is not a form. Four properties break positional extraction:

1. The layout is authored, not issued. Every audit firm sets its own house style, and the same firm changes it between years. Add a "Restated" comparative column and every zone to its right shifts.
2. Tables are frequently borderless. Indian and UAE audited accounts routinely present the P&L with no vertical rules at all. Column boundaries exist only as whitespace, which a coordinate rule cannot see.
3. Rows wrap. "Changes in inventories of finished goods, work-in-progress and stock-in-trade" occupies two or three physical lines. A zonal rule keyed to line position picks up the fragment, not the item.
4. Statements continue across pages. The expense schedule starts on page 14 and finishes on page 15. Positional rules have no notion of continuation.

Where classical OCR is genuinely excellent: fixed-format government and bureau output. An ITR-V acknowledgement, a Form 26AS extract, a GST return in the standard portal PDF, a bureau report in a fixed template. On those, template OCR runs at very high precision, costs almost nothing, and — importantly — is auditable line by line. Do not replace it with a model because models are fashionable. Our walkthrough of Form 26AS analysis for lenders covers exactly this class of document.

What does IDP add, and what does it cost you?

IDP adds tolerance for variation, and charges you in labelled data and in silence.

The training requirement is real and quantifiable. Azure's documentation is unusually candid about the trade-off: template models train in 1 to 5 minutes but need "a model per each variation"; neural models handle "structured, semi-structured and unstructured" documents with a single model, but take 30 minutes to 12 hours to train and accept up to 50,000 pages of training data. Both require a minimum of five labelled examples, and Microsoft explicitly recommends "more than five training documents if your images are low quality."

Translate that to a lending book. If you accept statements from 40 banks and each bank has three statement generations in circulation, template IDP is 120 models to build and maintain. Neural IDP is one model — but its accuracy on any given bank is a function of how many examples of that bank were in the training set, and nothing in the output tells you which banks were thin.

That is the hidden cost. IDP accuracy is a portfolio-weighted average over your document mix. Change the mix — enter a new geography, onboard a channel partner whose borrowers bank elsewhere — and the number you validated no longer describes your book.

Where does LLM and VLM extraction win, and where does it break?

It wins on exactly the case the other two cannot address: a layout nobody has ever seen, where the meaning is carried by language rather than position. A qualified audit opinion in prose. A related-party note that names three entities. A column headed "FY24 (as previously reported)". A statement in Arabic with English line items. A promoter's hand-written cash-flow annexure.

It breaks in three specific places.

Arithmetic. There is no calculator inside a language model; there is next-token prediction. Reading a number off a table and computing a ratio from four numbers are different tasks with very different reliability, a gap we set out in detail in how to stop an AI credit memo tool from hallucinating numbers.

Grounding. A model asked to cite will happily emit "page 14". That string is generated the same way the figure was. Only a citation produced by the extraction layer — a page number and a bounding box — is evidence rather than assertion.

Self-consistency. This is the underrated one. When a model encounters a column that does not foot, it tends to make it foot. That behaviour is helpful in a chatbot and dangerous in a spread, because it destroys the single most useful error signal a credit reviewer has.

Worked example: one page of a P&L, three ways

Page 14 of a scanned audited P&L. Mid-market manufacturer, ₹ in lakh, three columns: FY25 (audited), FY24 (restated), FY24 (as previously reported). The audit firm added the restated column this year. The expense schedule continues on page 15.

The true figures for FY25:

Component check: 5,238.90 − 142.30 + 812.40 + 268.70 + 341.20 + 1,406.80 = 7,925.70. PBT = 8,509.00 − 7,925.70 = 583.30.

Classical OCR with last year's template. The zone for "current year" was calibrated on the FY24 statement, which had two columns. The new statement has three. The current-year zone now sits over the middle column. Revenue reads 7,905.10. Total income reads 7,976.40. Total expenses reads 7,551.20. PBT reads 425.20.

Every figure is real, correctly read, and belongs to the wrong year. The column foots. The spread shows revenue flat year-on-year instead of up 6.4%, and PBT down 27%. Nothing in the output is flagged, because nothing is internally wrong.

IDP with a trained neural extractor. The layout model detects three columns and reads the headers, so the year attribution is correct: revenue 8,412.60. But the wrapped inventories row is split into two detected rows, and the trained extractor — whose training corpus was invoices and bank statements, where parenthesised negatives are rare — reads (142.30) as +142.30.

Component sum now = 5,238.90 + 142.30 + 812.40 + 268.70 + 341.20 + 1,406.80 = 8,210.30, against a printed total of 7,925.70. A break of 284.60, exactly twice the misread figure.

This is a worse extraction than the OCR run on one field and a far better outcome overall, because the error is loud. A footing check catches it in code, before a human sees the spread.

LLM / VLM in a single pass. The model reads the column headers semantically, gets FY25 right, gets the sign convention on inventories right, and stitches the page-15 continuation. On the six expense lines it is correct on five. The Finance costs line has a scan artefact across the last two digits; the model reads 263.70 instead of 268.70.

Components now sum to 7,920.70 against a printed 7,925.70 — a break of 5.00. The model resolves it. It adjusts Other expenses to 1,411.80 so the column foots, and returns a clean, complete, internally consistent P&L with the note "extracted from page 14".

Two figures are wrong. The page foots. Nothing is flagged. And interest coverage moves:

• Correct: (583.30 + 268.70) ÷ 268.70 = 852.00 ÷ 268.70 = 3.17x
• As extracted: (583.30 + 263.70) ÷ 263.70 = 847.00 ÷ 263.70 = 3.21x

Four basis points of coverage is not a credit decision. The precedent is. A pipeline that silently repairs its own inconsistencies has removed your detection mechanism, and the next artefact may land on Finance costs in a covenant test rather than a ratio table.

What the staged pipeline does with the same page. OCR returns characters with per-character confidence and coordinates. The layout model returns the table grid. The model resolves column semantics — which is the one job it is uniquely good at. Then, before anything is written to the spread: the component sum is compared to the printed total, in code; the low-confidence Finance costs field is flagged rather than inferred; and every accepted figure carries a page-and-box citation. Result: one field routed to an analyst, eleven fields clean and cited, zero silent errors.

How do the three actually compare?

What actually determines accuracy in practice?

Not the technique. These six, roughly in order of impact.

1. Native PDF vs scan vs photograph. A native PDF has a text layer with exact character positions; extraction from it is a parsing problem, not a recognition problem. A 200 dpi phone photograph of a stapled statement, shot at an angle under fluorescent light, is a different task entirely. Never quote one accuracy number across both.
2. Layout variance across your book. Count your distinct formats. Twelve bank formats is an IDP problem. Two hundred audit-firm house styles is not.
3. Table structure. Borderless tables, spanning cells, nested hierarchies and multi-level headers are where table recognition scores fall apart — and where the standard metrics mislead. OmniDocBench annotates six separate table-level tags precisely because "table accuracy" is not one thing.
4. Multi-page continuation. Schedules and notes that span pages require the pipeline to carry state. This is a pipeline design decision, not a model capability.
5. Handwriting. Promoter annotations, manual stock statements, hand-tallied cash books. Print and handwriting recognition are separate problems with a large accuracy gap between them.
6. Language and script. Arabic-English mixed statements in the UAE, regional-language headers in Indian filings, and Gulf and European thousands separators — 1.234.567 read as 1.234 is a 1,000x error that foots perfectly.

Why a single accuracy percentage is meaningless

"99% accurate" is not a claim. It is a shape.

Accurate at what unit? Character, field, table cell, page or document. Character accuracy of 99% on a page with 3,000 characters means roughly 30 wrong characters per page. That is a different world from 99% of fields.

Against which ground truth? Two experienced analysts spreading the same audited accounts will disagree on several judgement fields. If a vendor's ground truth is one junior's spread, the benchmark measures agreement with that junior.

On which document mix? The single biggest lever. A benchmark weighted towards clean native PDFs will beat one weighted towards photographed statements by a wide margin, on identical software.

And field-level is not document-level. The compounding is unforgiving if fields were independent:

Read the last row honestly, including against our own number. At 95.2% field-level across a 40-field spread, naive independence implies only 14% of documents come out perfectly clean — which is why we quote a field-level figure and say so.

But independence is the wrong assumption, and it cuts both ways. Extraction errors cluster: one badly scanned page produces eight errors in one document while thirty other documents come out spotless. Real document-level accuracy is therefore materially higher than the product of field-level accuracy — and there is no way to derive it. You have to measure it, on your files. Ask every vendor, including us, for both numbers and for the mix behind them. The clustering also tells you where to spend: fixing scan quality at intake moves the document-level number far more than any model change.

So what should a credit file pipeline look like?

Five stages, in this order.

1. Classify and route. Identify the document type first — ITR, bank statement, audited financials, bureau report, stock statement, GST/VAT return, trade licence — and send each to the technique that suits it. This is the decision the whole architecture turns on.
2. Assign to the right entity and period. In a group borrower this must happen before extraction, not after, or you will spread a subsidiary's statement into the parent. That is a substantial problem in its own right; see multi-entity document mapping.
3. Extract with grounding. Every accepted figure carries a page reference and a bounding box, produced by the extraction layer rather than written by a model.
4. Validate deterministically. Footing checks, balance-sheet balance, cross-statement ties, sign conventions, scale and unit checks. In code, before a human sees anything.
5. Route exceptions to a person. Low confidence, failed check or genuine ambiguity goes to an analyst with the page open and the field highlighted. A system that fails loudly is safer than one that is right more often and fails silently.

Then, and only then, financial spreading and ratio computation — in code, from cited inputs — feeding a credit assessment memo where every figure clicks back to its source page.

Where does YuSight sit?

YuSight's Document Intelligence classifies and validates tax returns, bank statements, financial statements and bureau reports, maps each to the right borrower entity in multi-entity structures, and hands cited figures to Financial Spreading, which computes DSCR, leverage, liquidity and profitability ratios deterministically and keeps every figure traceable to source document and page. Extraction accuracy is measured at 95.2% against a manual benchmark, at field level, and spreads stay analyst-editable with full version history.

We are not going to tell you a single technique wins. The file mix decides, the scan quality decides more than the model does, and the only number that describes your book is the one you measure on your own files.

FAQ

What is the difference between OCR, IDP and LLM extraction?

OCR turns pixels into characters and uses positional rules to say what they mean. IDP adds a layout model and a trained extractor so it tolerates variation within the formats it has learned. LLM and VLM extraction reads the page the way a person would, which is why it copes with layouts nobody trained for.

Why does OCR fail on financial statements?

Because a financial statement is authored, not issued on a fixed form. Audit firms change house style, tables are often borderless, long line items wrap across lines, and schedules continue onto the next page. A coordinate rule has no defence against any of that.

Which approach handles unseen layouts best?

LLM and VLM extraction, comfortably — it is the only one of the three that generalises by design. The catch is that it has no arithmetic guarantee and tends to make an inconsistent page foot rather than telling you it could not read a line.

Is a large language model more accurate than a purpose-built IDP model?

On unfamiliar documents, usually yes. On a high-volume format the IDP model was trained on, usually no, and the IDP model is far cheaper and faster. Published work also shows a staged pipeline beating a single large model by 8.8x on field accuracy, so the framing of one versus the other is the wrong framing.

How much does document extraction cost per page?

Plain text OCR runs from roughly $1.50 per 1,000 pages on hosted services, table extraction around $15, form extraction around $50. Model-based extraction is typically an order of magnitude more and varies with page length, so price it on your own document mix rather than on a list rate.

What accuracy should I expect on a scanned bank statement versus a native PDF?

Treat them as two different systems. A native PDF has a real text layer, so extraction is parsing; a photographed statement is recognition, and accuracy falls with resolution, skew, shadow and creasing. Any vendor quoting one number across both is averaging away the thing you need to know.

What is the difference between field-level and document-level accuracy?

Field-level is the share of individual figures extracted correctly. Document-level is the share of documents with no errors at all. On a 40-field spread at 98% per field, only about 45% of documents would be perfectly clean if errors were independent, so always ask for both numbers.

How should I run a proof of concept on document extraction?

Use 30 to 50 of your own files, stratified to your real mix, with ground truth frozen by two analysts before the vendor sees anything. Include your worst scan deliberately, and score not just whether the system got it right but whether it knew when it had not.

Do I still need a template-based approach for anything?

Yes. Fixed government and bureau output — ITR-V, Form 26AS, GST returns, standard bureau templates — is where template extraction is cheapest, fastest and most auditable. Replacing it with a model buys you nothing and costs you determinism.

Key takeaways

• A credit file is five or six document types with different physics. Asking which single technique is most accurate is asking the wrong question.
• Classical OCR is excellent on fixed government and bureau forms and fails silently the moment a layout shifts.
• IDP tolerates variation inside its training distribution, needs a minimum of five labelled examples per type, and quotes an accuracy that is really a weighted average over your document mix.
• LLM and VLM extraction is the only approach that handles unseen layouts, and its worst habit is making an inconsistent page foot instead of flagging it.
• Failure mode matters more than headline accuracy. A loud error is caught in code; a silent one reaches committee.
• Scan quality, layout variance, table structure, multi-page continuation, handwriting and language determine accuracy far more than the choice of technique.
• A single accuracy percentage with no unit, no ground truth and no document mix is unfalsifiable. Ask for field-level and document-level, and measure both on your own files.

Upload a messy document set and see it classified — [book a live demo](https://yuverse.ai/yusight) and watch the router pick a different path for the ITR, the bank statement and the audited accounts.