AI Hallucination Risk in Credit Analysis: The Architecture That Prevents It
The architecture that prevents it puts the language model outside the numeric path entirely. Figures are located by an extractor that emits a page-bound citation as part of the value. Ratios are computed in code. Sources are reconciled against each other. Low-confidence fields block rather than fill. The model writes prose around numbers it cannot alter.
That is a system design, not a setting. The measurable consequence in YuSight is that 100% of figures in a CAM carry a citation with one-click source verification — because there is no code path that produces a figure without one. This page is about the design that makes that true. The catalogue of what goes wrong is separate: see how do you stop an AI credit memo tool from hallucinating numbers for the failure modes and the controls. This page is for someone evaluating or building the system.
Key facts
- The regulator's own word for it is "confabulation." NIST defines it as "the production of confidently stated but erroneous or false content", and lists it as one of twelve generative-AI risk categories (NIST AI 600-1, *Generative AI Profile*, July 2024). Confident and erroneous is the combination the architecture has to defeat.
- Do not make the model the checker. In FinVerBench, a study of LLM financial-statement verification across 14 model runs on SEC 10-K filings from 43 S&P 500 companies, nine of fourteen models produced 95–100% false positives on clean statements under the original prompt — flagging errors in filings that had none (Panda, 2026).
- Architecture beats prompting, measurably. A three-stage pipeline of finance-aware retrieval, atomic claim decomposition and grounded regeneration cut hallucination by 78% relative to GPT-4o, and by 68% over the strongest baseline once retrieval quality was equalised — ending at a 4.1% average hallucination rate (Guo, Wu and Yiu, *FinGround*, 2025).
- There is no supervisory safe harbour to fall back on. SR 26-2 / OCC Bulletin 2026-13 (17 April 2026) states that "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance" (Federal Reserve; OCC). Your architecture is the control.
- YuSight's CAM Generation module cites every figure, computes ratios in code from those cited inputs, keeps spreads analyst-editable and records full version history.
What is the trust boundary, and where does it sit?
Every design decision below follows from one line: the language model is untrusted for values and trusted only for language.
That sounds like a policy. It is an engineering constraint, and it has a test. Draw a boundary around the generative component and ask what it is capable of writing into the memo. If it can emit a numeral that reaches the credit file without passing through a value object it did not create, the boundary is decorative. If it cannot — if numeric slots are filled by a renderer from typed objects and the model's output is only the connective text between them — the boundary is real.
Two consequences fall out immediately, and both are counter-intuitive.
Better models do not fix this. A model that hallucinates less still hallucinates, and the residual failures are the plausible ones — right order of magnitude, right format, consistent with their neighbours. Model improvement moves the rate; architecture moves the class of failure that can reach the file.
The model is a poor auditor of its own domain. The FinVerBench result is the important one here: nine of fourteen models flagged errors in clean statements almost every time. A design that adds "and then we ask the model to check it" has added noise and a false sense of coverage, not a control.
What does the reference architecture look like?
Seven components, three zones, and one direction of flow.
ZONE 1 — DETERMINISTIC INGEST ZONE 2 — TYPED VALUE STORE ZONE 3 — GENERATIVE
┌──────────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────┐
│ [1] Classifier │ │ [4] Value objects │ │ [6] Narrative │
│ doc type + confidence │───┐ │ {value, unit, scale, │ │ assembly │
│ [2] Entity resolver │ │ │ doc_id, page, region, │ │ templates + │
│ doc → legal entity/period│───┼───▶│ confidence, method} │────▶│ typed slots │
│ [3] Extractor │ │ │ [5] Compute layer │ │ READ-ONLY on │
│ value + page + region │───┘ │ formulas in code, │ │ every numeric │
│ + confidence per field │ │ units normalised, │ │ slot │
└──────────────────────────────┘ │ reconciliation graph │ └──────────────────┘
│ └──────────────────────────┘ │
│ │ │
└──────────── [7] GATE ───────────────┴──────────────────────────────┘
auto-accept │ analyst queue │ HARD BLOCK
│
▼
[8] Review surface + version history
The arrows only ever point right. Nothing in Zone 3 writes back into Zone 2. That is the whole design.
# | Component | Owns | Must never |
|---|---|---|---|
1 | Classifier | Document type, period, quality score | Guess a type below threshold rather than routing it |
2 | Entity resolver | Binding each document to a legal entity | Default to the primary borrower when uncertain |
3 | Extractor | Locating a value and emitting its provenance | Return a value without page, region and confidence |
4 | Value store | Typed, immutable value objects | Accept a numeric literal with no provenance |
5 | Compute layer | Ratios, normalisation, reconciliation | Call a language model |
6 | Narrative assembly | Prose, ordering, template selection | Write into a numeric slot |
7 | Gate | Routing on confidence and validation state | Downgrade a hard block to a caveat |
8 | Review surface | Making every figure clickable and every edit logged | Present machine output and human output identically |
Component 5's "must never" is the one vendors quietly break. A compute layer that calls a model to "interpret" an ambiguous line item has moved the trust boundary without telling anyone.
What is allowed to cross each boundary?
An architecture is only as good as its interface contracts. Three boundaries, three contracts.
Boundary | What may cross | What may never cross | How it is enforced |
|---|---|---|---|
Document → value store | A value object carrying value, unit, scale, doc_id, page, region, extraction method and confidence | A bare number; a value whose page reference was produced by a model | Type system: the constructor requires provenance, so a value without it cannot be instantiated |
Value store → compute | Value objects and a named formula from a versioned registry | An ad-hoc formula string; a value below the gate threshold | Formula registry lookup by ID; gate check before evaluation |
Compute → narrative | Resolved, rendered slot values and their provenance chains | Write access of any kind back into the store | Slot renderer runs after generation, replacing placeholders in the model's output |
The third row is the subtle one. If the model generates text containing numbers and a post-processor tries to verify them, you are doing post-hoc attribution — the architecture that the citation-faithfulness literature keeps finding unreliable. If the model generates text containing placeholders and the renderer substitutes bound values afterwards, the model never had the opportunity to invent a figure. The mechanics of that substitution are set out in how source-cited memo drafting actually works.
How does the compute layer remove a whole class of error?
By making scale and unit a property of the value rather than an assumption in the reader's head.
Consider a statement whose header reads "₹ in lakhs". The extractor reads the token 4,820 from the total-assets line. What it stores is not 4820. It is:
{ magnitude: 4820, unit: "INR", scale: "lakh", canonical: 482000000,
doc_id: FS_FY25, page: 10, region: [...], confidence: 0.981,
scale_source: "header, p.10, region [...]" }
The scale carries its own citation. If the header is misread as "millions", the canonical value becomes 4,820 × 1,000,000 = 4,820,000,000 instead of 4,820 × 100,000 = 482,000,000 — a uniform 10x inflation.
Here is why this needs an architectural answer rather than a validation rule. Every internal check still passes:
- Cross-foot: total assets 4,820 = total liabilities 2,940 + equity 1,880. Passes at any scale.
- Ratio consistency: TOL/TNW = 2,940 ÷ 1,880 = 1.5638 → 1.56x. Scale cancels. Passes.
- Year-on-year growth: 4,820 ÷ 4,310 = 1.1183 → 11.8%. Scale cancels. Passes.
Every ratio in the memo is correct. Every absolute figure is wrong by a factor of ten. Internal consistency cannot detect a uniform scaling error, because ratios are scale-invariant — which is exactly what makes them useful and exactly what makes them blind here.
Only an external magnitude anchor catches it: total bank credits for the year of ₹6.9 crore against declared assets of ₹482 crore is a 70x mismatch that no manufacturer's balance sheet supports. That anchor lives in the reconciliation graph, not in the statement. Which is the argument for the next component.
What does cross-source reconciliation look like as a component?
Not "the analyst checks turnover against the bank statement". A graph of claims about the same quantity, drawn from independent origins, with tolerance bands and an explanation search.
Worked example. FY2025, mid-market distributor. Three independent claims about annual turnover.
Claim A — audited financial statements. Revenue from operations: 62,400,000
Claim B — bank statements, all accounts, normalised.
Line | Amount |
|---|---|
Gross credits, all accounts | 71,240,000 |
less inter-account transfers | (8,610,000) |
less term loan disbursement received | (3,400,000) |
less promoter capital infusion | (1,300,000) |
Adjusted operating credits | 57,930,000 |
71,240,000 − 8,610,000 − 3,400,000 − 1,300,000 = 57,930,000
Claim C — tax filings. Aggregate taxable supplies declared: 61,150,000
Pairwise variance, against declared revenue as the base:
- A vs C: 62,400,000 − 61,150,000 = 1,250,000 → 1,250,000 ÷ 62,400,000 = 2.00%
- A vs B: 62,400,000 − 57,930,000 = 4,470,000 → 4,470,000 ÷ 62,400,000 = 7.16%
Tolerance bands (illustrative, set by policy): ≤ 5% auto-accept, 5–12% flag for explanation, > 12% hard block. Bands belong in your credit policy, not in a vendor default.
A vs C at 2.00% clears. A vs B at 7.16% flags. Two of three sources agree closely, and the third is low — which is a shape, not yet a finding.
The explanation search runs automatically. The most common benign cause of books-above-bank is a receivables build: revenue recognised but not yet collected. The engine pulls the balance sheet movement.
Line | FY2024 | FY2025 | Movement |
|---|---|---|---|
Trade receivables | 6,120,000 | 10,480,000 | +4,360,000 |
Residual after the receivables movement = 4,470,000 − 4,360,000 = 110,000 110,000 ÷ 62,400,000 = 0.18% → within the auto-accept band.
The exception closes with a stated, cited explanation: turnover variance of 7.16% against banked credits is accounted for by a ₹43.6 lakh increase in trade receivables (p. 10), leaving an unexplained residual of 0.18%. That sentence, with its two citations, is what a credit committee needs — and none of it was generated by a language model.
Note the second-order finding the reconciliation hands the analyst for free: receivables grew 71.2% (10,480,000 ÷ 6,120,000 = 1.7124) against turnover growth that was far slower. The variance is explained and the underlying signal is a working-capital deterioration. Reconciliation resolves the arithmetic; it does not resolve the credit question. The related discipline for bureau and banking data is in bureau vs bank statement reconciliation.
The design requirement that makes this work: independence of origin. Three claims are only three claims if they come from three places. A "reconciliation" between a spread and a ratio computed from that same spread is a tautology dressed as a control.
Where does confidence gating actually gate?
Gating is a state machine, and its value lies entirely in whether state 3 exists.
State | Trigger | Behaviour |
|---|---|---|
Auto-accept | Character confidence above threshold, layout match, cross-foot clean, reconciliation within band | Value flows to compute and narrative |
Analyst queue | Any single soft signal fails — poor scan quality, unusual layout, variance in the flag band | Value is held; the field renders in the review surface with the source page open beside it |
Hard block | Cross-foot failure, reconciliation beyond the block band, missing scale header, entity resolution below threshold | Downstream computation does not run. No ratio, no sentence, no memo section |
The hard block is where architectures separate. A system that produces a DSCR with a warning label has made the warning the analyst's problem; a system that refuses to produce it has made the missing input the analyst's problem, which is the correct problem. A cross-foot failure means the balance sheet the ratio would be built on does not balance — a DSCR computed on top of that is a number, not information.
The single most useful thing to ask at a demo is not "what is your accuracy". It is: show me a field your system refused to fill, and show me what the memo looks like with that hole in it. A tool that always produces an answer has no state 3. The workflow that sits around states 2 and 3 is designed in human-in-the-loop credit review.
What must the review surface expose?
The review surface is a load-bearing architectural component, not a UI layer. If a reviewer cannot exercise the provenance the pipeline generated, the pipeline generated it for nothing.
Four requirements:
- One-click resolution from figure to source region. Not to the document. Not to the page. To the highlighted region on the page, because "somewhere on page 41" is a forty-second task and forty seconds times a hundred figures is nobody's afternoon.
- The formula visible beside the ratio. DSCR at 1.31x means nothing until you can see whether the denominator used interest on term debt or total finance costs. Definitional variance is argued in DSCR formula: every variant lenders use.
- Machine output visually distinct from human output. A reviewer must be able to see, without asking, which figures the system produced and which an analyst overrode.
- A diff against the first machine draft. The examiner's question is not "is this memo right" but "what did the system say, what did the person change, and why". That is a record, and it has to be queryable.
Where does this architecture still fail?
Honestly, and in order of how often it matters.
Residual risk | Why the architecture cannot close it | Compensating control |
|---|---|---|
Definitional mismatch — every citation correct, wrong formula for your policy | The system computes what its registry says; the registry encodes someone's definition | Formula registry reviewed and signed off by credit policy, versioned, displayed with every ratio |
Wrong-entity binding — a genuine figure from a genuine document, mapped to the wrong company | Every citation resolves correctly, so no provenance check fires | Entity resolution confidence gated at a higher threshold than field extraction; see multi-entity document mapping |
Correlated sources — three "independent" claims that all originate from the borrower's own accounting system | Reconciliation assumes independence and cannot verify it | Source-origin tagging; require at least one third-party-issued source (bureau, tax authority, bank) in every reconciliation |
Forged or altered source document | Provenance proves where a figure came from, never that the document is genuine | Document authenticity checks upstream of extraction; see how do you detect a fake or tampered bank statement |
Interpretation error in the narrative | The model may write prose that misdescribes correct figures — "improving" a ratio that fell | Direction and comparison words resolved by computation, not word choice; reviewer reads the narrative, not the table |
The honest summary of all five: this architecture makes figures checkable and constrains what class of error can reach a credit file. It does not make the analysis correct. Any vendor claiming otherwise has told you something useful about the vendor.
How do you tell whether a vendor has actually built this?
Seed defects and watch which gate catches them. Six tests, each aimed at a specific component.
# | Seeded defect | Component that should catch it | Failure signal |
|---|---|---|---|
1 | Statement with header scale changed from lakhs to millions | Reconciliation graph, magnitude anchor | Memo produced with 10x figures and clean ratios |
2 | Balance sheet altered so assets ≠ liabilities + equity by 40,000 | Cross-foot validation → hard block | A DSCR is still produced |
3 | Prior-year column values swapped into current year | Period labelling in the value object | Trend commentary reads normally |
4 | A document from a sister entity added to the file | Entity resolver | Its turnover appears in the borrower's spread |
5 | A page removed from the middle of a bank statement | Sequence and closing-balance continuity check | Average balance computed on the gap |
6 | A figure that appears nowhere in the file requested in the memo | Narrative assembly boundary | A number appears in the prose |
Test 6 is the boundary test. Ask the system to comment on something the document set does not support — contingent liabilities where the notes disclose only prose, say. A correctly built system says the field is unavailable. A system with a leaky boundary produces a plausible number.
Run these on your own files, not the vendor's demo set. The measurement design — field-level versus document-level accuracy, and why the two numbers diverge — is set out in extraction accuracy vs straight-through rate and OCR vs IDP vs LLM extraction.
FAQ
How do you stop an AI credit memo tool from hallucinating numbers?
Architecturally. The model is kept off the numeric path: an extractor locates each figure and emits its page and region as part of the value, ratios are computed in code, and the narrative layer fills typed slots it cannot write to. Prompting is not a control.
Can an AI memo invent a ratio?
Not if the ratio is computed by a formula registry in code from cited inputs. It can invent one if the system asks a language model to compute or restate it, which is why the useful question at a demo is where the arithmetic actually happens.
How do citations prove a figure came from a document?
A citation is only evidence if the extractor produced it at the moment it read the value — page, region and confidence bound into the value object. A page reference written by a model, or found afterwards by searching for a matching number, proves nothing about origin.
Why not just ask the model to check its own work?
Because it is a poor checker of exactly this. In FinVerBench, nine of fourteen model runs flagged errors in clean financial statements 95–100% of the time. Self-checking adds noise and a false sense of coverage.
Does a low temperature setting reduce hallucination?
It reduces variation, not error. Temperature zero returns the same answer every time, including the same wrong answer. Determinism and correctness are different properties.
What should the system do when it cannot read a figure?
Refuse to produce anything that depends on it. A hard block on a cross-foot failure or a missing scale header is more useful than a ratio with a caveat, because caveats get read past and holes do not.
How does reconciliation catch what citations miss?
Citations prove where a number came from; reconciliation tests whether independent sources agree about the same quantity. Turnover in the audited accounts, credits in the bank statements and supplies in the tax filings come from three different origins, so a disagreement is a real signal.
Does any of this satisfy model risk management requirements?
Not automatically. SR 26-2 and OCC 2026-13 place generative and agentic AI outside their scope, which leaves the governance to you rather than granting a pass. Deterministic components — rules and formulas — are also outside the model definition, but still need change control and testing.
Can the analyst still override the system?
Yes, and the architecture depends on it. Overrides are the point of the review surface. What matters is that every override is recorded with author, timestamp, the field changed and, for figures, a reason.
Key takeaways
- The trust boundary is the design: the model is untrusted for values and trusted only for language. Test it by asking whether the model can emit a numeral that reaches the file.
- Provenance must be a property of the value object, created by the extractor. A citation added later is a search result, not evidence.
- Scale errors survive every internal consistency check, because ratios are scale-invariant. Only an external magnitude anchor catches them.
- Reconciliation is a component with tolerance bands and an automated explanation search — and it only works if the sources are genuinely independent in origin.
- Confidence gating is worthless without a hard block. Ask a vendor to show you a field their system refused to fill.
- The architecture makes figures checkable. It does not make the analysis correct — five residual risks stay open, and they are the ones worth staffing.
YuSight's CAM Generation module is built on this shape: extraction with provenance, ratios computed in code from cited inputs, cross-source reconciliation through the Repayment Tracker and Bank Statement Analyzer, analyst-editable spreads, and full version history. Every figure in a YuSight CAM carries a citation with one-click source verification — not a claim that it is never wrong, a guarantee that it is always checkable.
See your first CAM in 30 minutes — book a live demo.
This article is general information for credit and risk professionals. Tolerance bands, thresholds and the worked reconciliation are illustrative and should be calibrated to your own policy.
