A commercial lender at a mid-sized regional bank was reviewing an AI-generated credit memo for a $4.2 million equipment loan. The memo was well-organized, the prose was clean, and the debt-service coverage ratio analysis read like something a seasoned credit analyst had written. Then she reached the third paragraph of the collateral section, which stated that the borrower had pledged "a first-lien blanket UCC filing over all inventory, receivables, and equipment, consistent with the January 2024 master security agreement." There was no January 2024 master security agreement in the file. There was no blanket UCC filing. The AI had generated a collateral description that sounded exactly right for a file of that type and was entirely invented. Had that memo gone to the credit committee without verification, the bank might have approved a $4.2 million loan believing it had collateral protection that did not exist. This story is not unusual. It is, in fact, exactly what generative AI does when it is not properly grounded, and understanding why it does this is the entire point of this lesson.

What Generative AI Actually Is

A generative AI model, specifically a large language model (LLM), is a statistical system trained to predict what text should come next given the text that came before it. That is the complete mechanical description. The sophistication comes from the scale: modern LLMs are trained on hundreds of billions of words drawn from books, web pages, academic papers, code, and other text corpora. The model learns, through billions of training examples, the statistical relationships between words, phrases, sentences, and concepts. When you ask it a question, it uses those learned relationships to generate a response that is statistically consistent with the patterns in its training data.

This is not the same as retrieving information from a database. A database query finds a record that exists. An LLM generates text that is consistent with patterns it has learned. When those patterns align with reality, the output is accurate. When they do not (because the training data was wrong, because the situation is unusual, because the model is generalizing from similar but different cases), the output sounds exactly the same as when they do. The model has no internal flag that distinguishes accurate outputs from generated ones that happen to be plausible but wrong. It produces both with identical confidence.

The key term here is "token." A token is roughly a word or a word fragment (in practice, about 3 to 4 characters of text on average). An LLM processes text as a sequence of tokens and predicts the next token given all preceding tokens. The model's "attention mechanism" (the component that decides which earlier tokens are most relevant to predicting the next one) allows it to maintain coherence across a long document. When the model drafts a credit memo, it is generating tokens one at a time, each chosen as the statistically most likely continuation given the prompt and everything generated so far. There is no lookup. There is no verification. There is only prediction.

A large language model does not know things. It predicts what text a knowledgeable source would have written. In a regulated lending context, plausible and accurate are different standards, and only one of them survives an examination.

How Training Shapes What the Model Knows and Does Not Know

Understanding training is the key to understanding why generative AI produces the specific failure modes it does in lending contexts.

Training a large language model involves exposing it to enormous volumes of text and adjusting its internal parameters to improve its prediction accuracy. After training, the model's "knowledge" is encoded in those parameters: billions of numerical weights that together produce text responses. This has two important implications for a lender using the technology.

First, the model's training data has a cutoff date. Everything the model knows, it learned from text that existed before that cutoff. For a banking professional, this means the model does not know about regulatory guidance issued after its training cutoff, does not know about interest-rate environments that developed after cutoff, and does not know about your institution's specific credit policies unless those policies were included in its training or are provided in the prompt. A model trained through mid-2024 does not know the specific requirements of OCC Bulletin 2026-13. If asked about it, the model will either admit the gap or, in many cases, produce a confident description of what such guidance "would likely say" based on its understanding of the prior framework. Both responses are common. Only the first one is honest. The second one is the failure mode that matters.

Second, the model learned from publicly available text, not from your institution's internal documents. When a loan officer asks an LLM to draft a credit memo for a specific borrower, the model does not have access to the borrower's actual file unless the file is provided in the prompt. If the prompt includes the file, the model can ground its response on actual data. If the prompt does not include the file, the model generates what a credit memo for that type of borrower usually looks like, populated with plausible-sounding figures. The collateral section from the opening of this lesson is a perfect example: the model generated what a collateral section for a commercial equipment loan usually contains, because it had seen many such sections in its training data. It generated text that matched the pattern of real collateral sections without any underlying data to support the specific claims.

Fine-Tuning and Instruction Following

Most LLMs deployed in banking contexts have been fine-tuned through a process called instruction-following training, also sometimes called reinforcement learning from human feedback (RLHF). This process takes a pre-trained LLM and trains it further on examples of helpful, accurate, and appropriate responses to instructions. The result is a model that is better at following instructions (write a memo, summarize this document, answer this question) rather than simply completing text.

Fine-tuning is why a modern LLM can follow a prompt like "review this loan file and identify risks" rather than just completing the sentence. But fine-tuning does not change the fundamental architecture: the model is still predicting tokens, and it is still doing so without a truth-checking mechanism. Fine-tuning can reduce certain types of errors (instructed not to fabricate citations, the model makes fewer citation errors) but it cannot eliminate the statistical nature of generation. A well-fine-tuned model is a more reliable generator. It is not an accurate one in the sense that a database query is accurate.

Many vendors who sell LLM-based tools to banks add what is called a retrieval-augmented generation (RAG) layer. In a RAG setup, before generating a response, the system retrieves relevant documents from a database (the bank's credit policy, the borrower's actual loan file, regulatory guidance documents) and includes them in the prompt. This substantially reduces hallucination in the retrieved domain because the model now has actual text to draw on rather than generating from memory. A RAG system that retrieves the actual paystub before summarizing income is far less likely to fabricate income figures than one working from a prompt that does not include the file. RAG is not a complete solution (the model can still misread retrieved text, weight it incorrectly, or generate beyond what the retrieved text supports) but it is the most important architectural feature to understand when evaluating AI tools for document-heavy lending work.

Why Generative AI Invents Banking Details

The specific failure modes of generative AI in lending are predictable once you understand the architecture. The model produces what is statistically likely. In a credit memo for a commercial equipment loan, a blanket UCC security agreement is statistically common. In a mortgage file summary, an employment verification letter is statistically common. In an adverse-action notice for a high debt-to-income application, the debt-to-income ratio is a statistically common reason. The model puts these elements in because they fit the pattern, not because they are present in the specific file.

This means the model is most likely to hallucinate the specific details that require the most context to get right: the exact covenant language in a specific credit agreement, the specific income figures from a specific paystub, the specific adverse-action reason that matches this particular applicant's credit profile and this particular bank's underwriting criteria. These are exactly the details that matter most in a regulated lending context. The model's statistical confidence is highest precisely where the risk of error is highest, because these details are common in the pattern and the model has no way to distinguish "common in general" from "present in this file."

Consider the adverse-action notice failure mode in detail. Under ECOA and Regulation B, a denial notice must include the specific reasons that drove the decision. A generative AI model asked to draft the notice will produce the reasons that are statistically most common for that type of application (high debt-to-income ratio, insufficient credit history, high loan-to-value ratio). These reasons will be plausible. If the actual decision was driven by a non-standard factor (a specific underwriting exception that was not granted, an undisclosed liability the underwriter found in the income documentation), the AI-generated reasons may not match the actual reasons at all. The notice will be specific. It will not be accurate. The difference is a Regulation B violation.

The Unfair, Deceptive, or Abusive Acts or Practices (UDAAP) framework, enforced by the Consumer Financial Protection Bureau (CFPB) and state regulators, adds another layer of exposure. A borrower who receives an adverse-action notice stating incorrect reasons may later discover the true reason was different. A communication that creates a false impression about why credit was denied is potentially deceptive under UDAAP standards. What started as a model generating plausible text becomes, in a regulatory examination, a deceptive communication practice.

The Verification Imperative in Lending

Everything above leads to a single operational conclusion: in a lending context, every AI-generated output that touches a credit decision or a borrower communication must be verified against the actual source before it is used. This is not a recommendation. Under the legal framework that governs every bank's operations (ECOA, Regulation B, UDAAP, the Community Reinvestment Act (CRA, which requires banks to meet the credit needs of their entire community including low- and moderate-income neighborhoods), the Bank Secrecy Act and Anti-Money Laundering rules (BSA/AML, the federal framework requiring banks to detect and report suspicious financial activity)), the institution is accountable for the accuracy of every statement in a credit file and every notice sent to a borrower.

Verification in this context means three things. First, checking every number against its source document: does the income figure in the credit memo match the actual paystub? Does the debt-to-income ratio calculation use the correct monthly obligations? Does the collateral description match the actual security agreement? Second, checking every regulatory reference against the current text: if the credit memo cites a Regulation B requirement, does that citation match the current regulation? Third, checking that the stated reason matches the actual reason: if an adverse-action notice says the denial was due to insufficient income, was insufficient income actually the operative factor in the underwriter's decision?

This verification discipline is not unique to AI-assisted work. It is the same discipline that a good underwriter applies to any credit file, AI-generated or not. What changes with AI is the failure mode: a human analyst who invents a covenant is making an ethical or competence failure that is relatively rare. An AI model that invents a covenant is operating normally, doing exactly what it was designed to do, producing statistically consistent text. The frequency of the failure mode is higher. The surface area for errors is larger. The verification step becomes more critical, not less, as AI takes on more of the drafting work.

What This Means for How You Use AI Tools

Understanding the architecture shapes how a banker should engage with any LLM-based tool in a lending context. There are four operational habits that follow directly from what you now know about how generative AI works.

Ground the model on the actual file. Any prompt that asks the AI to summarize, analyze, or draft based on a specific loan should include the relevant documents or data. A prompt that says "summarize this borrower's income and employment" should include the paystub, the W-2, and the verification of employment in the prompt context. A model given the actual documents will generate from those documents. A model given only a description ("the borrower is a W-2 employee in the manufacturing sector") will generate from statistical patterns about W-2 employees in manufacturing. The difference between the two is the difference between extraction and fabrication.

Treat every number as unverified until checked. Every numerical figure that appears in AI-generated output (income, ratios, balances, rates, dates, addresses, account numbers) should be treated as a claim to be verified, not as a fact. The verification takes seconds when you have the source document in hand. It is impossible after the credit decision has been made and the notice has been sent.

Match the stated reason to the actual decision logic. For every adverse-action notice drafted with AI assistance, the reviewer must trace the stated reasons back to the actual underwriting notes and confirm that the AI-generated reasons are the same reasons the underwriter applied. This is the single check that prevents the fabricated-reason-code failure mode. It takes one to two minutes on a well-organized file. It is the difference between a compliant adverse-action process and a Regulation B examination finding.

Cite the bulletin, not the model. When AI output references a regulatory requirement (a Reg B timeframe, an OCC model-risk expectation, a UDAAP standard), verify the citation against the current regulatory text before relying on it in a compliance context. LLMs are trained on historical text. OCC Bulletin 2026-13 substantially changed the model-risk landscape. A model whose training predates April 2026 does not know the current requirements and will generate plausible-sounding descriptions of the prior framework. Those descriptions may be wrong in specifically the ways that matter most for your next examination.

Key Takeaways

• A large language model generates text by predicting the statistically most likely next token given the prompt and prior output. It does not look up facts, verify claims, or distinguish accurate outputs from plausible ones. Both types of output are produced with identical confidence and fluency.
• The model's knowledge is encoded in its training data, which has a cutoff date. It does not know about regulatory guidance issued after its cutoff, including OCC Bulletin 2026-13 (April 2026), unless that guidance is provided in the prompt.
• Hallucination in lending occurs because the model produces details that are statistically common for the type of document being drafted, even when those details are not present in the specific file. Invented covenants, fabricated income figures, and wrong adverse-action reason codes are all examples of statistically plausible text that is factually wrong.
• Retrieval-augmented generation (RAG) substantially reduces hallucination by grounding the model on retrieved documents before generating. A RAG system working from the actual paystub is far less likely to invent income figures than one working from a general description of the borrower. RAG is the most important architectural feature to ask about when evaluating any AI tool for lending document work.
• The ECOA and Regulation B requirement that adverse-action reasons be specific and accurate creates a direct legal obligation that collides with the generative model's failure mode: a model will generate the most statistically common adverse-action reasons for that loan type, not necessarily the reasons that actually drove the specific decision.
• The UDAAP framework adds a second layer of exposure: an adverse-action notice stating incorrect reasons may be deceptive if the true reasons were different. "The model generated it" is not a defense. The institution is accountable for the accuracy of every communication with a borrower.
• Every number, covenant, collateral description, regulatory citation, and adverse-action reason in AI-generated lending output must be verified against the actual source before the document is used or sent. This is the minimum professional standard, not an optional quality step.
• The job of the lending professional has shifted from "produce the draft" to "verify the draft." That shift requires exactly the domain expertise and regulatory knowledge that AI cannot replace. The most productive users of generative AI in lending are the most experienced professionals, because they know what accurate looks like.