The fair-lending officer placed the vendor's model explanation report on the conference table at 9 a.m. and asked the room which of the denial reasons was most likely to draw examiner scrutiny. The table had five people: a loan officer, two underwriters, a compliance attorney, and the fair-lending officer herself. The report showed a bar chart of SHAP values (SHapley Additive exPlanations, a method borrowed from game theory that attributes a credit score to contributions from individual input features) for a declined mortgage application. The bars were labeled with variable names: "revol_util_ratio," "months_since_most_recent_delinq," "installment_amt_ratio," "zip_segment_cluster_04." The bars had magnitudes and direction (positive bars pushed the score toward approval; negative bars pushed it toward denial). The largest negative bar, by a significant margin, belonged to "zip_segment_cluster_04." Nobody at the table knew what that variable was. Nobody knew what it measured or how it was constructed. Nobody could say whether a zip-based segmentation cluster introduced a proxy for a protected class. The compliance attorney said she would need to see the variable definition. The fair-lending officer said she would ask the vendor. The vendor's response, when it came back three days later, was that the cluster was constructed from "proprietary geospatial lending performance data" and was not further described in their documentation. By the end of the week, the institution had paused its use of the vendor model and notified its primary regulator. The problem was not the SHAP chart. SHAP was doing exactly what it was designed to do: showing which variables drove the decision. The problem was that nobody in the room had the skills to read what the chart was showing and immediately ask the right follow-up question. This lesson gives you those skills.

Why Explainability Is a Regulatory Artifact, Not a Technical Feature

Explainability in AI-assisted lending is not a nice-to-have feature that model developers add to make their tools more user-friendly. Under the regulatory framework governing credit decisions in the United States, explainability is a legal requirement, and the institution that uses the AI model is the party responsible for satisfying it, regardless of whether the model was built in-house or purchased from a vendor.

ECOA (the Equal Credit Opportunity Act, 15 U.S.C. 1691, the federal statute prohibiting credit discrimination on any basis listed in the Act) and Regulation B (Reg B, 12 CFR Part 1002, the CFPB rule implementing ECOA) require, under Section 1002.9, that a creditor who takes adverse action on a credit application must provide the applicant with a statement of specific reasons for the action, or a disclosure of the applicant's right to obtain such a statement. The statement of specific reasons must actually reflect the factors that drove the decision. "The model scored you below our threshold" is not a specific reason under Regulation B. "Your revolving credit utilization ratio of 87% exceeds our standard maximum of 30%" is a specific reason.

OCC Bulletin 2026-13, the April 2026 interagency model-risk guidance that superseded OCC 2011-12 and placed AI and generative AI explicitly under model-risk, fair-lending, third-party, and board-governance expectations, elevates the explainability requirement beyond adverse-action notices. The bulletin requires institutions to understand the AI models they use well enough to govern them: to test them for disparate impact, to validate their outputs, to document their inputs and logic, and to detect when a model's behavior deviates from its intended use. An institution that cannot read a model explanation output does not have the capability to fulfill these governance obligations, regardless of whether the model's vendor provides an explanation tool.

This is the precise gap the fair-lending officer's conference table illustrated. The institution had a model explanation output. It had a vendor-provided SHAP chart. What it lacked was the ability to read that chart, ask the right questions about what it showed, and identify the specific variable that required immediate escalation. This lesson closes that gap.

Explainability techniques are not a data science curriculum. They are a regulatory literacy requirement: the ability to read a model explanation, identify the variables driving a decision, and ask whether those variables are permitted, accurate, and free of proxy risk, before a decision goes into a credit file.

Reading SHAP and LIME Outputs Without a Statistics Degree

SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic Explanations) are the two most widely deployed model explanation methods in commercial lending AI as of 2026. Understanding their outputs does not require knowing the mathematics behind them. It requires understanding what each method measures and what its outputs mean for a specific credit decision.

SHAP: what it measures and how to read it. SHAP assigns to each input variable a numerical value representing how much that variable contributed to the model's output for a specific applicant, relative to the model's average prediction across all applicants. A positive SHAP value means the variable pushed the score in the direction of approval (increased the probability of approval relative to baseline). A negative SHAP value means the variable pushed the score in the direction of denial (decreased the probability of approval relative to baseline). The magnitude of the SHAP value indicates how large the contribution was: a SHAP value of -0.18 represents a larger contribution toward denial than a SHAP value of -0.03.

Reading a SHAP output for a specific declined application requires four questions, in order:

Question one: what were the three to five largest-magnitude variables, and in which direction did they push the score? The adverse-action reasons in a fair lending context should, at minimum, reflect the variables with the largest negative SHAP values (the factors that pushed most strongly toward denial). If the adverse-action reasons listed on the notice do not correspond to the highest-magnitude negative contributors in the SHAP output, the reasons may not accurately reflect the decision. That is a Regulation B problem.

Question two: can each high-magnitude variable be described in specific, human-readable terms? "revol_util_ratio" can be described as "revolving credit utilization ratio, which is the percentage of available revolving credit that is currently being used." "zip_segment_cluster_04" cannot be described in specific human-readable terms without a variable dictionary. Any high-magnitude variable that cannot be concretely described to a borrower in the adverse-action notice is a variable that cannot satisfy Regulation B's specificity requirement and that may require escalation for proxy-variable review.

Question three: does any high-magnitude variable have a geographic, demographic, or socioeconomic component that could introduce proxy risk? Variables constructed from zip codes, census tracts, neighborhood characteristics, school districts, or any geographically-based index require specific review for proxy risk under ECOA. A zip code variable is not automatically a prohibited factor, but a zip code segmentation cluster constructed from historical lending performance in that area could reflect historical patterns of discriminatory lending and function as a proxy for race or national origin. If the institution cannot obtain from the vendor a complete description of how the variable was constructed, that variable cannot safely remain in the model.

Question four: does the model's explanation match the adverse-action notice? The SHAP output for a denial is the institution's record of why the model scored the application as it did. The adverse-action notice is the institution's representation to the borrower of why the application was denied. These two documents should be consistent. If the adverse-action notice states that the primary denial reason was "insufficient income" but the highest negative SHAP contributor was "zip_segment_cluster_04," the notice does not accurately reflect the model's reasoning, and the institution has a Regulation B accuracy problem and potentially a fair-lending exposure.

LIME: what it measures and how to read it differently from SHAP. LIME (Local Interpretable Model-Agnostic Explanations) works differently from SHAP at the technical level. LIME builds a local surrogate model: it generates perturbed versions of the applicant's data, collects the complex model's predictions on those samples, and then fits a simple interpretable model (such as a linear model) to those local predictions. The coefficients of that surrogate model identify which variables mattered most for this specific decision. LIME explanations are typically presented as a list of the most influential features and their direction of influence for the specific decision being explained.

The key difference between SHAP and LIME outputs that matters for a loan officer or underwriter reading them: SHAP values are additive -- the sum of all SHAP values plus the model's baseline prediction (its average output across all applicants) equals the model's prediction for this applicant. This property makes it possible to check whether the identified reasons account for the full deviation from the baseline. LIME values indicate relative influence within the local surrogate model but are not reliably additive in the same sense. Both methods are post-hoc approximations of the underlying model's behavior and require validation rather than uncritical acceptance; SHAP's additivity property makes that validation more tractable. For the purpose of identifying the primary reasons for a denial, both methods produce usable outputs. For the purpose of checking whether the reasons account for the full departure from baseline, SHAP outputs are more reliable than LIME outputs.

Both methods are local explanation methods: they explain the model's decision for a specific applicant, not the model's general behavior across all applicants. This is the correct method for adverse-action purposes (you need to know why this applicant was denied) but is not sufficient for disparate-impact testing (which requires analyzing outcomes across demographic groups). The distinction matters because some vendors provide local explanations and describe them as a complete solution to explainability, when they address only the adverse-action requirement and not the fair-lending testing requirement.

Reading Reason Codes and Connecting Them to the Model

Many commercial credit scoring systems produce reason codes alongside a score: typically four to five standardized text descriptions of the factors that most negatively affected the score. Reason codes are familiar to anyone who has worked with FICO scores or similar consumer credit scoring products. In the context of AI-assisted lending, reason codes require the same critical reading as SHAP outputs, because the question is the same: do these reasons accurately reflect the model's decision, or are they a standardized approximation that may not match the specific factors that drove this applicant's result?

Standard reason codes from major credit bureaus are mapped to specific score factors and are required to meet a minimum accuracy standard for consumer credit reporting purposes. Custom model reason codes from AI vendors or internal model builds may or may not be held to the same standard. The institution must understand, for each model it uses, whether the reason codes produced are:

• Directly derived from the model's output (drawn from the model's own explanation method, such as SHAP values, for each specific applicant)
• Mapped from score buckets (reason codes that are assigned based on which score range the applicant falls into, rather than from the specific factors that drove the specific applicant's score)
• Standardized library codes (a set of approved reason codes from which the system selects based on general applicant characteristics, not from the specific model output)

The Regulation B requirement is that reasons must be specific and accurate. Reason codes derived directly from the model's output for a specific applicant satisfy this requirement most cleanly, because they reflect what actually drove the decision. Reason codes that are mapped from score buckets or selected from a standardized library may not accurately reflect the specific model factors for a specific applicant, which creates a Regulation B accuracy risk and, if the mismatch is systematic across protected-class applicants, a fair-lending risk.

The practical test for a loan officer or underwriter is this: take the reason codes produced for a denied application and ask whether each reason corresponds to a variable in the model explanation (SHAP or LIME output) with a significant negative contribution. If reason code one says "high revolving utilization" and the SHAP output shows that revolving utilization had a negative SHAP value of -0.21 (a large contributor), the reason is accurate. If reason code one says "high revolving utilization" but the SHAP output shows revolving utilization had a negative SHAP value of only -0.02 while "zip_segment_cluster_04" had a negative SHAP value of -0.31, the reason is not the primary factor in the decision. That discrepancy must be investigated before the denial is finalized and the adverse-action notice is sent.

The Proxy Variable Check: A Workflow Non-Quants Can Run

Proxy variable risk is the fair-lending concern that arises when a model uses a variable that is not itself a protected characteristic (like race) but that correlates strongly with a protected characteristic, effectively using the protected class as an indirect input. Zip code is the canonical example in mortgage lending: the zip code itself is not race, but because of historical residential segregation patterns, zip code can correlate strongly with race in many markets. A model that uses zip code as a high-weight input feature may be using race as a proxy, which constitutes disparate treatment risk and can produce disparate impact.

The proxy variable check is a workflow that a loan officer or compliance officer can run without statistical training, using the model explanation outputs and a systematic set of questions.

Step one: list the high-magnitude variables. From the SHAP output or model explanation, identify every variable with a SHAP value whose absolute magnitude exceeds a threshold you set based on context (for a model with 40 input features, a reasonable first-pass threshold might be the top 10 contributors by magnitude, regardless of direction).

Step two: classify each variable by its source category. For each high-magnitude variable, ask: where does this data come from? Possible categories include: borrower-specific credit behavior (payment history, utilization, account age), borrower-specific financial characteristics (income, assets, debt levels), property characteristics (LTV, property type, appraisal), and geographic or contextual inputs (zip code, census tract, neighborhood index, geospatial cluster). Geographic or contextual inputs require elevated scrutiny at the next step.

Step three: apply the correlation test question to geographic variables. For every geographic or contextual variable in the high-magnitude list, ask: could this variable's values in our market area correlate with the racial or ethnic composition of the neighborhood? In most markets, the answer for zip codes, census tracts, and neighborhood indices is likely yes, based on the documented history of residential segregation in American housing markets. That does not automatically mean the variable is prohibited, but it means the variable requires a specific documented analysis: the institution must be able to show that the variable's predictive value is not explained by its correlation with protected-class status, or that a less-discriminatory alternative achieves comparable predictive performance. This is the less-discriminatory-alternative analysis required by the ECOA disparate-impact defense framework.

Step four: require definitions for any variable you cannot describe. A variable you cannot describe in plain language is a variable you cannot defend to an examiner. For any high-magnitude variable in the model explanation that the submitting officer cannot describe in plain language from the variable name alone, require the model developer or vendor to provide a complete variable dictionary entry: what the variable measures, how it is calculated or constructed, what data sources feed it, and whether it has any geographic or demographic component. Refusal to provide this information is itself a model-risk finding under OCC 2026-13.

Step five: document the review. The proxy variable check must be documented in the model-risk record. For each application where a model explanation is reviewed, the reviewing officer should record: which variables were identified as high-magnitude, which required additional review, what the outcome of that review was, and whether any adverse-action reasons were modified based on the review. This documentation is the institution's evidence, in a fair-lending examination, that the model's explanations were actually reviewed and not simply passed through to the adverse-action notice without scrutiny.

Translating Model Explanations into Examiner-Ready Language

A SHAP chart or LIME output is a data visualization. An adverse-action notice is a legal document. An examiner's finding is a regulatory conclusion. Each of these three documents describes the same credit decision in different formats, and the loan officer or underwriter who works with AI-assisted decisioning tools must be able to move fluently between them, translating the technical output into compliant, specific, human-readable language.

Here is a worked example of that translation for a consumer mortgage denial.

The model's SHAP output for a specific applicant shows the following top contributors (negative values push toward denial):

Variable                       SHAP Value   Direction
revol_util_ratio               -0.31        Denial
months_since_recent_delinq     -0.18        Denial  
total_open_accounts            -0.09        Denial
annual_income_verified         +0.22        Approval
installment_to_income_ratio    -0.07        Denial

Step one: identify the primary denial drivers. The three variables with the largest negative contributions are revolving utilization (-0.31), months since most recent delinquency (-0.18), and total open accounts (-0.09).

Step two: translate to specific, human-readable factors. "revol_util_ratio" = the borrower's revolving credit utilization ratio (the total current revolving balances divided by the total revolving credit limit). If the institution's policy maximum is 30% and the borrower's ratio is 84%, the reason is: "Proportion of revolving balances to total revolving credit limits is too high." "months_since_recent_delinq" = the number of months since the borrower's most recent delinquency. If the borrower had a 30-day delinquency 14 months ago, the reason is: "Delinquent past or present credit obligations" or more specifically: "Time since most recent delinquency (14 months) does not meet the 24-month clean-history requirement under standard policy." "total_open_accounts" with a negative SHAP value means having more open accounts pushed the score down, which the institution must be able to explain in policy terms if it is to use this as an adverse-action reason.

Step three: verify each reason against the borrower's actual file. The reason code must reflect the actual value in the file. If the adverse-action notice says "high revolving utilization" but the file shows the borrower's utilization was 21%, the reason is wrong and must be corrected before the notice is sent.

Step four: assemble the Regulation B-compliant adverse-action notice reasons. Each reason should follow this structure: factor name, the specific value in the file, and the standard it fails to meet. Example of compliant language: "The proportion of balances to credit limits on your revolving accounts is too high. Your current revolving utilization ratio is 84%, which exceeds our standard maximum of 30%."

Step five: document the connection between the SHAP output and the adverse-action reasons in the credit file. The file should contain both the model explanation output and the adverse-action notice, with a notation confirming that the reasons on the notice correspond to the high-magnitude negative contributors in the SHAP output. This documentation satisfies the OCC 2026-13 requirement that AI-assisted adverse-action workflows be traceable from model output to borrower notification.

This five-step translation workflow does not require statistical training. It requires the ability to read a SHAP chart's variable names and magnitudes, access the variable dictionary to translate technical names into plain language, verify the values against the file, and draft Regulation B-compliant language from the verified values. Loan officers, underwriters, and compliance staff who build this skill eliminate the most common ECOA documentation failure in AI-assisted lending: an adverse-action notice that does not accurately reflect what the model actually computed.

Building Explainability into Vendor Relationships

The scenario that opened this lesson, in which the institution did not know what "zip_segment_cluster_04" was because the vendor would not describe it, is not an edge case. It is a recurring pattern in the deployment of third-party AI models in regulated lending. The vendor's proprietary model contains features the institution cannot fully inspect. The institution uses the model. A denial is made. An examiner asks about the primary denial driver. The institution cannot explain it because the vendor will not explain it.

OCC Bulletin 2026-13's third-party risk provisions address this directly. The institution's obligation to understand and govern the AI models it uses does not end at the vendor's proprietary boundary. The institution must, as a condition of using a third-party AI model in credit decisioning, obtain from the vendor documentation sufficient to: identify all input variables used in the model and their construction methodology, understand how the model produces its explanation outputs and what those outputs represent, test the model for disparate impact using the institution's own applicant data, and produce Regulation B-compliant adverse-action reasons that accurately reflect the model's specific computations for specific applicants.

If a vendor cannot or will not provide that documentation, the institution has a third-party risk problem under 2026-13 that is separate from any fair-lending concern. The practical implication for the fair-lending officer, the model-risk officer, and the chief credit officer who negotiate vendor contracts is that explainability commitments must be contractual, not aspirational. The vendor contract should require: a complete variable dictionary for all model inputs, a description of the explanation methodology (SHAP, LIME, or other), a representation that the explanation outputs will be updated if the model is retrained or updated, and the right for the institution to audit the model's inputs and explanation outputs as part of its ongoing model-risk and fair-lending program.

Vendors who resist these requirements are providing a tool that cannot be governed under the current regulatory framework. That is a business decision the institution makes at the vendor selection stage, not after a fair-lending examination has already begun.

Key Takeaways

• Explainability in AI-assisted lending is a regulatory requirement, not a technical feature. Under ECOA and Regulation B, adverse-action reasons must accurately reflect the specific factors that drove a specific applicant's denial, and the institution is responsible for satisfying this requirement regardless of whether the model was built internally or purchased from a vendor.
• SHAP (SHapley Additive exPlanations) is one of the most widely used model explanation methods in commercial lending AI. SHAP values are additive: they sum to the difference between the model's prediction for this applicant and the model's baseline (average) prediction, which allows a reviewer to verify that the identified reasons account for the full departure from baseline. Reading a SHAP output requires four questions: which variables were the largest contributors to denial, can each be described in specific human-readable terms, does any high-magnitude variable have geographic or demographic components that introduce proxy risk, and does the SHAP output match the adverse-action reasons actually sent to the borrower.
• LIME (Local Interpretable Model-Agnostic Explanations) builds a local surrogate model around each individual decision and produces similar practical outputs to SHAP for identifying primary decision drivers, but LIME values are not reliably additive in the way SHAP values are, making SHAP more suitable for checking whether the identified reasons account for the full departure from the model's baseline. Both methods are post-hoc approximations requiring validation.
• Reason codes must be verified against the model explanation output for each specific applicant. Reason codes mapped from score buckets or selected from a standardized library may not accurately reflect the specific model factors for a specific applicant, creating both a Regulation B accuracy risk and a potential fair-lending exposure.
• The proxy variable check is a five-step workflow non-quants can execute: list high-magnitude variables, classify by source category, apply the correlation test to geographic variables, require definitions for any variable that cannot be described in plain language, and document the entire review in the model-risk record.
• The translation from SHAP output to Regulation B-compliant adverse-action reasons requires identifying the primary denial drivers, translating variable names to plain language using the variable dictionary, verifying each value against the borrower's actual file, and documenting the connection between the model output and the notice in the credit file.
• Under OCC Bulletin 2026-13, vendor contracts for third-party AI models used in credit decisioning must require a complete variable dictionary, a description of the explanation methodology, and audit rights sufficient to support ongoing disparate-impact testing and adverse-action accuracy reviews. A vendor that will not provide this documentation is a third-party risk under the 2026 interagency model-risk standards, independent of any fair-lending concern.