You Can't Audit What You Don't Understand

Picture this: your organization's credit team has deployed an AI model that scores loan applicants. The model is fast, consistent, and the business loves it. You're asked to sign off on the controls. You open the vendor documentation and encounter phrases like "gradient boosting," "feature weighting," and "probabilistic output." The technical team assures you it's fine. Do you have enough to form an opinion?

If the answer is "not really," you're not alone—and you're in exactly the right place. Before you can ask smart questions about AI risk, before you can evaluate controls or interpret audit findings, you need a working mental model of what AI actually is. Not a computer science degree. A working mental model.

That's what this chapter gives you.

Why This Matters for Audit and Oversight

AI is no longer a technology curiosity confined to R&D labs. It is embedded in credit decisioning, fraud detection, hiring pipelines, clinical triage tools, customer service chatbots, and regulatory compliance workflows. When something in those systems goes wrong—and things do go wrong—audit and oversight professionals are expected to have a view.

The problem is that most audit frameworks were built for processes with clear, traceable logic. A human makes a decision; there's a record; you can follow the chain. AI systems operate differently. A model trained on historical data produces outputs that may be difficult to trace back to any single rule or decision point. The controls that worked for manual processes don't map cleanly onto AI systems.

That gap creates real exposure. Organizations have faced regulatory action, reputational damage, and material losses because no one with oversight authority understood enough about AI to ask the right questions early enough. Understanding what AI is—and crucially, what it is not—is the foundation for everything that follows in this credential.

Core Concepts

What AI Actually Is

Artificial intelligence, in practical terms, refers to systems that perform tasks that would ordinarily require human judgment—pattern recognition, language understanding, prediction, classification—by learning from data rather than following explicitly programmed rules.

The key distinction is learning from data. Traditional software operates on logic a programmer wrote: if X, then Y. AI systems, particularly modern machine learning models, derive their logic from patterns in training data. No one writes a rule that says "a loan applicant with these characteristics is higher risk." The model infers that relationship from thousands or millions of historical examples.

This is powerful. It allows AI to handle complexity that would be impossible to encode manually. But it also means the system's logic can be opaque, data-dependent, and potentially fragile in ways that traditional software is not.

The Main Categories You'll Encounter

Not all AI is the same. As an oversight professional, you'll encounter several distinct types, and they carry different risk profiles:

• Supervised learning—The most common type in business applications. A model is trained on labeled historical data (past loan outcomes, past fraud cases, past customer churn) to predict future outcomes. High stakes when the training data contains historical biases.
• Unsupervised learning—The model finds patterns in data without predefined labels. Used for anomaly detection, customer segmentation, and clustering. Harder to validate because there's no clear "right answer" to check against.
• Large language models (LLMs)—The technology behind tools like ChatGPT. Trained on vast text corpora to generate, summarize, and reason about language. Increasingly used in document review, report drafting, and internal Q&A tools. Introduces new risks around hallucination and output reliability.
• Reinforcement learning—A model learns by trial and error, optimizing for a reward signal. Less common in enterprise audit contexts but appears in algorithmic trading and some operational optimization systems.

What AI Is Not

Equally important is what AI is not—because misconceptions in both directions create audit blind spots.

AI is not magic. Every AI system has a specific scope, trained on specific data, to perform specific tasks. A fraud detection model trained on card transactions cannot tell you whether a contract clause creates legal exposure. Scope limitations are a genuine control consideration.

AI is not infallible. Models make errors. They can be confidently wrong. A well-designed AI system is accurate on average across a large population of cases—but individual outputs can be incorrect, and the errors may be systematically biased toward particular groups or conditions.

AI is not neutral. Because AI learns from historical data, it can encode historical patterns—including patterns that reflect past discrimination, past errors, or past operational conditions that no longer apply. "The data made the decision" is not a defense that regulators or courts have accepted.

AI is not a black box you can't examine. This is a common—and sometimes convenient—excuse. There are well-established techniques for interrogating model behavior: looking at feature importance, testing outputs across demographic groups, examining performance metrics on held-out data. The work is specialized, but it is absolutely accessible to a well-prepared audit function.

Real-World Examples

Where People Get This Wrong

Audit and oversight professionals make two opposite errors when it comes to AI, and both are costly.

Over-mystification. Some professionals treat AI as fundamentally unknowable—something to be deferred entirely to data scientists and vendors. This creates a dangerous accountability gap. If your audit function cannot form an independent view on AI risk, you are relying entirely on the people whose work you're supposed to be checking. That's not oversight; it's rubber-stamping.

Over-simplification. Others treat AI as just another piece of software—apply the standard IT general controls framework, check the change management log, done. This misses the category of risks that are specific to learned systems: training data quality, model drift over time, performance disparities across population subgroups, and the absence of explicit business rules to audit against.

The right posture is neither of these. It is informed skepticism: understanding enough about how AI systems work to know what questions to ask, what evidence to request, and when to escalate.

Practical Takeaways

• When an AI system is in scope for your audit, start by establishing its type (predictive model, LLM, anomaly detection, etc.) and its training data source. These two facts shape almost every subsequent question.
• Ask for the model's performance documentation—accuracy metrics, error rates, and critically, whether performance has been tested across different demographic or operational subgroups.
• Distinguish between AI making a decision and AI informing a decision. In the latter case, your human controls question shifts to: how are staff using the AI output, and what oversight exists at the human decision point?
• Do not accept "it's too technical to explain" as a complete answer. Reasonable plain-language explanations of how a model works and what it's optimizing for should be available from any responsible deployment team.
• Note the date of training data and the last model validation. AI systems require ongoing monitoring; a model validated three years ago against pre-pandemic data is not the same risk profile as one validated last quarter.

Before You Move On

Test your understanding with these questions. If you can answer them confidently, you're ready for the next lesson.

• What is the fundamental difference between traditional rule-based software and a machine learning model?
• Why is "the data made the decision" not a sufficient defense against a claim of biased outcomes?
• Name two risks specific to AI systems that standard IT general controls frameworks do not adequately address.
• What does model drift mean, and why is it an audit concern?

The next lesson, What Is Artificial Intelligence?, goes deeper into how machine learning models are built and trained—giving you the vocabulary to engage confidently with technical teams during fieldwork.