Skill.reCommon AI Failure Modes: Hallucination, Bias, and Distortion
Why Understanding AI Failure Modes Matters
Equip oversight professionals with a concrete understanding of how AI systems fail and why these failures are predictable, not anomalous.
At the Awareness level, your primary goal is to build a solid conceptual foundation. You do not need to operate AI systems yourself at this stage — but you must understand what they do, how they work at a high level, and why they matter for oversight. This knowledge will be the bedrock upon which all subsequent levels build.
This lesson is designed to be accessible to professionals at all experience levels while providing the depth needed for practical application. Whether you are encountering these concepts for the first time or building on existing knowledge, the material ahead will strengthen your ability to navigate AI governance challenges with confidence and competence.
Why This Matters for Risk, Compliance & Audit
Understanding concepts in the abstract is valuable, but the real test is whether you can apply them in professional practice. Recognizing how AI systems fail is the foundation for every oversight responsibility that follows.
In oversight work, the consequences of unrecognized AI failures are direct: misstated compliance obligations, biased risk scoring, and audit conclusions built on degraded models. Your governance responsibility is to recognize these failures and escalate them for investigation and correction.
Core Concepts
Common AI failure modes fall into a small set of recurring, predictable patterns. The three central ones for oversight are hallucination, bias, and drift (distortion).
- Hallucination is systemic in LLMs. It is not a rare bug; it is a feature of how they work. LLM output that sounds plausible may include facts, figures, or obligations that are not in the source material.
- Bias is learned from data. Even mathematical models are not "objective" if trained on biased data. Historical data reflects historical bias, and objectivity is not guaranteed by data volume.
- Drift is expected. AI systems degrade over time as the environment changes. A model that was accurate at deployment can become inaccurate as conditions shift.
- Distortion through feedback loops. When a model's own outputs influence the data it later learns from, recommendations can become self-reinforcing rather than reflecting actual risk.
These failures are predictable rather than random, which means governance can be designed to detect and mitigate them.
Practical Use Cases
Understanding concepts in the abstract is valuable, but the real test is whether you can apply them in professional practice. This section bridges the gap between theory and application with concrete scenarios drawn from oversight work.
In your organization, you might encounter:
- An LLM summary of a regulation that includes obligations not in the actual regulation
- A fraud detection model that flags certain vendors at much higher rates than others, despite similar risk profiles
- An audit sampling system that was accurate 5 years ago but now over-flags certain transaction types
- A vendor risk model trained on historical data where certain regions were over-investigated and now perpetually scores those regions higher
- A control effectiveness assessment system whose recommendations change dramatically from year to year without business explanation
Your governance responsibility: Recognize these failures and escalate for investigation and correction.
Detailed Examples
The following examples illustrate how the concepts from this lesson play out in real-world oversight scenarios. Each example is designed to help you recognize similar situations in your own work and respond with appropriate professional judgment.
Example 1: Hallucination in Policy Interpretation
Scenario: A compliance officer uses an LLM to summarize new data privacy regulations.
The LLM outputs: "Organizations must notify customers within 72 hours of a data breach affecting more than 100 records."
What happens: The compliance officer shares this requirement with the business. The organization implements a 72-hour notification process.
The problem: The regulation actually requires notification "without undue delay," not specifically 72 hours. The LLM hallucinated a specific timeframe that sounded plausible but wasn't in the actual regulation.
The failure: The organization implements a more stringent requirement than necessary, wasting resources. Or, if the real requirement is stricter (24 hours), the organization is now out of compliance.
Governance response: The compliance officer should have validated the LLM output against the actual regulation. Documentation should note: "LLM output reviewed and validated against source material on [date] by [person]." Future AI-assisted summaries should include this validation step before distribution.
Example 2: Training Data Bias in Fraud Detection
Scenario: A fraud detection model is trained on 5 years of transaction data and is deployed.
After 6 months of operation, audit review shows: Transactions from Vendor Type A: 4% flagged as high-risk; Transactions from Vendor Type B: 18% flagged as high-risk.
Investigation: Historically, Vendor Type B had more fraud cases detected (due to more intensive investigation, not necessarily more fraud). The training data reflected this, and the model learned "Vendor Type B is higher-risk."
The failure: The model perpetuates historical bias. Vendor Type B is now flagged at higher rates, creating a self-fulfilling prophecy: more investigation of Type B → more findings → stronger evidence of Type B risk.
Governance response: Bias analysis is performed: for each vendor type, measure false positive rate (how many flagged as fraud are actually fraud?). If Type B has higher false positive rate, the model is biased. Mitigation: adjust the threshold so that Type B is not systematically flagged at higher rates, or redesign the model to reduce bias. Going forward, bias analysis is performed quarterly.
Example 3: Data Drift in Control Assessment
Scenario: An AI system was trained to assess control effectiveness based on prior-year assessment, exception rates, and remediation rates.
For 3 years, the system's assessment matched management and audit assessment with high accuracy.
Year 4: The system rates a control as "Effective" that management and audit assess as "Needs Improvement"; the system consistently underscores controls in Region X compared to Region Y; false positive rate (controls rated Effective that later had findings) increases.
Investigation: The organization's controls environment improved, but the training data (from years 1-3) reflected an older, weaker environment. Region X has stricter auditing protocols; the historical training data didn't reflect this variation. The model's learned patterns are no longer representative.
Governance response: The model is retrained on current data. Annual retraining is implemented to ensure the model adapts to the changing controls environment. Accuracy is monitored quarterly; if it drops below 85%, retraining is triggered immediately.
Example 4: Feedback Loop Creating Distortion
Scenario: An audit sampling model recommends sample sizes based on prior findings and control risk rating.
Year 1: Controls with prior findings: model recommends larger sample (100 items); controls without prior findings: model recommends smaller sample (30 items).
Year 2: Audit follows the recommendations; controls with larger samples have more exceptions found (because more items were tested); the model learns "controls with prior findings have many exceptions."
Year 3: The model increasingly recommends large samples for controls with prior findings; the feedback loop has amplified: the model's recommendation drives testing intensity, which drives the findings the model learns from.
The failure: The model's increasing focus on controls with prior findings may not reflect actual control risk. The recommendations are self-reinforcing.
Governance response: The audit team recognizes the feedback loop. The model is redesigned to separate: (a) data it learned from (historical findings), and (b) data it will learn from going forward. Sample size recommendations are periodically validated against independent risk assessment. The audit team retains authority to override recommendations based on risk assessment.
Putting It Into Practice
As you complete this lesson, keep these guiding principles in mind for immediate application:
- Start with awareness: Begin observing where AI is currently being used — or proposed for use — in your organization. You do not need to evaluate it yet; simply notice it.
- Build your vocabulary: Use the terminology from this lesson precisely. Clear language prevents misunderstandings that lead to governance gaps.
- Ask questions: When colleagues mention AI, ask clarifying questions: What type of AI? What data does it use? How are outputs verified? Your questions alone improve organizational awareness.
- Document what you learn: Keep brief notes on AI-related observations and questions. This habit will serve you well in later levels when formal documentation becomes a professional requirement.
Anti-Patterns
Anti-pattern 1: "The model said so, so it must be right"
The claim: "The AI flagged this as high-risk, so we're escalating without further investigation."
The risk: The AI is wrong sometimes. Hallucination, bias, and drift are real. Every flagged item deserves investigation, not assumption of guilt.
Anti-pattern 2: Treating AI output as fact
The claim: "The LLM summarized the regulation; we'll distribute the summary."
The risk: Hallucination is systematic in LLMs. Summary distribution without validation risks compliance failure.
Anti-pattern 3: Assuming training data is unbiased
The claim: "We used 5 years of historical data, so the model is objective."
The risk: Historical data reflects historical bias. Objectivity is not guaranteed by data volume; it requires active bias detection and mitigation.
Anti-pattern 4: Deploying without ongoing monitoring
The claim: "We tested the model before deployment; ongoing monitoring is overkill."
The risk: Drift is expected. Systems degrade over time. Monitoring is required, not optional.
Human Judgment Checkpoints
- For hallucination-prone tasks (LLM use): Has the output been validated against source material by a subject matter expert?
- For prediction tasks (ML models): Has the model been tested for bias? Were results disaggregated by relevant groups or categories?
- For systems in production: Is the model's performance monitored over time? What triggers retraining?
- For all AI uses: What is the error rate? What is the cost of error? Is governance proportionate to that cost?
Responsible AI Considerations
Traceability and defensibility are central to responsible AI use in oversight work. The level of documentation should be proportionate to the cost of error.
In audit work: If AI assists in sampling or evidence assessment, document the AI method, validation that the method is sound, and results of ongoing monitoring. Example: "Audit sampling used an AI model to identify high-risk transactions. The model's accuracy was validated on prior-year data (87% recall, 89% precision). Performance is monitored quarterly."
In compliance work: If AI assists in summarization or interpretation, document human expert validation that occurred, any corrections made, and escalations for items where AI output conflicted with human judgment. Example: "LLM generated a summary of regulation X. The summary was reviewed by [expert] against the full regulation. Two inaccuracies were identified and corrected. The final summary was approved on [date]."
In risk work: If AI assists in assessment or scoring, document what the AI contributes (input data, scoring), how it is validated, and how human judgment is applied. Example: "Risk scoring uses an AI model for quantitative inputs (probability, impact estimates). Management reviews the AI scores and applies qualitative judgment to finalize risk assessment."
Practice and Reflection
- Hallucination awareness: In your organization, where would hallucination in AI output be most dangerous? How would you establish human validation?
- Bias detection: For an AI system in your organization (or a proposed one), how would you design a bias analysis? What groups or categories matter?
- Drift monitoring: If an AI system were deployed in your organization, what would you monitor to detect performance degradation? What triggers would prompt retraining?
- Historical precedent: Has your organization experienced a failure of an automated system or model in the past? What failure mode was it? Could it have been detected earlier?
Key Takeaways
- Hallucination is systemic in LLMs. It's not a rare bug; it's a feature of how they work. Always validate LLM output before using it for compliance, policy, or high-stakes decisions.
- Bias is learned from data. Even mathematical models are not "objective" if trained on biased data. Active bias detection is required.
- Drift is expected. AI systems degrade over time as the environment changes. Ongoing performance monitoring and retraining are required, not optional.
- Failures are predictable. These aren't random failures; they're systematic. Governance can be designed to detect and mitigate them.
- Human validation is a control. It may slow decisions but protects against failure modes that AI alone cannot prevent.
Frequently Asked Questions
Is hallucination a rare bug I can mostly ignore?
No. Hallucination is systemic in LLMs — it's a feature of how they work, not a rare bug. Always validate LLM output before using it for compliance, policy, or high-stakes decisions.
If a model is trained on years of historical data, isn't it objective?
No. Historical data reflects historical bias, and objectivity is not guaranteed by data volume. Even mathematical models can learn and perpetuate bias, so active bias detection is required.
If we tested the model before deployment, is ongoing monitoring really necessary?
Yes. Drift is expected: AI systems degrade over time as the environment changes. Ongoing performance monitoring and retraining are required, not optional.
Do I need to operate AI systems to add value at the Awareness level?
No. At this stage your goal is conceptual: understand what AI systems do, how they work at a high level, and why they matter for oversight.