Skill.reWhat AI Can and Cannot Do Reliably
Why Knowing AI's Limits Matters
Equip oversight professionals with a realistic assessment of AI capabilities and limitations so that AI is applied to work where it actually works, and rejected where it doesn't.
At the Awareness level, your primary goal is to build a solid conceptual foundation. You do not need to operate AI systems yourself at this stage — but you must understand what they do, how they work at a high level, and why they matter for oversight. This knowledge will be the bedrock upon which all subsequent levels build.
This lesson is designed to be accessible to professionals at all experience levels while providing the depth needed for practical application. Whether you are encountering these concepts for the first time or building on existing knowledge, the material ahead will strengthen your ability to navigate AI governance challenges with confidence and competence.
Why This Matters for Risk, Compliance, and Audit
Oversight professionals are accountable for decisions that AI may inform but should not unilaterally make. Knowing precisely what AI does reliably — and where it fails — lets you apply it where it adds value and reject it where it introduces risk.
This realistic assessment is the foundation for every later capability: applying AI to work where it actually works, and rejecting it where it doesn't, protects the organization from misplaced trust in unverified outputs.
Core Concepts
AI's reliability depends on the task, the data, and the context. Pattern recognition is a genuine strength; fact-checking and judgment are not.
- Confidence vs. accuracy. Confidence is how certain the model is in its prediction; accuracy is whether the prediction is actually correct. A model can be confidently wrong.
- Accuracy in context. Accuracy is measured on historical test data, usually in idealized conditions. For rare events such as true fraud or true violations, even high accuracy can mean many false positives. Always ask: accurate at what task, on what data, in what context?
- Generalization is limited. AI trained on one dataset may fail on data that differs from its training data.
- Transparency has limits. Some AI systems cannot explain their decisions. This is a governance problem, not a reason to trust them blindly.
- Hallucination. LLMs can invent details that sound plausible but are false, which makes human verification essential for high-stakes documents.
Practical Use Cases
Understanding concepts in the abstract is valuable, but the real test is whether you can apply them in professional practice. This section bridges the gap between theory and application with concrete scenarios drawn from oversight work.
What works:
- AI to detect and narrow: "Here are 50 anomalies out of 10,000 transactions; please investigate"
- AI to draft: "Here's a first-pass summary of the regulation; please validate with the full text"
- AI to categorize: "I've categorized these 100 policies by topic; please review for accuracy"
- AI to surface patterns: "These 8 audit findings have a common root cause (based on language analysis); please investigate"
What doesn't work:
- AI to replace regulatory interpretation (it will hallucinate)
- AI to determine materiality or control effectiveness (requires judgment)
- AI to make unreviewed decisions about compliance status
- AI to determine whether a control is operating effectively (too context-dependent)
The following examples illustrate how the concepts from this lesson play out in real-world oversight scenarios. Each example is designed to help you recognize similar situations in your own work and respond with appropriate professional judgment.
Example 1: Where AI helps — Anomaly Detection in Compliance Monitoring
Use case: Your compliance team monitors unusual payment activity to detect sanctions violations.
How it works:
- Historical transaction data is analyzed by ML to establish patterns for each payment category (vendors, geography, amount)
- New transactions are scored for how "unusual" they are
- Transactions above a threshold are flagged for human review by the compliance officer
Why it works:
- Pattern recognition is a strength of AI
- The final decision (is this a real violation?) remains with the human
- The human can override the AI if they have information the AI lacks
- False positives are acceptable because investigation costs are manageable
Why it's governed correctly:
- The threshold is a governance decision, not an AI decision
- The AI is a detection tool; humans are the decision-makers
- There is a process to validate the AI's accuracy over time
Example 2: Where AI fails — Hallucination in Policy Interpretation
Use case (wrong approach): Your compliance officer asks an LLM: "Summarize the key obligations under the new privacy regulation."
The problem:
- The LLM generates a confident, detailed summary
- It includes several obligations that are plausible but not in the actual regulation (hallucination)
- The compliance officer, trusting the summary, miscommunicates the requirements to the business
- The organization falls out of compliance because the AI invented an obligation
What went wrong:
- The compliance officer assumed the LLM fact-checks. It doesn't.
- There was no human expert validation of the LLM output against the actual regulation
Right approach:
- Use the LLM to generate a draft summary as a starting point
- A compliance expert cross-references the summary against the actual regulation, sentence by sentence
- The expert confirms or corrects each point
- Only then is the summary used for organizational guidance
Example 3: Confidence vs. Accuracy — Why Tone Matters
Scenario: Your compliance system runs an AI audit check: "Based on transaction patterns, this vendor may pose a sanctions risk."
The AI outputs this with high confidence (score: 0.94/1.0).
The trap: You interpret high confidence as high accuracy. Confidence and accuracy are different things.
- Confidence = how certain the model is in its prediction
- Accuracy = whether the prediction is actually correct
A model can be confidently wrong. Example: If the model was trained on a dataset where 5% of vendors were sanctions risks, and it learned "always predict high risk for unusual vendors," it could generate high-confidence scores that are often inaccurate.
Correct interpretation:
- The high confidence score means the AI found something unusual
- This is a signal for investigation, not proof of actual risk
- The human compliance officer must determine whether the AI's finding has merit
Anti-Patterns
Anti-pattern 1: "The AI is X% accurate, so we can trust it for this decision"
The claim: "Our model has 92% accuracy, so we'll automate this decision."
The problem:
- Accuracy is measured on historical test data, usually in idealized conditions
- Real-world performance often differs from test accuracy
- For rare events (true fraud, true violations), even high accuracy can mean many false positives
- You must know: accurate at what task, on what data, in what context?
Correct approach:
- Understand accuracy in your specific context: How many false positives will 92% accuracy generate?
- Measure real-world accuracy on your own data, not vendor benchmarks
- For high-stakes decisions, require human validation even with high accuracy
Anti-pattern 2: "If the AI can't explain it, neither can we"
The claim: "Since we can't audit the model's decision, we'll just treat it as a black box and accept its output."
The problem:
- Lack of transparency is not permission to skip governance
- It is actually a signal that human oversight must be stronger, not weaker
Correct approach:
- Require human review of AI outputs, especially for high-stakes decisions
- Measure the error rate empirically: How often is the AI wrong on your data?
- Establish thresholds for when human review is mandatory vs. optional
- Document the governance model clearly
Anti-pattern 3: Hallucination acceptance
The claim: "The summary looks good, so we'll use it."
The problem:
- You haven't verified it against source material
- The LLM may have invented details that sound plausible but are false
Correct approach:
- Treat all LLM outputs (especially for regulatory interpretation, policy, or high-stakes documents) as drafts
- Subject matter expert review is mandatory before use
- Spot-check the LLM output against original sources to verify accuracy
Human Judgment Checkpoints
Before using an AI output:
- What is the decision being made? Is it high-stakes or low-stakes?
- What is the cost of error? (This determines how much verification is needed.)
- Has the AI been tested on data like mine? (Transfer learning matters.)
- What is the false-positive rate? (How many incorrect flags will the organization tolerate?)
- Is there a human review process? (Who validates the AI's work? Under what conditions?)
- How is the AI updated? (Does it learn from corrections, or is it static?)
- What happens when the AI is wrong? (Is there escalation? Audit trail? Learning?)
Responsible AI Considerations
Honest capability statements:
- Vendors and internal teams should describe what the AI can and cannot do, with specificity.
- "92% accurate" without context is not honest. "92% accuracy on transaction classification in this merchant category, tested on 2024 data" is honest.
Threshold setting:
- The threshold for when AI assists vs. when it decides must be a governance choice, not a vendor default.
- For high-stakes decisions, the threshold should be "AI assists only; human decides always."
Ongoing monitoring:
- AI performance should be monitored over time to ensure it doesn't degrade.
- If performance drops, the system should be flagged for investigation or retraining.
Traceability and defensibility:
- In audit work: If you rely on an AI system to flag unusual transactions for investigation, document which items the AI flagged, which the auditor investigated, and what was found. This creates a traceable link between the AI output and the audit conclusion.
- In compliance work: If an AI assists in classifying documents or identifying obligations, retain the AI input, the AI output, and the human validation that was performed. This allows reconstruction of how the compliance determination was made.
- In risk work: If AI is used to score risk or detect emerging issues, retain the input data, the score, the threshold for escalation, and documentation of human review.
- Key principle: The organization must be able to explain and defend how an AI output was used in a governance decision. This requires traceable documentation.
Practice and Reflection
- In your organization: Is there a process for validating AI outputs before they are used in governance decisions? What would that process look like?
- Hallucination risk: You receive an AI-generated summary of a new regulation. What steps would you take before distributing it to the business?
- False positive tolerance: An AI system flags 10% of transactions for review. Is that acceptable? (The answer depends on the cost of review. What would change your assessment?)
- Context transfer: Your organization uses a fraud detection model trained by a third-party vendor. The vendor claims 95% accuracy. What questions would you ask before accepting that benchmark?
- Your judgment checkpoint: Describe a decision in your professional work that is high-stakes enough that you would never rely on an AI output without human validation.
Application Exercise
As you complete this lesson, keep these guiding principles in mind for immediate application:
- Start with awareness: Begin observing where AI is currently being used — or proposed for use — in your organization. You do not need to evaluate it yet; simply notice it.
- Build your vocabulary: Use the terminology from this lesson precisely. Clear language prevents misunderstandings that lead to governance gaps.
- Ask questions: When colleagues mention AI, ask clarifying questions: What type of AI? What data does it use? How are outputs verified? Your questions alone improve organizational awareness.
- Document what you learn: Keep brief notes on AI-related observations and questions. This habit will serve you well in later levels when formal documentation becomes a professional requirement.
Key Takeaways
- Pattern recognition is AI's strength. Use AI to find signals in noise.
- Fact-checking is AI's critical weakness. Never assume an LLM has verified information.
- Confidence and accuracy are different. A high confidence score does not guarantee correctness.
- Generalization is limited. AI trained on one dataset may fail on different data.
- Transparency has limits. Some AI systems cannot explain their decisions. This is a governance problem, not a reason to trust them blindly.
- Human review is a structural control. Even with high-accuracy AI, human judgment checkpoints protect the organization.
Frequently Asked Questions
Is a high confidence score the same as high accuracy? No. Confidence is how certain the model is in its prediction; accuracy is whether the prediction is actually correct. A model can be confidently wrong.
If an AI system can't explain its decision, can we still rely on it? Lack of transparency is not permission to skip governance. It is a signal that human oversight must be stronger, not weaker, with mandatory human review for high-stakes decisions.
Can we trust an AI-generated summary of a regulation? Treat it as a draft only. A subject matter expert must cross-reference the summary against the actual regulation before it is used for organizational guidance, because LLMs can hallucinate plausible but false obligations.
Glossary
- Hallucination: When an AI generates false information presented as fact.
- Confidence score: The model's estimate of certainty in its own prediction. Not the same as accuracy.
- Accuracy: The proportion of correct predictions in test data.
- False positive: When the AI flags something as high-risk/problematic that is actually normal.
- False negative: When the AI misses something that is actually high-risk/problematic.
- Threshold: The confidence level at which the AI flags something for human review.
- Generalization: The ability to perform well on data different from training data.
- Opaque / Black box: An AI system whose reasoning cannot be explained to humans.