Practical Use Cases

Understanding concepts in the abstract is valuable, but the real test is whether you can apply them in professional practice. This section bridges the gap between theory and application with concrete scenarios drawn from oversight work.

As an oversight professional, you might:

• Discover during audit that a team has been using ChatGPT to analyze sensitive data
• Learn that a business unit deployed a vendor risk-scoring tool without IT approval
• Find that a process that seemed manual is actually relying on an undocumented AI model
• Uncover that a team is using a public AI service for internal work that should be confidential

Your response:

• Assess the risk
• Require governance and documentation
• Evaluate whether the use can be approved going forward or must stop
• Remediate any harm (e.g., address exposed data)
• Implement controls to prevent future shadow AI

The following examples illustrate how the concepts from this lesson play out in real-world oversight scenarios. Each example is designed to help you recognize similar situations in your own work and respond with appropriate professional judgment.

Example 1: Shadow AI in Audit

Scenario: During an audit of the audit function itself, you discover that audit staff have been using an AI classification tool to sort audit evidence documents.

Details:

• The tool is a public SaaS product
• Audit staff found it helpful for organizing evidence
• No IT approval was sought
• Audit evidence (including sensitive control descriptions, test findings) has been uploaded to the tool
• The tool's accuracy has not been validated
• Performance is not monitored

Assessment:

• Data risk: Medium-High (audit evidence is confidential; the vendor's data handling is unknown)
• Accuracy risk: High (the tool has not been validated on audit evidence)
• Governance risk: High (an audit function is using an unvetted tool)

Response:

1. Stop use of the tool immediately for new evidence
2. Contact the vendor to understand what happened to data already uploaded (request deletion if possible)
3. Assess whether the vendor's data practices comply with the organization's data governance
4. Evaluate the vendor's tool formally: test accuracy, assess bias, review security practices
5. Determine: can the tool be approved going forward with controls, or must it be prohibited?
6. Document the discovery and remediation in the audit file

Example 2: Shadow AI in Compliance

Scenario: A compliance officer mentions that they use ChatGPT to help draft compliance summaries of new regulations.

Assessment:

• The officer feeds regulatory text and sometimes organization-specific compliance information into ChatGPT
• No approval from information security or legal
• No understanding of whether confidential information is being used as training data
• No validation of ChatGPT's accuracy (hallucination risk is not being managed)
• Summaries are distributed to business units without verification

Risks:

• Data exposure: confidential compliance information may become training data
• Hallucination: ChatGPT may have invented obligations or misinterpreted regulations
• Regulatory risk: if distributed summaries are inaccurate, the organization may be out of compliance
• Liability: if the organization relies on an inaccurate AI-generated summary, the organization is liable for any resulting harm

Response:

1. Meet with the compliance officer to understand the current use
2. Assess: which regulatory areas is ChatGPT being used for? How often? Is confidential information being shared?
3. Determine acceptable and unacceptable uses: Acceptable — ChatGPT is used to help understand publicly available regulatory text, with output always validated by a compliance expert before use; Unacceptable — confidential compliance information is fed into ChatGPT, or ChatGPT output is used without expert validation
4. Provide guidance: "You may use ChatGPT to help understand regulatory text, but always verify the output against the actual regulation before using it in compliance decisions. Do not share confidential compliance information with ChatGPT."
5. Monitor: periodically ask if ChatGPT is still being used and check for compliance with guidance

Example 3: Shadow AI in Risk Assessment

Scenario: The risk function has been using a vendor tool to help score operational risk. The tool is a machine learning model that predicts risk based on organizational data (incident history, control exceptions, staffing changes, etc.).

Discovery:

• The tool is not on the approved vendor list
• No contract governs data handling
• No performance validation was done before use
• No ongoing monitoring of model accuracy or bias
• Risk scores from the tool are input to the risk register without clear documentation of the tool's contribution
• Stakeholders may believe the scores are the organization's professional judgment, not AI-assisted

Assessment:

• Accuracy risk: Unknown (model not validated)
• Bias risk: High (model may have learned from historical data and perpetuated bias)
• Governance risk: High (scores are used in risk reporting without adequate transparency)
• Vendor risk: High (no contract, no audit rights)

Response:

1. Immediately assess: is the vendor tool being used to make decisions, or to provide input? (This affects urgency.)
2. If used for decisions: begin transition to an approved process or tool while the vendor tool is assessed
3. Conduct formal vendor assessment: request security assessment, data handling terms, and model training data; test model accuracy on organizational data; assess for bias
4. Determine: can the tool be approved going forward with controls, or must it be replaced?
5. If approved: establish contract with appropriate terms, document governance, monitor ongoing accuracy
6. Remediate: assess prior risk scores that may have been affected by undocumented AI input