Overview

Vilma Žukauskas was Head of AI Governance at a Lithuanian bank when her team deployed a credit scoring model that, on paper, performed better than the human process it supplemented. Six months later, an internal audit noticed that approval rates for applicants from one postal code cluster were 23 percentage points lower than for statistically equivalent applicants elsewhere. The model was technically sound. Its training data reflected historical lending patterns that encoded decades of geographic bias. Nobody had reviewed the model as high-risk before deployment. "We treated it like a productivity tool," Vilma told me. "It was making decisions about people's financial futures."

Not all AI is equal. A system that recommends meeting times has essentially no downside risk. A system that determines credit eligibility, medical treatment priority, hiring decisions, insurance premiums, or criminal sentencing scores carries significant potential for harm - not just to the individuals affected, but to the organizations deploying it, in terms of legal liability, regulatory sanction, and reputational damage.

High-risk AI requires a different level of scrutiny before deployment and a more intensive oversight regime during operation. This lesson explains how to identify high-risk AI, what enhanced oversight looks like in practice, and how to build the organizational processes to sustain it.

What Makes AI High-Risk

Risk level is determined by two variables: the significance of the decisions the AI influences, and the vulnerability of the people affected by those decisions.

The European Union AI Act - the most comprehensive AI-specific legislation enacted to date - classifies AI as high-risk when it is used in specific domains that carry inherent potential for serious harm. These domains include:

• Credit scoring and financial services decisions
  - Employment and recruitment decisions (CV screening, interview assessment, performance monitoring)
  - Educational and vocational access decisions
  - Critical infrastructure management (energy, water, transport)
  - Law enforcement and border control applications
  - Healthcare diagnosis and treatment recommendation
  - Access to essential public services and benefits

Even if you are not subject to the EU AI Act specifically, this classification framework is useful. It identifies the domains where the consequences of AI error or bias are most severe. If your AI touches any of these areas, treat it as high-risk regardless of what your current regulatory environment requires.

Two additional risk factors apply across domains:

Scale. A model that makes 100 decisions per day creates limited aggregate risk. A model that makes 100,000 decisions per day, each affecting a different person, creates the potential for systemic harm even if its error rate is low. At scale, a 2% error rate means 2,000 people per day are receiving incorrect outcomes.

Automation level. A model whose recommendations are always reviewed by a human decision-maker carries lower risk than a model whose outputs trigger automatic action. The higher the degree of automation, the more important pre-deployment scrutiny becomes.

Pre-Deployment Requirements for High-Risk AI

High-risk AI systems should not go to production until they have passed a structured pre-deployment review. The review should address five areas.

Purpose and scope documentation. What exactly does this system do? What decisions does it inform or make? What is the intended population it will affect? This documentation should be specific enough that someone who was not involved in building the system can understand precisely what it does.

Data audit. What data was used to train and test the system? Does that data accurately represent the population the system will be applied to? Does it contain historical patterns that, if learned, would produce discriminatory outcomes? Vilma's credit model was trained on 12 years of historical lending data - data that reflected a period when the bank's branch network was concentrated in specific neighborhoods and had specific customer demographics. The model learned those patterns faithfully. The problem was that the patterns encoded historical bias, not risk.

Bias and fairness evaluation. Test the system's outputs for disparate impact across protected groups - race, gender, age, disability status, and other characteristics relevant to your context and regulatory environment. This requires disaggregating performance metrics. A model that achieves 92% accuracy overall may achieve 85% accuracy for one subgroup and 97% for another. The overall figure obscures the disparity.

Human oversight design. Who reviews AI recommendations before consequential actions are taken? How are edge cases escalated? What is the documented process for a human to override the system? These questions should be answered and documented before deployment, not improvised during operation.

Legal and compliance review. Has the system been reviewed by legal counsel in the relevant jurisdictions? Are there consent, disclosure, or documentation requirements associated with the decisions this system influences? High-risk AI without legal review is not a technology problem. It is a liability problem.

Operating High-Risk AI: Enhanced Oversight in Practice

Pre-deployment review is the entry point. Enhanced oversight is an ongoing operational commitment.

Continuous bias monitoring. Bias is not a one-time test. The population the model serves changes. The data it processes changes. New patterns can emerge that create disparate impact where none existed at launch. Sample-based bias monitoring - reviewing a random sample of decisions quarterly, disaggregated by relevant subgroup - is the minimum. For high-volume, high-stakes systems, automated drift detection tools can flag emerging disparities faster than manual sampling.

Decision audit logging. Every decision made or influenced by a high-risk AI system should be logged with enough information to reconstruct the reasoning: what inputs were used, what the system's output was, whether a human reviewed it, and what the final decision was. This log has two functions. It enables investigation of specific complaints or adverse outcomes. It also enables statistical auditing of overall system performance.

An explainability requirement. Affected individuals should be able to understand the basis for consequential decisions. "The algorithm said so" is not an acceptable explanation for a credit denial or an employment rejection. If the system produces outputs that cannot be explained in plain language, either a more explainable model is needed or a human reviewer must be able to articulate the basis for the decision without referencing the model's internal state.

Incident escalation and response. When a high-risk AI system produces an outcome that appears incorrect, biased, or harmful, there must be a clear escalation path. Who is notified? Within what time frame? What powers does the escalation team have (suspend the system, override specific decisions, recall affected decisions)? These processes need to exist before an incident occurs, not be improvised during one.

Building the Organizational Capacity

Enhanced oversight requires organizational infrastructure. Three elements are non-negotiable.

A designated responsible role for each high-risk AI system. This is the person who owns the system's ongoing compliance, monitors its performance, and is accountable if something goes wrong. Not a team. A person. Shared accountability is a mechanism for avoiding accountability.

A review board or committee with the authority to approve, delay, or require modification of high-risk AI deployments. This body should include technical staff, legal, risk or compliance, and at least one representative from the business functions affected. It should meet on a defined schedule and have documented criteria for approval decisions.

An appeals mechanism. Individuals who believe they have been adversely affected by a high-risk AI decision should have a process to raise a concern and receive a human review. This is not just an ethical requirement. It is increasingly a legal one, particularly under the EU AI Act and analogous frameworks being developed in other jurisdictions.

Key Takeaways

• High-risk AI is defined by what it affects, not what it does technically. Decisions about credit, employment, healthcare, public benefits, and similar domains are high-risk regardless of technical sophistication.
  - Scale and automation level amplify risk. A low error rate at high volume creates systemic harm. High automation with no human review creates accountability gaps.
  - Five pre-deployment requirements apply to high-risk AI: purpose documentation, data audit, bias and fairness evaluation, human oversight design, and legal and compliance review.
  - Bias monitoring is continuous, not a one-time test. Populations and data change. Quarterly disaggregated sampling is a minimum for high-volume systems.
  - Every high-risk decision must be auditable. Log inputs, outputs, human review status, and final decisions for every consequential AI-influenced decision.
  - Assign a named accountable owner. Shared accountability for high-risk AI is a mechanism for avoiding accountability. One person must own ongoing compliance.
  - Affected individuals need an appeals path. The ability to contest an AI-influenced decision is both an ethical requirement and an increasingly legal one.