The call comes at 2:17 AM. A 230 kV circuit has tripped and the operator is asking why the AI-assisted switching system recommended the configuration that left the bus exposed. In the next 90 seconds, the operator needs to restore service. In the next 90 days, a NERC investigation team will need to reconstruct exactly what the AI recommended, what the operator did, and whether the AI system's role in the decision was appropriate. This lesson gives you the runbook for both windows.

The Two Clocks in an AI Grid Incident

Every AI-related grid event runs on two clocks simultaneously. The first is the operational clock: restore service, protect equipment, maintain reliability. This clock is measured in seconds and minutes. The second is the governance clock: document what happened, notify the appropriate parties, investigate root cause, and determine what changes are needed. This clock runs for weeks to months.

Most utility incident response frameworks were designed for the first clock. They are excellent at directing operators through restoration. They are less equipped for the governance clock on events involving AI, because AI introduces a new class of documentation requirement and a new category of contributing factor analysis. When the root cause of an event includes a question about an AI recommendation, the investigation needs to answer questions that a traditional relay failure or operator error analysis does not address: what did the model recommend and why, what data did it use, was the model operating within its validated domain, did the operator understand the recommendation's limitations, and was the AI system appropriately approved for use in the scenario where it was deployed.

This lesson builds a runbook that addresses both clocks for utilities with AI systems operating in grid functions. The five-phase structure follows the sequence of events from the first moment of detection through the final governance review.

In an AI-related grid event, "the model recommended it" is never a complete answer and never an exculpation. The governance investigation's job is to determine whether the AI system was appropriately deployed, whether the operator's response was appropriate, and what needs to change. Both the AI system and the human operator are evaluated.

Phase 1: Immediate Response (The First 90 Minutes)

In the first 90 minutes of any grid event involving an AI-assisted system, the operational priority is restoration and the documentation priority is preservation. These two priorities do not conflict, but the documentation priority is often forgotten in the urgency of restoration.

Operational Actions: First 30 Minutes

The operator's first obligation is reliability: restore service, isolate faulted equipment, protect remaining assets. AI recommendations do not change this priority. If an AI-assisted switching system issued a recommendation immediately before the event, the operator must assess whether executing that recommendation is still appropriate given the current system state. The operator retains full authority to reject, modify, or ignore the AI recommendation. "The model recommended it" does not provide legal or regulatory cover for an action that an experienced operator would not take.

If an operator determines during the event that the AI system is producing unreliable recommendations (recommendations that contradict SCADA readings, recommendations that would violate N-1 criteria, recommendations that do not reflect the current system configuration), the operator must disable the AI advisory tool and operate conventionally. The procedure for disabling the AI advisory tool without disrupting the SCADA or EMS display must be documented in the system's standard operating procedure and rehearsed in drills. Operators should never be in a situation during a live event where they are trying to figure out how to turn off an AI tool for the first time.

Documentation Actions: First 30 Minutes

The critical documentation action in the first 30 minutes is a manual log entry noting that an AI-assisted system was active during the event. This sounds trivial but is important: most utilities' incident logs do not have a field for "AI system active: yes/no." The operator's verbal log, the electronic SCADA log, and the control room recording should all capture that the AI tool was operating and what its last recommendation was before the event.

If at all possible during the event, the AI system's operational log should be flagged for preservation before it is overwritten. Modern AI advisory systems typically log recommendations, input data, and model version. If the log buffer is finite, there is a risk that the event data is overwritten before it can be extracted. The operator should know the procedure for flagging the AI log for preservation, and this procedure should be included in the system's standard operating procedure from deployment day.

Notification: Within 90 Minutes

Within 90 minutes, the control room supervisor must notify the shift lead and the relevant operations manager. If the event requires NERC reporting (significant events require notification within certain windows), that process is initiated through the standard reporting chain. The AI system's deployment owner (the person responsible for the AI governance file) should be notified within 90 minutes regardless of whether NERC notification is required, because the governance clock starts regardless of event severity. A minor event that did not require NERC reporting may still require an AI governance investigation if the AI system's role in the event raises questions that the governance committee needs to assess.

Phase 2: Initial Investigation (Hours 1 to 24)

Once the operational crisis has passed and the system is restored to a stable state, the initial investigation begins. The first task is to secure the evidence. The second is to make a preliminary determination about the AI system's role in the event.

Evidence Preservation

Evidence in an AI-related grid event is distributed across multiple systems and must be preserved before it is overwritten or lost. The evidence list includes: the AI system's operational log (recommendation history, input data, model version, and timestamp for the event period); the SCADA historian record of grid state during the event (pre-event loading, the triggering fault, switching events, and restoration); the operator action log from the EMS/ADMS (each operator keystroke, command, and confirmation during the event period); the control room voice recording (most utilities record control room communications); weather data for the event period from the relevant weather service; and any external inputs to the AI system (day-ahead market prices, weather API feeds, recent large-load changes) that might affect what the model was optimizing for during the event.

Evidence preservation requires a defined extraction process with designated personnel. The AI governance lead extracts the AI system log. The SCADA historian extraction goes through the control system administrator. The operator action log is extracted from the EMS administrator. These steps should be documented in the incident runbook so that in a 3 AM event, someone knows exactly who calls whom and in what order. Improvising evidence extraction at 3 AM produces incomplete records and creates forensic challenges in the subsequent investigation.

Preliminary AI Role Assessment

Within 24 hours, the AI system's deployment owner should provide a preliminary assessment of the AI system's role in the event. This assessment answers five questions: Was the AI system's recommendation within its validated operating domain at the time of the event? Did the operator receive, understand, and act on the AI recommendation? Was the AI recommendation consistent with what a human expert would have recommended given the same inputs? Are there any indications of model failure (a recommendation that contradicts SCADA state, a recommendation that was produced with incomplete input data)? Was the AI system operating within the conditions under which it was approved for use by the governance committee?

The preliminary assessment does not need to be the final answer. It is a 24-hour snapshot that determines the scope of the full investigation and the level of urgency. If the preliminary assessment finds no indication that the AI system contributed to the event, the full investigation can proceed at a normal pace. If the preliminary assessment finds evidence that the AI recommendation contributed to the operator's decision in a way that may have affected the event outcome, the full investigation is escalated to priority status.

Phase 3: Full Investigation (Days 2 to 14)

The full investigation for an AI-related grid event is a structured root-cause analysis that runs in parallel with the standard grid event investigation. It has two tracks: the technical track (what did the AI system actually do) and the governance track (was the AI system properly deployed and supervised).

Technical Investigation Track

The technical track is conducted by the AI development team in collaboration with the reliability engineering team. It reconstructs the AI system's behavior during the event period: what data was available to the model at the time of the recommendation, what the model's internal state was, what recommendation it produced, and what its confidence level or uncertainty estimate was for that recommendation. This reconstruction is only possible if the operational log was preserved; it is the primary reason why evidence preservation in Phase 2 is non-negotiable.

The technical investigation should answer the domain-validity question specifically. AI models are validated on historical data and are most reliable within the domain of that data. A model trained on three years of load data before the data-center buildout is operating outside its validated domain when it recommends switching configurations on circuits serving new 200 MW data-center loads. The technical investigation should determine whether the event's operating conditions were within or outside the model's validated domain, and this finding is a key input to the governance decision about whether the AI system can resume operation without modification.

Governance Investigation Track

The governance track is conducted by the AI governance committee lead, the NERC compliance lead, and the regulatory affairs representative. It reviews the AI system's governance file against the event facts: Was the system operating within the approved use boundaries defined at Stage 5 of the deployment approval process? Did the reliability assessment at Stage 3 adequately anticipate the failure mode that occurred? Was the pre-committed metrics monitoring detecting the model's behavior in the period leading up to the event (and if not, why not)? Were there any override rate anomalies in the weeks before the event that should have triggered a governance review?

This track often produces the most important findings for future governance improvement. If the governance review reveals that the AI system was operating outside its approved use boundaries (it was being used in scenarios the governance committee did not evaluate), or that the pre-deployment reliability assessment did not consider the failure mode that occurred, or that the metrics monitoring was not functioning as required, these are governance gaps that must be remediated before the system is allowed to resume operation.

Phase 4: Notification, Reporting, and Regulatory Engagement

AI-related grid events create notification and reporting obligations across multiple regulatory channels, and failing to complete these obligations within required windows creates secondary regulatory problems on top of the primary event investigation.

NERC Reporting

If the event meets NERC's criteria for a reportable disturbance, the standard reliability event report (NERC OE-417 or the applicable form) must be completed. The report should describe the AI system's role in the event in whatever level of detail is appropriate for the event's severity. NERC has been developing frameworks for AI-related event reporting; the utility should consult with its NERC compliance lead on current reporting guidance before filing.

If the event triggers a NERC CIP compliance concern (for example, if the investigation reveals that the AI system's connection to the EMS was not properly documented in CIP change management), a separate CIP compliance notification may be required. The NERC compliance lead is responsible for making this determination within the 24-hour initial investigation window.

State Commission Notification

Many state commissions have reliability event reporting requirements that apply to events affecting customers, regardless of whether the event also triggers a NERC report. If the AI-related event resulted in customer outages, the applicable state commission notification must be filed within the required window. The event description in the state filing should include the AI system's role if it is relevant to the event's cause or duration, because the commission's reliability oversight staff will need this information if they open their own review.

If the AI system involved in the event is one for which the utility has sought or received rate-case cost recovery, the regulatory affairs lead should assess whether the event creates a disclosure obligation in any open or pending rate proceedings. An adverse event involving an AI system that the utility has publicly characterized as a reliability improvement may create an obligation to update or supplement the rate-case record.

Internal Notifications

The AI governance committee must be convened within five business days of any AI-related event with a material reliability impact. The committee reviews the preliminary investigation findings, determines whether the AI system should remain in service, suspended, or modified, and initiates any governance policy updates required by the event's findings. This meeting is documented in the committee's governance record, not just in the incident investigation file.

Phase 5: Remediation and Lessons Learned

Phase 5 begins after the full investigation is complete and the regulatory reporting obligations are met. It has three components: technical remediation of the AI system, governance policy updates, and operator training.

Technical remediation addresses whatever the technical investigation identified as contributing to the event. If the model was operating outside its validated domain, remediation includes retraining or expanding the validation domain to cover the event's operating conditions before the system resumes operational use. If the evidence preservation process failed, remediation includes changes to the AI system's logging architecture to extend the log retention window. If the model's confidence estimate was not available in the operator interface during the event, remediation includes a display update that makes confidence information visible.

Governance policy updates address whatever the governance investigation identified as policy gaps. If the system was operating outside its approved use boundaries, the governance committee must either update the approved use boundaries (after a new reliability assessment for the expanded scope) or issue an operational directive restricting the system to its originally approved scope. If the Stage 3 reliability assessment did not consider the failure mode that occurred, the reliability assessment template must be updated to include this failure mode category for all future assessments. If the metrics monitoring was not functioning as required, the monitoring system must be repaired and the monitoring policy must be updated with a compliance check process.

Operator training is the component most often skipped in post-incident remediation. If the investigation found that operators had misunderstood the AI recommendation, were uncertain about when to override, or had not rehearsed the procedure for disabling the AI tool, these training gaps must be addressed before the system resumes operation. The training update should be documented as a governance record, with operator acknowledgment signatures, so that the next time the governance committee's records are reviewed (in a NERC audit or a rate case), there is evidence that the training was actually conducted.

The Runbook: A One-Page Reference

A runbook that requires reading a full lesson to use is not a useful runbook. Here is the operational summary that belongs in the control room binder and in the AI governance file.

Worked Example: The 2 AM Switching Event

Return to the opening scenario: 2:17 AM, a 230 kV circuit trips, the operator is looking at an AI-assisted switching recommendation that was issued 14 seconds before the trip.

The operator's immediate actions are operational: isolate the fault, assess the system state, initiate restoration. The operator does not act on the AI recommendation without first confirming that the current system state is consistent with the state the AI used when it generated the recommendation 14 seconds ago. The circuit trip changed the system state; the recommendation may no longer be valid. The operator notes in the control room log: "AI switching advisory system was active at time of event. Last recommendation issued at 02:16:53. Not executed due to changed system state." This 15-second documentation action is the basis for the governance investigation that starts in the morning.

By 3:00 AM, the AI governance lead has been notified, the AI system's log has been flagged for preservation, and the NERC compliance lead has been texted (they will assess notification requirements in the morning). By 8:00 AM, the evidence package is assembled: AI log preserved, SCADA historian segment extracted for the 02:00 to 03:00 hour, operator log exported, control room recording secured.

The preliminary AI role assessment at 24 hours finds that the AI system's recommendation was for a switching configuration that would have been valid given the pre-trip system state but does not properly account for the dynamic loading that occurs when the large data-center load on that bus ramps from 60% to 95% of nameplate in the 30 seconds before the trip. The model was not trained on this load ramp rate. This is a domain-validity finding: the event conditions were outside the validated training domain.

The governance investigation finds that the governance committee's Stage 1 technical submission for this AI system did not include this load ramp scenario in the described failure modes. The Stage 3 reliability assessment did not test the fallback procedure under conditions of rapid large-load ramp. Both of these are governance gaps. The committee suspends the AI system for the relevant circuit cluster pending retraining and a new Stage 3 reliability assessment that specifically includes large-load ramp scenarios. Within 45 days, the system is back in service with an expanded training dataset and updated operator training that includes a large-load-ramp scenario drill.

The event is documented in the NERC reliability event report with a description of the AI system's role, the domain-validity finding, and the corrective actions. The governance committee's suspension and remediation actions are documented in the committee record. When the rate-case team prepares the next year's annual AI performance report for the commission, the event is disclosed in the adverse-events section with the full corrective action record. The disclosure is five paragraphs. The alternative, discovering the event through commission discovery without prior disclosure, would have been significantly more damaging.

Key Takeaways

• An AI-related grid event runs on two clocks: the operational clock (restore service in minutes) and the governance clock (investigate, report, and remediate over weeks). Both clocks start at the same moment and must both be managed.
• Evidence preservation in the first 30 minutes is the non-negotiable Phase 1 documentation action. The AI system's operational log, SCADA historian record, operator action log, and control room recording must be secured before they are overwritten.
• The preliminary AI role assessment within 24 hours answers five questions: was the system within its validated domain, did the operator understand the recommendation, was the recommendation technically sound, are there signs of model failure, and was the system within its approved use boundaries.
• The full investigation has two parallel tracks: the technical track (model reconstruction and domain validity) and the governance track (approved use boundary review, metrics monitoring review, Stage 3 reliability assessment adequacy).
• Proactive disclosure of AI-related events in the commission's annual AI performance report is far less damaging than having the event discovered through commission discovery. Transparency builds regulatory credibility; concealment destroys it.
• Phase 5 remediation must include documented operator training with acknowledgment signatures. Skipping the training update, the most commonly omitted step, leaves the same human-AI interaction vulnerability in place for the next event.
• Every AI system must have a standard operating procedure that includes the log-preservation procedure and the AI-tool-disable procedure. Operators should never encounter either of these for the first time during a live reliability event.