โ†
AI for Public Safety & First Responders
Proficient ยท M18 ยท lesson 18 of 18 ยท queued
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
When Not to Use AI
๐Ÿ“–
now learning

When Not to Use AI

15 min

Lieutenant Carlos Mendez is reviewing the complaint file at 9 AM on a Tuesday when he realizes the problem. A domestic violence incident that occurred three weeks ago was handled with AI-assisted report drafting. The officer who took the call ran the draft through the standard verification pass. The report is technically accurate as far as the footage shows. But the victim's name appears in the body of the AI-generated narrative in a way the officer did not catch in review, alongside the responding officer's personnel ID number and the specific address of a shelter the victim had mentioned in an offhand remark that the model incorporated into its structured output. The report went into the records management system (RMS) unredacted. Three days later it was released in response to a public-records request because no one had flagged it as a sensitive file requiring review prior to release. Mendez is now looking at a situation that involves a privacy violation, a potential safety risk to a victim, and an officer who followed the standard workflow exactly as trained. The workflow was wrong for this incident type. The kill criteria, the written list of cases where AI-assisted drafting should not be used at all, did not exist. It does now.

Why Kill Criteria Must Be Written Down

The decision to not use AI on a particular case is a judgment call. Judgment calls made by tired people under time pressure, without a reference framework, are inconsistent. One officer will use AI on a domestic violence case and produce the scenario above. Another officer on the next shift will decline to use AI on a similar case and produce a careful, hand-typed report. Neither officer is following policy, because policy does not specify when AI use is prohibited. The disparity in outcomes is not a personnel problem. It is a governance gap.

A written kill-criteria list converts the judgment call into a structured decision. It does not eliminate judgment. It provides a framework that focuses judgment on the right questions at the right time: before the officer opens the AI tool, not after the draft is already in the system. The list is a professional discipline, the same kind of discipline that governs when an officer can and cannot use certain force options, when a detective must seek a warrant rather than relying on consent, and when a records clerk must obtain supervisory approval before releasing a file. The written form is not bureaucracy for its own sake. It is accountability by design.

A kill-criteria list is not a list of cases where AI failed. It is a list of cases where the potential consequences of AI failure are too high to accept, regardless of whether failure actually occurs.

This lesson provides a model kill-criteria list built from the legal, evidentiary, and operational principles this program has developed across all three levels. Agencies should adapt it to their specific context, submit it to legal review, and publish it as part of their formal AI use policy. The list in this lesson is a starting point, not a substitute for that process.

The Seven Categories of No-AI Cases

The following categories define the cases where AI-assisted drafting should not be used, and where the officer should produce the report, summary, or documentation entirely by hand without an AI draft. Each category includes the specific reason the category exists, grounded in the legal and evidentiary principles this program has taught.

Category 1: Domestic Violence and Sexual Assault Incidents

The kill criterion: Do not use AI-assisted drafting for domestic violence, intimate partner violence, sexual assault, or child abuse reports.

Why: These incidents involve victim information that is protected under state confidentiality statutes, address confidentiality programs, and federal law (specifically the Violence Against Women Act framework). The specific location of a victim, the identity of a minor, and contact information for a shelter or support resource are categories of information that a language model will incorporate into a coherent narrative without flagging their sensitivity. The model does not know that "the victim mentioned she was staying at a house on Alcott Street" is a safety risk. It knows that a coherent narrative includes location details. The persona engineering and the verification pass are not reliable safeguards for this category because the risk is not inaccuracy but inappropriate disclosure of accurate information.

Additionally, domestic violence and sexual assault reports are frequently subject to specialized disclosure rules. Many jurisdictions require specific language, specific exclusions, and specific victim notification procedures that an AI tool is not configured to apply consistently. A report in this category that does not comply with those requirements is not just inaccurate. It may expose the agency to liability and the victim to harm.

The practice: These reports are hand-typed. The officer uses the agency's standard narrative template for the incident type and does not run any portion of the narrative through an AI drafting tool. AI transcription of the body-worn camera (BWC) audio may be acceptable if the transcript is not shared with a drafting tool, but any use of AI in the pipeline for these incident types should be specifically approved in writing by the agency's AI policy and legal team.

Category 2: Incidents Involving Minor Victims or Witnesses

The kill criterion: Do not use AI-assisted drafting for any incident where a minor (a person under the age of 18) is a victim, a witness, or a subject of a child welfare report.

Why: Minor victim and witness identities are protected by law in most jurisdictions. The officer's obligation to protect that information extends to every document in the chain of custody, including the draft that is fed to an AI tool. In the scenario of a cloud-based AI drafting tool, the recording or transcript is transmitted to the vendor's servers as part of the drafting process. Even where the vendor has signed a CJIS (Criminal Justice Information Services) Security Policy agreement, the transmission of a recording containing a minor's identity or statements is a risk that the agency's legal team must specifically authorize, not one that can be assumed to be covered by a general AI use policy.

Beyond data transmission, the narrative risk is similar to domestic violence cases: an AI model trained to produce coherent narratives will include detail about minor victims without recognizing that those details are legally protected and must be excluded from or carefully handled in the report.

Category 3: Officer-Involved Shootings and Significant Use of Force

The kill criterion: Do not use AI-assisted drafting for officer-involved shootings (OIS), in-custody deaths, or incidents where the level of force was significant enough to require administrative or external review.

Why: These are the incidents with the highest legal stakes and the highest AI risk simultaneously. The use-of-force section of any report is where language models are most likely to produce gap-fill content: phrases drawn from common use-of-force narrative patterns that are plausible but not grounded in this specific footage. A gap-filled use-of-force narrative in a standard disturbance call is a serious problem. A gap-filled use-of-force narrative in an officer-involved shooting is a case that can result in criminal prosecution, civil rights litigation, and termination.

These reports will receive exhaustive scrutiny from the prosecuting attorney's office, defense counsel, oversight boards, internal affairs, civil plaintiffs, and the media. Every sentence will be tested against the footage. The standard for the human author is already the highest in policing. Introducing an AI draft into that process adds a layer of uncertainty about provenance that these cases cannot absorb. The Brady v. Maryland (1963) disclosure obligations, the Giglio v. United States (1972) credibility disclosure requirements, and the chain of custody standards all apply with maximum force to OIS documentation.

The officer involved in the use of force writes the report. The officer's supervisor reviews it. The agency's legal team reviews it. The process is slow, careful, and deliberately human. AI does not improve that process. It introduces risk that the stakes do not permit.

Category 4: Incidents Where Credibility Is Already Disputed

The kill criterion: Do not use AI-assisted drafting for any incident where a complaint has been filed against the responding officer, where there is a known conflict between officer and witness accounts at the time of drafting, or where the agency's legal team has flagged the incident for special handling.

Why: When an incident is already the subject of a credibility dispute, the origin and drafting process of the officer's report becomes evidence in the dispute. A defense attorney, a civil plaintiff, or an oversight investigator examining a report that was AI-assisted has a straightforward argument: the officer did not write the account independently. They used a tool that draws on statistical patterns from other reports. How do we know the account reflects what this officer observed rather than what the AI inferred from the audio? In a standard case, the answer to that question is the verification pass and the disclosure documentation. In a case where credibility is already being contested, that answer may not be sufficient, and the effort spent establishing it may exceed the time the AI tool saved in the first place.

The cleaner answer, and the one that avoids the credibility question entirely for these cases, is the hand-typed report by the officer, reviewed by a supervisor, with a disclosure note confirming no AI assistance was used.

Category 5: Mental Health Crisis Incidents and Vulnerable Adult Contacts

The kill criterion: Do not use AI-assisted drafting for incidents primarily involving a person in mental health crisis, or incidents involving a vulnerable adult (a person whose status may create additional confidentiality or disclosure obligations).

Why: Mental health information is protected under the Health Insurance Portability and Accountability Act (HIPAA) framework and under state mental health confidentiality laws in most jurisdictions. The officer documenting a mental health crisis contact must be particularly careful about what information enters the official record, because that record may be accessible in contexts where mental health information disclosure is inappropriate or illegal. An AI tool does not know the difference between a reportable fact about a person's behavior and a confidential fact about a person's diagnosis or treatment history. If a subject mentioned a medication, a provider, or a diagnosis in the course of the contact, the AI will incorporate it into a coherent narrative. Whether it belongs in the official record requires a legal judgment the model cannot make.

Vulnerable adult protections add a similar layer. A person whose capacity to consent or object is compromised by age, disability, or condition is entitled to additional protections in many jurisdictions. The specific rules vary, but the principle is consistent: more protection is required, not less, and the officer applying those protections must be the one authoring the document.

Category 6: Incidents Involving Confidential Informants, Undercover Operations, or Active Investigations

The kill criterion: Do not use AI-assisted drafting for any incident that involves a confidential informant (CI), an undercover officer, or an active investigation where the report's contents could compromise operational security.

Why: This is a data security kill criterion as much as an accuracy one. When an officer feeds a BWC recording or a case file excerpt to a cloud-based AI drafting tool, that material is transmitted to the vendor's infrastructure. A recording that contains information about an active undercover operation, the identity of a CI, or the direction of a sensitive investigation is material that the agency should not be transmitting outside its controlled environment without specific legal and operational approval. CJIS Security Policy governs how criminal justice information is handled, and the agency bears that obligation regardless of vendor contractual commitments.

Even where the data security risk is managed, the narrative risk is real: an AI tool that produces a coherent narrative from the recording will include operationally sensitive details. The officer may not catch all of them in the verification pass. One missed detail in a report that enters the public records system, the defense disclosure package, or the RMS can compromise an investigation, endanger a CI, or expose an undercover officer.

Category 7: Incidents Where the Footage Is Incomplete, Missing, or Disputed

The kill criterion: Do not use AI-assisted drafting for incidents where the BWC recording is absent, significantly incomplete, or where the quality is so degraded that the model cannot be reliably grounded in the footage.

Why: The footage is the source of truth for AI-assisted report drafting. The persona engineering that constrains the model to describe only what the footage shows, and the verification pass that checks every claim against the footage, both depend on a usable recording. When the recording is absent or unusable, both safeguards are unavailable. What the model produces in those conditions is not an AI-assisted description of the footage. It is a hallucination from the audio alone, from the model's inference about what the call type typically involves, or from the CAD entry. That output is not grounded in an evidentiary record. It is a sophisticated guess.

A hand-typed report based on the officer's contemporaneous field notes, the CAD entry, and the officer's direct observations is a more reliable document in these circumstances than an AI draft that has no footage to be grounded in. The report should accurately describe what recording is available ("the BWC recording activated at 3:17 into the call and captures the second half of the contact; the first portion of the contact is documented based on the officer's field notes"), what it shows, and what its limitations are. That is honest documentation. An AI draft that fills the recording gap with plausible language is not.

Building and Publishing Your Agency's Kill Criteria List

The seven categories above constitute a starting point. Every agency operates in a specific legal, operational, and community context that will add categories, modify the criteria, or specify additional conditions. The process for building the agency's formal list should include the following steps.

The agency's legal counsel should review every proposed kill criterion against the applicable state and federal law. The domestic violence and sexual assault criterion, for example, should be reviewed against the specific confidentiality statutes in the agency's jurisdiction. The mental health crisis criterion should be reviewed against state mental health law and HIPAA guidance. The CI and undercover operations criterion should be reviewed against state investigative security statutes and department operational security policies. Legal review is not optional. It is what gives the list enforceable standing.

Supervisory Input

Supervisors who review reports daily have pattern knowledge about which incident types produce the highest risk from AI-assisted drafting. Their input should be solicited before the list is finalized. A sergeant who has seen three cases in the past year where an AI-assisted report created problems in a specific category has operational knowledge that belongs in the policy. The list is a living document. Supervisor observation is how it improves over time.

Publication and Training

The kill-criteria list is only effective if every officer, every telecommunicator, and every records staff member knows it. It should be published in the agency's formal AI use policy, incorporated into in-service training, and posted in every workstation environment where officers draft reports. It should be simple enough to be consulted in real time, at the end of a shift, when an officer is making the call about whether to use the AI tool. A list that requires a policy manual search is a list that will not be used when it matters.

The list should also specify the fallback: when AI-assisted drafting is not permitted, what is the officer required to do instead? Hand-typed narrative from field notes and direct observation is the standard fallback. For records staff, a supervisor review prior to any AI-assisted processing of a sensitive file type is the standard fallback. The list should name the alternative, not just the prohibition.

Exception Process

There will be cases that fall in the grey zone: an incident that is primarily a routine disturbance but includes a detail that touches one of the kill criteria. The exception process defines who can authorize AI use in a category-adjacent case, what documentation is required to justify the exception, and what additional review applies to the resulting output. An exception process that requires a supervisor's written authorization and a legal team notification is an exception process that takes the decision seriously. An exception process that allows officers to self-authorize without documentation is not an exception process. It is a gap in the policy.

The Relationship Between Kill Criteria and the Full Program

The kill-criteria list does not stand alone. It is the final safeguard in a system that includes persona engineering, verification passes, bias monitoring, disclosure protocols, and audit trail documentation. Those elements work together. An agency that has all of them deployed is an agency that has taken AI-assisted policing seriously at every layer of the evidentiary pipeline.

The King County, Washington, prosecutor's ban on AI-written police reports was a response to a situation where those layers did not exist. The tool was deployed. The reports were AI-generated. The human review process was not standardized. The disclosure documentation did not exist. The kill criteria had not been defined. The response to that situation, a blanket ban, is understandable. It is also a failure mode for agencies that need the time savings that AI assistance provides: officers in high-volume agencies spend 30 to 40 percent of each shift on paperwork, and the 82 percent decrease in report-writing time that Axon Draft One testing showed is a real benefit that serves both officers and the communities they work in.

The answer to the King County problem is not to abandon the tool. It is to build the infrastructure that the tool requires to be used responsibly: persona engineering, verification passes, bias monitoring, disclosure protocols, audit trails, and kill criteria. The kill criteria are the final line of that infrastructure, the point at which the agency says: "For these specific cases, the stakes are too high and the risks are too specific for AI assistance. We do this work by hand." That line, stated clearly and enforced consistently, is what makes the rest of the system trustworthy.

The EFF's transparency concerns about AI in policing are answered, in part, by a published kill-criteria list. An agency that can say "here is where we use AI, here is where we do not, here is why, and here is the policy that governs both" has answered the transparency question directly. The concerns that remain are about execution: does the agency actually follow its own list? That is an oversight and accountability question, answered by audit trails, supervisor review, and the willingness to investigate and correct violations of the policy when they occur.

Key Takeaways

  • A written kill-criteria list converts an inconsistent individual judgment call into a structured decision framework. Without it, AI use on sensitive cases varies by officer, shift, and mood, which is a governance failure, not a personnel one.
  • The seven foundational kill categories are: domestic violence and sexual assault, incidents involving minor victims or witnesses, officer-involved shootings and significant use of force, incidents where credibility is already disputed, mental health crisis and vulnerable adult contacts, incidents involving confidential informants or active investigations, and incidents where the footage is absent or severely degraded.
  • Each category is grounded in a specific legal or operational risk: confidentiality statutes, Brady and Giglio disclosure obligations, CJIS data security requirements, operational security for active investigations, or the absence of the footage foundation that makes AI-assisted grounding possible.
  • The kill-criteria list must be published in the agency's formal AI use policy, incorporated into training, and accessible in real time at the workstation. A list that exists in a policy manual and is not used at the moment of decision is not a safeguard. It is a document.
  • The fallback for kill-criteria cases is always specified: hand-typed narrative from field notes and direct observation. The list names the prohibition and the alternative, so the officer always has a clear path.
  • An exception process for category-adjacent cases should require supervisor written authorization and, where appropriate, legal team notification. Self-authorization by the officer is not an exception process.
  • Kill criteria are the final layer of an AI governance system that includes persona engineering, verification passes, bias monitoring, and disclosure documentation. They work because the other layers work. An agency that deploys kill criteria without the rest of the system has one gate and no fence.
  • A published kill-criteria list is part of the transparency infrastructure that answers the EFF's documented concerns about AI in policing and the King County-style objection to AI-written reports. It demonstrates that the agency knows where AI assistance is appropriate and where it is not, and that the line has been drawn deliberately and legally.