The quarterly report was due to the state by Friday, and the program director had spent two days of her week pulling the numbers. Her food-and-shelter program served roughly 600 families a quarter, and the funder wanted the same figures it always wanted: families served, services delivered, outcomes achieved, the percentage who reached stable housing within 90 days. This quarter she had a new helper. The agency had turned on an AI reporting feature that read across the case-management system, summarized the case notes, and produced a draft outcomes report in minutes instead of days. The draft was clean. It was well organized. It put the housing-stability rate at 71 percent, up from 64 percent the prior quarter, a result that would look very good to a funder deciding whether to renew a grant that paid eight caseworkers' salaries. The director read it twice. The number was plausible. It was the kind of number you want to be true. And that is exactly why she stopped, opened the case-management system, and started counting families by hand, because a reporting number that distorts the work does not just mislead a funder. It bends the program away from the people it is supposed to serve.

What Outcomes Reporting Is and Why It Is Different

Most of this program has been about documentation that describes a single case: a case note from a home visit, a court report for one family, an eligibility determination for one household. Outcomes reporting is a different animal. It aggregates. It takes hundreds or thousands of individual case records and rolls them up into counts, rates, and trends that go to a funder, a state oversight body, a legislature, or the public. A foster-care agency reports how many children reached permanency. A benefits program reports how many households were enrolled and retained. A re-entry program reports how many participants found employment within six months. These aggregate numbers decide whether a grant is renewed, whether a program is expanded or cut, and whether the agency is seen as effective or failing.

The shift from single-case documentation to aggregate reporting changes the AI risk in a specific and important way. When AI drafts a single case note, a hallucinated detail harms one family, and a careful worker comparing the draft to field notes can catch it. When AI drafts an aggregate report, an error is not one fabricated observation; it is a wrong number that summarizes the work of an entire program. A housing-stability rate that is off by seven points is not a typo. It is a claim about how well hundreds of families were served, and it travels to a funder who will make a renewal decision on the strength of it. The verification problem does not go away at the aggregate level. It gets larger and harder to see, because a single wrong number hides the hundreds of records it was supposed to represent.

There is a second difference that matters even more. A case note is read by people who can, in principle, check it against the record. An outcomes report is read by people who cannot. The state analyst reviewing your quarterly submission does not have your case files. The funder deciding on renewal was not in the room for any home visit. They take your numbers as given, because the numbers are the only window they have into the work. That trust is the entire basis of accountability reporting, and it is also what makes a distorted number so damaging. The reader has no way to catch the error. You are the last line of verification, and in aggregate reporting there is no judge, no advocate, and no family downstream who will notice that the figure was wrong.

Where AI Helps and Where It Quietly Distorts

AI genuinely helps with outcomes reporting, and it is worth being clear about where, because the help is real and the field needs it. Pulling a quarterly report by hand is exactly the kind of slow, repetitive, error-prone work that eats a program director's week. Counting families across hundreds of records, categorizing case notes by outcome, formatting the same tables every quarter, and writing the narrative summary that funders expect: this is documentation burden in its aggregate form, and it carries the same cost as the case-level burden the rest of this program addresses. A director who spends two days every quarter assembling a report is a director who spent two days not supervising caseworkers, not reviewing safety decisions, not in the field. AI that drafts the report structure, summarizes the case-note narratives, and formats the tables can return that time.

But notice precisely what AI is doing in each of those tasks, because the risk is different for each. Summarizing narrative case notes into a program-level story is a generation task, and it carries the hallucination risk this program has taught throughout: the model can produce a fluent summary that includes a service or an outcome that the underlying notes do not support. Counting and categorizing is different. When AI counts how many families reached housing stability, it is making a series of classification judgments: it reads each case and decides whether that case counts as a "stable housing" outcome. That classification is where outcomes reporting quietly distorts, because the model is applying a definition, and if the definition is fuzzy or the model applies it inconsistently, the count will be wrong in a way that looks exactly like a clean number.

The Classification Problem

Consider the housing-stability rate from the opening. To compute it, something or someone has to decide, for each of 600 families, whether that family achieved "stable housing within 90 days." That sounds like a fact. It is actually a definition being applied to messy reality. Does a family that moved into transitional housing on day 85 count? Does a family that found an apartment but lost it on day 92 count? Does a family whose case note says "client reports she has secured housing" but with no verification count the same as a family with a signed lease in the file? Each of those is a judgment call, and the answer depends on the program's exact definition of the outcome. When a program director counts by hand, she applies the definition consistently because she knows what it means and she made the rules. When an AI counts, it applies whatever definition it inferred from the prompt and the case notes, and it may apply that definition differently to similar cases, or apply a definition that is subtly more generous than the program's official one. The result is a rate that is internally plausible and externally wrong.

This is the aggregate version of the policy-misapplication failure mode. In a single eligibility determination, the model applies an eligibility rule incorrectly to one household. In outcomes reporting, the model applies an outcome definition inconsistently across hundreds of cases, and the error compounds. A definition that is ten percent too generous, applied across 600 families, produces a rate that is meaningfully inflated. No single case looks wrong. The number looks fine. Only counting against the official definition, case by case or with a careful sample, reveals the gap.

An aggregate number is a definition applied hundreds of times. If the definition drifts, the number lies, and it lies in clean, confident, well-formatted prose.

The Three Distortions That Bend a Program

Outcomes reporting can go wrong in three distinct ways when AI is in the loop, and each one bends the program in a different direction. Naming them separately makes them easier to catch.

The Inflated Outcome

The first distortion is the inflated outcome: a reported result that is better than the work actually was. The housing-stability rate at 71 percent when the true rate is 64 percent is an inflated outcome. It can come from a too-generous classification, from a hallucinated success in a narrative summary, or from the model double-counting families who appear in multiple case episodes. The inflated outcome is the most dangerous distortion because it points the same direction as wishful thinking. Everyone in the program wants the number to be good. The funder wants to renew. The director wants to keep eight people employed. When the AI produces a number that is better than reality, the incentive to verify it hard is at its weakest, which is exactly when verification matters most. An inflated outcome that survives into a funder report is a false claim that the program is doing better than it is, and it can win a renewal the program would not have won on its true numbers. That is not a victory. It is a debt that comes due when the next quarter cannot reproduce the result.

The Buried Disparity

The second distortion is subtler and, in this field, more serious. An aggregate number can be accurate overall and still hide an inequity underneath it. A housing-stability rate of 64 percent across all families can be 74 percent for one group and 48 percent for another. The aggregate is true. It is also a lie of omission, because it buries a disparate outcome that the program has an equity obligation to see. AI reporting tools tend to produce the topline number the funder asked for, and they will not surface the disaggregated breakdown unless someone asks for it. A program that reports only the topline is reporting a number that conceals exactly the equity information the program most needs. This connects to the equity-first non-negotiable that runs through this entire program: a reporting practice that never disaggregates by race, ethnicity, language, disability, or geography is a reporting practice that is structurally blind to disparate outcomes. The buried disparity is not an AI failure mode in the narrow sense. It is a reporting design failure that AI makes easier, because the tool gives you the clean topline and asking for more feels like extra work.

The Distorted Incentive

The third distortion operates not on the number but on the work. When a program is measured on a metric, the metric starts to shape behavior, and that pressure can bend the work away from the people it serves. If a re-entry program is funded on a six-month employment rate, there is pressure to count any job, however brief or unsuitable, as a success, and pressure to enroll participants who are easiest to place rather than those who need the most help. This is an old problem in human services, older than AI, and it has a name in the research literature: the metric becomes a target, and once it is a target, it stops measuring the thing it was supposed to measure. AI does not create this distortion. But AI can accelerate it, because an AI reporting tool optimized to produce the numbers the funder wants will, if pointed at the goal of a good-looking report, find the most favorable reading of every case. The director's job is to make sure the reporting tool is pointed at accuracy, not at the number, and that the program is not quietly reshaping who it serves to make the metric easier.

The Verification Practice for Aggregate Numbers

Single-case verification means tracing every factual claim to a source. Aggregate verification is different, because you cannot trace every one of 600 families by hand every quarter and still get the time savings AI promised. The practice has to be smart about where it spends scrutiny. Here is what that looks like.

Verify the Definition First

Before checking any number, check the definition the number claims to represent. The single highest-value verification step in outcomes reporting is confirming that the AI applied the program's official outcome definition, not a plausible-sounding approximation. If the report says "housing-stability rate," the director must know exactly what the program counts as housing stability, whether the AI used that exact definition, and whether the definition matches what the funder's grant agreement requires. A surprising share of reporting errors are not arithmetic errors at all. They are definition errors, where the model counted the right cases against the wrong rule. Confirming the definition up front catches the largest category of distortion before a single case is checked.

Trace the Number to the Cases

For any headline number that drives a consequential decision, the director must be able to get from the number back to the specific cases it counts. If the report says 426 of 600 families reached housing stability, the director should be able to pull the list of those 426 families and confirm that the count is real and that each one genuinely meets the definition. A reporting tool that produces a number you cannot trace back to a list of named cases is a tool you cannot verify, and an unverifiable number should never go into a funder report. This is the aggregate equivalent of the single-case rule that any claim that cannot be traced to a source should be removed. At the program level, any number that cannot be traced to a list of cases should not be reported.

Sample and Recount

When tracing all 600 is impossible, sample. Pull a random sample of the cases the AI classified as a success and a random sample it classified as not a success, and check each one by hand against the official definition. If the AI agrees with your hand count on the sample, your confidence in the full number rises. If the AI and your hand count disagree on even a few cases in the sample, that disagreement rate, applied across the full population, tells you roughly how wrong the headline number is. A 10 percent disagreement rate on a sample is a warning that the headline number could be off by a margin that changes the story. Sampling is the practical bridge between the impossibility of checking everything and the unacceptability of checking nothing.

Always Disaggregate

For every topline outcome, ask the reporting tool to break the number down by the groups the program serves: by race and ethnicity, by language, by disability status, by geography, by age. Then read the breakdown for disparities. This step is not optional in a field bound by equity obligations. It is the step that turns a clean topline into honest reporting. If a disparity appears, it does not mean the report is wrong. It means the program has found something it needs to see, and the right response is to report it honestly and act on it, not to bury it under the aggregate. The funder that learns about a disparity from your honest report will trust you more than the funder that discovers it later in an audit you did not run.

The Decision-Aid Rule at the Program Level

The cardinal rule of this program is that AI informs and humans decide, and it applies to outcomes reporting as fully as it applies to a removal decision. At the program level, the consequential decisions are decisions about the program: whether to claim a result to a funder, how to characterize the quarter's work, what story the numbers tell. An AI reporting tool can draft the report, compute the candidate numbers, and write the narrative. It cannot decide what the program will assert as true. That decision belongs to the program director, who signs the report and is accountable for it the way a caseworker is accountable for a case note. "The reporting tool generated the number" is no more a defense in a funder audit than "the AI wrote it" is a defense in a court. The director who submits an inflated outcomes report is accountable for the inflation regardless of which tool produced the first draft.

This accountability has a sharp edge in human services because outcomes reports are not marketing. They are often legally binding representations to a government funder, and a knowingly or negligently false report can carry consequences that range from clawed-back funding to findings of fraud. A program that reports a 71 percent housing-stability rate it cannot substantiate has made a representation it may have to defend in an audit, and the audit will not accept the explanation that the AI counted it that way. The verification practice is not bureaucratic caution. It is what stands between a program and a false representation to the government that funds it.

The same logic protects the program from the opposite error. A director who blindly trusts an AI report that understates the program's true results may report a number worse than the work, costing the program a renewal it earned. Verification protects the program in both directions. The point is never to make the number look good or bad. It is to make the number true, because a true number is the only kind a director can stand behind in front of a funder, a board, an auditor, and the families the program serves.

Clean Data Is an Equity and Due-Process Issue

It is tempting to treat outcomes reporting as the dry, back-office end of human-services work, a matter of spreadsheets and grant compliance far removed from the home visit and the fair hearing. That framing misses what is actually at stake. Program data decides which programs survive, and which programs survive decides which families get served. When an inflated report wins a renewal for a program that is not actually working, the families who needed a working program are harmed, even though no single case note was wrong. When a buried disparity hides the fact that one community is being served far worse than another, the program continues failing that community with the funder's blessing, because the report never showed the gap. Aggregate reporting is where the program's equity record is written, and a reporting practice that does not look for disparity is a practice that lets disparity persist unexamined.

There is also a due-process dimension that is easy to miss. The same case records that feed outcomes reports also document the determinations that affected real families: who was enrolled, who was denied, who reached an outcome and who did not. When AI reads across those records to produce a report, it is reading the documented history of consequential decisions. A reporting practice that is sloppy about how it summarizes those decisions can mischaracterize the program's treatment of the people in it, and that mischaracterization can obscure exactly the patterns an oversight body or an advocate would need to see. Clean, accurate, disaggregated reporting is not just good grant management. It is part of how a program stays accountable to the people it serves and to the public that funds it. The director counting families by hand in the opening was not being a perfectionist. She was protecting the only thing the report is for: a true account of whether the work helped.

Key Takeaways

• Outcomes reporting aggregates hundreds or thousands of case records into counts, rates, and trends that decide grant renewals and program survival, which raises the stakes of an AI error from one harmed family to a distorted account of an entire program.
• The reader of an outcomes report cannot check it against the case files. The funder and the oversight body take the numbers as given, which makes the program director the last line of verification, with no judge, advocate, or downstream family to catch a wrong number.
• AI genuinely helps by drafting report structure, summarizing narratives, and formatting tables, returning the days a director loses to manual reporting. But counting and categorizing are classification tasks where the model applies an outcome definition that can drift from the program's official one.
• Three distortions bend a program: the inflated outcome (a result better than the work, dangerous because it aligns with wishful thinking), the buried disparity (an accurate topline that hides a disparate outcome the program is obligated to see), and the distorted incentive (a metric that reshapes the work and who gets served).
• Aggregate verification differs from single-case verification: verify the official definition first, trace any headline number back to a list of named cases, sample and recount when full tracing is impossible, and always disaggregate by race, language, disability, and geography.
• A number that cannot be traced to a list of cases should not be reported, just as a single-case claim that cannot be traced to a source should be removed from a draft.
• The decision-aid rule governs at the program level: AI can draft and compute, but the director decides what the program asserts as true and is accountable for it. "The reporting tool generated the number" is no defense in a funder audit, where a false outcomes report can carry clawback or fraud consequences.
• Clean, disaggregated reporting is an equity and due-process issue, not back-office paperwork. Which programs survive decides which families get served, and a reporting practice that never looks for disparity lets disparity persist unexamined.