Overview

Chisom Sethi had been running production AI systems for two years when he discovered that one of his fraud detection models had been quietly underperforming for eleven weeks. The model was still running, still scoring transactions, still generating alerts. But it had drifted - the input data distribution had shifted after a payment processor change, and no one had noticed the model's false negative rate climb from 2.1% to 8.7%. Fraud losses in that period were $340,000 higher than they would have been had the drift been caught early. "The model was trusted because it was running," Chisom told me. "We had confused uptime with integrity."

Model and data integrity is the practice of ensuring that AI systems continue to do what they were designed to do - not just that they continue to run. A model that is operating but has been compromised, degraded, or tampered with is not an asset. It is a liability masquerading as one.

The Integrity Problem

Traditional software integrity is relatively well understood. A program either runs or it does not. Its code is either intact or it has been changed. Automated checksums and access controls handle the core cases.

AI model integrity is more complex. A model can be running correctly from a software perspective while behaving incorrectly from a performance perspective. Its code and weights can be intact while its outputs have degraded. The model has not failed. It has drifted.

There are three distinct integrity risks for AI systems, and they require different monitoring approaches.

*Model compromise*: the model has been deliberately tampered with - weights modified, a backdoor introduced, or the serving infrastructure altered. This is an active security threat.

*Model drift*: the model's performance has degraded over time because the real world has changed in ways the training data did not anticipate. Chisom's fraud model was a classic example. This is a natural maintenance challenge.

*Data integrity failure*: the data flowing into the model has been corrupted, altered, or changed in structure without corresponding model updates. The model is doing the right computation on the wrong inputs.

Model Provenance and Signing

*Provenance* answers the question: where did this model come from? For AI systems, provenance includes the training data (what was used and when), the training process (what code, parameters, and compute resources), the evaluation results (what performance was measured and accepted), and the people involved (who trained it, who reviewed it, who approved it for deployment).

Provenance documentation serves two purposes. First, it enables forensic investigation when something goes wrong - you can determine what changed between a model that performed well and a model that is performing poorly. Second, it enables trust - a model with documented, auditable provenance is more defensible under regulatory scrutiny than one that appeared from a data science notebook with no record of its origins.

*Model signing* extends the concept of cryptographic code signing to AI models. A digital signature is generated from the model weights at the point of approval for production. Any subsequent alteration to the weights - whether accidental or deliberate - invalidates the signature and triggers an alert. Implementing model signing requires only a few additional steps in your deployment pipeline and provides strong protection against tampering.

Practical minimum: maintain a model registry - a controlled system that records every model version, its provenance metadata, its evaluation results, and its deployment history. No model should reach production without a registry entry. Many organisations use MLflow, Weights & Biases, or cloud-native equivalents for this purpose.

Model Versioning and Rollback

Model versioning is the practice of maintaining labelled, accessible copies of each production-ready model version. If a newly deployed model performs poorly - or if an integrity issue is discovered - versioning enables rollback to the last known good version in minutes rather than hours.

The minimum viable versioning practice: every model deployed to production is assigned a version number, the model weights are stored in a designated, access-controlled location, and the deployment record specifies which version is currently live. This sounds simple, and it is - but a surprising number of organisations skip it and discover they cannot roll back when they need to.

More sophisticated versioning uses A/B serving infrastructure: routing a percentage of traffic to a new model version while keeping the previous version in place. This limits the blast radius of a poor-performing new model - you discover the issue affects 10% of traffic before it affects 100%.

Detecting Model Drift

Drift is the most common integrity problem, and it is entirely predictable - the world changes, and models trained on past data become less accurate over time. The question is not whether your models will drift, but how quickly you will detect it.

There are two types of drift worth monitoring.

*Data drift*: the statistical distribution of inputs to the model has changed compared to the training data distribution. This does not always cause performance degradation - but it is an early warning signal. Chisom's payment processor change altered the distribution of transaction fields. That change should have been caught at the data level before the performance impact became visible.

*Performance drift*: the model's outputs have become less accurate relative to ground truth. This is the ultimate measure of integrity, but it requires ground truth - actual outcomes - to measure, which may only become available weeks or months after the prediction was made. For fraud detection, you know whether a transaction was actually fraudulent eventually. For customer churn prediction, you know in 90 days whether the customer churned.

The practical monitoring approach combines both. Set statistical thresholds for input feature distributions and alert when any feature deviates significantly from training-time distributions. Maintain a shadow evaluation dataset that is periodically run through the model to check performance metrics. Schedule full performance reviews at regular intervals - monthly for high-stakes models, quarterly for lower-stakes ones - using accumulated ground truth data.

Data Integrity Verification

The data pipeline feeding a model is as much a security surface as the model itself. Data that has been corrupted, accidentally altered, or maliciously poisoned in transit produces incorrect model outputs even if the model itself is intact.

Data integrity verification means establishing checksums or hashes at key points in the data pipeline and validating them downstream. If a file that should have a specific hash arrives with a different hash, the pipeline should halt and alert. This catches both accidental corruption (a network error during file transfer) and deliberate tampering.

Schema validation - checking that incoming data matches the expected structure, data types, and value ranges - is the simpler first step. A column that should contain integer values between 0 and 1000 but suddenly contains strings is an integrity failure that schema validation catches immediately. Chisom's payment processor change introduced a new field encoding that violated the schema his model expected - schema validation would have flagged it within hours rather than weeks.

Backup Models and Graceful Degradation

For AI systems in critical operational paths - fraud detection, credit decisioning, clinical support tools - maintaining backup models is essential. If the primary model must be taken offline for integrity investigation or emergency retraining, a backup model allows operations to continue.

A backup model does not need to be as sophisticated as the primary. It may be a simpler model with lower accuracy but greater robustness. It may be a previous version of the model kept in a warm-standby state. The criterion is that it is preferable to the alternative - either no AI support, or a system operating on an integrity-compromised model.

>
"We ran for eleven weeks on a drifted model because no one's job was to notice the drift. The model was owned by no one in production. The team that built it had moved to the next project." - Chisom Sethi, describing the governance gap behind his fraud loss event

The governance lesson here is as important as the technical one: every production AI system needs a named owner responsible for ongoing integrity monitoring. Not a team, not a committee. A person. Without that accountability, integrity monitoring is the thing that is everyone's responsibility and no one's job.

Key Takeaways

• Uptime is not integrity. A model that is running can be drifted, tampered with, or operating on corrupted data. Monitoring must track performance, not just availability.
  - Model provenance documentation enables forensic investigation and builds regulatory defensibility. Maintain a model registry that records training data, process, evaluation results, and approval chain for every production model.
  - Model signing detects tampering at the weight level. Cryptographic signatures on model weights, verified at deployment and periodically during operation, catch both accidental and deliberate alterations.
  - Versioning with rollback capability is a minimum operational requirement. Store every production model version in an accessible, access-controlled location. Test rollback procedures before you need them under pressure.
  - Monitor both data drift and performance drift. Input distribution monitoring provides early warning; performance monitoring against ground truth provides definitive assessment. Use both, with different alert thresholds.
  - Every production AI system needs a named owner responsible for integrity monitoring. Collective responsibility is no responsibility. Assign a person, give them time allocation, and hold them accountable for catching drift before it becomes a loss event.