Overview

Zara Khalil was head of digital products at a mid-sized insurance company when her team deployed an AI underwriting assistant. The model was genuinely good - it flagged risk factors human underwriters had been missing, and early results showed a 12% improvement in loss ratios. Then a broker complained. "Your system rejected my client," she said, "and no one can tell me why." Zara's team spent three days trying to reconstruct the reasoning. They couldn't. "We had a model that worked," she told me. "We just had no way to explain it."

That gap - between a system that performs well and a system that people can trust - is what this lesson is about. Trust is not an automatic outcome of technical accuracy. It is earned through honest communication, meaningful explanation, and accountability when things go wrong.

Why Trust Is a Design Requirement

An AI system that users do not trust will not be used. Or it will be worked around - people finding informal ways to override it while maintaining the appearance of compliance. Either outcome wastes the investment and, more importantly, misses the actual business goal.

Trust has three components that are often conflated but need to be addressed separately.

*Competence trust*: does the system actually do what it claims to do? This is the technical performance question - accuracy, reliability, consistency.

*Integrity trust*: does the system behave honestly and within appropriate limits? Does it admit uncertainty? Does it stay within its intended scope?

*Benevolence trust*: does the system operate in the user's interest, not just the organisation's? This is where AI systems most often fail, because they are designed to optimise organisational metrics - efficiency, cost, throughput - which do not always align with what is good for the individual user or customer.

A trust programme needs to address all three. Focusing only on competence trust - demonstrating accuracy - while ignoring integrity and benevolence will produce a system users recognise as technically capable but still do not trust.

Communicating Honestly About AI Capabilities

Organisations have a consistent tendency to overstate what their AI systems can do. The reasons are understandable - marketing wants to generate excitement, leadership wants to demonstrate ROI, product teams want to create a compelling story. The result is dangerous: users who trust the system more than they should.

Zara's underwriting assistant was described internally as "AI-powered risk analysis." Brokers heard this as "the system makes the decision." The reality was more nuanced - the model surfaced risk signals, but a human underwriter still made the final call. That nuance was never clearly communicated.

Honest capability communication means specifying what the system does and does not do. It means quantifying confidence - "this model correctly classifies 87% of cases in this risk category, with more uncertainty in novel cases." It means being explicit about what the system was not designed to handle. And it means updating that communication when performance changes.

The practical format for this is a *system disclosure statement* - a brief, plain-language document that describes the system's purpose, its known accuracy and limitations, and what decisions it does and does not make autonomously. This should be available to any person who interacts with or is affected by the system.

Explainability: What It Actually Means

Explainability is often presented as a technical problem - how do you extract human-readable reasoning from a complex model? That framing is too narrow. Explainability is a communication problem: can you give a person a meaningful account of why a decision was made?

The level of explanation needed depends on the stakes. A recommendation for a movie requires almost no explanation. A denial of an insurance claim, a rejection of a loan application, or a flag for a performance review requires a substantive account that the affected person can understand and, if necessary, challenge.

Three types of explanation are useful in practice:

Feature attribution: which inputs most influenced this outcome? For the insurance underwriting case, this might mean telling the broker: "The application was flagged primarily because of three risk factors - prior claims history, property age, and location risk score." This does not require a technically perfect explanation of the model's internal mathematics. It requires an honest account of what drove the result.

Counterfactual explanation: what would need to change for the outcome to be different? "If the property had been built after 2005 and had no prior claims in the last five years, it would likely have qualified for standard underwriting." This is often the most useful explanation for someone who wants to understand what they can do differently.

Confidence communication: how certain is the system? Presenting every AI output as a definitive result is misleading. A model that says "this application falls in the borderline zone - I recommend human review" is more honest than one that silently forces a binary output.

Appeals Processes and Human Override

Any AI system that makes or influences decisions about people needs an appeals process. This is a legal requirement in some jurisdictions - GDPR Article 22 gives individuals the right not to be subject to solely automated decisions that significantly affect them, and the right to request human review.

An appeals process has three elements. First, a clear and accessible way for the affected person to request review. Second, a human reviewer with genuine authority to override the system. Third, a documented outcome - either the decision is upheld with explanation, or it is reversed with explanation.

The second element is where many organisations fail. They create an appeals path that nominally involves a human but where the human has been trained to follow the model's recommendation. That is not a meaningful appeal. The human reviewer needs to be empowered to disagree with the model - and to have the process and information they need to make an independent judgment.

Feedback Loops and Continuous Improvement

Trust is not a static property. It erodes when systems make mistakes and those mistakes are not acknowledged. It builds when systems demonstrate that they learn and improve over time.

A transparency programme should include a mechanism for collecting feedback from users and affected individuals. This does not have to be elaborate - a structured way for underwriters to flag cases where the AI recommendation seemed wrong, for example. But it needs to be genuinely used: someone must be responsible for reviewing that feedback, identifying patterns, and translating them into model improvements.

Zara's team built a simple feedback button into their underwriting interface. Within six months they had collected 340 flagged cases. Analysis revealed a consistent pattern: the model underweighted the impact of recent building renovations, which reduced risk. They retrained with that feedback incorporated and saw a measurable improvement in the cases where brokers had been most frustrated.

Equally important: communicate what you have done with the feedback. A quarterly update - even a brief one - telling users "you flagged 340 cases; we identified three systematic issues and have updated the model" closes the loop and demonstrates that the feedback mechanism is genuine, not cosmetic.

Trust Assessments

How do you know whether users actually trust your AI system? The answer is not to assume they do because they are using it. Use and trust are not the same. People can use a system they distrust because they have no alternative.

A *trust assessment* is a structured survey - administered to users and, where possible, to people affected by the system's decisions - that measures competence trust, integrity trust, and benevolence trust separately. It asks: do you think this system is accurate? Do you think it operates within appropriate limits? Do you think it has your interests in mind?

Run trust assessments at deployment and every six months thereafter. Track the scores over time. A declining score is an early warning signal. An improving score tells you that your transparency and accountability efforts are working.

Key Takeaways

• Trust has three components. Competence (does it work?), integrity (does it behave honestly?), and benevolence (does it serve the user's interests?) must all be addressed - accuracy alone is not enough.
  - Honest capability communication prevents over-trust. Publish a plain-language disclosure statement for every AI system that describes what it does, what it does not do, and how confident it is - and update it when performance changes.
  - Explainability is a communication problem, not just a technical one. The goal is giving affected people a meaningful account of why a decision was made - feature attribution, counterfactuals, and confidence levels are more useful than model internals.
  - Appeals processes need genuine human authority. A review path where the human is trained to follow the model's recommendation is not meaningful. Reviewers must have the information and authority to disagree.
  - Feedback loops build trust over time. Collect structured feedback from users, act on it, and communicate what you changed. This demonstrates that the system improves rather than stagnates.
  - Measure trust directly. Use regular trust assessments - not usage metrics - to distinguish genuine trust from compelled adoption. Track scores over time and investigate declines.