Overview

Dragos Popescu signed what he considered an excellent AI platform contract in January. By October the same year, the vendor had changed their model three times, two of the integrations his team had built were broken, the accuracy on his core use case had dropped by 11 percentage points, and the vendor's account manager had changed twice without a handoff. "The contract was fine," he told me. "What we did not have was a relationship management process. We were treating it like buying software. It is not like buying software."

AI vendor relationships are dynamic. The systems you purchase change constantly - model updates, API changes, pricing revisions, deprecations. The compliance landscape around them shifts. Your own use of the system evolves. Managing this actively is the difference between a vendor relationship that compounds value over time and one that generates surprises you are never quite ready for.

Ongoing vendor management is not the same as the contract negotiation that preceded it. It is a continuous practice, structured around regular checkpoints, clear ownership, and defined escalation paths.

The Three Layers of Vendor Governance

Effective AI vendor governance operates at three levels simultaneously. Organizations that manage only one or two layers create blind spots.

Operational governance is the day-to-day layer. It covers system performance monitoring, integration health, user issue resolution, and routine communication with the vendor's technical team. This is the layer Dragos was missing. When the model updates broke his integrations, there was no operational monitoring process to catch the degradation quickly. By the time someone noticed, 90 days of production decisions had been affected.

Operational governance requires a named internal owner for each significant vendor relationship - someone whose job includes watching key performance indicators and escalating when they move outside acceptable ranges. This does not have to be a full-time role. For most organizations, a vendor manager who covers three to five AI vendors with a few hours per week of monitoring work is sufficient.

Strategic governance is the quarterly and annual layer. It covers whether the vendor relationship is still delivering the value it was selected to deliver, whether the vendor's product roadmap still aligns with your direction, and whether the commercial terms remain competitive. At least annually, you should be asking: if we were choosing a vendor for this function today, would we choose this one again?

That question is not a signal that you are planning to switch. It is a discipline that keeps you honest about whether inertia is carrying the relationship past its useful life.

Risk governance is the ongoing monitoring of vendor-specific risks: financial health, security posture, regulatory compliance, and concentration risk. Concentration risk is the exposure created when too much of your AI capability depends on a single vendor. If one vendor provides your core AI platform, your AI writing tools, your data labeling service, and your model monitoring solution, a change in that vendor's terms, a security incident at their platform, or a financial distress event creates a cascading impact across your entire AI operation.

Performance Monitoring in Practice

AI systems are not static products. The model underlying a generative AI tool you deployed six months ago may be different from the model running today. Without monitoring, you may not know when that change has affected your outcomes.

Four metrics worth tracking for any AI system in production:

Accuracy or output quality against a baseline. When you first deployed the system, what was its performance on your core task? That baseline should be documented. Sample-based evaluation at regular intervals - weekly for high-volume systems, monthly for lower-volume ones - tells you whether performance is stable, improving, or degrading. A degradation of more than 5% relative to baseline should trigger a conversation with the vendor.

API latency and uptime. If the system is embedded in a production workflow, response time degradation affects your users before it shows up on any vendor dashboard. Monitor latency from your environment, not the vendor's reported metrics. The vendor's infrastructure may be fast; the path between their infrastructure and yours may be slow.

Cost per unit. AI platform pricing often includes usage-based components that can drift significantly as adoption grows. Track cost per API call, per active user, or per processed document on a monthly basis. Sudden cost increases often signal either an adoption change (more users than expected) or a pricing change (the vendor quietly revised their tiers).

Terms and policy changes. Most vendors can unilaterally change their terms of service with 30 days notice. Set up automated alerts for changes to the vendor's terms of service, acceptable use policy, and data processing annexes. Review these changes when they occur, not six months later when a compliance audit surfaces them.

Cadence and Communication

A structured communication cadence with each significant vendor prevents the drift that Dragos experienced. A practical framework:

Weekly or bi-weekly: Operational check-in with the vendor's technical account team. Covers open issues, upcoming changes (model updates, API changes, maintenance windows), and anything requiring action from either side. This can be asynchronous - a shared Slack channel or weekly status email - for vendors where the relationship is stable.

Quarterly: Business review with the vendor's account team. Covers performance against SLAs, usage trends, upcoming product roadmap, and any commercial or contractual matters. This should be a documented meeting with a shared agenda and written follow-up. Dragos now requires his team to produce a one-page quarterly health scorecard for each major vendor before the meeting.

Annually: Strategic review with senior representatives from both organizations. Covers the overall state of the relationship, alignment between the vendor's direction and yours, and any renegotiation of commercial terms. This is also the moment to revisit the question: if we were selecting this vendor today, would we choose them?

Managing Model Changes and Deprecations

This is the area where AI vendor management differs most sharply from managing traditional software vendors. Traditional software versions change on a schedule you can plan for. AI models change continuously, and the change may have no visible signal except a shift in output quality or behavior.

Three practices reduce the risk of unmanaged model change:

First, lock to a model version wherever possible. Most enterprise AI APIs allow you to pin to a specific model version. This means your integration runs on the same model until you actively choose to update. The tradeoff is that you do not automatically get improvements. The benefit is that your baseline stays stable and you control the timing of change.

Second, maintain a test suite of representative use cases. When a model update does occur - either forced by the vendor or chosen by you - run your test suite against the new version before migrating production workloads. A test suite of 50 to 100 representative cases, with documented expected outputs, takes a few hours to run and can catch significant regressions before they affect real work.

Third, subscribe to the vendor's technical change log. Most enterprise AI vendors publish release notes and deprecation notices. These are often more informative than account manager communications. Read them when they arrive, not when something breaks.

Key Takeaways

• AI vendor relationships require active management, not passive monitoring. The systems change continuously. The relationship needs to keep pace.
  - Governance operates at three levels: operational (day-to-day performance and issue resolution), strategic (quarterly and annual value assessment), and risk (ongoing monitoring of vendor health and concentration exposure).
  - Track four metrics for every production AI system: output quality relative to baseline, API performance, cost per unit, and changes to vendor terms and policies.
  - Maintain a structured communication cadence: operational check-ins frequently, business reviews quarterly, strategic reviews annually.
  - Pin to model versions in production unless you have a specific reason to accept continuous model updates. Control the timing of change.
  - Maintain a test suite. Before any model migration, run representative test cases to catch regressions before they affect production.
  - Ask annually: would we choose this vendor today? Inertia is not a strategy. Regularly re-earning the relationship keeps it honest.