Deployment metrics are seductive because they are easy to collect and they always improve. Licences issued goes up. Training completions go up. Weekly active users goes up, at least for a quarter. None of these tell you whether the organisation is doing anything differently, and a benefits review will eventually notice.

The gap between a deployed capability and an adopted one is behavioural, and it is where most AI investment in delivery quietly evaporates. The tool works. People have access. And the decision that the tool was bought to improve is still being made the old way, in a different meeting, on different evidence.

What to measure instead

Adoption measures answer one question: has the decision changed? They are harder to collect and they do not always improve, which is precisely why they are worth having.

  • Proportion of relevant decisions where the model output is on the record as an input.
  • Time from signal to decision, trended — the thing the capability was supposed to compress.
  • Rate at which the output is overridden, and whether overrides are documented.
  • Number of forums where the output appears as a standing agenda item rather than on request.
  • Whether anyone asks for the output when it is late — the clearest evidence it is relied upon.

The override rate deserves particular attention, because both extremes are bad. A very low override rate usually means people are deferring to the model without scrutiny, which is a governance problem waiting to happen. A very high one means it is being ignored. Somewhere in between, with documented reasoning, is the healthy state.

The clearest sign a capability has been adopted is that someone complains when it is late.

Adoption is designed, not encouraged

Communication campaigns and training do not produce adoption on their own. What produces adoption is changing the conditions under which the old behaviour was rational. If a programme director is still judged on the RAG status they submit, they will keep managing the RAG status. If the portfolio review still opens with the manual pack, the manual pack is still the source of truth.

The practical levers are structural:

  • Put the output on an existing agenda, at a fixed point, owned by a named person.
  • Retire the artefact it replaces, on a date, with the retirement announced in advance.
  • Make the override path explicit and low-friction, so disagreement is recorded rather than silent.
  • Change what is asked for in the forums where people are held to account.

Retiring the predecessor artefact is the single highest-leverage action and the one most often deferred. As long as both exist, the familiar one wins — and the new capability is judged on being redundant.

The first difficult quarter

Every adoption curve has a point, usually one to two quarters in, where the novelty has gone and the capability has not yet proved itself on something consequential. Usage dips. Someone senior asks whether the investment was worthwhile. This is normal and predictable, and the programmes that survive it are the ones that planned for it.

What helps is having decided in advance what evidence would count. If the target was to compress time-from-signal-to-decision, and that measure has moved, the conversation is straightforward even if enthusiasm has dipped. If nobody agreed a measure at the outset, the conversation becomes about sentiment, and sentiment at that point in the curve is always poor.

Reporting the failures

A final discipline, and an unpopular one. Use cases that do not move the baseline should be reported as such and retired. There is a strong institutional pull toward keeping them alive — someone sponsored them, someone built them, and quietly dropping them looks like admitting a mistake.

Retiring them publicly does the opposite. It is what makes the surviving use cases credible, and it is the clearest possible signal that the measurement is real rather than decorative. A portfolio of AI capabilities where nothing has ever been retired is not a portfolio that is being measured.