The pilot worked. That was the easy part
Pilots are run on the happy path by the people who wanted them to work. Production is run by everyone else, on the awkward cases, at month end.
A pilot is scoped by whoever wanted it, run on examples they chose, and judged on whether the output looked impressive. Those conditions make success close to the default. Production is a different test, and a pilot designed only to demonstrate capability was never built to pass it.
What changes between the two
How it usually goes
- 01Clean inputs chosen by the person running it
- 02Output reviewed by someone who wants it to work
- 03No deadline riding on the result
- 04Failures discussed, not logged
- 05One user, high context
How it should go
- 01Whatever arrives, including the malformed
- 02Output used by someone who did not ask for this
- 03Month end, and the number has to be right
- 04Failures need to be visible and recoverable
- 05Many users, no context, no patience
The gap is not model quality. It is everything around the model, which is the part a pilot deliberately omits in order to be quick.
The four things that stop it
Nobody decided what happens when it is wrong. This is the big one. A model that is right ninety per cent of the time is genuinely useful if the other ten per cent is caught, and genuinely dangerous if it is not. That design decision (flag for review, fail closed, escalate to a person) is the difference between a system and a liability, and it never comes up in a pilot because in a pilot a person is already reading every output.
The integration was faked. Pilots run on an export. Production runs on a live connection to a system that goes down, rate-limits, and occasionally returns something nobody expected. Building that properly is frequently more work than the AI part.
No one owns it. The pilot had a champion. Production needs an owner, a named person responsible when it misbehaves at eleven at night. A project can reach the end of a successful pilot and stall on the discovery that the champion was never going to be that person.
It was never measured against a number. If nobody recorded how long the task took before, there is no way to demonstrate it now takes less. The project becomes a matter of opinion, and opinions lose budget arguments.
Scoping the pilot so it can survive
The fix is not a bigger pilot. It is a pilot scoped to answer the production question rather than the capability question.
- Pick the ugliest real inputs, not the cleanest. Capability is not in doubt; robustness is.
- Have it used by someone who did not ask for it, early.
- Decide the wrong-answer behaviour before building, not after the first incident.
- Write down the before number on day one.
- Agree who owns it in production before the pilot starts, and if nobody will, treat that as the finding.
A pilot designed this way sometimes fails, which is the point. A cheap no is a good outcome, and a pilot built only to impress cannot produce one.
What to do this week
Find the AI pilot in your organisation that everyone agrees went well, and ask three questions about it: what happens when it is wrong, who owns it once it is live, and what the before number was. If the answers are not ready, the pilot has not finished, whatever the demo showed.
Where this goes next
Whether the thing should be built at all is a separate and earlier question: build an AI agent or buy an AI tool. Establishing whether it paid for itself is covered in how to tell whether an AI project paid for itself. And the workstreams that turn a working prototype into a system are the same ones any software implementation has to cover.
Scoping a pilot that can answer the production question is part of AI consulting.