Why most AI pilots return nothing — and what separates the ones that don't

If you have run an AI pilot in the last two years, the result was probably a demo that impressed the room and then quietly stopped being used. That is not a local failure. MIT's Project NANDA put the share of enterprise generative-AI pilots with no measurable P&L impact at roughly 95%, and BCG found 74% of companies struggling to achieve and scale value from AI at all.

The useful question is not why AI fails. It is why these particular projects fail while a minority get to production and stay there. The failure modes are boringly consistent — and every one of them is detectable before a line of code is written.

The number everyone quotes, and the part they skip

The 95% figure travels well because it is bleak. The more actionable finding sits next to it: the gap is not about model quality. McKinsey's state-of-AI work found that redesigning workflows has the biggest effect on whether gen AI moves EBIT at all, while only about one in five organisations using gen AI had fundamentally redesigned any workflow. Everyone bought the same models. The difference is what they changed around them.

Failure mode 1: the pilot had no owner in operations

Most pilots are owned by innovation, IT, or a strategy function — people whose success is measured by having run the pilot, not by the throughput of the process it touched. When the pilot ends, nobody's targets change, so nobody fights to keep it.

The diagnostic is blunt: name the person whose weekly numbers improve if this works. If that person is not in the room, in the demo, and in the sign-off, the project is already an exhibit rather than a change.

Failure mode 2: the process was described, never mapped

Ask three people how an invoice exception gets resolved and you get three different processes. Build against any one of them and the automation is correct for a third of reality. Deloitte's global intelligent-automation survey has put process fragmentation at the top of the barriers to scaling automation for years running, with only a minority of organisations reporting mature process definitions.

Mapping is unglamorous and cheap: sit with the people doing the work, watch twenty real cases end to end, and write down every branch, every system touched, and every judgement call. The map is also the specification. Skipping it does not save time, it moves the discovery into the build, where it costs ten times more.

Failure mode 3: the pilot automated a task, not a workflow

Summarising a document is a task. Getting the document, extracting the fields, checking them against the system of record, routing the exceptions, and closing the ticket is a workflow. Task-level automation produces an output a human still has to carry to the next step — which means the human is still in the loop for the coordination, which is where most of the time actually goes.

Workers in operations, finance, and admin roles report spending over nine hours a week moving data between systems by hand, at an estimated cost of around $28,500 per employee per year. Almost none of that time is the thinking part. It is the fetching, pasting, checking, and chasing between the thinking parts.

Failure mode 4: no exception path, so one bad output ended it

Every automation involving language or documents will be wrong sometimes. A system designed for that is unremarkable when it happens: low-confidence cases land in a queue, a human resolves them, and the resolution is logged. A system not designed for that produces a silent error, someone downstream finds it, and the story becomes "the AI got it wrong" — which is the end of the internal political runway, regardless of the error rate.

• A confidence threshold per decision, tuned so the exception rate is something the team can actually absorb.
• An exception queue that is somebody's job, not a shared inbox.
• A full log of what the system saw, what it decided, and why — retrievable per case.
• A rollback path for anything that writes to a system of record.

Failure mode 5: success was "it works", not a number

If nobody measured the process before the pilot, there is nothing to compare afterwards, and the decision to scale becomes a matter of taste. Two weeks of baselining — cases per week, minutes per case, error rate, rework rate, cycle time — converts the post-pilot conversation from an argument into arithmetic.

A pre-mortem you can run in an afternoon

Before committing budget, answer these out loud. Any question you cannot answer is where the project will fail.

1. 1Whose weekly operating numbers improve if this works, and have they said so?
2. 2How many cases per week does this process handle, and what does one case cost in minutes today?
3. 3Has anyone watched twenty real cases end to end, including the ugly ones?
4. 4Which steps are judgement, and which are pure data movement?
5. 5What happens to a case the system is not confident about, and who owns that queue?
6. 6What systems does it write to, and what is the rollback if a write is wrong?
7. 7What is the number that has to move, by how much, by when?
8. 8Who maintains this in month six, and is that time budgeted?

Pilot conditions versus production conditions

Most pilots are not scaled-down versions of production. They are different systems with different requirements, which is why the jump so often fails.

What the ones that survive do differently

Nothing exotic. They pick one high-volume, low-variance process with a named owner. They baseline it. They map it from observation rather than description. They automate the whole flow including the handoffs, and they design the exception path first. They scope maintenance as a permanent line item rather than a surprise. It is much closer to operations engineering than to data science, which is precisely why so many AI pilots — staffed and framed as experiments — never make the jump.