Why most AI pilots never ship, and five decisions that change the odds

In 2025, researchers at MIT reported that roughly 95% of the enterprise generative AI pilots they studied had produced no measurable effect on profit and loss. That figure was quoted everywhere, usually as proof that the technology is overhyped.

We read it differently. In our experience the models are rarely what fails. Pilots stall because of choices made in the first fortnight, long before anyone writes a prompt. Get those choices right and the odds change completely.

Here are the five we now insist on before we build anything.

1. Pick a process with a number already attached

The most common pilot we see is "let's try AI on customer service" or "let's see what it can do with our documents". That isn't a project. It's a hobby with a budget.

Start instead with a process that already has a measurable cost: hours spent, response times, error rates, leads lost after hours. Write down today's figure and the figure that would make the project worth paying for. If nobody can name either number, choose a different process.

If you can't say what success looks like in one sentence, the pilot will end in a debate rather than a decision.

2. Use real data from the first week

Demos are built on clean, invented examples. Real customer messages are misspelt, angry and written in three languages at once. Real invoices are scanned sideways.

Collect a few hundred genuine examples before the build starts, including the ugly ones. An agent that performs well on your worst cases will cope with everything else. One that has only seen tidy data will fall over in its first hour of live traffic, and the project will lose the trust it needs to survive.

3. Decide what the agent may do on its own

This is the decision teams most often postpone, and the one that kills the most pilots. Halfway through, someone senior asks, "Wait, it can issue refunds?" and everything stops for a month of meetings.

Settle it up front. List every action the agent could take and sort each one into three groups:

• Alone: safe, reversible, low value. Answering a question, checking a status.
• With approval: anything involving money, personal data or a commitment. A person clicks yes.
• Never: the things only a human should do, however good the agent gets.

Sign it off with the people who carry the risk. Then build exactly that.

4. Write the exam before you build the student

Most pilots are judged on impressions. Someone tries a few questions, likes the answers and says it's ready. Someone else tries different questions, hits a bad answer and says it isn't. Neither is evidence.

Build a test set first: real inputs with the correct outcome agreed by the people who do the job today. Score every version of the agent against it. Agree the pass mark in advance. When the agent clears it, you launch. When a model update arrives next year, you run the same test and know within minutes whether anything got worse.

5. Plan for the day after launch

A pilot has a team and a deadline. Production has neither, unless you plan for it. Before launch, answer three questions:

• Who owns this agent's performance, by name?
• Who watches the numbers each week, and what happens when they slip?
• What does it cost to run per month, and who approved that budget?

Agents drift. Products change, policies change, customers find new ways to ask old questions. Without an owner and a routine, a good agent quietly becomes a bad one, and nobody notices until a customer complains.

And one more: own what you build

Whatever you build, make sure the code, the prompts, the test set and the accounts sit in your name. AI moves quickly. The model that is best today won't be best in eighteen months, and the vendor you start with may not be the one you finish with.

If you own the pieces, switching is a weekend's work. If you don't, it's a new project from scratch.

The short version

Pick a process with a number. Use real data. Decide the rules of approval. Test before you trust. Give it an owner. Then keep the keys.

None of this is glamorous, and none of it needs the latest model. It's simply the difference between a pilot that ends with a slide deck and one that ends with an agent doing a job every day.