Why AI Pilots Die Between the Demo and the Factory Floor

The pilot was shown at the board meeting, everyone applauded, budget was approved for ‘scaling.’ Six months later, the computer vision system that caught 98% of defects during the demo is catching maybe half, quality inspectors have stopped trusting it, and the project has quietly migrated into the ‘deferred initiatives’ column. This is not a rare mishap or the fault of a particular integrator. According to RAND, this is how more than 80% of corporate AI projects end – and almost always for reasons that were visible before the project even started.

The most uncomfortable part of that statistic is not the failure rate itself but the fact that technology is almost never to blame. RAND, in a 2024 study, interviewed 65 data scientists and engineers with 5 to 30 years of experience and dissected the reasons AI projects never reach production. The conclusion sounds almost insulting: the single most common cause of failure is that executives and technical teams understood the problem differently. Not data, not the model, not GPUs. Problem definition.

For a manufacturing leader, this is actually good news. If the issue were the technology, all you could do is wait for the next generation of models. But since the issue is management decisions – the outcome is within your control. Below are seven signs that a pilot is doomed, drawn from RAND, McKinsey, and post-mortems in automotive manufacturing. Each sign is a question worth asking your team before you sign the budget for scaling.

First, about scale: why ’the pilot worked’ means nothing

Getting a pilot off the ground today is not hard. Getting it to a state where it is used daily across three shifts and shows up in the P&L – that is hard. The gap between those two points is the main trap.

McKinsey, in a 2025 report on scaling AI in manufacturing, offers a number that should sober anyone up: only 2% of manufacturers believe AI is fully embedded in their operations. About two-thirds are stuck at the exploration and point-solution stage – endless pilots, in other words. The MIT report we covered separately paints a similar picture: for 95% of companies, generative AI has not produced measurable profit impact.

In the auto industry, this plays out in its purest form. According to analysis by Automotive Manufacturing Solutions, most computer vision projects for quality control never make it past the test cell – not because they fail to spot defects during the demo, but because the demo and the real shop floor are two different environments.

Now, the signs.

Sign 1. Nobody can put a dollar figure on the problem

RAND places misunderstanding the problem at the top of its failure list, and in manufacturing, this shows up in a specific way. The pilot gets launched because ‘we need to adopt AI’ or because a competitor showed something off at a conference. No one can articulate what specific financial loss the project is meant to address.

McKinsey documents the same disease quantitatively: the majority of manufacturers have no link between their AI initiatives and concrete value targets. A project without a price tag on the problem cannot be prioritized, cannot be shut down, and cannot be declared a success – it simply has no criterion. We covered how to build that business case with real data in a separate piece.

Before launching a pilot, try filling in one line: ’this problem costs us $X per month because …’ If you cannot calculate X, that does not mean the task is bad. It means you do not yet understand it well enough to spend your AI budget on it. Pick one where X is obvious: line downtime, rework costs, penalties for missed deliveries.

Sign 2. The data for the task does not actually exist

The second most common cause on RAND’s list: the organization lacks the necessary data. On the factory floor, this sounds especially familiar. Data supposedly exists – SCADA writes telemetry, MES stores work orders, quality control keeps logs. But when it comes time to train or prompt a model, it turns out that half the logs are on paper, SCADA has no labeled failure history, and every inspector describes defects in their own words.

This is the same data fragmentation that automotive manufacturers complain about: data exists, but it is scattered across systems, unlabeled, and not in a common format. A model trained on a clean sample for the demo meets reality in production – and its accuracy drops.

Before commissioning a pilot, run an honest audit: where does the data for this task live, what format is it in, who enters it, and how consistently. If things start feeling uncomfortable at this stage, it is better to clean up the data first and launch AI second. The paper-layer approach we recommend starting with works precisely because it does not require a clean historical dataset – the model works with whatever you give it right now.

Sign 3. The team is in love with the technology, not the problem

RAND’s third cause is stated bluntly: the project is more excited about deploying the latest technology than solving a real problem. In manufacturing, this looks like a rush to build computer vision with fine-tuned neural networks where a simple text assistant could have solved the task in a week.

MIT’s report flags a related imbalance: more than half of generative AI budgets go toward flashy showcases – sales and marketing – while the greatest returns come from the unglamorous back office. On the factory floor, the showcase equivalent is the ‘smart shop’ built for investor tours; the back-office equivalent is automating shift reports and supplier claims processing, which will never impress a boardroom but genuinely saves hours.

One test question for whoever is championing the pilot: ‘If this technology did not exist, how would we solve this problem?’ If there is no clear answer, the solution was invented before the problem. Steer the conversation from the model back to the task.

Sign 4. The task is beyond what today’s AI can handle

The fifth cause on RAND’s list: AI is applied to tasks that are too complex for it. This is not a verdict on the technology – it is a question of boundaries. AI is good at structuring text, recognizing common defects, finding analogues for parts. It struggles where rare events with no history are involved, where physical reasoning about unfamiliar equipment is needed, or where the cost of error is a human life and there is no safety net.

It helps to sort tasks into three buckets: ‘AI does it alone,’ ‘AI drafts, a human approves,’ and ’leave AI out for now.’ Most manufacturing tasks will honestly land in the middle bucket – and that is fine. Problems start when a task from the third bucket gets forced into the first under deadline pressure.

Sign 5. The drift nobody remembers six months later

This is a trap specific to manufacturing that office deployments barely encounter. A computer vision model that performed brilliantly at launch starts making mistakes a few months later – not because someone broke it. The lighting in the shop changed, the supplier switched to a steel grade with a slightly different sheen, a camera wore out, the season shifted humidity levels. The model stayed the same; the world around it moved.

An analysis of this effect in the auto industry reveals a pattern: inspection systems degrade not immediately but months after a successful pilot, when the implementation team has already been disbanded and nobody thought about retraining. A pilot with no plan for life after launch is doomed regardless of how good the start was.

The project plan should cover not just launch but ongoing operation: who checks accuracy and how often, what threshold triggers retraining, and who is accountable. If the answer is ’the implementation team will leave, and we will figure it out later,’ you are paying for a pilot that will die on schedule.

Sign 6. Everything depends on two or three people

RAND’s fourth cause: lack of infrastructure for data management and model deployment. In practice, this often masquerades as a staffing dependency – the pilot works because two enthusiastic engineers are holding it together with duct tape. They go on vacation, change jobs, or get pulled onto another project, and the system stops – because the infrastructure that would keep it alive without them does not exist.

Check whether the project would survive the departure of any single person. If the answer is ’no,’ you do not have a deployed system – you have a demo that accidentally made it to production. Before scaling, you need minimal infrastructure: a documented process, access controls, monitoring, and an update schedule.

Sign 7. The project has no business-side owner

This is the cross-cutting cause that ties all the previous ones together. RAND and BCG converge on one point: successful projects have a product owner from the business side – someone accountable for results in dollars, not for ’the model has been trained.’ For IT and product managers who are building the same kind of process within their teams, the pattern is identical. When the pilot sits entirely with IT or an outside integrator, there is nobody to frame the task in shop-floor terms, nobody to make the call to stop, and nobody to ensure the floor actually uses it every shift.

BCG describes a working ratio for implementation effort as 70/20/10: seventy percent of effort goes to people, processes, and changing how the shop floor operates; twenty to technology and data; and just ten to algorithms. If your project is the other way around and 90% of the budget is licenses and integration, the imbalance is obvious.

A business-side owner should be appointed before the project starts, not after. They need the authority to kill the project – and the motivation to carry it through to daily use.

Pulling it into a single checklist

Before signing off on the budget to scale a pilot, run through seven questions. Each ’no’ is not a reason to abandon AI – it is where the project will break if you do not fix it in advance.

• Can we put a dollar figure on the problem?
• Do we have usable data for this task?
• Are we solving a problem, or admiring the technology?
• Is this task within reach of today’s AI?
• Is there a plan for the system’s life after launch, accounting for drift?
• Would the project survive the departure of any single person?
• Is there a business-side owner with the authority to kill the project?

The RAND pattern reads clearly between the lines of this list: six of the seven questions are about management, and only one is about technology. A factory that can answer ‘yes’ to the management questions will get even a modest model into production. A factory banking on technology to carry the day will fail even with the best model available.

And here is what is encouraging amid the grim statistics. The ability to define a problem, price it, and separate ‘within AI’s reach’ from ’not yet’ – this is not an innate talent or a rare skill that only expensive consultants possess. It is a trainable management discipline. The easiest way to build it is not on a costly failed pilot but on safe practice tasks where mistakes are visible immediately and cost nothing.

Related articles

Managers and AI: The Most Frequent Users – But Not for Managing

5 Layers of an AI Agent: What Every Manager Needs to Know

40 GigaChat Case Studies vs the Benchmark: Checking Sber's Numbers

How to Implement AI in Manufacturing: A Step-by-Step Guide for Plant Leaders