Why R&D AI Pilots Stall and How to Get Out of Pilot Purgatory

This content was originally discussed in the presentation, From PoC to Production: Bridging the Gap in Scientific Software Development.

AI Adoption Is Real and so is the ROI Gap

AI tool use among physical scientists jumped from 57% to 84% in a single year (Wiley, 2025), and the 2024 Nobel Prizes in both Physics and Chemistry recognized AI-driven research, which is a historically fast turn from cutting-edge technique to mainstream validation. The investment has followed: AI in materials discovery is projected to grow from $536 million today to $5.58 billion by 2034. By most measures, the technology has arrived. 

And yet Gartner predicted that over 30% of generative AI projects would be abandoned after proof-of-concept in 2025, an S&P report found that 42% of companies scrapped most of their AI initiatives that year, and the average organization abandoned roughly 46% of proofs-of-concept before they ever reached production. In R&D settings, this failure mode is easy to recognize: the formulation optimizer that only works on one dataset, the property predictor that only one person knows how to run, the Jupyter notebook for processing instrument data that nobody has touched in six months. Models that perform well on public benchmarks and fail immediately when pointed at real company data. This is what is commonly called pilot purgatory, and it is where most AI investment quietly disappears.

Why Projects Stall

Wrong problem selection

AI tools make it fast and easy to build a prototype, so many teams skip the harder question of which problem to solve first. A researcher gets excited about a new technique, builds a PoC around it, and then tries to find a business problem it solves. That is backwards.

Underestimating what scaling requires

A working prototype is not a production tool. The code from early prototyping typically carries significant technical debt. Getting from "works on my laptop" to "anyone in the organization can rely on it" requires skills in version control, containerization, cloud infrastructure, CI/CD pipelines, and user feedback loops. These are not typical parts of a researcher's toolkit. Boston Consulting Group found that only 26% of companies have the capabilities to move beyond proof of concept.

Data that is not ready

A 2024 Forrester survey of 500 enterprise data leaders found that 73% named data quality and completeness as their primary barrier to AI success, ranking above model accuracy, compute costs, and talent shortages. Only 20% of business-critical information lives in structured data. The rest lives in synthesis notes, PDF reports, email threads, instrument output files, and lab notebooks. Companies with strong data integration achieve more than 10x the ROI from AI tools compared to those with poor integration. 

A Five-Phase Path to Production

Phase 1: Ideation. Scientists and developers work together to identify the highest-value problems. Data gaps get surfaced before building starts.

Phase 2: Prototyping. Scientist-led, rapid iteration, AI-assisted development. Organize and label key data, build something testable, get it in front of real users.

Phase 3: Validation. Test against real data. Get eyes on the code. Make sure data integration holds up under realistic conditions.

Phase 4: Hardening. Containerize, document, secure, set up CI/CD. Unglamorous work. Also what separates a prototype from something people can actually rely on.

Phase 5: Deployment. The tool is live, but iteration continues. Collect user feedback. Feed findings back into new ideation.

Scaling is 80% of the work. If your project plan ends at the prototype, you are planning for pilot purgatory.