Key Takeaways
- AI projects have a high failure rate, around 80%, often due to their complex, probabilistic nature compared to traditional IT projects.
- Success hinges on carefully navigating five critical steps: selection, development, evaluation, adoption, and management.
- Building user trust in both the AI and its developers is essential for adoption, even if the technology is effective.
- Thorough project selection, considering business impact and ethical implications, is crucial before development begins.
- Ongoing evaluation, experimentation, and auditing are necessary to manage AI systems, identify unintended consequences, and ensure they deliver real value.
Embarking on an artificial intelligence project can feel like a gamble, with many initiatives unfortunately falling short. If you’ve ever seen an AI project meet confusion or outright fail after launch, you’re certainly not alone.
Iavor Bojinov, an assistant professor at Harvard Business School and a former LinkedIn data scientist, highlights that approximately 80% of AI projects don’t succeed. He discussed these challenges and best practices on an HBR IdeaCast episode, insights also detailed in his HBR article “Keep Your AI Projects on Track.”
So, why do AI projects stumble so often? Bojinov explains it starts with a fundamental difference: AI isn’t deterministic like standard IT projects. An IT system usually gives the same output for the same input, but AI can produce different results even with identical prompts, as seen with tools like ChatGPT. This adds layers of uncertainty and complexity.
Failures can happen at various stages. Sometimes, a project is doomed from the start if it doesn’t address a real business need. Other times, months are spent on data and algorithms, only for the AI to have poor accuracy or exhibit unfair biases. These multiple failure points make AI uniquely challenging.
Even a technically successful AI can fail if users don’t trust it. Bojinov shared a personal experience from LinkedIn where a powerful AI tool he built to speed up data analysis by weeks was largely ignored post-launch simply because people didn’t trust it. This “if you build it, they will not come” scenario is common.
To improve the odds, Bojinov outlines five critical steps. The first is **selection**: picking the right project. This involves balancing strategic impact with feasibility, which includes having the right data, infrastructure, and, importantly, considering ethical implications like privacy and fairness upfront.
Too often, Bojinov notes, data science teams pick projects based on exciting new technology rather than genuine business needs. For most organizations, especially those new to AI, the latest tech isn’t always where the real value lies.
Thinking about trust from the very beginning is vital. This means trust in the algorithm’s fairness and transparency, but also trust in the developers—that they understand and are building for the end-users. Identifying whether the AI is for internal employees or external customers helps tailor this engagement.
For internal projects, involving employees early and often ensures the AI solves their actual problems. For customer-facing AI, focus groups and experimentation are key to refining the product.
When it comes to balancing speed and effectiveness, especially for large companies, Bojinov suggests that continuous experimentation is key. He mentions a LinkedIn study showing that leveraging experimentation improved key business indicators by about 20%. This means learning from trials and incorporating those lessons rapidly, often by using opt-in groups for beta testing new features safely.
Effective **evaluation** goes beyond just checking predictive accuracy. Bojinov shared an example from Etsy, which spent months building an infinite scroll feature that ultimately had no effect. A simpler, quicker test of their underlying hypotheses (more items per page or faster loading) could have saved significant effort.
It’s crucial to evaluate AI on real people because most products can have neutral or even negative impacts on the very metrics they aim to improve. AI doesn’t exist in a vacuum; it interacts with the entire company ecosystem, leading to trade-offs you might not foresee until deployment.
Once an AI product is adopted, the **management** stage kicks in. This involves continuous monitoring, improvement, and, crucially, auditing for unintended consequences. Audits, while sometimes daunting, are vital. Bojinov described how LinkedIn’s “People You May Know” algorithm, designed to increase connections, was found through an audit (published in Science) to also unexpectedly impact job applications and placements by increasing users’ weak ties.
This discovery led LinkedIn to adjust its internal practices to monitor these longer-term effects, showcasing the power of such audits. Algorithms have knock-on effects, and most organizations aren’t yet studying these broader impacts.
The process is cyclical. You might reach the end of the five steps only to reassess and find new opportunities. The main takeaway for leaders, according to Bojinov, is that while AI projects are significantly harder than most business initiatives, their potential payoff is tremendous. Investing time to navigate these stages with a structured approach can greatly increase the chances of creating AI products that are adopted and deliver substantial value.
