Four Challenges CDOs Must Overcome to Deliver Comprehensive Machine Learning

In November and December last year, I had the pleasure of moderating two executive roundtable discussions on the topic of “Comprehensive Machine Learning’s Competitive Advantage” at Evanta’s CDO Executive Summit series. Earlier in the year, members of our Lucidworks Customer Advisory Board had also shared issues they faced as they tried to go beyond scatter spot projects to make machine learning comprehensive throughout their respective organizations.

The biggest takeaway across both groups? The hopes, dreams, challenges and complications of delivering comprehensive machine learning are universal, even across industries and geographies. 

Turning to ML to Improve Content Findability

Employees still have a hard time finding the content they need to do their jobs. Existing enterprise search functionality has a hard time seeing across the various data repositories within the organization. When employees can’t quickly find information to answer their questions, they often assume that the information doesn’t exist.

Search is the best way to derive value from the world’s data. I described to the roundtable participants how the Lucidworks enterprise search platform leverages machine learning in many ways to make it far easier for colleagues to find information. Machine learning personalizes the data discovery experience for each user.

The hopes, dreams, challenges and complications of delivering comprehensive machine learning are universal, even across industries and geographies. 

This post shares the four challenges that those CDOs discussed and debated, as they make machine learning comprehensive in their respective organizations.

Challenge #1: Collecting and Curating Data

Those CDOs agreed that the availability of training data often slowed progress towards comprehensive adoption of machine learning. They shared how the data needed to initially train models can be hard to collect and curate, and keeping them tuned once they go into production is an added hurdle.

I recommended adopting a platform that continuously generates its own training data and then reuse those capabilities in other areas of the business. That’s the beauty of AI-powered search. It comes with a large army of employees who will give you the training data as they type in their search queries and click on the results. For example, the Lucidworks enterprise search application creates millions of signals per month (at large enterprises) that can be used to train other ML applications like question-answering systems.

Challenge #2: Application and Model Development

Suppose you’re one of the lucky ones and you do have a pristine, relevant, current set of training data. Many enterprises get tied up in knots, trying to pick the best first application to train with that data. Here’s the good news: none of our organizations is as different or unique as we think it is (at least in terms of business problems we’d like to solve with ML).

The best applications for augmenting workplace intelligence have already been identified, so why reinvent the wheel? We encourage our clients to take a “product-based” approach to adopting comprehensive machine learning. The traditional “project-based” approach is vulnerable to the “slings and arrows of outrageous fortune” (not to mention cross-functional misunderstanding between business leaders, data scientists and IT teams.) Start with a working application that’s already tried and true–like enterprise search, predictive merchandising, chat bots–and then let your data scientists create new models and applications based on a platform that’s generating relevant user signals.

Search is the best way to derive value from the world’s data.

Challenge #3: Operationalizing Machine Learning

Lucidworks promotes the product-based approach to operationalizing machine learning because we’ve all seen or heard of the high failure rate with the project-based approach. Many of the failed ML projects didn’t work because they relied on successful human collaboration between three groups with different perspectives and objectives:

• Business leaders
• Data scientists
• DevOps teams

Machines may not be very creative, but at least they reliably do what they’re told. Not so with we homo sapiens. Because Lucidworks Fusion has “AI inside”, cross-functional teams have a common point of reference for fruitful collaboration. Data scientists have a harder time building models the business doesn’t want or that the IT team cannot operate with a reasonable amount of resources. Collaboration is far easier around improving a working model, rather than building one from the ground up. This accelerates enterprise adoption of comprehensive ML.

For more on this, read “Accuracy vs Speed – what Data Scientists can learn from Search” by Radu Miclaus, Lucidworks’ Director of Product for AI, for advice on helping data scientists and search developers work together.  

Challenge #4: Making Augmented Intelligence Accessible to the Masses

All of the enterprise ML pioneers involved in those roundtable conversations wanted to use artificial intelligence to augment the human intelligence of as many of their employees as possible. This means it must be easy to use, with little to no training or advanced technical skills.

The CDOs that were furthest along on this journey, still had challenges encouraging adoption of machine learning by the hundreds or thousands of employees who are not data scientists and only know python as a dangerous snake. Organizations can make a sincere commitment to incorporating machine learning into decision-making processes, but they still need their team members to use it. That behavioral change can be slow. 

Search speeds adoption, because it is universally understood. Even young children know how to search text or browse content catalogs. AI-powered search augments everyone’s intelligence, from front-line employees to data scientists to C-level executives.

There are ways to overcome or sidestep these four common challenges to the adoption of comprehensive machine learning, and the innovative leaders who do will have a sustainable advantage over those who do not.