Welcome to AI book reviews, a series of posts that explore the latest literature on artificial intelligence.
Machine learning can do great things, from detecting faulty parts in manufacturing to predicting health outcomes and generating text and images. However, integrating ML into a product and business is easier said than done. This is why despite the excitement, most machine learning projects fail.
Part of these failures is because business professionals don’t know enough about the intricacies of machine learning, and ML professionals don’t know about the practical needs of businesses.
In his new book The AI Playbook, Eric Siegel, a leading consultant and former Columbia University professor, helps bridge the gap between ML as a science and a business practice. Siegel delves into the reasons ML projects fail and provides a framework for implementing machine learning in business.
His framework, called bizML, comprises six steps that help you plan, implement, deploy, and adjust ML models into real-world applications. Here is a brief recap of the bizML framework and how you can apply it to your organization.
The business/tech culture gap
There are a variety of reasons for the failures of machine learning projects. But Siegel underlines one that is almost present in every case: the need for improvement in the humans running and using the ML systems.
“It takes a holistic view—one that integrates business- and technology-side perspectives—to sell, educate on, socialize, and lead ML projects,” he writes. “Lacking this, organizations often fail to bridge the business/tech ‘culture gap.’”
On the one hand, data scientists focus on improving the accuracy of their models without thinking about the production environments where the models will be deployed. They ignore the business processes and go directly to modeling.
On the other hand, business professionals often ignore the technical details of machine learning models. As a result, they tend to have unrealistic expectations from machine learning and underestimate the changes they must make to their business processes and operations to take advantage of the models.
Assembling the right ML team
To bridge this gap and successfully launch ML projects, Siegel makes three key recommendations. First, business professionals must gain knowledge and familiarity with key ML fundamentals.
Second, organizations must create an environment where data scientists and business professionals team up to work on ML projects. This can help organizations discover and handle the radical changes that must be applied to current operations and establish important business goals that would go unnoticed if the data scientists were left to their own devices.
“To keep things on track and grease the wheels for operationalization, business-side stakeholders must be enlisted to deeply collaborate with data scientists and weigh in at each project step, end to end,” Siegel writes.
Finally, organizations should plan ML projects backward. The first step to the ML project should be to define the end goal and how the company intends to improve its operations. All the other work, including gathering data, training models, and evaluating results, comes after.
BizML
Siegel’s end-to-end framework for ML projects is called bizML, and as he recommends, starts from the end goal and moves backward:
1- Establish the deployment goal: Define clearly how ML will affect your operations
2- Establish the prediction goal: Determine what the model will predict and how it relates to the deployment goal
3- Establish the evaluation metrics: Determine the metrics that matter and the performance level required to achieve the deployment goal
4- Prepare the data: Define what the data must look like and prepare the datasets
5- Train the model: Use the data to train your machine learning model
6- Deploy the model: Integrate the model into your product to make predictions on new data coming from business operations
After finishing step 6, you are just at the beginning of the process of refining and improving your machine learning operations. “BizML only begins an ongoing journey, a new phase of running improved operations—and of keeping things working,” Siegel writes. “Once launched, a model requires upkeep: monitoring it, maintaining it, and periodically refreshing it.”
Rethinking ML projects
What is interesting about bizML is that model training and deployment are the very last steps of the process. This is in contrast to what you experience in ML courses, where the data is usually ready and you immediately get to modeling.
Before working on the models, you must prepare the data. Data engineering and preparation are among the main challenges of real-world machine learning applications. You usually don’t have ready-made datasets to train and test your models. Your data might not even be in one location and has to be gathered from different sources. It has to be cleaned, filtered, and formatted in a way that can be consumed by machine learning models. And you have to carry out many experiments to make sure your data is representative of the real-world environment and has the right balance of different types of examples the model will face in production.
But even before you think about data and models, you must establish your business goal and metrics. “Before the technical magic, you must perform business magic,” Siegel writes.
This is why you must have a deployment goal, a prediction goal, and the right choice of metrics before preparing the data and developing a model. These business-oriented decisions help bring stakeholders on the same page on all aspects of the project, including what to expect from the ML system, how to integrate it into everyday operations, and how to evaluate its performance.
These steps will ensure that business professionals dive more deeply into the mechanics and arithmetics of the ML system. Accordingly, they will push data scientists beyond their technical comfort zone and encourage them to work more closely with business people.
By working backward, you are redirecting focus from doing a technology project to solving a real problem for your organization. “Don’t propose an ‘ML project,’” Siegel writes. “Instead, pitch and lead a project that will improve operations, with no more than a side note mentioning ML as part of the solution.”
Why is this important?
The tight collaboration between data scientists and business professionals will help you avoid many pitfalls that can lead to failed projects. Consider the example of an ML project that aims to reduce customer churn.
A data scientist assigned with this project would grab a bunch of data rows from the company’s database, split it into train, test, and validation datasets, train a model, and hand it over to the product team for integration. That is a sure way to see the project fail.
But under the bizML framework, the data scientist will first have to work with the product team to define the ultimate goal, which will be something like “Develop a new marketing process to retain customers that are likely to cancel their subscription.” This formulation implies that the company’s operations will have to change, which will require buy-in from all stakeholders. Machine learning will be one of several tools used in this new workflow. The ML team will realize that they will need help from other teams, such as the database administrators and software developers who will be building the new marketing system and integrating it into their existing products.
Next, the team will have to establish what the model must predict. In this case, it will usually be the probability that a customer will churn within a certain timeframe. The business people will be able to determine the timeframe based on their experience in reaching out and hearing back from customers. Maybe according to their experience, it takes one month to turn a customer around and convince them to stay. Therefore, the metric will be something like “The probability that the customer will cancel their subscription next month.”
The discussions between the business and data science teams will also reveal important information such as the frequency of predictions, the costs that the team can incur on the hardware and infrastructure required to train and run the models, and more. After all, what is the use of a customer churn prediction system if the costs are higher than the revenue?
The team will then have to agree on the percentage of accuracy expected from the churn prediction model. Again, the business team will play a vital role here because they will have a more realistic view of the benefits and tradeoffs of the actions taken to prevent churn. For example, your new marketing process might involve providing a discount to customers who are likely to cancel their subscriptions. In this case, by working together, the business and data science teams might determine to reach out to all customers who are more than 80% likely to churn next month. This way, they will be able to balance the lost revenue from giving discounts to customers who would have stayed and the revenue gained from keeping customers who would have canceled their subscriptions.
With these goals established, the data scientist will have to work with the subject matter experts and data engineers to create the right dataset for training the model. For example, subject matter experts can help the data scientist divide customers into relevant segments and find and remove features that are spurious and irrelevant to churn modeling. Data engineers will be key to consolidating data from various sources and establishing the pipeline required to feed the model with continuous data.
Only then will the data scientists be ready to try out different algorithms, train their models, and test them against the validation dataset. Finally, when it comes to the deployment stage, the model will be integrated into a suite of tools that have been developed in parallel. The team will have established a set of key performance indicators (KPI) and baseline metrics to evaluate and improve the model as it is integrated into production. They will also have a pipeline that will enable them to gather new data to retrain and improve their models as new information comes in and customers interact with their system.
Successful ML projects do not necessarily replace the humans doing the work. They change the way we work to improve our productivity. “For this advanced technology to succeed, we now need improvements in humans—in the way of understanding and leadership—more than in the technology itself,” Siegel writes.
Read more about bizML and the challenges of applied machine learning in Eric Siegel’s book The AI Playbook.
(This article contains affiliate links, meaning, at no additional cost to you, we may earn a commission if you purchase something through our links. This helps us to continue providing free content and to keep the site running.)