Big Data and Machine Learning are still the most popular IT-trends: demand for Data Science specialists is growing, Big Data and Machine Learning are even discussed on government conferences. Nevertheless, with all advantages of these technologies, not near every company uses them in practice: the professionals in that sphere are costed too much. At the same time, it is hardly to predict, when the spent investment will return. However, businessmen want to understand clearly, what objective benefits that certain tools will bring.  

From a business point of view, any technology is only a means of increasing its efficiency. Machine Learning (ML) is not an exclusion. New algorithms and non-trivial models on practice are interested only for ML-specialists, analyst or data scientist. Business needs money and rates, that will control them, for example, the number of visitors to the site, the duration of visits, consumer loyalty index, average bill, conversion, bounce rate, etc. The article discusses how to measure the effectiveness of Churn Rate Prediction of service visitors using ML.

ML or cheaper alternatives: what does business need? 

Before the implementation of ML into business process, it is necessary to answer a few simple questions:

1. What is the expected increase in business metrics in monetary terms or percents? For example, increased profits, conversions, average checks or reduction of churns.  
2. What is the growth of ML-metrics needed to improve the business indicators? How the improvement of the prediction accuracy will have an impact on the real increase of company venue?
3. What are the costs of ML:the salary of specialists, equipment, time for searching and adaptation of professionals, taxes and other expenses?
4. What alternatives could be and how much do they cost?For example, it is possible to segment consumers with RFM-analysis, manually set the rules for generating recommendations on products offers, and hire a man to look through emails quickly, in a quality manner and inexpensive in order to filter them from spam. 

Choosing an efficient ML-solution that will help to receive considerably more money, rather than cheaper alternatives, will pay off faster, automate a complex business process or bring some other practical benefits.

It is also important to mention that it is difficult to measure results precisely before the ML implementation, so a stress-test of that plan can be done in addition.

Evaluation of the effectiveness of ML implementation on the example of Churn Prediction 

The issue of churn occurs really often, from telecom to retail: in programs, internet-shops and other services. Churn is a behavior of users, when he or she stops bringing benefits to the service, in other words pay for using service or a product. In this case the aim of ML is to predict that situation before it will happen. Generally, if a business loses a customer, it is hardly to return him, almost unreal. The aim of business is to take such steps that will keep consumers and do not let them stop using the service or product, because the customer acquisition is more expensive and time-consuming than customer retention. It is considered that ML is good for resolving such problems.

Firstly, for model specification it is necessary to define the hypothesis that is tested: when a customer left or retained (Fig.1). Let us consider the typical Confusion matrix, that will help to understand the correlation of the values of the predicted and actual situations (tabular 1). 

Tabular 1. Confusion matrix

From the point of business, if a customer/user is at a high risk of churning, it is necessary to retain a current paying customer: offer a discount or something else that will not let him or her churn. And if a client is not at the risk of churning, there is no need in spending time or money in order to retain a person. Choosing the best ML-metric requires an understanding the task from the point of business, in other words, it is necessary to count money. 

There are 4 variants:

• If the ML-model predicts truly, that a customer is at risk of churning, it is possible to offer a person a Discount (D), so business will earn the sum of money, that is equal to the LifeTime Value (LTV) minus the rate of discount;
• If the ML predicts the churn, but a customer are not leaving, business will lose the rate of the offered discount;
• If according to the ML-model a customer remains, but actually a person go to churn, so business will lose all money it could make - LTV.
• If a customer remains as the ML-model predicts, so business will not get loss or profit.

Tabular 2. Business-metric calculation according to the Confusion matrix 

Then the tabular 3 shows the results of the confusion matrix after the validation of the ML-model.

Tabular 3. The results of the confusion matrix after the validation of the ML-model.

Actually, the type I error has grown: on the model validation 6% of clients remain. How to estimate the result from such prediction? On the one hand, types I and II errors are not serious. On the other hand, how can the valuation be converted into real money? Pay attention, the simple multiplying by the prediction accuracy and the relevance of income at the particular cells will not give a right answer.

How to evaluate the result from the implementation of ML?

To evaluate the effectiveness correctly, it is important to estimate, how many percents of customers, who was at risk of churning, will accept the discount and remain after that.It is necessary to make the A/B-test:

• variant А – discounts are not offered to anyone, but we are predicting, if a client will go to churn or will remain with the help of ML-recommendations;
• variant В – discounts are offered to the people, who are at risk of churning, as the ML predicts.

The resulra of the A/B-test are given in the the tabular 4.

Tabular 4. The results of А/В-test

Totally, without any discounts (variant А) 30% of customers went to churn, with discounts (variant В) – 28%. Therefore, 2% (30-28=2) of all users or 6,6% from those who was at risk of churning, managed to keep with ML and recommendations according to the model.  The real effect is:

(2%*LTV-8%D) = X*LTV-Y*D, where:

• X – is a difference betweenАиВin percents of users,who remained as a result of ML;
• Y - percentage of users, who got the discount and remained. 

The economic impact for that example is: (2%*LTV-8%D)*U – С,

where U – the number of users of service, and C – implementation costs, considering all expenses incurred on that action: А/В-test, team, ML-instruments, etc.

The first important thing that if the LTV is four times less than the rate of the discount, than the project automatically is unprofitable.

To prove it, the simple example. Suppose that we have the following figures:

• LTV=7 000 rubles;
• discountD=500 rubles;
• implementation expenses onML amounted2 millionrubles.

With such indicators valuesyou need at least 20 000 users just to recoup costs.

The figure 2 shows the example of diagram correlation of churn with the number of indicators.

Instead of conclusion

It is worth remembering, that LTV are «long money», that are coming from customers gradually. So it is important to choose carefully the discount rate in order not to worsen the whole amount of revenues by the measure of remaining clients.

Sometimes the task of the Churn Rate Prediction with ML, despite  its comprehensibility and attractiveness, can almost not be offset. Therefore, it is necessary to estimate the feasibility of the project, before launching it. For example, try to offer the discount for a small group of customers and see, how many percents are accepting it and how it is improving the business.

In that way, we are ending up right back where we started: specialists of machine learning have to understand the business and count money, that business could earn or save on the implementation of ML. Only when preliminary calculations would prove, that applying the Machine Learning for solving the particular problem is cost-effective and potentially profitable, so it is worth launching that IT-project.