This article discusses what Explainable Artificial Intelligence (AI) is, why the transparent interpretation of ML models is so important, and what tools are available to describe and visualize machine learning results in a comprehensible way. There are many off-the-shelf tools for interpreting and visualizing ML models, packaged in special libraries. For example, Yellowbrick is an extension of scikit-learn, ELI5 is another visual library for eliminating errors in models and explaining predictions (compatible with scikit-learn, XGBoost, and Keras), MLxtend for data visualization, solution comparison, and its constituent classifiers, LIME is a package for prediction interpretation [1] and SHAP, which will be discussed below.
Gartner analytics agency has included Explainable AI (Explainable AI) in the top 10 Data&Analytics trends of 2020. This feature set, which describes the ML model with its strengths and weaknesses, is expected to help more accurately predict model behavior and identify any potential errors. This will improve the transparency and reliability of AI decisions and results, reducing regulatory and reputational risk [2].
Indeed, a lack of understanding of the process of obtaining results using Data Science tools, such as neural networks or other machine learning methods, is a serious obstacle to their mass adoption in business and everyday life. Usually, people are afraid of what they don't understand and tend to avoid it. Thus, the areola of mystery around AI in the minds of consumers slows down the practical development of this scientific field. Therefore, the transparency of ML modeling processes and the comprehensibility of their results is important for every Data Scientist. This task is described by the term "interpretability," which refers to the extent to which one understands the reason why a particular decision was made by an ML algorithm. This is how AI from "complicated magic" or "black box" becomes explainable and predictable, and hence not scary, becoming a day-to-day phenomenon. In addition, there are several other key factors why the interpretation of ML models is particularly relevant today [3].
The interpretability of the ML model increases its potential value. In reality, however, the realization of any added value incurs additional costs. In particular, it increases the working time of the Data Scientist, who is trying to show the potential of his research to the users. Therefore, one should not chase the maximum interpretability of the results without considering the context. For example, if the cost of prediction error is low (e.g., recommending a movie to a user), then it may not be worth the enormous effort to make the model more interpretable. In this case, one can limit oneself to the standard model quality validation procedure [3].
The complexity of real ML modeling is that some predictors influence the result more than others. It is possible to reveal such dependence at the stage of model cross-validation, but it does not give an exact answer to the question about the contribution of each attribute to the obtained result. However, such problems are often observed in game theory, a branch of applied mathematics devoted to the study of transactions. The key concept of this mathematical method is a game as a process involving two or more parties fighting for their interests. Each side has its own goal and uses its own strategy, which can lead to winning or losing, depending on the behavior of the other players. The point of Game Theory is to choose optimal strategies, taking into account perceptions of the other participants, their resources, and their possible actions. Mathematicians John von Neumann, Oscar Morgenstern, and John Forbes Nash, Jr. made great contributions to the development and popularization of these ideas in the 40s and 50s of the 20th century. Since the mid-1980s the mathematical apparatus of Game Theory has been actively used in economics, management, politics, sociology, and psychology [4]. The Game theory itself, as a developed mathematical field, includes many categories, such as cooperative games, in which groups of players can combine their efforts, forming coalitions to achieve the best result. In turn, we can determine the optimal distribution of winnings between players by means of the vector of Shepley (). It represents the allocation in which each player's winnings are equal to his average contribution to the general welfare under a certain mechanism of coalition formation [5]:
Where n is the number of players, k is the number of members of the coalition K
Applying the above-mentioned provisions of Game Theory to the interpretation of ML-models, the following conclusions can be made [6]:
When calculating the vector of Shepley, it is necessary to form coalitions from a limited set of attributes. However, not every ML-model allows to simply remove a trait without learning the model again "from scratch". Therefore, to form coalitions one does not usually remove "superfluous" features, but replaces them with random values from the "background" dataset. The averaged result of the model with random values of the feature is equivalent to the result of the model, in which this feature is absent at all [6].
The practical implementation of this approach is represented by a special library SHAP (Shapley Additive exPlanations), which is supported by tree ensemble models on XGBoost, LightGBM, CatBoost, scikit-learn, and pyspark. Like any Python library, installing SHAP is very easy: just write a "pip install shap" script at the command line [6].
To be fair, it is worth noting that the results produced by the library are not quite exact Shapley vectors, but an approximation of them. At the same time, ML tree ensembles algorithms such as Gradient Tree Boosting and neural networks use additional information about the model structure, performing the computation in a reasonable time [6]. SHAP is also supported by Deep Learning, currently popular in the form of Deep SHAP, an algorithm for the high-speed approximation of SHAP values. In this case, the distribution of background samples is used instead of a single reference value and the Shepley equations are applied to linearize components such as max, softmax, etc. TensorFlow and Keras models using the TensorFlow backend are also supported, and there is also prior support for PyTorch [7].
Speaking of interpreting the results of ML modeling, it is worth noting the rich functionality of the SHAP library for data visualization. In particular, it supports the main types of graphs that are most commonly used in Data Science: histograms, line graphs, scatter plots, etc. This allows the results of ML modeling to be visualized and explained clearly to Business users. Data visualization also helps Data Scientists themselves in assessing the adequacy of the model [8].
Let's consider several real cases of SHAP application. For example, to analyze employee attrition in a company using xgboost, the following Python code plots the importance of the attributes used in the model (Fig. 1) [8]:
import shap shap_test = shap.TreeExplainer(best_model).shap_values(df) shap.summary_plot(shap_test, df, max_display=25, auto_size_plot=True)
Amount of pay rise
Pay rise on the 1st day of work
The average amount of emails after 6 p.m.
Median pay raise
The average amount of emails from 0 a.m. to 8 a.m.
Lack of A grades for 2015-2017
Average pay rise
Time to write the first email of the day
Job area
Function
The average amount of emails from 12 p.m. to 6 p.m.
Time to write the last email of the day
Amount of emails during morning hours
Age
The average amount of emails from 12 p.m. to 6 p.m.
The number of SMS is more during working hours than after
Time on position
Communication drops in all channels
Invitations to meetings by mail
The interval between internet sessions
Time of receiving the first email of the day
Number of incoming emails
Moving after work
Amount of e-mails from colleagues of the same grade
Employee position
The obtained graph is interpreted as follows [8]:
Thus, we can draw interesting conclusions from this graph and check their adequacy [8]:
Based on such findings, we can draw a portrait of the quitting employee: he or she is quite young, single, has been in the same position for a long time without a salary increase, does not receive high annual grades, and has had little communication with his or her colleagues.
Similarly, we can identify the most significant factors of individual bankruptcy and predict the probability of this event for an individual client. Let's consider the SHAP chart from the financial scoring area (Fig.2) [3].
Previous loans
Interest payable
Age
NBСР request
Days since last payment
Years of experience
Income
The graph in Figure 2 is interpreted as follows [3]:
This chart allows a quick assessment of whether the simulation results are in line with expectations or common sense. In the example under consideration, the low number of previous loans increases the predicted probability of bankruptcy for most clients. And a long track record and high income lead to a lower probability of default. Notably, SHAP not only reveals trends common to the entire sample but also explains the results of the ML modeling for each particular case.
In particular, explain the prediction of bankruptcy of a particular customer using a separate chart that distributes the variables that affect the forecast. The predicted probability of default for this client is 19%, which is quite high. Low income and open credit are the most important variables. In Figure 3, the features that increase the target variable are red, and the features that decrease it are blue. Such a diagram is useful for a bank employee or loan officer when assessing the degree of confidence in the decision to grant a loan, which is formed by the ML model [3].
SHAP can also be used for web analytics, for example, to identify which factors have the greatest impact on on-site conversion. Besides, scatter charts (boxes plot) show significant variables more clearly than typical Yandex.Metrika or Google.Analytics reports (fig. 4) [6].
It is worth noting that Shepley vectors are not the only method for interpreting ML models to show the effect of individual variables on the outcome. In practice, Partial Dependence Plots (PDP), Individual Conditional Expectation (ICE) plots, and LIME (Local Interpretable Model-agnostic Explanations) are widely used [3]. However, for example, LIME does not guarantee a fair distribution of the target variable from predictors. Moreover, the SHAP value is the only method that provides a complete explanation. This is especially important when the law requires explainability, such as the right to explain at the GDPR mentioned at the beginning of this article.
Also, Shepley vectors permit contrastive explanations, allowing a prediction to be compared to a subset of the data or a single case instead of the average of the entire sample. This feature distinguishes the SHAP method from LIME, which offers a local interpretation of the linear behavior of the ML model. Finally, a key advantage of SHAP is the availability of a solid mathematical basis with axioms and proofs [9].
The flipside of all these merits are the following disadvantages [9]:
As a conclusion, it is worth noting that, for all its capabilities in analyzing and visualizing ML models, SHAP is not a silver bullet that can be applied always and everywhere without regard to circumstances. In particular, SHAP values do not identify causality, which is best determined experimentally or by analogy [10]. This corresponds to the statement "after is not a consequence", meaning that when interpreting machine learning results using the SHAP library, it is worth remembering that it only shows the degree of influence of individual predictors on the target variable, but does not explain other aspects of Machine Learning: choosing the model architecture, adjusting neural network parameters, etc. That's why SHAP is a great tool for a competent Data Scientist at the final stage, that can partially automate some tasks without reducing the complexity of the whole work.