Explainable AI is king, but not Jesus.
“General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models” by Christoph Molnar, Gunnar König, Julia Herbinger and more is a must-read for everyone working with model-agnostic methods from the field of explainable #AI. As the authors outline, while methods of interpretable machine learning can set free valuable insights, many xAI approaches share common pitfalls which can lead to fundamentally flawed conclusions.
📢 Here are my 𝘁𝗼𝗽 𝟱 𝘁𝗮𝗸𝗲-𝗮𝘄𝗮𝘆𝘀 from the paper: 📢
✅ 𝗼𝗻𝗲 𝘅𝗔𝗜 𝗺𝗲𝘁𝗵𝗼𝗱 𝘁𝗼 𝗿𝘂𝗹𝗲 𝘁𝗵𝗲𝗺 𝗮𝗹𝗹: which xAI method is superior heavily depends on the concrete use case at hand. Even a relatively simple xAI method, like permutation feature importance, can outperform a relatively complex xAI method, like SHAP, depending on the goal.
✅𝘅𝗔𝗜 𝘁𝗼 𝗿𝗲𝗮𝘀𝗼𝗻 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝘄𝗼𝗿𝗹𝗱 𝗿𝗮𝘁𝗵𝗲𝗿 𝘁𝗵𝗮𝗻 𝗮𝗯𝗼𝘂𝘁 𝘆𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹: many practitioners are using xAI to learn about the relationship between a feature and the target in the real world rather than simply within your model. If you train a complex machine learning model to mimic a data generating process and then use an xAI method with the goal of reasoning about that process, model accuracy is key. xAI methods used in order to learn about the underlying data can only be successful if the model is an accurate representation of the data.
✅𝗻𝗼 𝗿𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝘀𝘁𝗮𝘁𝗲𝗺𝗲𝗻𝘁𝘀 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗮𝗰𝗰𝗼𝘂𝗻𝘁𝗶𝗻𝗴 𝗳𝗼𝗿 𝘂𝗻𝗰𝗲𝗿𝘁𝗮𝗶𝗻𝘁𝘆: many xAI methods do only provide you with one estimate of a feature effect and no uncertainty. This is troublesome because even when the actual effect of a feature is close to zero, it might still show a large score when training your model on one particular dataset. When the goal is to get a realistic picture about a feature effect in the data, repeatedly computing your xAI method with a given model but different data permutations is key.
✅𝘁𝗿𝗼𝘂𝗯𝗹𝗲 𝗱𝘂𝗲 𝘁𝗼 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗰𝗼𝗿𝗿𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝘀: many xAI methods such as LIME and SHAP rely on permutations of the data - artificial data points used for generating predictions from your trained model, which in turn are used to understand the model’s behavior under different scenarios. When features show dependencies, many permutation approaches produce unrealistic data points where actual data is not observed – and interpret the model in areas where it was trained with little or no training data.
✅𝘁𝗿𝗼𝘂𝗯𝗹𝗲 𝗱𝘂𝗲 𝘁𝗼 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻𝘀: be careful when relying on global interpretation methods like partial dependence plots (PDP) or accumulated local effects (ALE) when the features of your model interact, which they will most certainly do. PDP and ALE will provide you with one aggregate picture of the effect while completely missing out on interactions.
What is the common pitfall that you deem most relevant in your work?
