InterpretML

InterpretML is an open-source toolkit for understanding why machine learning models make their predictions. Data scientists, auditors and researchers use it to debug models, explain individual predictions and support regulatory reviews.

It covers glass-box models that are interpretable by design, including its Explainable Boosting Machines, linear models and decision trees, as well as black-box explainers such as LIME and SHAP for neural networks and other opaque models. Explanations can be global, local or for a chosen subset of predictions.

Interactive visuals let you compare several models, study errors and dataset distributions, and run what-if analysis by editing a data point's features. The project started at Microsoft and is now maintained by a community of contributors on GitHub.