Captum

Captum is an open-source PyTorch library for explaining model predictions with attribution methods such as Integrated Gradients. Researchers and ML engineers use it to see which inputs, layers or neurons drive a model's output.

It works with most PyTorch models with little or no change to the network, and supports vision, text and other modalities. Installation is through conda or pip, and the docs include tutorials and a full API reference.

Because it is generic and extensible, researchers can implement new interpretability algorithms and benchmark them against existing ones. The project is part of the PyTorch ecosystem and hosted on GitHub.