Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces

I Introduction

Machine learning techniques, especially deep neural networks, have been successful at converting large amounts of data into complex and highly accurate predictive models. As a result, these models have been considered for a growing number of applications. Yet, their complex nonlinear structure makes their decisions opaque and the model behave as a black-box. In the context of sensitive and high-stakes applications, the necessity to thoroughly verify the decision strategy of these models before deployment is crucial. This important aspect has contributed to the emergence of a research field known as Explainable AI that aims to make ML models and their predictions more transparent for the user.

A popular class of Explainable AI techniques, commonly referred to as ‘attribution’, identifies for a given data point the contribution of each input feature to the prediction . Attribution techniques have demonstrated usefulness in a broad range of applications. They can identify contributing features in nonlinear relations of scientific interest , or enable further validation of the models at hand . However, for certain applications and data types, a simple attribution of the decision function on input features may be of limited use. Specifically, it may fail to expose the multiple reasons why a particular input feature contributes or which component of the decision strategy is responsible for that contribution.

These limitations have led to the advance of richer structured explanations. The development encompasses ‘higher-order explanations’ that aim to extract the contribution of input features in relation to other input features, and ‘hierarchical explanations’ where concepts (e.g. directions in activation space) are first extracted and then leveraged to identify joint feature-concept contributions. Proposals for hierarchical or concepts-based explanations typically construct a latent space that maximally correlates with some ground-truth annotations or learn a latent space that maximizes some statistics of projected activations . These approaches, however, do not guarantee a specific focus on features that are most relevant for the model to arrive at its decision; they may in some cases extract directions in activation space to which the model is partly invariant.

To address the general need for more structured and focused explanations, we propose to equip Explainable AI with a new form of representation learning: extracting a latent space that is both disentangled w.r.t. the input features, and relevant for the model predictions. The approach is illustrated and compared to non-disentangled and feature-only approaches in Fig. 1. Technically, we contribute two novel analyses: principal relevant component analysis (PRCA) and disentangled relevant subspace analysis (DRSA), that achieve two particular flavors of the representation learning objective. PRCA can be seen as an extension of the well-known PCA in that it combines both activations and model response into the analysis . In a similar fashion, DRSA extends subspace analysis by incorporating model response into the process of finding subspaces. As a side effect, PRCA and DRSA inherit advantageous properties of the methods they build upon, such as simplicity and ease of optimization.

Furthermore, our contributed PRCA and DRSA methods integrate transparently into a number of popular attribution techniques, in particular, Integrated Gradients , Shapley Values , and Layer-wise Relevance Propagation . Hence, any explanation produced by these common attribution techniques can be disentangled by our method into several meaningful components. Moreover, our proposed methods preserve useful properties of the underlying attribution techniques such as conservation (aka. completeness or efficiency) and their computational/robustness profile (for robustness see e.g. ). Fig. 2 shows examples of disentangled explanations produced by our PRCA/DRSA approach, where we observe that the overall prediction strategy of a VGG16 network for the class ‘basketball’ decomposes into multiple sub-strategies including detecting the outfits, the faces, and the ball.

Through an extensive set of experiments on state-of-the-art models for image classification, we demonstrate qualitatively and quantitatively that our approach yields superior explanatory power compared to a number of contributed baselines or other approaches from the literature. In particular, we observe that our disentangled explanations highlight more distinctly the multiple visual patterns used by the model to predict, and that these explanations have a low pixel-wise overlap.

Lastly, we present three use cases for the proposed disentangled explanations: (1) We show that disentangled explanations enable a simple and efficient interface for the user to identify and remove Clever Hans effects in some model of interest. (2) We demonstrate on a subset of ImageNet containing different butterfly classes how disentangled explanations help us to enrich our understanding of the relation between visual features and butterfly classes. (3) We apply PRCA to analyze explanations that are adversarially manipulated by subjecting the input image to a perturbation (see e.g. ). We find that PRCA sheds light on the manipulation by disentangling the original explanation from its adversarial component.

II Related Work

We discuss below the work on Explainable AI that is most closely related to our contribution, specifically, higher-order explanations, Explainable AI in latent spaces, and hierarchical explanations. For a broader overview of Explainable AI techniques, general discussions of Explainable AI, and applications, we refer to reviews, e.g. .

Building on findings that deep neural networks encode useful intermediate concepts in their intermediate layers , a first set of related works considers the problem of explanation in terms of abstract concepts that are represented well in intermediate layers. For example, the TCAV and IBD methods build an interpretability basis in activation space in a supervised manner, and they identify contributing interpretability units using directional derivatives in activation space or by identifying the terms of a linear model respectively. extends the TCAV framework by using clustering algorithms to find directions in latent space, bypassing the need of having the concept dataset. Alternatively, finds that using non-negative factorization yields higher fidelity than using clustering approaches. views concepts as subspaces of the representation formed in some intermediate layer and proposes a sparse clustering algorithm to extract such subspaces. The NetDissect framework offers a way of matching hidden neurons to concepts (obtained from the Broden dataset ). It enables to partition the space of activation into multiple subspaces, each of them representing a distinct concept.

Another line of work aims to extract structured explanations, in terms of groups of input features, or joint contributions of input features and concepts. Higher-order methods enable explanation in terms of pairs or larger groups of input features, thereby better accounting for interaction effects present in ML models such as graph neural networks . builds an interpretable model, called prototype networks, that support joint explanations in terms of concepts and input features. enables such structured explanation in a post-hoc manner, by extending the LRP framework to filter explanation signal that passes through different activation maps representing different concepts. Another propagation-filtering approach is applied at each layer in in order to build a hierarchical explanation. learns a surrogate graph-based model at multiple layers of a trained neural network in order to produce hierarchical explanations. proposes the context decomposition approach to extract hierarchies of input features that explain the prediction of an NLP model. The contextual decomposition approach is further extended in , in particular, addressing the question of how to explain combinations of two phrases. extracts a hierarchical explanation through the use of a second-order attribution method.

Other works leverage the joint usage of explanation techniques and intermediate representations for different tasks. detects Clever Hans phenomena in a trained model by application of a LRP procedure and builds models of LRP attribution in activation space in order to ‘unlearn’ these Clever Hans effects. inspects hidden layer representations by applying a clustering algorithm on activation and backpropagating cluster assignment onto input features using LRP. proposes a data-agnostic framework that uses synthetic images to investigate whether the intermediate representation of a trained model exhibits any potential Clever Hans effects.

Beyond the field of explainable AI, a broad range of works have addressed the question learning disentangled representations . Related to our focus on relevant subspaces, some of these works take the labels or model response into account . Further works focus not on learning disentangled representations, but on evaluating them (e.g. ). In contrast to these works, our paper proposes disentanglement techniques that specifically address open problems in Explainable AI.

III Towards Disentangled Explanations

for $i\in\{1,\dots,d_{x}\}$. That is, $R_{i}$ represents the share of the model output that has been attributed to feature $x_{i}$. A common axiomatic property that many attribution techniques embody is conservation: $\textstyle\sum_{i=1}^{d_{x}}R_{i}=y.$ Attribution techniques operate in various ways. Methods based on the Shapley value analyze the effect of multiple input perturbations on the output of the ML model. Integrated Gradients analyzes the function and its gradient along some path in the input space. Layer-wise Relevance Propagation (LRP) leverages the neural network structure of the ML model and attributes to the input features by backpropagating through the multiple layers of the network using purposely designed propagation rules. All these methods can be called ‘standard’ attribution methods because they are solutions to the well-studied problem of decomposing the output score into contributions of individual features. Yet, these standard attribution methods do not highlight why a particular input feature is relevant.

The first step attributes the output of the network onto the $K$ factors. One achieves this first step by expressing $y$ as a function of $(\boldsymbol{h}_{k})_{k=1}^{K}$ and then running some standard attribution technique on the prediction of this function. The second step takes the resulting attribution $(R_{k})_{k}$, expresses its elements as functions of the neural network input $\boldsymbol{x}$, and repeats the attribution process for each of these functions. This results in relevance scores $R_{ik}$ expressing the joint contributions of input features and factors. These joint contributions form the disentangled explanation. We provide a concrete example of this two-step procedure for LRP which we use in most of our experiments.

where $\sum_{k}$ pools contributions from all units $k$ in the layer above. Application of LRP rules from layer to layer results in scores $R_{i}$’s assigned to input features. Instead, the two-step procedure outlined in Eqs. (2)–(3) can be implemented by omitting the pooling step $\sum_{k}$ in the equation above, i.e.:

and retaining the index $k$ in the subsequent propagation steps until the input is reached in order to arrive at the scores $R_{ik}$.

This way of filtering the LRP backward pass is also found in e.g. . We provide the derivations of the two-step procedure for other attribution techniques (e.g. the Shapley value and Integrated Gradients ) in Supplementary Note \zrefsup-note:two-step. When the attribution technique used at each step obeys the conservation principle, one gets the conservation equation $\sum_{ik}R_{ik}=y$. Furthermore, under certain assumptions about the model and the attribution technique, one gets the stronger form of conservation $\forall i\colon\sum_{k}R_{ik}=R_{i}$ (cf. Supplementary Note \zrefsup-note:two-step). Assuming this stronger form, the joint explanation becomes a decomposition of the standard explanation into multiple sub-explanations, and conversely, the standard explanation can be seen as a reduction (or coarse-graining) of the disentangled explanation.

There are a number of ways to learn matrices $U_{k}$’s. For example, one can learn them using the classical PCA technique on the statistics of a set of activation vectors $\boldsymbol{a}$’s. However, in order to learn directions that match data and are also relevant to the decision function, we propose to optimize the relevance scores $R_{k}$’s themselves. To achieve this, we need to link $R_{k}$’s to the activation layer. We find that for a number of common attribution methods, one can express the relevance score as:

i.e. a product in projected space between the activation $\boldsymbol{a}$ and ‘context’ vector $\boldsymbol{c}$ that represents the model response to the activations. We first provide the demonstration for LRP:

Assume we have propagated using LRP up to the activations $(a_{j})_{j}$ and obtained the relevance scores $(R_{j})_{j}$. Propagating $R_{j}$’s on the layer below (onto concepts $\boldsymbol{h}_{k}$’s) can be achieved e.g. by the propagation rule:

which has the form of Eq. (5) and where $c_{j}=R_{j}/(\epsilon_{j}+a_{j})$.

Other attribution methods such as Gradient$\,\times\,$Input and Integrated Gradients (with reference $\widetilde{\boldsymbol{h}}=\boldsymbol{0}$) also produce relevance scores of the same form. This is a consequence of the structure of the chain rule for derivatives, and we provide the demonstration in Supplementary Note \zrefsup-sec:examples-subspace-attributions. Methods based on the Shapley value do not yield the form of Eq. (5), however, one can consider a variant of it where Shapley values are first computed w.r.t. the activations $a_{j}$ (or groups of it), and performing the very last propagation step onto concepts $k$’s using LRP.

IV Learning Relevant Subspaces

Having expressed $R_{k}$’s in terms of known quantities and projection matrices $U_{k}$’s, we can formulate an optimization objective. We first propose principal relevant component analysis (PRCA) that extracts a subspace that is maximally relevant for the model prediction (Section IV-A). We then extend the former idea to disentangled relevant subspace analysis (DRSA) whose goal is to learn a set of subspaces that identify multiple distinct concepts relevant to the prediction (Section IV-B).

We propose to ask the question “what subspace $U$ of fixed dimensions $d^{\prime}$ is maximally relevant for the prediction”. Starting from the expression of relevance in Eq. (5), we can formulate the search for a maximally relevant subspace via the optimization problem:

Note that several related approaches to refocus PCA on task-specific features have been proposed, although in a different context than Explainable AI. This includes ‘directed PCA’ , where a subset of task-related features are selected before running PCA. It also includes ‘supervised PCA’ which formulates a trace maximization problem involving both input and labels, and further methods based on partial least squares .

IV-B Disentangled Relevant Subspace Analysis (DRSA)

We now look for a set of components that are both relevant to the predictions and disentangled, i.e. expressing distinct factors of the model’s decision strategy. For example, in image classification settings, disentanglement may correspond to a low spatial overlap between the multiple heatmaps forming the explanation.

be the positive part of the relevance associated to subspace $k$ according to Eq. (5) for a given data point $n$. The rectification operation allows us to focus our subsequent analysis on extracting salient positive contributions. In particular, we define our disentanglement-inducing objective as:

The soft max-pooling over data points serves to encourage subspaces to align with instances $n$’s with particularly high relevance scores. These instances be interpreted as prototypes for each identified component of the decision strategy. The soft min-pooling over subspaces, on the other hand, serves to favor solutions that balance the total relevance attributed to the different subspaces. Other nested pooling structures are found in the Independent Subspace Analysis algorithms of . These nested structures have also been studied in more depth in .

While the optimization problem above is non-convex and does not have a closed form solution, a local optimum can be obtained by starting from a random set of orthogonal vectorsOne can construct a set of orthogonal vectors from the orthogonal group. We sample such a set using the ‘ortho_group’ function from SciPy ., and then—similarly to —repeatedly applying a gradient ascent step followed by an orthogonalization step, specifically $\boldsymbol{U}\leftarrow\boldsymbol{U}(\boldsymbol{U}^{\top}\boldsymbol{U})^{-1/2}$ .

IV-C Theoretical Properties

The proposed relevant subspace analyses have a number of desirable theoretical properties:

Let $\boldsymbol{U}=(U_{k})_{k}$ be the set of projection matrices learned by DRSA. By construction, the matrix $\boldsymbol{U}$ is orthogonal. Using the formulation of relevance $R_{k}=(U_{k}^{\top}\boldsymbol{a})^{\top}(U_{k}^{\top}\boldsymbol{c})$, we obtain the conservation property $\sum_{k}R_{k}=\sum_{j}R_{j}$. Furthermore, when $\boldsymbol{c}=\xi\boldsymbol{a}$ with $\xi\geq 0$, then we necessarily have $R_{k}\geq 0$.

(A proof can be found in Supplementary Note \zrefsup-section:propositions.) These properties are due to the orthogonality constraint on the matrices $(U_{k})_{k}$. The first property (conservation) ensures that the two-step explanation produced by our method retains the conservation properties of the original explanation technique it is based on. The second property (positivity) ensures that an absence of contradiction in the decision function (e.g. a perfect alignment between activations and model response) results in a similar absence of contradiction w.r.t. concepts.

The following result links the proposed PRCA and DRSA algorithms to well-known analyses such as PCA and ICA.

(A proof can be found in Supplementary Note \zrefsup-section:propositions.) In other words, with some restrictions on the parameters, our proposed algorithms reduce to PCA and ICA when the model response is perfectly aligned with the activations. Unlike PCA and ICA, our analyses are able to extract subspaces that are relevant to the prediction even when the model response does not align well with the activations.

V Quantitative Evaluation

To evaluate the performance of the proposed PRCA and DRSA methods at extracting relevant subspaces and producing disentangled explanations, we consider three publicly available ImageNet pretrained models. These models are two VGG16 models—which are from the TorchVision (TV) and NetDissect (ND) repositories, denoted by VGG16-TV and VGG16-NDWe remark that VGG16-ND is a PyTorch version of the original model (in Caffe ’s format) provided at http://netdissect.csail.mit.edu/dissect/vgg16_imagenet/. respectively—and a more recent architecture called Normalizer-Free Networks (NFNets) . For NFNets, we use NFNet-F0—the smallest variant of the family—and it is available via Pytorch Image Models . We refer to Supplementary Note \zrefsup-section:pixel-flipping for additional details of the models.

We evaluate our proposed methods with Shapley Value Sampling—an approximation of the classic Shapley value—and LRP; these two attribution techniques are chosen based on pixel-flipping experiments (see Supplementary Note \zrefsup-sec:selection-explanation-backend). We use the implementation of Shapley Value Sampling from Captum . Our LRP implementation for VGG16 is based on LRP-$\gamma$ used in . For NFNets, we contribute its LRP implementation (see Supplementary Note \zrefsup-sec:implementing-lrp-nfnet). We provide the values of these attribution methods’ hyper-parameters in Supplementary Note \zrefsup-sec:selection-explanation-backend.

In the following, we focus on evaluating the proposed approaches using activations from VGG16 at Conv4_3 (after ReLU) and NFNet-F0 at Stage 2We adopt the layer-name conventions of VGG16 from and of NFNet-F0 from .. We refer to ablation studies on the choice of layers and the number of subspaces in Supplementary Note \zrefsup-sec:ablation-studies.

Our evaluation is based on a subset of 50 classesFor ease of reproducibility and maximizing class coverage, we choose classes with indices $\{0,20,40,\dots,980\}$. from the ImageNet dataset . For each of these 50 classes, we randomly choose 500 training images and compute their feature map activations in the intermediate layer. For each image, we randomly pick 20 spatial locations in the feature maps, resulting in a dataset of $500\times 20=10000$ activation vectors for each class. For PRCA and DRSA, we also take the corresponding context vectors (w.r.t. the target class); therefore, each training data point becomes a pair of activation and context vectors. We summarize these details in Supplementary Note \zrefsup-sec:training-irca. All our evaluations are performed on a validation set disjoint from the data used for training the networks and optimizing the PRCA/DRSA subspaces.

The scores $R_{i,U}$’s are the contributions of input features to the model output for the target class—computed by a given explanation technique, e.g. LRP or Shapley Value Sampling—but filtered to only incorporate the contribution of the latent space $U$. We compute the expectation in Eq. (10) over all validation images $\mathcal{D}$ from the target class.

To the best of our knowledge, there are no existing baselines from the literature that are designed to achieve a high TC score or some related metricReference studies a related problem: completeness in latent space, although its objective is to verify whether extracted concept vectors can restitute the full accuracy of the model. In contrast, our objective is to extract a subspace that maximally expresses the predicted evidence for the target class.. Hence, for comparison, we consider several baselines: 1) a random orthogonal subspace (first $d^{\prime}$ columns of a random orthogonal matrix sampled from the orthogonal group $O(d)$); 2) standard (uncentered) PCA on the activations; and 3) retaining the $d^{\prime}$ most relevant feature maps, similar to . One can also view the choice of baselines as a special ablation study of PRCA. Specifically, PCA corresponds to PRCA with context vector $\boldsymbol{c}$ representing model response set to the activation vector $\boldsymbol{a}$ (Proposition 2); the most relevant feature maps corresponds to PRCA with the orthogonal basis $U$ constrained to be aligned with the standard basis vectors; and the random approach can be thought of as an ‘untrained’ PRCA.

We set $d^{\prime}=1$ and evaluate the subspaces $U$’s learned from PRCA and the baseline approaches by applying the evaluation metric in Eq. (10) on the 50 validation images of each class. For each model, we first average results over images and then over classes. We present the results in Table I; we note that Reference in our benchmark, as for other baselines methods, is not designed specifically for that task, and we only include it for indicative purpose.

From Table I, we observe that, as to be expected, PRCA strongly surpasses baseline methods on this task. This observation confirms that two properties of PRCA: 1) maximizing relevance, and 2) doing so over any orthogonal projection, are both important in order to concisely capture the relevant part of the decision strategy.

As a further experiment, we analyze the relevance $R_{i,U}$ when varying the number of components extracted by each method (in a similar way as rendered in ). Results for the VGG16-TV model are given in Fig. 5. We observe that PRCA is able to extract in the top-few principal components a large amount of positive evidence for the output neuron and also strongly suppresses negative contributions. PRCA is superior to other baselines for every subspace size.

V-B Extracting Subspaces for Disentangled Explanations

The second question we have considered in this paper (and for which we have proposed the DRSA analysis) is whether the explanation can be disentangled into multiple components that are distinctly relevant for the prediction. In practice, measures of disentanglement based on statistical independence are affected by the curse of dimensionality. Instead, we propose two surrogate metrics for disentanglement that measure whether different components of the explanation map to different input features (i.e. in the context of image classification, form distinct patterns in pixel space).

Recall that $\boldsymbol{U}=(U_{k})_{k}$ denotes a partition of the space of activation into orthogonal subspaces. Furthermore, methods in our benchmark are only allowed to differ by the choice of $\boldsymbol{U}$. We define our first metric:

A low separability score occurs for example when only a single explanation factor (i.e. a standard explanation) is available or when all components of the explanation are the same. Conversely, the separability score is high when the contributions associated to different factors correspond to different input features; in other words, these factors are spatially separated. More formally, the separability metric satisfies a certain number of desirable properties:

Eq. (11)’s separability score (i) is divided by two if duplicating all heatmaps and halving their relevance scores and (ii) increases with splitting a component into two spatially disjoint components.

To prove (i), one shows that duplicating all heatmaps $k$’s neither changes the $\sum_{i}\max_{k}\{\cdot\}$ nor $\max_{k}\sum_{i}\{\cdot\}$ terms of Eq. (11) due to the max function being invariant to duplication. Halving all $R_{ik}$’s in Eq. (11) halves the separability score. For (ii), one observes that splitting in disjoint components keeps the left term unchanged, but reduces the second term, leading to an increase of separability. ∎

As a further evaluation, we consider another simpler metric, which we call ‘$K$-peakness’, and define it as:

It extracts the maximal relevance score in each component of the explanation, retains the top-$K$ such scores, and sums them. High peakness indicates that components of the explanation focus on distinct aspects of the decision strategy. An illustration of our separability and peakness metrics is given in Fig. 6.

We evaluate our DRSA method against three baselines. The first baseline is random subspaces obtained by sampling a random $d\times d$ orthogonal matrix $\boldsymbol{U}$ from the orthogonal group and splitting this matrix into $K$ submatrices. The second baseline is NetDissect , a state-of-the-art framework linking neurons to a large set of real-world concepts extracted from the Broden database . The approach associates each filter in the layer’s feature map to a concept available in the dataset. We reproduce these filter-concept associations for VGG16-TV and -ND using the code provided by (see Supplementary Note \zrefsup-sec:baselines-netdissect for the details), and for each identified concept we define its subspace as the span of the standard basis vectors of the associated filters. Our third baseline, called DSA, is an ablation of the objective of DRSA where we replace the context vector with the activation vector itself.

We set the number of subspaces in DRSA, DSA, and Random Subspace to $K=4$. We set each subspace to have the same number of dimensions, which we deduce from the numbers of subspaces and the dimensions of activation vectors. To build the DSA and DRSA models, we use each class’s set of activation (and context) vectors similar to the setup in Section V-A. We perform the optimization of DSA and DRSA for 5000 iterations; we empirically find that the optimization converges (see Supplementary Note \zrefsup-sec:training-irca). We discuss additional training details in Supplementary Note \zrefsup-sec:training-irca. For the NetDissect baseline, because a concept subspace is associated to a collection of neurons, we consider all concepts identifiedThe NetDissect framework identifies more than 100 concepts (or subspaces) in Conv4_3 of VGG16-TV and -ND (see Supplementary Note \zrefsup-sec:baselines-netdissect). by the framework in order to satisfy the invertibility condition. For NetDissect, the peakness score is calculated from the top-4 concepts of each example. Similar to Section V-A, we evaluate the considered methods on VGG16-TV and -ND at Conv4_3 and NFNet-F0 at Stage 2. We compute separability and peakness scores by 1) averaging over examples within each class and 2) normalizing the class averages by the scores of the corresponding random subspaces, which we average over three seeds. The normalization mitigates the effect of unbalanced logit values between classes on the scores. We evaluate the setups of Shapley Value Sampling on only 10 validation images of each class due to its computational demand, whereas we use all 50 validation images for the setups of LRP.

Table II shows the separability and peakness scores across setups. From the table, we observe that our proposed approach (DRSA) outperforms baseline methods by reaching the highest separability and peakness scores. This result also aligns with the visual inspection of Fig. 2 earlier in the paper, where spatially distinct concepts could be identified from the DRSA explanations. When using different layers or number of subspaces to extract subspaces, the conclusion from Table II still holds, with separability and peakness scores of DRSA remaining consistently higher than DSA. We provide the results in Supplementary Note \zrefsup-sec:ablation-studies.

VI Application Showcases

We showcase three possible applications of the proposed PRCA and DRSA methods, namely (1) building a more trustworthy ML model by detection and removal of Clever Hans strategies in the model, (2) getting better insights into the data by highlighting multiple ways input and output variables are related, and (3) bring further understanding about the problem of adversarially manipulated explanations.

A common issue with ML models is that they sometimes rely not on the true features—that should support the ML decision—but on artifactual features that spuriously correlate with the true features on the available data. Such flawed strategies of the ML model are commonly referred to as ‘Clever Hans’ . Clever Hans models evade traditional model validation techniques, such as cross-validation, when the spurious correlation is present both in the training and test data. Nevertheless, Explainable AI can reveal these Clever Hans strategies; specifically, the user would inspect the explanation of a number of decision strategies and verify that artifactual features are not highlighted as ‘relevant’ in the explanation.

We demonstrate in this showcase how the proposed DRSA analysis enables us to detect and mitigate Clever Hans effects in a highly efficient manner. In contrast to existing state-of-the-art approaches to Clever Hans detection and mitigation , our approach can leverage the multiple sub-strategies readily identified by DRSA, some of which can be of Clever Hans nature. In particular, for detecting Clever Hans strategies, one can let the user inspect one or a few representative examples of each decision strategy identified by DRSA. When it comes to mitigating Clever Hans strategies, one can again leverage DRSA by removing from the overall prediction those strategies identified to be of Clever Hans nature, by zeroing their contributions.

Specifically, assuming we have identified for a given class a set of Clever Hans (CH) subspaces, one can in principle build a refined prediction by subtracting the relevance scores from the prediction:

We test our approach on some known example of Clever Hans strategy: the reliance of VGG16-TV on Hanzi watermarks for predicting ‘carton’ . We investigate whether we can mitigate the influence of watermarks occurring on images of classes that VGG-TV tends to confuse as ‘carton’, specifically, ‘crate’, ‘envelope’, ‘packet’, and ‘safe’. Those are the classes whose instances are likely to be of class ‘carton’ according to VGG16-TV, more precisely, the class ‘carton’ appears in the VGG16-TV’s top-3 prediction at least 10% of the time. Using validation images of these classes and class ‘carton’, we then construct a classification problem in which some of non-carton images are poisoned with a random Hanzi watermark (from one of the three we have prepared; cf. Supplementary Note \zrefsup-sec:showcase-1-additional).

We apply 25% poisoning, i.e. 25% of non-carton images are inpainted with Hanzi watermarks. We observe that the classification accuracy of the original model decreases on the poisoned data (from around 82% to 76%). The decrease indicates that our poisoning procedure effectively fools VGG16-TV.

Fig. 7 (top) shows three training imagesWe select the examples based on the procedure outlined in Supplementary Note \zrefsup-sec:algo-prototype. from the class ‘carton’ and their standard and DRSA subspace heatmaps using LRP (DRSA is applied at Conv4_3 with $K=4$). From the heatmaps, we can see that, unlike other subspaces, Subspace 4 captures the Hanzi watermark quite prominently when the watermark is salient (e.g. the first and second example). We therefore identify that Subspace 4 is a Clever Hans subspace. Fig. 7 (bottom) shows the difference of accuracy between the original and refined models on the 25%-poisoned data. We see that the refined model based on excluding the contribution of Subspace 4 has the highest classification accuracy (adding 3.6% to the accuracy score of the unrefined model). We further investigate the structure of error in Fig. 8 which shows the confusion matrices between predicted and target classes for the original and refined model. After refinement we observe that the number of misclassified non-carton examples decreases substantially. We refer to Supplementary Note \zrefsup-sec:showcase-1-additional for the details of the experiment and the confusion matrices on clean and 50%-poisoned data.

Overall, this showcase has demonstrated that DRSA can be an effective tool for detecting and mitigating Clever Hans effects in complex models. Furthermore, we stress that our approach is purely unsupervised: It requires neither assembling a dataset of examples labeled according to the strategy the model employs to predict them, nor to generate synthetic examples where the Clever Hans features have been stripped or artificially added. (Here, synthetically generated examples were used only for evaluation purposes.) Furthermore, our Clever Hans mitigation approach is ‘post-hoc’: except for the DRSA analysis, our method does not require any training or retraining of the neural network model.

VI-B Better Insights via Disentangled Explanations

Explainable AI has been shown to be a promising approach to extract insights in the data and in the systems or processes that generates this data . Several recent works have shown successful usages in biomedical or physics applications. For example, Explainable AI enabled a better understanding of what geometrical aspects of molecules are predictive of toxicity (or ‘toxicophores’). It also allowed to predict proteins interactions in a human cell , thereby supporting the research on identifying signaling pathways. There are many further examples of successful uses of Explainable AI for extracting scientific insights in geology , hydrology , quantum chemistry , neuroscience , histopathology , etc. In these works, the authors often resort to standard heatmaps highlighting the extent by which one feature or a group of features contributes to the overall prediction.

The amount of insights one can extract from a standard explanation is however restrained by the fact that multiple concepts are entangled, and it is therefore difficult to gain a structured understanding of the relation between input and output. We showcase in the following how our proposed DRSA-LRP method enables the extraction of more sophisticated insights. We consider for an illustrative purpose the task of gaining insights into the visual differences between six classes of butterflies present in the ImageNet dataset: ‘admiral’, ‘ringlet’, ‘monarch’, ‘cabbage’, ‘sulphur’, and ‘lycaenid’ butterflies.

For this showcase where the objective is for the user to gain insights from the model, it is natural to choose the best model available. We choose NFNet-F0, which achieves an overall top-1 accuracy of 82% compared to VGG16-TV and -ND that achieve 72% and 70% respectively. We select 125 training images from each of these butterfly classes to form a training set. We use activation and context vectors from NFNet-F0 at Stage 1, and use LRP (with parameter $\gamma=0.1$ to compute the explanations). We extract eight subspaces using DRSA with the optimization details similar to Section V-B (see also Supplementary Note \zrefsup-sec:training-irca).

First, we would like to build a correspondence table between classes and concepts, indicating for each class which concepts are specific to it. We propose the following simple statistical test, which accounts for the fact that concepts are typically expressed only in a subset of images from the given class. Denote $\mathcal{D}_{\omega}$ to be the set of class $\omega$’s validation images and $\mathcal{D}$ the set of all validation images from the investigated classes (in our showcase, all butterfly images). We consider Subspace $k$ to be specific to class $\omega$ if

where $Q_{\alpha}$ is the $\alpha$-quantile of the given distribution and $\alpha<\beta$. In our experiments, we choose $\alpha=0.75$ and $\beta=0.85$. In this equation, scores $R_{k}$ are measured via $\sum_{i}R_{ik}$.

Figure 10 illustrates the process of matching classes with DRSA subspaces. The right border of the rectangles and the dashed lines correspond to the left and right hand sides of Eq. (14). The analysis reveals 10 class-subspace matchings (highlighted in red). We observe that each DRSA subspace is associated to one type of butterfly, except for subspaces S1 and S4, which matches multiple classes, thereby indicating visual concepts that are shared between multiple classes. Furthermore, the number of concepts associated to a particular class vary from one (sulphur butterfly) to three (cabbage butterfly).

Fig. 9 (left) explores using a three-dimensional scatter plot, how the relation between butterflies and their respective classes is resolved by subspaces S4, S5, and S7 of our DRSA analysis. Each point in this plot corresponds to one example, and its coordinate is given by the scores $R_{k}$’s. As already noted in Fig. 10, we observe that ‘monarch’ is jointly expressed along axes S4 and S5, and ‘admiral’ is jointly expressed along axes S4 and S7. These subspaces are not relevant for other classes, hence, their respective examples appear near the origin.

Fig. 9 (right) shows pixel-wise explanations for the most prototypical example of a few selected classesWe show for the selected classes the example $\arg\max_{n}\min_{k\in\mathcal{K}}R_{k,n}$, where $\mathcal{K}$ is the set of subspaces associated to the given class, and $R_{k,n}$ is the contribution of Subspace $k$ for example $n$ and its associated class.. We observe that Subspace S1 corresponds to yellow colored surfaces which seems to be common of ringlet and sulphur butterflies. S4 corresponds to white-dot texture, which is found on monarch’s wings and body and admiral’s wings. S5’s pattern is specific to the orange/black texture on the wings of monarch specie. S7 captures the prominent orange pattern on the wings of the admiral butterfly. Lastly, we find that Subspace 8 captures the distinct dotted pattern that appears on the wings of the ringlet species. We provide the complete set of these subspace heatmaps in Supplementary Note \zrefsup-section:showcase2.

Overall, throughout this showcase, we have demonstrated that our method is capable of providing further insights into the complex relation between visual features and class membership. In addition to highlighting features that are predictive of class membership, we have identified distinct visual concepts such as dotted patterns or yellow textures, that are shared between multiple classes. These shared visual patterns provides a structured understanding the nonlinear relation between butterfly visual features and butterfly species.

VI-C Analyzing Manipulated Explanations using PRCA

One of the premises of Explainable AI is to facilitate trust to stakeholders, but previous works show that explanation techniques are vulnerable to manipulation. More concretely, a slight perturbation of the input could lead to substantial changes in its explanation while maintaining the similarity between the original and perturbed input and other statistics (e.g. model output). Crucially, shows that such perturbation can lead to arbitrary changes in explanations, having neither relation to the input nor the original heatmap. Fig. 11 contrasts such a scenario (where a perpetrator perturbs an image to manipulate its explanation) and a regular Explainable AI scenario.

Certainly, the vulnerability to perturbation does not only raise practical concerns but also theoretical questions on how such a phenomena could happen. As a result, a number of theoretical analyses have been conducted to investigate the cause of the perturbation vulnerability. In particular, the investigation of elucidates that the degree to which an explanation can change is partially upper-bounded by the principal curvature evaluated the data point. Furthermore, shows that, for neural networks with ReLU, the principal curvature can be reduced by approximating ReLU with the softplus activation. By controlling the smoothness parameter of the softplus function, shows that the robustness of explanation manipulation can be effectively increased in a post-hoc manner.

Nevertheless, from the perspective of layer-wise representation, it is still unclear how perturbation causes such dramatic changes in explanation or how such changes manifest at a certain layer. We therefore aim to demonstrate that PRCA might provide a clue to answer such questions.

As a proof of concept, we study the PRCA decomposition of LRP explanations from validation images of class ‘tibetan terrier’ in the ImageNet dataset on VGG16-TV at Conv4_3. More precisely, we perform PRCA on a set of activation and context vectors from 500 training images of the class (details similar to the setup of Section V-A).

To manipulate explanations, we use the optimization procedure proposed by to find a perturbation that causes arbitrary changes in the explanation of each image, while retaining the same level of model response and visual similarity between the original and perturbed images. The arbitrary changes are induced by a target explanation, which is the explanation of a random image from a different class. In addition, we also constrain the original and manipulated explanations to have similar total relevance scores. We summarize the details of the algorithm in Supplementary Note \zrefsup-sec:showcase3.

Qualitatively, Fig. 11 (bottom) shows that the heatmap generated from the first PRCA component preserves features highlighted in the original heatmap, while the residual heatmap (orthogonal complement of the first PRCA component) contains features from both the original and target heatmaps.

Looking at the positive and negative parts of the residual heatmap, we observe that the former substantially resembles the target heatmaps, while the latter is closely similar to part of the original heatmap expressed in the PRCA heatmap with opposite sign. When using more PRCA components, the PRCA heatmap becomes similar to the target heatmap (see Supplementary Note \zrefsup-fig:showcase3-varying-prca-components). The behavior suggests that, for VGG16-TV at Conv4_3 and class ‘tibetan terrier’, the first PRCA component is the direction affected the least by perturbation.

Quantitatively, Fig. 12 shows the mean squared error between manipulated heatmaps (and their PRCA decomposition versions) and original or rescaled target heatmaps: the error is averaged over the 50 validation images of class ‘tibetan terrier’. We first observe that the manipulated heatmaps have lower error when comparing to the target heatmaps than the original heatmaps. It confirms that the optimization proposed by is indeed effective and also works well with the additional constraint we impose.

Secondly, when looking at the error from the manipulated heatmaps on the first PRCA component (PRCA-1), we observe that these heatmaps are closer to the original heatmaps than the target ones. This difference between the two errors is interesting because it indicates that PRCA indeed captures parts of the class-specific representation that is less affected by the perturbation. The insight may provide a new perspective towards understanding and increasing the robustness of explanation manipulation (cf. also ).

VII Conclusion and Discussion

In this work, we have proposed to disentangle explanations of neural network models into multiple components in order to provide more useful information to the user compared to standard explanation.

Technically, the desired disentanglement is achieved via an unsupervised analysis at some intermediate layer of the neural network model. A singular aspect of the proposed method is that it analyzes jointly the data and model response to the data. Hence, unlike a purely data-driven approach, our method enables a more focused disentanglement that efficiently ignores aspects of the data to which the model is invariant. Besides, our approach does not require any specialized datasets or concept annotations and can be applied to any deep neural network model. Our method also works together with a broad range of state-of-the-art attribution frameworks such as the Shapley value and LRP.

We have demonstrated the high performance of our disentanglement approach on state-of-the-art image classifiers trained on the ImageNet dataset. Our method scores significantly higher compared to all baseline methods. Furthermore, the practical benefit of our method was demonstrated on three showcases: 1) detection and defusion of Clever Hans strategies in the popular VGG16 image classifier, 2) in-depth exploration of a complex nonlinear relation of interest, subsumed by a state-of-the-art ML model, in order to extract new domain insights, and 3) investigation of the problem of adversarially manipulated explanations, for which we could gain new understanding.

In future work, we plan to investigate the capabilities of our methods to analyze complex scientific data and generate new insights. Furthermore, we anticipate a number of applications from our methods: First, extracting the PRCA/DRSA subspaces may not only enable the removal of identified ‘Clever Hans’ strategies from the model, but also provide immunity to new unobserved components of the decision strategy (potentially also of Clever Hans nature). Second, our disentangling framework may provide tools for studying more advanced adversarial scenarios, for example, defending against falsified yet plausible explanations.

Acknowledgments

This work was supported by the German Ministry for Education and Research (BMBF) under Grants 01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18025A, 031L0207D, and 01IS18037A, and by BASLEARN—TU Berlin/BASF Joint Laboratory, co-financed by TU Berlin and BASF SE. P.C. is supported by the German Federal Ministry of Education and Research and the Max Planck Society. K.R.M. was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation).

This work used the Scientific Compute Cluster at GWDG, the joint data center of Max Planck Society for the Advancement of Science (MPG) and University of Göttingen.

We thank Thomas Schnake, Lorenz Linhardt, Simon Letzgus, and Ali Hashemi for helpful comments and feedback to improve the manuscript.

References