Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Introduction

Multi-modal Transformers may change the way that computer vision is practiced. While the state of the art computer vision models are often trained as task-specific models that infer a fixed number of labels, Radford et al. have demonstrated that by training an image-text model that employs Transformers for encoding each modality, tens of downstream tasks can be performed, without further training (“zero-shot”), at comparable accuracy to the state of the art. Subsequently, Ramesh et al. used a bi-modal Transformer to generate images that match a given description in unseen domains with unprecedented performance.

These two contributions merge text and images differently. The first encodes the text with a Transformer , the image by either a ResNet or a Transformer, and then applies a symmetric contrastive loss. The second concatenates the quantized image representation to the text tokens and then employs a Transformer model. There are also many other methods of combining text and images . What is common to all of these is that the mapping from the two inputs to the prediction contains interaction between the two modalities. These interactions often challenge the existing explainability methods that are aimed at attention-based models, since, as far as we can ascertain, all existing Transformer explainability methods (e.g., ) heavily rely on self-attention, and do not provide adaptations to any other form of attention, which is commonly used in multi-modal Transformers.

Another class of Transformer models that is not restricted to self-attention is that of Transformer encoder-decoders, i.e. generative models, in which the model typically receives an input from a single domain, and produces output from a different one. These models are used in an emerging class of object detection and image segmentation methods, and are also widely used for various NLP tasks, such as machine translation . In these object detection methods, for example, embeddings of the position-specific and class-specific queries are crossed with the encoded image information.

We propose the first explainability method that is applicable to all Transformer architectures, and demonstrate its effectiveness on the three most commonly used Transformer architectures: (i) pure self-attention, (ii) self-attention combined with co-attention, and (iii) encoder-decoder attention. We use an exemplar model from each architecture, and prove our method’s superiority over existing Transformer explainability methods, adapted from their single modality origin. Our explainability prescription is easier to implement than existing methods, such as , and can be readily applied to any attention-based architecture.

Related work

Explainability in computer vision Interpreting computer vision algorithms usually entails the synthesis of a heatmap that depicts the computed relevancy at each image location. This can be class-dependent (for every possible label), or class-agnostic, in which case it depends only on the input and the model. Unlike most methods below, our method is of the first type. There are multiple families of explainability methods, including saliency-based methods , methods that consider activations using the forward pass or the backprop , perturbation based methods , and methods based on Shapley-values . The latter enjoy clear theoretical motivation. Theoretical justification is also given to attribution-based methods, through the theory of the Deep Taylor Decomposition . Such methods assign relevancy recursively from the top layer, backward, such that the sum of relevancies remains fixed. The LRP method , is one such prominent method. Since LRP and most variants are class agnostic , class-specific extensions were introduced .

Gradient-based methods directly consider the gradient of the loss with respect to the input of each layer, as computed through backpropagation. Examples include class agnostic methods . A related class-specific approach is the Grad-CAM method , which considers the input features with the class-dependent gradient at the top layers.

Explainability for Transformers Most attempts to explain Transformers directly employ the attention maps. This, however, neglects the intermediate attention scores, as well as the other components of the Transformers. As noted by Chefer et al , the computation in each attention head mixes queries, keys, and values and cannot be fully captured by considering only the inner products of queries and keys, which is what is referred to as attention.

LRP was applied to capture the relative importance of the attention heads within each Transformer block by Voita et al. . This method, however, does not propagate the relevancy scores back to the input to produce a heatmap.

Abnar et al. propose a way to combine the attention scores across multiple layers. Two methods are suggested: attention rollout and attention flow. The first combines attention linearly along alternative paths in the pairwise attention graph. It is shown in that this method fails to distinguish between positive and negative contributions to the decision, leading to an accumulation of relevancy scores across the layers in cases for which these should be cancelled out. The attention flow method is formulated as a max-flow problem on the same pairwise attention graph. While it was shown in to somewhat outperform rollout in specific scenarios, this method is too slow to support large-scale evaluations.

In contrast to these methods, Chefer et al. provide a comprehensive treatment of the information propagation within all components of the Transformer model, which back propagates the information through all layers from the decision back to the input. The solution is based on Layer-wise Relevance Propagation , with gradient integration for the self-attention layers, and is shown to be very effective for single modality Transformer encoders, such as . This method, however, does not provide a solution for attention modules other than self-attention, thus can not provide explanations for all Transformer architectures.

Transformers in computer vision Transformer technology has become increasingly prevalent for bi-modal tasks, such as image captioning and text-based image retrieval. We distinguish between networks that rely on self-attention, such as VisualBERT and Oscar and those that also employ co-attention modules, such as LXMERT and ViLBERT . Our method provides suitable visualization for both types.

Our method also provides the first complete solution, as far as we can ascertain, for Transformer encoder-decoders , which have been increasingly prevalent in computer vision. In the DETR Transformer-based detection method , the image is encoded by a Transformer encoder, and the obtained information is co-attended together with queries that are both positional and class-based. Our method can be also applied to encoder-based visual Transformers, such as those used for image recognition , and image segmentation with a CNN decoder . However, in this case, existing Transformer explainability methods can also be applied.

Method

Our method uses the model’s attention layers to produce relevancy maps for each of the interactions between the input modalities in the network. In this work, we focus on image and text interactions, and attention modules for generative models, i.e., encoder-decoder attention. However, our method is easily applicable to any Transformer-based architecture, and can also be generalized to address more than two modalities. In the following, we discuss the method’s propagation rules under the assumption of two modalities, e.g. text and image for simplicity, followed by a detailed description of how to apply our method to each of the model types used in this work.

Let $t,i$ be the number of text and image input tokens respectively. To simplify notation, we use the same symbols ($t,i$) to identify variables that are associated with the two domains. Multi-modal attention networks contain four types of interactions between the input tokens: $\mathbf{A}^{tt}$ and $\mathbf{A}^{ii}$ are the self-attention interactions for the text and image tokens, respectively. $\mathbf{A}^{ti}$, $\mathbf{A}^{it}$ are the multi-modal attention interactions, where $\mathbf{A}^{ti}$ represents the influence of the image tokens on each text token, and $\mathbf{A}^{it}$ represents the influence of the text tokens on each image token.

In accordance with the attention interactions described, we construct a relevancy map per interaction, i.e. $\mathbf{R}^{tt}$, $\mathbf{R}^{ii}$ for self-attention, and $\mathbf{R}^{ti}$, $\mathbf{R}^{it}$ for bi-modal attention.

The method calculates the relevancy maps by a forward pass on the attention layers, with each layer contributing to the aggregated relevance matrices using the update rules we will describe in the following subsections.

Relevancy initialization Before the attention operations, each token is self-contained. Thus, self-attention interactions are initialized with the identity matrix. For bi-modal interactions, before the attention layers, each modality is separate and does not contain context from the other modality, therefore, the relevancy maps are initialized to zeros.

As the attention layers contextualize the tokens, our method modifies the relevancy maps that are impacted by the mixture of token embeddings. Recall the attention mechanism presented in :

In Eq. 6 we account for the fact that the tokens were already contextualized in previous attention layers by applying matrix multiplication with the aggregated self-attention matrix $\mathbf{R^{ss}}$, as done in . For Eq. 7, notice that the previous bi-modal attention layers inserted context from $q$ into $s$, therefore, when the self-attention mixes tokens from $s$, it also mixes the context $q$ in each token from $s$. The previous layers’ mixture of context is embodied by $\mathbf{R}^{sq}$. Thus, we calculate the added context from the self-attention process.

where $/$ stands for matrix division element by element. In the above, we normalize each row in $\mathbf{\hat{R}}^{xx}$ by dividing each element in the row by the sum of the row. Next, we define the following aggregation rules for bi-modal attention units:

Eq. 10 accounts for the fact that the tokens of each modality were already contextualized in previous attention layers by applying matrix multiplication with the normalized aggregated self-attention matrices $\mathbf{\bar{R}}^{ss},\mathbf{\bar{R}}^{qq}$.

For Eq. 11, notice that the previous bi-modal attention layers integrate the embeddings of the two modalities, thus when contextualizing $s$ with $q$, $q$ also contains information from $s$, embodied in $\mathbf{R}^{qs}$.

Note that the above rules are described w.r.t. input from modality $s\in\{i,t\}$, and context from modality $q\in\{i,t\}$ i.e. the rules are symmetrically applied to both modalities, image and text.

In order to make the final classification, Transformer-based models usually regard the [CLS] token, which is a token that is added to the input tokens and constructs a general representation of all the input tokens. To retrieve per-token relevancies for classification tasks, one can consider the row corresponding to the [CLS] token in the corresponding relevancy map. For instance, assuming the [CLS] token is the first token in the text modality, to extract relevancies per text token, one should consider the first row of $\mathbf{R}^{tt}$, and to extract the image token relevancies, consider the first row in $\mathbf{R}^{ti}$ which describes the connections between the [CLS] token and each image token.

2 Adaptation to attention types

In this work, we examine our method on three different types of attention mechanisms used in Transformer-based networks. The architectures and matching propagation rules are visualized in Fig. 2. The first architecture type is a multi-modal Transformer, where the two modalities are concatenated and separated by the [SEP] token , as demonstrated in Fig. 2(a). Such networks only use self-attention to contextualize the modalities, i.e. only Eq. 6. Since the model is based on pure self-attention, we produce one relevancy map $\mathbf{R}^{(t+i,t+i)}$ which defines connections between the modalities, as well as within each modality. In order to visualize the tokens related to the classification, one should consider the row of $\mathbf{R}^{(t+i,t+i)}$ which corresponds to the token used for classification. This row $\mathbf{R_{cls}}^{(t+i)}$ yields a relevancy score per image token and per text token.

The second type is a multi-modal attention network that incorporates co-attention modules that contextualize each modality with the other modality , as can be seen in Fig. 2(b). Such networks require all propagation rules described above, for each modality. To produce relevancies for the classification, we simply follow the example in Sec 3.1, since as Fig. 2(b) depicts, the [CLS] token in this case is the first token of the text modality.

The third and last type is a generative model where there is one input modality, and the output is from a different domain , which is visualized in Fig. 2(c). Such networks contain an encoder that utilizes self-attention on the input and a decoder. The decoder has two types of inputs. The first is the encoded data, which remains unchanged, and the second are inputs from the decoder’s domain. The decoder proceeds to utilize self-attention on the decoder domain’s tokens, followed by a co-attention layer contextualizing them with the encoder’s output. To clarify, in this case, the relevance update rules are as follows: notate by $e$ the encoder’s tokens, and by $d$ the decoder’s tokens. The relevancy matrices are: $\mathbf{R}^{ee},\mathbf{R}^{dd}$ for the self-attention interactions, and $\mathbf{R}^{de}$ for the bi-modal interactions between the decoder’s tokens and the encoder’s tokens. Notice that since the encoder is not contextualized, we do not have a relevancy matrix $\mathbf{R}^{ed}$. The encoder’s self-attention calculation for $\mathbf{R}^{ee}$ simply follows Eq. 6. For the decoder’s self-attention, we apply Eq. 6, 7. For the bi-modal attention in the decoder, we follow Eq. 10 to account for self-attention in the encoder and the decoder. Notice that Eq. 11 is irrelevant since we do not have a relevancy map for $\mathbf{R}^{qs}=\mathbf{R}^{ed}$. In order to extract relevancies in this case, we consider the relevancy map $\mathbf{R}^{de}$. In this work, we use an object detection model as our exemplar encoder-decoder architecture. For such models, each token from $d$ is a query representing an object in the input image. In order to produce relevancy for each of the image regions w.r.t. an object $j$ that was detected, one should consider the $j$-th row of $\mathbf{R}^{de}$, which corresponds to the $j$-th detection. $\mathbf{R}_{j}^{de}$ contains a relevancy score per each encoder token, which is in this case an image region.

Baselines

We focus on methods that are both common in the explainability literature, and applicable to the extensive tests we report in this work. We present baselines of three classes, following : attention map baselines, gradient baselines, and relevancy map baselines. Our attention map baselines are raw attention and rollout. Raw attention regards only the last layer’s attention map as the relevancy map, e.g. $\mathbf{R}^{tt}=\mathbf{A}^{tt}$, where $\mathbf{A}^{tt}$ is the last text self-attention map. The second is rollout, which follows for all the self-attention layers. Since the rollout baseline is based solely on self-attention, to distinguish from raw attention, we employ the following for $\mathbf{R}^{sq},s,q\in\{t,i\}$:

where the only difference compared to Eq. 5, is that uses the LRP relevancy values of $\mathbf{A}$, i.e. $\mathbf{R^{A}}$, instead of using the raw attention maps as done in Eq. 5. Additionally, uses Eq. 6 for all self-attention layers. For non self-attention layers, our version of takes the last attention map, and averages across heads using Eq. 13. Note that while applying our method only requires a few simple hooks for the attention modules, LRP requires a custom implementation of all network layers.

Experiments

Our experiments include three Transformer-based models, each representing one of the three types of architectures we refer to in this work. See Fig. 2 for illustrations of each of the architectures. In addition, to compare with previous work in the same setting for which these methods were conceived, we also consider ViT . The relevancy propagation for each model follows Sec. 3.2.

The first model we examine is VisualBERT , which represents a self-attention based architecture, and the second model is LXMERT , which represents an architecture combining self-attention and co-attention in a Transformer encoder for two modalities.

For both models, we perform positive and negative perturbation tests on each modality separately to evaluate the quality of the relevancy matrices produced by the methods. We use the visual question answering task in testing the explanations since this task requires the models to demonstrate an understanding of both input modalities and the connections between them.

The perturbation tests are performed as follows: first, a pre-trained network is used for extracting relevancy maps for $10,000$ randomly picked samples from the validation set of the VQA dataset. Second, we gradually remove the tokens of a given modality and measure the mean top-1 accuracy of the network. In positive perturbation, tokens are removed from the highest relevance to the lowest, while in the negative version, from lowest to highest. In positive perturbation, one expects to see a steep decrease in performance, which indicates that the removed tokens are important to the classification score. In negative perturbation, a good explanation would maintain the accuracy of the model while removing tokens that are not related to the classification. In both cases, we measure the area-under-the-curve (AUC), to evaluate the decrease in the model’s accuracy.

We note that in all perturbation tests, the accuracy does not reach $0\%$, even when removing $100\%$ of the tokens of each modality. This is since the input from the other modality remains intact therefore the models can rely on a single modality to provide a reasonable answer.

Notice that the LXMERT image perturbation test results, which are depicted in Fig. 4(a,b), demonstrate a clear advantage to our method compared to other methods. For negative perturbation, the AUC using our method is the largest by a sizeable margin, and the accuracy is well-preserved even after removing more than $80\%$ of the image tokens, and for positive perturbation, notice the very steep decrease in accuracy, and the low AUC.

As can be seen in Fig. 2(b), the [CLS] token for LXMERT is the first token of the text modality, thus following Sec. 3.2, $\mathbf{R^{ti}}$ is the map used for extracting relevancies in the image perturbation case. Since $\mathbf{R^{ti}}$ is a multi-modal relevancy map, the image perturbation tests best demonstrate the advantage of using our method over all existing methods, which fall short in evaluating relevancies from the co-attention modules.

For the LXMERT text perturbation tests which are depicted in Fig. 4(c,d), notice that by Sec. 3.2, we visualize $\mathbf{R^{tt}}$ which is a self-attention map, where the dominating update rule is Eq. 6. This rule is identical to the rule employed by the Transformer attribution baseline, except for the head averaging in Eq. 13. Therefore, the main difference between our proposed method and the method described in is the choice to use LRP in the head averaging process. This results in very similar results for both methods. For completeness, we provide in the supplementary results for our method when adding LRP, as is done in Eq. 13. The rest of the methods fall far behind.

Fig. 4 presents typical results for our method and for Transformer attribution . The rest of the methods are not competitive and their matching samples are presented in the supplementary. As can be seen, the text results are similar, as predicted by the quantitative results. Our image attention results are much more focused on the relevant image parts than those of the baseline method.

Note that since VisualBERT is based on pure self-attention, the difference between our method and the Transformer attribution method stems from the choice of whether or not to use LRP for head averaging in Eq. 5, similarly to the LXMERT text (but not image) perturbation tests. As can be seen in Fig. 5, our method outperforms all methods and achieves very similar results to those of , and in some cases, such as the text perturbation test, even outperforms by a sizeable margin. This demonstrates that the use of LRP is unnecessary, even for pure self-attention architectures.

The third model we experiment on is DETR , which is an encoder-decoder model, as seen in Fig. 2(c). We use a pre-trained DETR model with the ImageNet pre-trained backbone ResNet-50, which is trained for object detection on the MSCOCO dataset. Importantly, this model has only been trained for object detection, i.e., producing bounding boxes and classifications for each object in the input image. To evaluate the different explainability methods, our test uses each of the methods on the $5,000$ samples of the MSCOCO validation set to produce segmentation masks, i.e. we consider the output of each method to be a segmentation mask. We first filter the queries to include only ones where the classification probability is higher than $50\%$ and then employ Otsu’s thresholding method to separate the foreground and the background of the segmentation. See supplementary for the full details.

Our generated segmentation masks visualize the bounding boxes predicted by DETR, therefore it should be noted that the produced masks are inherently dependent on the quality of the corresponding bounding boxes, i.e., when the predicted bounding box is not sufficient, naturally, the mask produced for it will be at least equally inaccurate. In addition, since the explainability methods are not aimed at producing segmentation maps, they often do not output contiguous masks, and the Otsu threshold may also create ”holes” in the produced masks. For all the reasons above, we decrease the minimal IoU used for MSCOCO evaluation from $0.5$ to $0.2$, which significantly benefits all the methods, and we present the results of the MSCOCO segmentation evaluation for the categories where the produced bounding boxes are good enough for the generation of segmentation masks, e.g., we do not present results for small objectsWe choose this working point since using a stricter threshold leads to baseline results that are slightly better than chance and our method outperforms but provides a score that is only 2-3 times better than chance.. As can be seen in Tab. 1, our method outperforms all other methods by a very large margin, which indicates that our novel formulations are necessary for non self-attention architectures. Notice the correlation in Tab. 1 between the bounding box evaluation for DETR and our segmentation. See Fig. 6 for visualizations of the masks.

Lastly, in order to compare our method with existing single-modality baselines, we present the positive and negative perturbation tests on ViT-Base , as performed by . As mentioned, since ViT-Base is a single-modality Transformer encoder, the only difference between our method and the Transformer attribution method of is the use of LRP in Eq. 5, as shown in Eq. 13. Therefore, as can be seen in Tab. 2, the differences between our method and the method proposed in are very mild, which is another indication that LRP can be removed. Tab. 2 also shows improvement in performance when using the target class instead of the predicted class for gradient propagation in Eq. 5, which, as stated in , indicates that our method is able to produce class-specific visualizations.

Ablation study We present in the supplementary three variations of our method that demonstrate the effectiveness of our normalization (Eq. 8,9), the necessity of the aggregation in all our rules 6, 7, 10, 11, and the need for the self-attention updates to the bi-modal rule 10.

Conclusions

Transformers play an increasingly dominant role in computer vision, with image-text Transformers and Transformers that perform tasks that have output domains that are more complex than the labels provided by a classifier, presenting groundbreaking results. In order to debug such models, as well as to support downstream tasks, and the increasing demand for model-interpretability, it is required to have complete and accurate explainability methods. However, the current explainability literature for Transformers is limited, overly focuses on pure attention maps, and lacks the methodology for treating co-attention maps.

Our method carefully tracks the evolution and mixing of the attention maps. It provides a generic prescription that is applicable to all attention models we are aware of. Empirically, it outperforms the existing methods across Transformer architectures and evaluation metrics. In some cases, when self-attention is prominent, the recent method by Chefer et al. is the only method that can provide comparable results. However, in the majority of the experiments, our method leads over all methods by a very sizable margin.

Acknowledgment

This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974). The contribution of the first author is part of a Master thesis research conducted at Tel Aviv University.

References

Appendix A Code

The code contains Jupyter notebooks with the examples presented for LXMERT and DETR. Both notebooks also allow using images from the internet. For LXMERT, we also support the option of asking a free form question.

Appendix B Extended LXMERT VQA visual results

In Fig. 3 we present extended results for Fig. 4 in the paper, i.e. we present the explanations extracted by each method for typical samples from the VQA dataset using the LXMERT model for question answering.

Appendix C Preparing the DETR relevancy maps for the COCO segmentation evaluation code

In this section, we elaborate on the process of extracting segmentation masks from DETR’s object detection results. The extracted segmentation masks are then used for our DETR tests, as presented in Sec. 5 of the paper.

DETR has been trained for object detection, i.e., producing a bounding box and a classification for each object in the input image. In order to evaluate the different explainability methods, we refer to the $\mathbf{R^{qi}}$ relevancy map, where the $j$-th row defines the relevance of each image feature to the $j$-th query, i.e. the $j$-th bounding box, as described in Sec. 3.2 of the paper. Our test uses each of the explainability methods on the $5,000$ samples of the MSCOCO validation set to produce segmentation masks, as described in Alg. 1. We first filter the queries to include only ones where the classification probability is higher than $50\%$ (Alg. 1, L. 3). Then, for each query $j$ that is left, we use the relevancy matrix $\mathbf{R^{qi}}$ in row $j$ as a heatmap of the image features (Alg. 1, L. 6), noting the important pixels for the $j$-th predicted bounding box. Since most of our baselines, as well as our method, produce non-negative relevancies, we use Otsu’s thresholding method to separate the foreground and the background of the segmentation mask (Alg. 1, L. 7). Then, the DETR segmentation evaluation code upsamples the masks to the target mask size, followed by a sigmoid operation, which only leaves the strictly positive values of the segmentation map (Alg. 1, L. 8-9). Finally, the DETR segmentation evaluation code upsamples the generated map back to the size of the original image (Alg. 1, L. 10).

Appendix D Ablation Study

We present three variations of our method. Firstly, we verify the effectiveness of our normalization for the self-attention relevancies presented in Eq. 8,9. Since the normalization is applied to rule 10, we expect it to affect mostly bi-modal relevancies, i.e. the image perturbation experiments for LXMERT, and the DETR tests. The second ablation we present studies the necessity of the aggregation in all our rules 6,7,10,11, i.e. instead of adding the former relevancy matrix to the newly constructed one, we only keep the new one, e.g. for rule 6 the update becomes: $\mathbf{R}^{ss}=\mathbf{\bar{A}}\cdot\mathbf{R}^{ss}$. Lastly, we explore the need for the self-attention updates to the bi-modal rule 10 by changing the update rule to: $\mathbf{R}^{sq}=\mathbf{R}^{sq}+\mathbf{\bar{A}}$. All our ablations are done on the LXMER, DETR experiments since, as mentioned several times, VisualBERT is based on pure self-attention, which yields similar results to the Transformer attribution baseline.

As can be seen from Tab. 3, all the components included in our method are crucial to its success on DETR, and the ablations cause a sizeable decrease in performance. It should be noted that for the reasonable ablations of not using normalization and not using self-attention in Eq.10, our ablations still outperform all other methods significantly for the DETR experiment.

For the image perturbation test on LXMERT, presented in Tab. 4, we observe relatively mild differences between our method and the ablations of no normalization and no self-attention, this can be attributed to the fact that in contrast to DETR, LXMERT only uses 36 image regions that had gone through Non-maximum Suppression (NMS), therefore the added context from the self-attention to the multi-modal attention is not as crucial, since usually the top-1 image region is identical to that of the ablations, and is sufficient to make the classification.

Appendix E Using LRP with our method

We present the results for the LXMERT perturbation tests evaluated by the area-under-the-curve measure for our method with LRP for completeness, i.e. with head averaging as presented in the Transformer attribution method and in Eq. 13 instead of the head averaging in Eq. 5. The results in Tab. 5 support and substantiate the conclusions presented in the paper: for the image perturbation tests, whether or not LRP is used, our method’s contributions lead to a large gap in performance over all baseline methods. LRP itself leads to a small degradation in performance. For the text perturbation tests which are, as mentioned in the paper, self-attention based, our method is similar in performance to the Transformer Attribution method. Here, too, the choice of whether or not to use LRP is insignificant. Given the complexity of implementing LRP (see Sec. 4 of the main text), we advocate to eliminate it.

Appendix F Perturbation experiments graphs

In Fig. 4, 5, we present enlarged graphs corresponding to our perturbation experiments for better clarity.