Effective Attention Sheds Light On Interpretability

Introduction

Attention mechanism Bahdanau et al. 2015 is an essential component of many NLP models, including those that are built on the ubiquitous transformer architecture Vaswani et al. 2017. As a result, visualizing attention weights is a widely used technique to interpret models’ behavior Belinkov and Glass 2019. Despite that, the validity of this analysis method is a subject undergoing intense discussion and study in NLP (Jain and Wallace 2019; Wiegreffe and Pinter 2019; Serrano and Smith 2019; Moradi et al. 2019; Mohankumar et al. 2020; Tutek and Snajder 2020, i.a.).

Related to this discussion, Brunner et al. 2020 show that, under mild conditions, the attention matrix of a transformer self-attention sublayer can be written as a sum of two components. One of them is irrelevant for the model output because its product with the value matrix is zero. They term the other component as effective attention (formally defined in §2). We study whether effective attention gives interpretations that differ from conclusions we get by analyzing standard attention. If this is the case, interpretation of effective attention is better suited for studying transformers’ internals because it is more pertinent to the model output by design.

Brunner et al. 2020 briefly discuss this by comparing standard and effective attention matrices from a single BERT head Devlin et al. 2019 for one example. They observe that: (i) standard attention is largely concentrated on the delimiter tokens ([SEP], [CLS]) or on near-diagonal elements; (ii) effective attention is more dispersed; (iii) effective attention disregards the delimiters. They stress that we should not extrapolate too much from these observations since they are based on a single example, and that further research is needed on this topic.

In this work, we aim to reliably answer whether effective attention disregards the [SEP] and [CLS] tokens, and if so, are effective attention weights dispersed to linguistic features? To address these questions, we embrace the methodology for a quantitative analysis of the attention patterns produced by individual transformer heads proposed by Kovaleva et al. 2019. We carry out their experiments on a subset of the GLUE tasks with BERT’s standard and effective attention. We show that effective attention “ignores” [SEP] and punctuation symbols (§3.1, §3.2), but not [CLS] (§3.2), and that it highlights end-task features instead (§3.1, §3.2, §3.3). Our code is available at https://github.com/KaiserWhoLearns/Effective-Attention-Interpretability

Background: Effective Attention

Each transformer layer consists of multi-head self-attention and feedforward sublayers (Vaswani et al. 2017, see Appendix A). Brunner et al. 2020 show that the standard attention matrix $A$ can be decomposed into two components, if a mild condition is satisfied. Specifically, if the left nullspace of the value matrix $V$:

is not trivial (contains vectors other than $\vec{0}$). This is satisfied when the maximum input sequence length is larger than the value matrix dimension (see Appendix A). The two components are: the component in the left nullspace of $V$ ($A^{\parallel}$) and the component orthogonal to the nullspace ($A^{\perp}$). Notably, $A^{\parallel}$ does not contribute to the output of the self-attention sublayer:

The effective attention matrix is defined as $A^{\perp}$. If visualizations of standard and effective attention differ, interpretation of effective attention is an accurate interpretation because effective attention is what contributes to the model output (per Eq. 1).

We explain how to compute $A^{\perp}$ since that was not described in Brunner et al. 2020. We first compute the singular value decomposition (SVD) of the value matrix $V=U\Sigma W^{T}$. The rows of U that correspond to singular values equal to zero span $\nullspace(V)$:

where $\langle\cdot,\cdot\rangle$ denotes the dot product. Finally, effective attention equals to:

Effective attention is not guaranteed to be a probability distribution as some of its weights might be negative and larger than 1.

We observe that effective attention is slower to compute due to the SVD decomposition of $V$ for each out of 144 BERT-base heads, and additional matrix multiplications (Table 3; §B). If speed is bottleneck, we recommend doing quantitative analyses with effective attention on a subset of the dev set. For qualitative analyses, common practice is already to select a subset for a manual analysis.

What Does Effective Attention Reveal?

We compare visualizations of standard and effective attention following the methodology for analysis of the attention patterns Kovaleva et al. 2019. We carry out our analyses using five English-language datasets in the GLUE benchmark (Wang et al. 2019): RTE Dagan et al. 2005; Haim et al. 2006; Giampiccolo et al. 2007; Bentivogli et al. 2009, MRPC Dolan and Brockett 2005, QNLI Rajpurkar et al. 2016; Wang et al. 2019, SST-2 Socher et al. 2013, and STS-B Cer et al. 2017. We omit larger datasets (QQP, MNLI), due to the limit of our computation budget (a single Nvdia GTX1070 with 8GB memory), and CoLA/WNLI following Kovaleva et al. 2019. See Table 1 for their specifications. For each dataset, we train BERT-base with standard attention, a batch size of 8, maximum sequence length of 128, and 3 training epochs. All other hyperparameters are set to default values in the transformers library Wolf et al. 2020. For analyzing effective attention, we replace standard with effective attention at the test time.

In this section, we start studying whether effective attention disregards the delimiter tokens.

The visualizations of attention matrices exhibit patterns Clark et al. 2019; Vig and Belinkov 2019. Kovaleva et al. 2019 identified five frequently occurring pattern categories:

vertical (associated with the delimiters tokens)

diagonal (either syntactic features between neighbouring words in the English language or the previous/following token attention coming from the language modeling pretraining)

block (intra-sentence attention for the tasks with two distinct sequences; potentially encodes semantic and syntactic information)

heterogeneous (as “block”, more likely to capture interpretable linguistic features).

They annotated 400 BERT’s attention matrices using these categories, and used them to train a ConvNet for pattern classification of 1K random test set attention matrices. We replicate their results for standard attention (using their code), and classify effective attention matrices for a comparison. We thank the authors for sharing their code and model weights for this experiment.

Table 2 (Fig. 4 in Appendix B) shows a drop in the percentage of the “vertical” and “vertical + diagonal” patterns when we replace the standard with effective attention. Since the vertical patterns are associated predominantly with attention to the delimiters tokens, this result supports the hypothesis that effective attention disregards the delimiter tokens. Moreover, although the amount of “heterogeneous” patterns did not change notably, the amount of “block” and “diagonal” patterns increased. This suggests that we are better positioned to find end-task linguistic features captured by the model by visualizing effective attention.

As an illustration, Figure 2 presents the attention matrices for one sentence from one attention head. In this example, effective attention highlights all mentions of the noun “antibiotics” that the adjective “new” modifies and that is also the object of the preposition “against”, instead of giving prominence to the [SEP] token as standard attention.

2 Delimiter Tokens vs. Linguistic Features

We showed that the “vertical” pattern, associated with the delimiter tokens, is less dominant with effective attention (§3.1). To verify that both delimiter tokens are indeed less relevant with effective attention, following Kovaleva et al. 2019, we report the standard and effective attention weights of specific token types when processing the [CLS] token in the final layer. Namely, the attention weights of linguistic features (nouns, pronouns, verbs), the delimiter tokens ([SEP], [CLS]), and punctuation symbols that are conceptually similar to [SEP]. If there are multiple tokens of the same type in the input, we use the one with the maximum weight. If a word consists of the multiple subtokens, we use the weight of the first subtoken.

Figure 1 shows that [SEP] is among the two most relevant features for all tasks except QNLI according to standard attention (upper two rows in each subfigure, colored green). For all but one task (SST-2), it loses its dominance with effective attention and its weights are apparently shifted to linguistic features. This is also the case for punctuation symbols. This result shows that the [SEP] token and punctuation symbols are not as important for understanding how the model solves the end-task as standard attention suggests.

We observe that [CLS] is attended similarly with effective and standard attention, contrary to what Brunner et al. 2020 suggested. To rule out this is because we plot the attention assigned to [CLS] when processing [CLS], we report the attention assigned to [CLS] when processing other input words (regardless of their type) in Fig. 5 in Appendix B. Again, we do not observe differences between standard and effective attention, unlike for [SEP] (Fig. 6 in §B). These results confirm the hypothesis of Brunner et al. 2020 that effective attention disregards [SEP], but not [CLS] as they also hypothesized. Notably, [SEP] is associated with the LM pretraining and [CLS] only with the task-specific finetuning.

3 Effects of Task-Specific Finetuning

To provide our final evidence that effective attention captures end-task features, we investigate how attention changes with finetuning layer-wise; again following Kovaleva et al. 2019. They calculate the cosine similarity between pretrained and finetuned flattened attention matrices. The layers that change the most, encode most task-specific features. To reiterate, effective attention is the part of standard attention that contributes to the model output (Eq. 1; §2), and we showed that it is less associated with the pretraining feature [SEP] and more with linguistic features (§3.1, §3.2). Thus, changes of standard attention from task-specific finetuning should be the product of changes of effective attention, and the outcome of this analysis should be the same, regardless of the attention “type”.

As expected, we come to the same conclusion with effective attention as Kovaleva et al. 2019 did with the standard: the last two layers change the most with finetuning (Fig. 3). This soundness check suggests once again that effective attention is the component of standard attention that manifests end-task features.

Conclusions

We study whether effective attention, the part of the transformer attention matrix that does not get canceled out with the value matrix, gives different interpretations than standard attention. We present a comparison of the two attentions and show that they differ in weights assigned to delimiter tokens such as [SEP] and punctuation marks, but not [CLS] as it was previously thought. Instead, effective attention gives more weight to linguistic features. Given the differences, and that effective attention is more pertinent to the model output by design, we urge to use it for studying transformers’ internals.

As an alternative to analyzing attention weights, Kobayashi et al. 2020 propose anayzing the norm of vectors produced by multiplying the outputs of the value matrix with the attention weights. Following the experimental setting of Clark et al. 2019, i.e., by analyzing 992 sequences extracted from Wikipedia, their norm-based analysis also shows that the contributions of [SEP] and punctuations are actually small. However, unlike us, they report the same observation for [CLS]. Future work might consider a more formal study between the norm-based analysis and effective attention, especially since the norm-based analysis could circumvent the problem of costly SVD.

Acknowledgments

The authors thank Noah A. Smith, members of Noah’s ARK, as well as anonymous reviewers for their helpful feedback, and Olga Kovaleva for sharing the code and model weights for classification of attention patterns.

References

Appendix A Background: On The Rank Of The Value Matrix

The output $Z$ of an individual self-attention head is given by:

where $d_{s}$ is the maximum length of the input sequence (in number of subtokens), $Z_{l-1}$ is the output of the previous transformer layer, $W^{Q},W^{K},W^{V}$ are the query, key, and value weight matrices, respectively. For BERT-base, $d_{q}=d_{k}=d_{v}=64$, $n_{\text{heads}}=12$, $d_{s}=512$, and $d_{v}\cdot n_{\text{heads}}=768$.

Brunner et al. 2020 show that the upper bound of the rank of the value matrix $V$ is given by:

As a result, the left nullspace of $V$, defined as:

which contribute exactly the same to the output as the attention matrix A:

Appendix B Additional Results

We provide the following additional results that complement the discussions in Section 3:

A comparison of the evaluation time with standard vs. effective attention.

In Figure 4, visualization of results presented in Table 2.

Attention to the [CLS] token in Figure 5.

Attention to the [SEP] token in Figure 6.