Adapting a Language Model While Preserving its General Knowledge

Introduction

Pre-trained general-purpose language models (LMs) like BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), and GPT-3 Brown et al. (2020) have become a standard component in almost all NLP applications. Researchers have also found that domain-adaptive pre-training (or DA-training for short) using an unlabeled corpus in a specific domain to adapt an LM can further improve the end-task performance in the domain Gururangan et al. (2020); Xu et al. (2019a, b); Sun et al. (2019); Alsentzer et al. (2019). Note that domain-adaptive pre-training is also called post-training Xu et al. (2019a).

Existing DA-training methods simply apply the same pre-training objective, i.e., the mask language model (MLM) loss, to further train an LM using a domain corpus. These methods are sub-optimal because they do not explicitly identify what should be preserved and what should be updated in the LM by the domain corpus.

This paper argues that a good DA-training method has two needs. On the one hand, the general language knowledge learned in the LM should be preserved as much as possible because the target domain data is typically not large enough to be sufficient to learn the general knowledge well. For example, some words and their contexts may appear infrequently in a particular domain. The knowledge about them cannot be learned accurately based on the domain data alone. When these words and contexts appear in an end-task, the system will have difficulties. Thus, we need to rely on the knowledge about them in the LM. Since existing DA-training updates the LM with little guidance, such useful general knowledge may be corrupted. On the other hand, due to polysemy (same word with different meanings in different domains) and the fact that different domains also have their special word usages and contexts, the LM should be specialized or adapted to the target domain. A good DA-training should balance these two needs to adapt the LM to the target domain with minimal corruption to the good general knowledge in the LM.

This paper proposes a novel technique to enable a more informed adaptation to (1) preserve the general knowledge in the LM as much as possible, and (2) update the LM to incorporate the domain-specific knowledge of the target domain as needed. The focus of the existing DA-training research has been on (2). As we argued above, (1) is also important as focusing only on (2) may destroy some useful general knowledge and produce sub-optimal results for end-tasks. To achieve (1), the system should constrain the gradient update of each attention headWe will see in Sec. 4 that constraining the neurons in other layers is unnecessary. based on its importance to the general knowledge so that the general knowledge in LM can be preserved as much as possible. With (1), (2) will be able to change the part of the general knowledge that needs to be updated to adapt the LM to suit the target domain.This is very different from continual learning (CL) Chen and Liu (2018) as CL needs to preserve the past knowledge to deal with catastrophic forgetting McCloskey and Cohen (1989). DA-training can and should change/adapt the general knowledge in the original LM to suit the target domain.

In this paper, we propose a novel model called DGA (DA-training - General knowledge preservation and LM Adaptation) for the purpose. The key idea of the proposed method is to preserve the general language knowledge in the LM while adapting the LM to a specific domain. However, it is not obvious how this can be done, i.e., how to find those parameters that are important for the general knowledge and how to protect them. This paper proposes a novel proxy-based method to achieve the objectives. It works as follows. DGA first estimates the importance of each attention head in the LM via the newly proposed proxy KL-divergence loss (Sec. 3.1). This importance score reflects how important each attention head is to the general knowledge. Based on the importance scores, it performs two key functions: The first function uses the scores to soft-mask (rather than binary-mask or completely block) the gradient update to prevent important general knowledge in LM from being unnecessarily corrupted. This is related to pruning of unimportant attention heads Michel et al. (2019). However, pruning is not directly applicable to DA-training as we will show in Sec. 2. The proposed soft-masking constrains only the backward gradient flow in training. It is not necessary to soft-mask the forward pass in either training or inference. This is important because using the knowledge in the full network encourages maximal integration of pre-trained general knowledge and the target domain-specific knowledge. The second function contrasts the representation for the general knowledge in the LM and the full (including both the general and the domain-specific) knowledge to learn an integrated representation (Sec. 3.2).Contrasting the general and only the domain-specific knowledge gives poorer results (see Sec. 4.3) as it causes the two types of knowledge to split rather than to integrate.

In summary, this paper makes two key contributions.

(1). It proposes the idea of informed adaptation to integrate the specialized knowledge in the target domain into the LM with minimal corruption to the useful general knowledge in the original LM.

(2). It proposes a new model DGA with two novel functions to enable better DA-training. DGA estimates the attention head importance to protect the important general knowledge in the LM and integrates the specialized knowledge in the target domain into the LM through contrasting the general and the full knowledge.

To the best of our knowledge, none of these has been reported in the literature before.

Extensive experiments have been conducted in 6 different domains and on 10 baselines to demonstrate the effectiveness of the proposed DGA.

Related Work

Domain-adaptive pre-training (DA-training). Researchers have applied DA-training to many domains, e.g., reviews Xu et al. (2019a, b), biomedical text Lee et al. (2020), news and papers Gururangan et al. (2020), and social media Chakrabarty et al. (2019). However, they all use the same mask language model (MLM) loss. We argue that it is sub-optimal and it is also important to preserve the general knowledge in the LM as much as possible and integrate it with the target domain knowledge.

Network pruning as importance computation. It is known that many parameters in a neural network are redundant and can be pruned Li et al. (2021); Lai et al. (2021). This has also been shown for pre-trained Transformer Chen et al. (2020a); Lin et al. (2020); Gao et al. (2021b); Michel et al. (2019); Voita et al. (2019). A popular pruning method is to discard the parameters with small absolute values Han et al. (2015); Guo et al. (2016). Other methods prune the network at a higher level. In a Transformer-based model, these include pruning the attention head Michel et al. (2019); Voita et al. (2019); McCarley et al. (2019) and pruning sub-layers in a standard Transformer layer Fan et al. (2020); Sajjad et al. (2020). However, the above methods are not directly applicable to us as we need to compute the head importance for the LM using unlabeled domain data, while the above approaches are all for supervised end-tasks. We propose to use a proxy KL-divergence loss for our purpose. Note that it is possible to prune other sub-layers in the Transformer. However, as shown in Sec. 4.3, estimating the importance for other layers does not improve the performance.

Contrastive learning. Contrastive learning Chen et al. (2020b); He et al. (2020) can learn good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs:

where $N$ is the batch size, $\tau$ is a temperature parameter, $\text{sim}(\cdot)$ is a similarity metric, and $q_{i}$ and $q^{+}_{i}$ are representations for positive pairs $x_{i}$ and $x_{i}^{+}$ (typically, $x_{i}^{+}$ is an augmented sample of $x_{i}$, e.g., generated via cropping, deletion or synonym replacement Gao et al. (2021a)). In the unsupervised contrastive loss, the negative samples are the other samples in the batch, indicated in the denominator.

We mainly use contrasive loss to contrast the representations of the important general knowledge in the original LM and the full knowledge (both the general and domain-specific knowledge) to achieve a good integration of the general knowledge and the domain specific knowledge.

Proposed DGA System

As discussed earlier, DGA goes beyond the MLM loss to perform two more functions: (1) preserving the important general knowledge in the LM by soft-masking the attention heads based on their importance. This helps avoid potential corruption of the general knowledge in the LM in DA-training (Sec. 3.1). However, the challenge is how to identify the general knowledge in the LM and how to protect it. We will propose a method to do that. (2) encouraging the model to learn integrated representations of the target domain and the general knowledge in the LM (Sec. 3.2). It is also not obvious how this can be done. We propose a contrastive learning based method to do it. Figure 1 gives an overview of DGA.

where $h$ indicates the $h^{th}$ attention head. Note that the input $\bm{x}$ is different in each layer since the input of a given layer is the output of last layer. To ease the notation, we use the input $\bm{x}$ for all layers.

Head importance. Researchers have found that not all attention heads are important Michel et al. (2019). We introduce a gate vector, $\bm{g}_{l}$, where each cell is a gate variable, $g_{lh}$, to the attention head summation for detecting the importance of attention heads. The resulting importance scores are used to soft-mask the heads in DA-training.

where $\otimes$ is the element-wise multiplication. A gradient-based head importance detection method is proposed in Michel et al. (2019). Given a dataset $D=\{(\bm{y}_{m},\bm{x}_{m})\}_{m=1}^{M}$ of $M$ samples ($\bm{y}_{m}$ is the label of $\bm{x}_{m}$ as Michel et al. (2019) worked on supervised learning), the importance of a head is estimated with a gradient-based proxy score

where $\nabla_{g_{lh}}$ is the gradient of gate variable $g_{lh}$,

where $\mathcal{L}_{\text{impt}}$ is a task-specific/domain-specific loss function. The gradient can be used as the importance score because changing ${g_{lh}}$ is liable to have a large effect on the model if $I_{lh}$ has a high value.

Although Eq. 6 offers a way to compute the importance of attention heads w.r.t. a given loss $\mathcal{L}_{\text{impt}}$, we are unable to directly apply it: If we use the domain data at hand and the MLM loss as $\mathcal{L}_{\text{impt}}$, $\nabla_{g_{lh}}$ only indicates the importance score for domain-specific knowledge. However, our goal is to estimate the attention heads importance for the general knowledge in LM which requires the data used in training the LM to compute the $\mathcal{L}_{\text{impt}}$. In practice, such data is not accessible to users of the LM. Further, label is needed in Eq. 6 but our domain corpus is unlabeled in DA-training. To address these issues, we propose to compute a proxy KL-divergence loss for $\mathcal{L}_{\text{impt}}$.

Proxy KL-divergence loss. We need a proxy for $\mathcal{L}_{\text{impt}}$ such that its gradient ($\nabla_{g_{lh}}$) can be used to compute head importance without using the LM’s original pre-training data. We propose to use model robustness as the proxy, i.e., we try to detect heads that are important for LM’s robustness. Its gradient, $\nabla_{g_{lh}}$, then indicates the robustness and thus the importance to the LM model. Our rationale is as follows: If an $I_{lh}$ (the average of $|\nabla_{g_{lh}}|$, see Eq. 6) has a high value, it indicates that it is important to the LM’s robustness because its change can cause the LM to change a great deal. It is thus an important head to the LM. In contrast, if $I_{lh}$ has a small value, it is a less or not important head to the LM.

To compute the robustness of the LM, we take a subset (a hyper-parameter) of the target domain data $\{\bm{x}^{\text{sub}}_{m}\}$ (no label in DA-training) and input $\bm{x}^{\text{sub}}_{m}$ twice to the LM and compute the KL-divergence of the two resulting representations,

where $f_{1}$ and $f_{2}$ are the LM with different dropout masks. Note that we don’t need to add any additional dropouts to implement $f$ because independently sampled dropout masks are used as input in the Transformer. In training a Transformer, there are dropout masks placed on fully-connected layers and attention probabilities. Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks. Since dropout is similar to adding noise, the difference between the two representations can be regarded as the robustness of the Transformer model. Figure 1 (A) shows how we compute the importance of each attention head using the gradient of the gate vector $\bm{g}_{l}$.

Soft-masking attention heads in DA-training. Recall we want to preserve the general knowledge in the LM during DA-training using head importance $I_{lh}$. Given the attention head $\text{att}(\bm{x},q)$ and DA-training loss $\mathcal{L}_{\text{DA-train}}$ (typically the MLM loss; we also propose an additional loss in Sec. 3.2), we can “soft mask” its corresponding gradient ($\nabla_{\text{att}_{lh}}$ $\nabla_{\text{att}_{lh}}$ indicates the gradient for attention head $\text{att}_{lh}(\bm{x},q)$, distinguished from $\nabla_{g_{lh}}$ in Eq. 6 which is the gradient for the gate variable $g_{lh}$) using the head importance value $I_{lh}$,

where $I^{\text{norm}}_{lh}$ is from $I_{lh}$ via normalization

Normalize makes the $I_{lh}$ have a mean of 0 and standard deviation of 1. Absolute value of Tanh ensures that $I_{lh}$ takes values in the interval $$. Eq. 9 means to constrain the gradient of the corresponding head$\text{att}_{lh}(\bm{x},q)$by element-wise multiplying one minus the head importance to the gradient. It is “soft-masking” because$I_{lh}$is a real number in$$ (instead of binary {0, 1}), which gives the model the flexibility to adjust the attention head. This is useful because although some heads are important to the LM, they may conflict with the knowledge in the target domain and thus need adjusting. Also note that the soft masks here affect only the backward pass and are not used in forward pass (so that forward pass can use the full network and encourage maximal integration of pre-trained general and domain-specific knowledge) except for feature learning using contrastive learning (see below). Figure 1 (B) shows that attention heads are soft-masked during training.

2 Contrasting General and Full Knowledge

We now present how to integrate the general knowledge in the LM and the domain-specific knowledge in the target domain by contrasting the general knowledge and the full knowledge (both general and domain-specific). We first introduce how we obtain such knowledge from the LM for the input $\bm{x}$, and then discuss how we contrast them.

Obtaining the general knowledge for the input sequence $\bm{x}$ from the LM is by extracting the representation of combining the attention heads and their importance scores ($I^{\text{norm}}_{lh}$ in Eq. 10) in the forward pass. The intuition is that since the importance scores show how important each attention head is to the general knowledge, the resulting representation reflects the main general knowledge used by $\bm{x}$. Formally, we plug $I^{\text{norm}}_{lh}$ (soft-masks) as the gate variable $g_{lh}$ in Eq. 5,

Given the attention heads for general knowledge, we can plug it into the whole Transformer to obtain the final general knowledge (taking the average of each token’s output in the input sequence).

(See $\bm{o}^{\text{gen}}$ also in Figure 1 (B)).

Obtaining the full (both general and domain-specific) knowledge in $\bm{x}$ is similar. The only difference is that we extract the representation of $\bm{x}$ without applying the importance (soft-masks) on attention heads in the forward pass,

Similarly, we can plug it into the Transformer,

(See $\bm{o}^{\text{full}}$ also in Figure 1 (B)). Note that it is possible to use $(1-I_{lh}^{\text{norm}})$ as the importance of domain-specific knowledge and contrast it with the general knowledge. However, this produces poorer results (see Table 3) as explained in footnote 4.

Contrasting general and full knowledge. It is known that contrastive learning helps learn a good isotropic representation that is good for down-stream tasks, with the help of positive and negative instances. We contrast the general ($\bm{o}^{\text{gen}}$) and full ($\bm{o}^{\text{full}}$) representations (as positive and negative instances) for the same input $\bm{x}$ to make them different, which encourages the learning of domain-specific knowledge in $\bm{o}^{\text{full}}$ that is not already in the general knowledge and yet related to and integrated with the general knowledge ($\bm{o}^{\text{gen}}$) of the input.

We construct contrastive instances as follows: for an input $\bm{x}_{m}$, three contrastive instances are produced. Anchor $\bm{o}_{m}$ and positive instance $\bm{o}_{m}^{+}$ are both full knowledge from Eq. 14, obtained based on two independently sampled dropout masks in the Transformer (recall that this can be achieved by inputting $\bm{x}_{m}$ twice (see Sec. 3.1). We regard $\bm{o}_{m}^{+}$ and $\bm{o}_{m}$ as positive instances because the dropout noise has been shown to be good positive instances for improving alignment in training sentence embedding Gao et al. (2021a). Negative instance $\bm{o}_{m}^{-}$ is the general knowledge for $\bm{x}_{m}$ from the LM obtained via Eq. 12. With $\bm{o}_{m}$, $\bm{o}_{m}^{+}$, and $\bm{o}_{m}^{-}$, our contrastive loss is ($\text{sim}(\cdot)$ is the cosine similarity),

Compared to Eq. 1, the second term is added in the denominator, i.e., general knowledge representations as additional negative samples/instances. Figure 1 (B) shows a red arrow pointed from $\bm{o}^{\text{full}}$ to itself, indicating the positive instances are from inputting twice. The dashed red arrow pointing to $\bm{o}^{\text{gen}}$ indicates the negative instances contrasting the specialized and general knowledge.

3 DGA Objectives

DGA is a pipelined model: First, a subset of the domain data is used to estimate the attention head importance ($I_{lh}$ in Sec. 3.1). Second, given the attention head importance, we compute the final domain-adaptive loss by combining the conventional Masked Language Model (MLM) loss (include the proposed soft-masking for general knowledge) and the proposed contrastive loss:

where $\lambda_{1}$ is the hyper-parameter to adjust the impact of the added term.

Experiments

We follow the experiment setup in Gururangan et al. (2020). RoBERTa Liu et al. (2019)https://huggingface.co/roberta-base is used as the LM. In each experiment, we first DA-train the LM and then fine-tune it on the end-task. The final evaluation is based on the end-task results.

Datasets: Table 1 shows the statistics of the unlabeled domain datasets for DA-training and their corresponding end-task classification datasets. We use 6 unlabeled domain datasets:We down-sampled the PubMed due to its huge original size. In general, our datasets are much smaller comparing to previous work Gururangan et al. (2020) (which used more than 11GB of data for each domain). Our experiments showed that a smaller dataset is sufficient and more data does not help. It also requires much less computation resource. 3 of them are about reviews: Yelp Restaurant Xu et al. (2019a), Amazon Phone Ni et al. (2019), Amazon Camera Ni et al. (2019); 3 of them are academic papers: ACL Papers Lo et al. (2020), AI Papers Lo et al. (2020), and PubMed Papershttps://pubmed.ncbi.nlm.nih.gov/. Each unlabeled domain dataset has a corresponding end-task classification datasetNote that our results are different from those presented in Table 5 of Gururangan et al. (2020) because we observe very high variances due to very small original test sets and thus re-partition the training and test set (by enlarging the test set and reducing the training set slightly): RestaurantTo be consistent with existing research Tang et al. (2016), examples with conflict polarities (both positive and negative sentiments are expressed about an aspect term) are not used. Xu et al. (2019a), Phone Ding et al. (2008); Hu and Liu (2004), Camera Ding et al. (2008); Hu and Liu (2004)Note that Ding et al. (2008) and Hu and Liu (2004) contain 9 and 5 domains, respectively. We extract those domains related to “Phone” and “Camera” from them., ACL (ACL-ARC in Jurgens et al. (2018)), AI (SCIERC in Luan et al. (2018)), and PubMed (CHEMPORT in Kringelum et al. (2016)).

(1). Non-DA-training (RoBERTa) Liu et al. (2019) uses the original RoBERTa for the end-task fine-tuning without any DA-training.

(2). DA-training using masked language model loss (MLM) is the existing DA-training method. To our knowledge, existing DA-training systems are all based on the MLM loss.

(3). DA-training using adapter-tuning (MLM (Adapter)) adds adapter layers between layers of Transformer for DA-training. An adapter Houlsby et al. (2019) has two fully connected layers and a skip connection. During DA-training, the Transformer is fixed, only the adapters are trained. The bottleneck (adapter) size is set to 64 Houlsby et al. (2019). During end-task fine-tuning, both RoBERTa and adapters are trainable for fair comparison.

(4). DA-training using prompt-tuning (MLM (Prompt)) Lester et al. (2021) adds a sequence of prompt tokens to the end of the original sequence. In DA-training, RoBERTa (the LM) is fixed and only the prompt tokens are trained. In end-task fine-tuning, both LM and the trained prompt are trainable. We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in DA-training, following the setting in Lester et al. (2021).

(5). Knowledge distillation (MLM+KD) Hinton et al. (2015) minimizes the representational deviation between the general knowledge in LM and the specialized knowledge in DA-training. We compute the KL divergence between the representations (the output before the masked language model prediction head) of each word of the two models (LM and DA-trained) as the distillation loss.

(6). Adapted distillation through attention (MLM+AdaptedDeiT) is derived from DeiT Touvron et al. (2021), a distillation method for visual Transformer (ViT) Dosovitskiy et al. (2020). We adapt DeiT to a text-based and unsupervised model by distilling the LM representationWe take the average of its token’s output as sequence’s representation. The same for SimCSE baseline. to the added distillation token and change ViT to RoBERTa.

(7, 8). DA-training using sequence-level contrastive learning (MLM+SimCSE and MLM+InfoWord). SimCSE is a contrastive learning method for sentence embedding Gao et al. (2021a). We use its unsupervised version where positive samples are from the same input with different dropout masks and negative samples are other instances in the same batch. InfoWord Kong et al. (2020) is another contrastive learning method contrasts the span-level local representation and sequence-level global representation.

(9, 10). DA-training using token-aware contrastive learning (MLM+TaCL and MLM+TaCO). TaCL Su et al. (2021) and TaCO Fu et al. (2022) are two recent methods to improve BERT pre-training with token-aware contrastive loss.TaCL and TaCO are not a DA-training model. It pre-trains an LM to improve it using the same data as that for training the LM. We switch the data to our target domain data. We change the backbone to RoBERTa for fair comparison.

2 Implementation Detail

Architecture. We adopt $\text{RoBERTa}_{\textbf{BASE}}$ as our backbone LM (12 layers and 12 attention heads in each layer). A masked language model head is applied for DA-training. The end-task fine-tuning of RoBERTa follows the standard practice. For the three ASC tasks (see Table 1), we adopt the ASC formulation in Xu et al. (2019a), where the aspect (e.g., “sound”) and review sentence (e.g., “The sound is great”) are concatenated via .

Hyperparameters. Unless otherwise stated, the same hyper-parameters are used in all experiments. The maximum input length is set to 164 which is long enough for all datasets. Adam optimizer is used for both DA-training and end-task fine-tuning. The max sequence length is set to 164, which is long enough for our end-tasks and only needs moderate computational resources.

Domain-adaptive pre-training (DA-training). The learning rate is set to 1e-4 and batch size is 256. We train 2.5K steps for each domain, roughly a full pass through the domain data, following Gururangan et al. (2020); Xu et al. (2019a). The subset of data $\{\bm{x}^{\text{sub}}_{m}\}$ for computing $\mathcal{L}_{\text{impt}}$ to determine head importance in Sec. 3.1 is set to 1.64 Million tokens, which is sufficient in our experiments. $\lambda_{1}$ in Eq. 16 is set to 1 and $\tau$ in Eq. 15 is set to 0.05.

End-task fine-tuning. The learning rate is set to 1e-5 and batch size to 16. We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera. We simply take the results for the last epoch as we empirically found that the above number of epochs gives us stable and convergence results.

3 Evaluation Results and Ablation Study

We report the end-task results of the 10 baselines on the 6 datasets in Table 2.

Superiority of DGA. Our DGA consistently outperforms all baselines. Thanks to the proposed more informed adaptation, DGA improves over the widely used traditional DA-training baseline MLM. We also see that MLM markedly outperforms RoBERTa (non-DA-training) on average (see the last column). We discuss more observations about the results bellow.

(1). Training the entire LM in DGA helps achieve much better results. Using adapter (MLM (adapter)) and prompt (MLM (prompt)) have mixed results. This is because adapter and prompt do not have sufficient trainable parameters, which are also randomly initialized and can be difficult to train.

(2). DGA is also better than distillation-based systems: MLM+AdaptedDeiT and MLM+KD, which try to preserve the past knowledge. This is not surprising because the goal of DA-training is not simply preserving the previous knowledge but also to adapt/change it as needed to suit the target domain. DGA is specifically designed for this with soft-masking and contrasting of knowledge.

(3). The contrastive learning in DGA is more effective than the other contrastive alternatives (MLM+SimCSE, MLM+TaCL, MLM+TaCO and MLM+InfoWord). This indicates contrasting the general and full knowledge for knowledge integration is important.

Effectiveness of the proxy KL-divergence loss. We use the proposed proxy KL-divergence loss to compute the head importance to identify the general language knowledge in the LM without using the LM’s original pre-training data (Sec. 3.1).

For evaluation, we are interested in how good the proxy is. Since we don’t have the data that pre-trains RoBERTa, it is not obvious how to assess the quality of the proxy directly. Here, we provide some indirect evidences to show the effectiveness of the proxy for computing the importance of units to the general knowledge in the LM.

We conduct a separate experiment to compare the attention heads’ importance score vectors after applying the proxy using the data from different domains. For each domain $i$, we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain $i$. We get 0.92 for Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera. We see that different domains give similar importance values, which indirectly show that our proxy can identify the common general knowledge.

We also compute the importance score distributions of the proxy. For each of the 6 domains, after applying the proxy, around 20% of the attention heads are heavily protected ($0.8\leq I^{\text{norm}}_{lh}\leq 1.0$) and another 20% moderately protected ($0.6\leq I^{\text{norm}}_{lh}<0.8$), which indicate the general knowledge. While Phone, AI, Camera and Restaurant share a similar distribution, ACL and PubMed protect slightly less. This is understandable as PubMed and ACL (medical or NLP publications) are probably less common than the other domains and the general knowledge in the LM covers them less.

Ablation study. To better understand DGA, We want to know (1) whether constraining the neurons in other layers are helpful (the proposed DGA only constrains the attention heads), and (2) where the gain of DGA is from. To answer (1), we constrain the training of different layers in a standard Transformer. In Table 3 (rows 3-5), “H”, “I”, and “O” refer to attention head, intermediate layer, output layer in a standard Transformer layer, respectively. “E” refers to the embedding layers. The brackets with combination of “H, I, O, E” indicate the location we apply the soft-masking (DGA only applies soft-masking in the attention head). We can see their results are similar or worse than DGA, implying that attention heads are more indicative of important knowledge. To answer (2), we conduct the following ablation experiments: (i) DGA (w/o contrast), without the contrastive loss, but only soft-masking the backward pass according to the attention head importance. (ii) DGA (random masking) with randomly generated attention head importance scores and using them to do soft-masking and contrastive learning. (iii) Ensemble (LM+MLM) performs the end-task fine-tuning on both the MLM DA-trained RoBERTa (conventional DA-training) and the original RoBERTa (LM) by concatenating their outputs and taking the average. (iv) DGA (domain-specific) refers to the variant that contrasts domain-specific and general knowledge (see Sec. 3.2).We don’t have DGA(w/o soft-masking) because our contrastive learning relies on soft-masking. If removed, contrastive loss will not have the additional negative samples and our DGA becomes MLM+SimCSE.

Table 3 shows that the full DGA always gives the best result, indicating every component contributes. Additional observations are as follows:

(1) DGA’s gain is partially from the novel soft-masking: we can see that on average, DGA (w/o contrast) outperforms conventional DA-training (MLM). Besides, our gradient-based mask is informative: we can see DGA (random mask) is worse than DGA (w/o contrast) on all datasets. DGA (w/o contrast) is even better than Ensemble, which directly combines the information given by both the original LM and the traditional DA-trained model during end-task fine-tuning

(2) Besides soft-masking, contrasting the general and full knowledge also helps. We can see DGA outperforms DGA (w/o contrast) and DGA (domain-specific) in all datasets.

Conclusion

This paper argued that an effective DA-training method should effectively integrate the target domain knowledge to the general knowledge in the LM. Existing approaches do not explicitly do this. This paper proposed a novel method DGA to achieve it (1) by estimating the attention heads importance in LM and using the importance scores to soft-mask the attention heads in DA-training to preserve the important knowledge in LM as much as possible, and (2) by contrasting the general and the full knowledge. Extensive experiment results demonstrated the effectiveness of the proposed approach DGA.

Limitations

While effective, DGA has some limitations. First, the main focus of DGA is to adapt an LM to a given target domain. It does not consider the generalization to other domains. For example, it will be interesting to incrementally or continually adapt an LM to more and more domains to make the LM more useful Ke et al. (2020, 2021, 2022). Second, the importance of parameters for general knowledge in the LM is computed using a proxy method based on model robustness. Although it is quite effective, it is interesting to explore other approaches to further improve it. We will work on these in our future work as specializing and improving an LM is an important problem.

Acknowledgments

The work of Zixuan Ke and Bing Liu was supported in part by three National Science Foundation (NSF) grants (IIS-1910424, IIS-1838770, and CNS-2225427).

References

Appendix A Datasets Details

Table 2 in the main paper has given the number of examples in each dataset. Here we provide additional details about the 4 types of end-tasks.

(1) (Phone, Camera and Restaurant) Aspect Sentiment Classification (ASC) is defined as follows Liu (2015): given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in a domain or product category (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for Phone and Camera, there are only negative and positive polarities in the data).

(2) (ACL) Citation Intent Classification is defined as follows: given a citing sentence (a sentence contains a citation), classify if the sentence expresses a citation function among “background”, “motivation”, “uses”, “extension” and “comparison or contrast future”.

(3) (AI) Relation Classification is defined as follows: given a within-sentence word sequence spans containing a pair of entities, classify if the span expresses a relation among “feature of”, “conjunction”, “evaluate for”, “hyponym of”, “used for”, “part of” and “compare”.

(4) (PubMed) Chemical-protein Interaction Classification is defined as follows: given a span containing a pair of chemical and protein, classify if the span expresses a chemical-protein interaction among “downregulator”, “substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”, “activator”, “product of”, “agonist-activator”, “inhibitor”, “upregulator”, “substrate product of”, “agonist-inhibitor”and “antagonist”.

Appendix B Standard Deviations

Table 4 reports the standard deviations of the corresponding results in Table 2 (in the main paper) of DGA and the considered baselines over 5 runs with random seeds. We can see the results of DGA are stable. Some baselines (e.g., RoBERTa in AI, MLM in Camera and MLM+TaCL in ACL) can have quite large standard deviations.

Table 5 reports the standard deviations of the corresponding results in Table 3 (in the main paper) of DGA and the considered baselines over 5 runs with random seeds. We can see the results of DGA are stable. Some baselines (e.g., DGA (random mask) and DGA (w/o contrast) in Camera) can have quite large standard deviations.