An Analysis of Simple Data Augmentation for Named Entity Recognition

Introduction

Modern deep learning techniques typically require a lot of labeled data [Bowman et al., 2015, Conneau et al., 2017]. However, in real-world applications, such large labeled data sets are not always available. This is especially true in some specific domains, such as the biomedical and materials science domain, where annotating data requires expert knowledge and is usually time-consuming [Karimi et al., 2015, Friedrich et al., 2020]. Different approaches have been investigated to solve this low-resource problem. For example, transfer learning pretrains language representations on self-supervised or rich-resource source tasks and then adapts these representations to the target task [Ruder, 2019, Gururangan et al., 2020]. Data augmentation expands the training set by applying transformations to training instances without changing their labels [Wang and Perez, 2017].

Recently, there is an increased interest on applying data augmentation techniques on sentence-level and sentence-pair natural language processing (NLP) tasks, such as text classification [Wei and Zou, 2019, Xie et al., 2019], natural language inference [Min et al., 2020] and machine translation [Wang et al., 2018]. Augmentation methods explored for these tasks either create augmented instances by manipulating a few words in the original instance, such as word replacement [Zhang et al., 2015, Wang and Yang, 2015, Cai et al., 2020], random deletion [Wei and Zou, 2019], or word position swap [Şahin and Steedman, 2018, Min et al., 2020]; or create entirely artificial instances via generative models, such as variational auto encoders [Yoo et al., 2019, Mesbah et al., 2019] or back-translation models [Yu et al., 2018, Iyyer et al., 2018].

Different from these sentence-level NLP tasks, named entity recognition (NER) does predictions on the token level. That is, for each token in the sentence, NER models predict a label indicating whether the token belongs to a mention and which entity type the mention has. Therefore, applying transformations to tokens may also change their labels. Due to this difficulty, data augmentation for NER is comparatively less studied. In this work, we fill this research gap by exploring data augmentation techniques for NER, a token-level sequence labeling problem.

Our contributions can be summarized as follows:

We survey previously used data augmentation techniques for sentence-level and sentence-pair NLP tasks and adapt some of them for the NER task.

We conduct empirical comparisons of different data augmentations using two English domain-specific data sets: MaSciP [Mysore et al., 2019] and i2b2-2010 [Uzuner et al., 2011]. Results show that simple augmentation can even improve over a strong baseline with large-scale pretrained transformers.

Related Work

In this section, we survey previously used data augmentation methods for NLP tasks, grouping them into four categories:

Various word replacement variants have been explored for text classification tasks. ?) and ?) replace words with one of their synonyms, retrieved from an English thesaurus (e.g., WordNet). ?) replace words with other words that are predicted by a language model at the word positions. ?) replace uninformative words with low TF-IDF scores with other uninformative words for topic classification tasks.

For machine translation, word replacement has also been used to generate additional parallel sentence pairs. ?) replace words in both the source and the target sentence by other words uniformly sampled from the source and the target vocabularies. ?) search for contexts where a common word can be replaced by a low-frequency word, relying on recurrent language models. ?) replace a randomly chosen word by a soft word, which is a probabilistic distribution over the vocabulary, provided by a language model.

In addition, there are two special word replacement cases, inspired by dropout and masked language modeling: replacing a word by a zero word (i.e., dropping entire word embeddings) [Iyyer et al., 2015], or by a [MASK] token [Wu et al., 2018].

Mention replacement:

?) augment a question answering training set using an external knowledge base. In particular, they extract nominal groups in the training set, perform string matching with entities in Wikidata, and then randomly replace them with other entities of the same type. In order to remove gender bias from coreference resolution systems, ?) propose to generate an auxiliary dataset where all male entities are replaced by female entities, and vice versa, using a rule-based approach.

Swap words:

?) randomly choose two words in the sentence and swap their positions to augment text classification training sets. ?) explore syntactic transformations (e.g., subject/object inversion) to augment the training data for natural language inference. ?) rotate tree fragments around the root of the dependency tree to form a synthetic sentence and augment low-resource language part-of-speech tagging training sets.

Generative models:

?) train a question answering model with data generated by back-translation from a neural machine translation model. ?) and ?) use a sequence-to-sequence model to generate diversely augmented utterances to improve the dialogue language understanding module. ?) convert data from a high-resource language to a low-resource language, using a bilingual dictionary and an unsupervised machine translation model in order to expand the machine translation training set for the low-resource language.

Data Augmentation for NER

Inspired by the efforts described in Section 2, we design several simple data augmentation methods for NER. Note that these augmentations do not rely on any externally trained models, such as machine translation models or syntactic parsing models, which are by themselves difficult to train in low-resource domain-specific scenarios.

For each token, we use a binomial distribution to randomly decide whether it should be replaced. If yes, we then use a label-wise token distribution, built from the original training set, to randomly select another token with the same label. Thus, we keep the original label sequence unchanged. Taking the instance in Table 1 as an example, there are five tokens replaced by other tokens which share the same label with the original tokens.

Synonym replacement (SR):

Our second approach is similar to LwTR, except that we replace the token with one of its synonyms retrieved from WordNet. Note that the retrieved synonym may consist of more than one token. However, its BIO-labels can be derived using a simple rule: If the replaced token is the first token within a mention (i.e., the corresponding label is ‘B-EntityType’), we assign the same label to the first token of the retrieved multi-word synonym, and ‘I-EntityType’ to the other tokens. If the replaced token is inside a mention (i.e., the corresponding label is ‘I-EntityType’), we assign its label to all tokens of the multi-word synonym.

Mention replacement (MR):

For each mention in the instance, we use a binomial distribution to randomly decide whether it should be replaced. If yes, we randomly select another mention from the original training set which has the same entity type as the replacement. The corresponding BIO-label sequence can be changed accordingly. For example, in Table 1, the mention ‘headache [B-problem]’ is replaced by another problem mention ‘neuropathic pain syndrome [B-problem I-problem I-problem]’.

Shuffle within segments (SiS):

We first split the token sequence into segments of the same label. Thus, each segment corresponds to either a mention or a sequence of out-of-mention tokens. For example, the original sentence in Table 1 is split into five segments: [She did not complain of], [headache], [or], [any other neurological symptoms], [.]. Then for each segment, we use a binomial distribution to randomly decide whether it should be shuffled. If yes, the order of the tokens within the segment is shuffled, while the label order is kept unchanged.

All:

We also explore to augment the training set using all aforementioned augmentation methods. That is, for each training instance, we create multiple augmented instances, one per augmentation method.

Experiments and Results

We present an empirical analysis of the data augmentation methods described in Section 3 on two English datasets from the materials science and biomedical domains: MaSciP [Mysore et al., 2019]https://github.com/olivettigroup/annotated-materials-syntheses and i2b2-2010 [Uzuner et al., 2011].https://portal.dbmi.hms.harvard.edu/

MaSciP contains synthesis procedures annotated with synthesis operations and their typed arguments (e.g., Material, Synthesis-Apparatus, etc.). We use the train-dev-test split provided by the authors. i2b2-2010 focuses on the identification of Problem, Treatment and Test from patient reports. We use the train-test split from its corresponding shared task setting and randomly select 15% of sentences from the training set as the development set.

To simulate a low-resource setting, we select the first 50, 150, 500 sentences which contain at least one mention from the training set to create the corresponding small, medium, and large training sets (denoted as S, M, L in Table 3, whereas the complete training set is denoted as F) for each data set. Note that we apply data augmentation only on the training set, without changing the development and test sets.

2 Backbone models

We model the NER task as a sequence-labeling task. Let $\mathbf{x}=\left\langle x_{1},\ldots,x_{T}\right\rangle$ be a sequence of ${T}$ tokens, the model aims to predict a label sequence $\mathbf{y}=\left\langle y_{1},\ldots,y_{T}\right\rangle$, where each label is composed of a position indicator (e.g., BIO schema) and an entity type. The state-of-the-art sequence-labeling models roughly consist of two components: a neural-based encoder which creates contextualized embeddings $r_{i}$ for each token, and a conditional random field output layer, which captures dependencies between neighboring labels:

We consider two encoder variants in our study: one based on LSTM [Graves et al., 2013] and one based on BERT [Devlin et al., 2019]. The LSTM-based encoder consists of a context-independent token embedding layer (e.g., GloVe [Pennington et al., 2014]) and a bidirectional LSTM layer, whose weights are learned from scratch. The representations $r_{i}$ are obtained by concatenating the hidden states of the forward and backward LSTMs at each token position. The BERT-based encoder consists of a sub-token embedding layer and a stack of multi-head self-attention and fully-connected feed-forward layers. The final hidden state corresponding to the first sub-token within each token is used as the representation $r_{i}$. Studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when the BERT models are further pretrained on in-domain data [Gururangan et al., 2020, Dai et al., 2020]. We thus choose SciBERT [Beltagy et al., 2019], which is pretrained on scholar articles, and fine-tune it on the NER task. In our preliminary experiments, we observe that SciBERT achieves significant better results than BERT [Devlin et al., 2019].

We use the Micro-average string match $F_{1}$ score to evaluate the effectiveness of the models. The model which is most effective on the development set, measured using the $F_{1}$ score, is finally evaluated on the test set.

3 Hyperparameters

For each augmentation method, we tune the number of generated instances per training instance from a list of numbers: {1, 3, 6, 10}. When all data augmentation methods are applied, we reduce this tuning list to: {1, 2, 3}, so that the total number of generated instances given each original training instance is roughly the same for different experiments. We also tune the $p$ value of the binomial distribution which is used to decide whether a token or a mention should be replaced (cf., Section 3). It is searched over the range from 0.1 to 0.7, with an incrementation step of 0.2. We perform grid search to find the best combination of these two hyperparameters on the developement set.

4 Results

Table 3 provides the evaluation results on the test sets. The first conclusion we can draw is that all data augmentation techniques can improve over the baseline where no augmentation is used, although there is no single clear winner across both recurrent and transformer models. Synonym replacement outperforms other augmentation on average when transformer models are used, whereas mention replacement appears to be most effective for recurrent models.

Second, applying all data augmentation methods together outperforms any single data augmentation on average, although, when the complete training set is used, applying single data augmentation may achieve better results (c.f., MaSciP-Recurrent and i2b2-2010-Transformer). This scenario may reflect a trade-off between diversity and validity of augmented instances [Hou et al., 2018, Xie et al., 2019]. On the one hand, applying all data augmentation together may prevent overfitting via producing diverse training instances. This positive effect is especially useful when the training sets are small. On the other hand, it may also increase the risk of altering the ground-truth label, or generating invalid instances. This negative effect may dominate for larger training sets.

Third, data augmentation techniques are more effective when the training sets are small. For example, all data augmentation methods achieve significant improvements when the training set contains only 50 instances. In contrast, when the complete training sets are used, only three augmentation methods achieve significant improvements and some even decrease the performance. This has also been observed in previous work on machine translation tasks [Fadaee et al., 2017].

Last but not least, we notice that previous studies mainly investigate the effectiveness of data augmentation with recurrent models where most of the parameters are learned from scratch. Considering the significant improvements when using pretrained transformer models, we argue that it is important to investigate the effectiveness of techniques also on pretrained models, such as BERT [Devlin et al., 2019], because they are supposed to capture various knowledge via self-supervision learning.

Conclusion

We survey previously used data augmentation methods for sentence-level and sentence-pair NLP tasks and adapt them to NER, a token-level task. Through experiments on two domain-specific data sets, we show that simple data augmentation can improve performance even over strong baselines.

References