GECToR -- Grammatical Error Correction: Tag, Not Rewrite

Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, Oleksandr Skurzhanskyi

Introduction

Neural Machine Translation (NMT)-based approaches Sennrich et al. (2016a) have become the preferred method for the task of Grammatical Error Correction (GEC)http://nlpprogress.com/english/grammatical_error_correction.html (Accessed 1 April 2020).. In this formulation, errorful sentences correspond to the source language, and error-free sentences correspond to the target language. Recently, Transformer-based Vaswani et al. (2017) sequence-to-sequence (seq2seq) models have achieved state-of-the-art performance on standard GEC benchmarks Bryant et al. (2019). Now the focus of research has shifted more towards generating synthetic data for pretraining the Transformer-NMT-based GEC systems Grundkiewicz et al. (2019); Kiyono et al. (2019). NMT-based GEC systems suffer from several issues which make them inconvenient for real world deployment: (i) slow inference speed, (ii) demand for large amounts of training data and (iii) interpretability and explainability; they require additional functionality to explain corrections, e.g., grammatical error type classification Bryant et al. (2017).

In this paper, we deal with the aforementioned issues by simplifying the task from sequence generation to sequence tagging. Our GEC sequence tagging system consists of three training stages: pretraining on synthetic data, fine-tuning on an errorful parallel corpus, and finally, fine-tuning on a combination of errorful and error-free parallel corpora.

Related work. LaserTagger Malmi et al. (2019) combines a BERT encoder with an autoregressive Transformer decoder to predict three main edit operations: keeping a token, deleting a token, and adding a phrase before a token. In contrast, in our system, the decoder is a softmax layer. PIE Awasthi et al. (2019) is an iterative sequence tagging GEC system that predicts token-level edit operations. While their approach is the most similar to ours, our work differs from theirs as described in our contributions below:

1. We develop custom g-transformations: token-level edits to perform (g)rammatical error corrections. Predicting g-transformations instead of regular tokens improves the generalization of our GEC sequence tagging system. 2. We decompose the fine-tuning stage into two stages: fine-tuning on errorful-only sentences and further fine-tuning on a small, high-quality dataset containing both errorful and error-free sentences. 3. We achieve superior performance by incorporating a pre-trained Transformer encoder in our GEC sequence tagging system. In our experiments, encoders from XLNet and RoBERTa outperform three other cutting-edge Transformer encoders (ALBERT, BERT, and GPT-2).

Datasets

Table 1 describes the finer details of datasets used for different training stages.

Synthetic data. For pretraining stage I, we use 9M parallel sentences with synthetically generated grammatical errors Awasthi et al. (2019)https://github.com/awasthiabhijeet/PIE/tree/master/errorify.

Training data. We use the following datasets for fine-tuning stages II and III: National University of Singapore Corpus of Learner English (NUCLE)https://www.comp.nus.edu.sg/~nlp/corpora.html Dahlmeier et al. (2013), Lang-8 Corpus of Learner English (Lang-8)https://sites.google.com/site/naistlang8corpora Tajiri et al. (2012), FCE datasethttps://ilexir.co.uk/datasets/index.html Yannakoudakis et al. (2011), the publicly available part of the Cambridge Learner Corpus Nicholls (2003) and Write & Improve + LOCNESS Corpus Bryant et al. (2019)https://www.cl.cam.ac.uk/research/nl/bea2019st/data/wi+locness_v2.1.bea19.tar.gz.

Evaluation data. We report results on CoNLL-2014 test set Ng et al. (2014) evaluated by official $M^{2}$ scorer Dahlmeier and Ng (2012), and on BEA-2019 dev and test sets evaluated by ERRANT Bryant et al. (2017).

Token-level transformations

We developed custom token-level transformations $T(x_{i})$ to recover the target text by applying them to the source tokens $(x_{1}\ldots x_{N})$ . Transformations increase the coverage of grammatical error corrections for limited output vocabulary size for the most common grammatical errors, such as Spelling, Noun Number, Subject-Verb Agreement and Verb Form (Yuan, 2017, p. 28).

The edit space which corresponds to our default tag vocabulary size = 5000 consists of 4971 basic transformations (token-independent KEEP, DELETE and 1167 token-dependent APPEND, 3802 REPLACE) and 29 token-independent g-transformations.

Basic transformations perform the most common token-level edit operations, such as: keep the current token unchanged (tag $KEEP), delete current token (tag$ DELETE), append new token $t_{1}$ next to the current token $x_{i}$ (tag $APPEND_$ t_{1} $) or replace the current token$ x_{i} $with another token$ t_{2} $(tag$ REPLACE_ $t_{2}$ ).

g-transformations perform task-specific operations such as: change the case of the current token (CASE tags), merge the current token and the next token into a single one (MERGE tags) and split the current token into two new tokens (SPLIT tags). Moreover, tags from NOUN NUMBER and VERB FORM transformations encode grammatical properties for tokens. For instance, these transformations include conversion of singular nouns to plurals and vice versa or even change the form of regular/irregular verbs to express a different number or tense.

To obtain the transformation suffix for the VERB_FORM tag, we use the verb conjugation dictionaryhttps://github.com/gutfeeling/word_forms/blob/master/word_forms/en-verbs.txt. For convenience, it was converted into the following format: $token_{0}\_token_{1}:tag_{0}\_tag_{1}$ (e.g., $go\_goes:VB\_VBZ$ ). This means that there is a transition from $word_{0}$ and $word_{1}$ to the respective tags. The transition is unidirectional, so if there exists a reverse transition, it is presented separately.

The experimental comparison of covering capabilities for our token-level transformations is in Table 2. All transformation types with examples are listed in Appendix, Table 9.

Preprocessing. To approach the task as a sequence tagging problem we need to convert each target sentence from training/evaluation sets into a sequence of tags where each tag is mapped to a single source token. Below is a brief description of our 3-step preprocessing algorithm for color-coded sentence pair from Table 3:

Step 1). Map each token from source sentence to subsequence of tokens from target sentence. [A $\mapsto$ A], [ten $\mapsto$ ten, -], [years $\mapsto$ year, -], [old $\mapsto$ old], [go $\mapsto$ goes, to], [school $\mapsto$ school, .].

For this purpose, we first detect the minimal spans of tokens which define differences between source tokens $(x_{1}\ldots x_{N})$ and target tokens $(y_{1}\ldots y_{M})$ . Thus, such a span is a pair of selected source tokens and corresponding target tokens. We can’t use these span-based alignments, because we need to get tags on the token level. So then, for each source token $x_{i}$ , $1\leq i\leq N$ we search for best-fitting subsequence $\Upsilon_{i}=(y_{j_{1}}\ldots y_{j_{2}})$ , $1\leq j_{1}\leq j_{2}\leq M$ of target tokens by minimizing the modified Levenshtein distance (which takes into account that successful g-transformation is equal to zero distance).

Step 2). For each mapping in the list, find token-level transformations which convert source token to the target subsequence: [A $\mapsto$ A]: $KEEP, [ten$ \mapsto $ten, -]:$ KEEP, $MERGE_HYPHEN, [years$ \mapsto $year, -]:$ NOUN_NUMBER_SINGULAR, $MERGE_HYPHEN], [old$ \mapsto $old]:$ KEEP, [go $\mapsto$ goes, to]: $VERB_FORM_VB_VBZ,$ APPEND_to, [school $\mapsto$ school, .]: $KEEP,$ APPEND_{.}].

Step 3). Leave only one transformation for each source token: A $\Leftrightarrow$ $KEEP, ten$ \LeftrightarrowMERGE_HYPHEN, years $\Leftrightarrow$ $NOUN_NUMBER_SINGULAR, old$ \LeftrightarrowKEEP, go $\Leftrightarrow$ $VERB_FORM_VB_VBZ, school$ \LeftrightarrowAPPEND_{.}.

The iterative sequence tagging approach adds a constraint because we can use only a single tag for each token. In case of multiple transformations we take the first transformation that is not a $KEEP tag. For more details, please, see the preprocessing script in our repositoryhttps://github.com/grammarly/gector.

Tagging model architecture

Our GEC sequence tagging model is an encoder made up of pretrained BERT-like transformer stacked with two linear layers with softmax layers on the top. We always use cased pretrained transformers in their Base configurations. Tokenization depends on the particular transformer’s design: BPE Sennrich et al. (2016b) is used in RoBERTa, WordPiece Schuster and Nakajima (2012) in BERT and SentencePiece Kudo and Richardson (2018) in XLNet. To process the information at the token-level, we take the first subword per token from the encoder’s representation, which is then forwarded to subsequent linear layers, which are responsible for error detection and error tagging, respectively.

Iterative sequence tagging approach

To correct the text, for each input token $x_{i}$ , $1\leq i\leq N$ from the source sequence $(x_{1}\ldots x_{N})$ , we predict the tag-encoded token-level transformation $T(x_{i})$ described in Section 3. These predicted tag-encoded transformations are then applied to the sentence to get the modified sentence.

Since some corrections in a sentence may depend on others, applying GEC sequence tagger only once may not be enough to fully correct the sentence. Therefore, we use the iterative correction approach from Awasthi et al. (2019): we use the GEC sequence tagger to tag the now modified sequence, and apply the corresponding transformations on the new tags, which changes the sentence further (see an example in Table 3). Usually, the number of corrections decreases with each successive iteration, and most of the corrections are done during the first two iterations (Table 4). Limiting the number of iterations speeds up the overall pipeline while trading off qualitative performance.

Experiments

Training stages. We have 3 training stages (details of data usage are in Table 1):

Pre-training on synthetic errorful sentences as in Awasthi et al. (2019).

Fine-tuning on subset of errorful and error-free sentences as in Kiyono et al. (2019).

We found that having two fine-tuning stages with and without error-free sentences is crucial for performance (Table 5).

All our models were trained by Adam optimizer Kingma and Ba (2015) with default hyperparameters. Early stopping was used; stopping criteria was 3 epochs of 10K updates each without improvement. We set batch size=256 for pre-training stage I (20 epochs) and batch size=128 for fine-tuning stages II and III (2-3 epochs each). We also observed that freezing the encoder’s weights for the first 2 epochs on training stages I-II and using a batch size greater than 64 improves the convergence and leads to better GEC performance.

Encoders from pretrained transformers. We fine-tuned BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), GPT-2 Radford et al. (2019), XLNet Yang et al. (2019), and ALBERT Lan et al. (2019) with the same hyperparameters setup. We also added LSTM with randomly initialized embeddings ( $dim=300$ ) as a baseline. As follows from Table 6, encoders from fine-tuned Transformers significantly outperform LSTMs. BERT, RoBERTa and XLNet encoders perform better than GPT-2 and ALBERT, so we used them only in our next experiments. All models were trained out-of-the-boxhttps://huggingface.co/transformers/ which seems to not work well for GPT-2. We hypothesize that encoders from Transformers which were pretrained as a part of the entire encoder-decoder pipeline are less useful for GECToR.

Tweaking the inference. We forced the model to perform more precise corrections by introducing two inference hyperparameters (see Appendix, Table 11), hyperparameter values were found by random search on BEA-dev.

First, we added a permanent positive confidence bias to the probability of $KEEP tag which is responsible for not changing the source token. Second, we added a sentence-level minimum error probability threshold for the output of the error detection layer. This increased precision by trading off recall and achieved better$ F_{0.5}$ scores (Table 5).

Finally, our best single-model, GECToR (XLNet) achieves $F_{0.5}$ = 65.3 on CoNLL-2014 (test) and $F_{0.5}$ = 72.4 on BEA-2019 (test). Best ensemble model, GECToR (BERT + RoBERTa + XLNet) where we simply average output probabilities from 3 single models achieves $F_{0.5}$ = 66.5 on CoNLL-2014 (test) and $F_{0.5}$ = 73.6 on BEA-2019 (test), correspondingly (Table 7).

Speed comparison. We measured the model’s average inference time on NVIDIA Tesla V100 on batch size 128. For sequence tagging we don’t need to predict corrections one-by-one as in autoregressive transformer decoders, so inference is naturally parallelizable and therefore runs many times faster. Our sequence tagger’s inference speed is up to 10 times as fast as the state-of-the-art Transformer from Zhao et al. (2019), beam size=12 (Table 8).

Conclusions

We show that a faster, simpler, and more efficient GEC system can be developed using a sequence tagging approach, an encoder from a pretrained Transformer, custom transformations and 3-stage training.

Our best single-model/ensemble GEC tagger achieves an $F_{0.5}$ of 65.3/66.5 on CoNLL-2014 (test) and $F_{0.5}$ of 72.4/73.6 on BEA-2019 (test). We achieve state-of-the-art results for the GEC task with an inference speed up to 10 times as fast as Transformer-based seq2seq systems.

Acknowledgements

This research was supported by Grammarly. We thank our colleagues Vipul Raheja, Oleksiy Syvokon, Andrey Gryshchuk and our ex-colleague Maria Nadejde who provided insight and expertise that greatly helped to make this paper better. We would also like to show our gratitude to Abhijeet Awasthi and Roman Grundkiewicz for their support in providing data and answering related questions. We also thank 3 anonymous reviewers for their contribution.