MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset

Introduction

Translation quality estimation (QE) is the task of evaluating a translation system’s quality without access to reference translations (Blatz et al. 2004; Specia et al. 2018b). This task has numerous applications: deciding if a sentence or document that has been automatically translated is ready to be sent to the final user or if it needs to be post-edited by a human, flagging passages with potentially critical mistakes, using it as a metric for translation quality when a human reference is not available, or in the context of computer-aided translation interfaces, highlighting text that needs human revision and estimating the human effort.

Due to its high relevance, QE has been the subject of evaluation campaigns in the Conference for Machine Translation (WMT) since 2014 (Bojar et al. 2014; Specia et al. 2018a; Fonseca et al. 2019; Specia et al. 2020), where datasets in various language pairs have been created containing source sentences, their automatic translations, and human post-edited text. However, the currently existing data has several shortcomings. First, the MT system used to produce the translations is not publicly available, which makes it impossible to develop the so-called glass-box approaches to QE and exploit model confidence (or conversely, uncertainty) of the MT system or look into its internal states. Second, the quality assessments have been either produced based on the difference between the MT output and the post-edited text (e.g., through the human translation error rate metric, HTER, or by marking individual words with ok or bad labels), or by direct human assessments, but not both—which raises the question of how much these two quality assessments correlate. Third, most datasets have focused exclusively on high-resource language pairs, where it is often the case that many sentences are correctly translated; however, medium and low-resource settings are the ones where QE would be particularly useful, since it is where MT currently presents serious challenges. Finally, most of these datasets focus on a specific domain, such as IT or life sciences, where translations are generated by a domain-specific MT model, which also tends to result in most sentences being translated with high-quality.

To overcome the limitations stated above, we introduce MLQE-PE, the first multilingual quality estimation and post-editing dataset that combines the following features:

It includes access to the state-of-the-art neural MT (NMT) models built with an open-source toolkit (fairseq, Ott et al. 2019), that were used to produce the translations in the dataset. This opens the door to uncertainty-based and glass-box approaches to QE.

It combines both direct assessments of MT quality and post-edits. This allows combining two sorts of quality assessments: how good a translation is and how much effort is necessary to correct it. Moreover, the post-edited sentences can be used for training and evaluating automatic post-editing systems, another important task considered in WMT campaigns Chatterjee et al. 2019.

It contains the titles of the Wikipedia articles where the original sentences were extracted from, thus allowing to take document-level context into account when predicting sentence-level or word-level MT quality.

It includes 11 language pairs, mixing high-resource language pairs (English-German – En-De and English-Chinese – En-Zh, and Russian-English – Ru-En), medium-resource (Romanian-English – Ro-En, and Estonian-English – Et-En) and low-resource ones (Nepali-English – Ne-En, Sinhala-English – Si-En, Pashto-English – Ps-En, Khmer-English – Km-En, English-Japanese – En-Ja, and English-Czech – En-Cs).

This dataset was created with contributions from different institutions: Facebook, University of Sheffield and Imperial College selected the Wikipedia articles and sentences, built the NMT models, prepared and outsourced data for DA annotation in 10 language pairs (En-De, En-Zh, Ro-En, Et-En, Ne-En, Si-En, Ps-En, Km-En, En-Ja, En-Cs). IQT Labs led the same efforts for collecting and DA-annotating the Ru-En data. Facebook, University of Sheffield and Imperial College also outsourced data for all language pairs except En-De and En-Zh for post-editing, and created reference translations for Et-En. Unbabel and Instituto de Telecomunicações outsourced the post-editing of En-De and En-Zh sentences. The current version of MLQE-PE is publicly available from https://github.com/sheffieldnlp/mlqe-pe.

Data Collection and Statistics

We briefly describe the data collection and preparation process. Table 1 presents some statistics about the MLQE-PE dataset. As shown in Table 1, we collected 10K sentences split into train, dev and two test partitions (test20 and test21) for nine language pairs. In addition, we collected 2K sentences for 4 language pairs, which are meant to be used for testing QE in a zero-shot setting where no training or development data is provided. 1K of these sentences will be kept as a blind test set and released later.

For the most part, the dataset is derived from Wikipedia articles (with exception of Russian-English, described below). The source sentences were collected from Wikipedia articles following the sampling process outlined in FLORES Guzmán et al. 2019. First, we sampled documents from Wikipedia for English, Estonian, Romanian, Sinhala, Nepali, Khmer and Pashto. Second, we selected the top 100 documents containing the largest number of sentences that are: (i) in the intended source language according to a language-id classifier https://fasttext.cc and (ii) have the length between 50 and 150 characters. In addition, we filtered out sentences that have been released as part of recent Wikipedia parallel corpora Schwenk et al. 2019, ensuring that our dataset is not part of parallel data commonly used for NMT training.

For every language, we randomly selected the required number of sentences from the sampled documents and then translated them using SOTA NMT models (see below). For German and Chinese, we followed an additional procedure in order to ensure sufficient representation of high- and low-quality translations for these high-resource language pairs. We selected the sentences with minimal lexical overlap with respect to the NMT training data. Specifically, we extracted content words for each sentence in the data used for training the NMT models and in the Wikipedia data. We computed perplexity scores for the Wikipedia sentences given the NMT training data. Finally, we sampled 20K from available Wikipedia sentences weighted by the perplexity scores.

In addition, we collected human reference translations for a 1K subset of Estonian-English dev/test data. Two reference translations were generated independently by two professional translators. This part of the dataset allows for comparing reference-free MT evaluation with reference-based approaches (see Fomicheva et al. 2020 for details).

The Russian-English data collection followed a slightly different set up collected by collaborators from IQT Labs. We note that Facebook was not involved in the collection of the Russian-English data. The original sentences were collected from multiple sources in order to gather a varied sample of data in different domains that are still challenging for current NMT systems. Data sources include: Russian proverbs and Reddit data from various subreddits, particularly those focused on topics of politics and religion. We included Reddit data since colloquial text is a challenge for MT. We included Russian proverbs from WikiQuotes to test MT on short sentences with unconventional grammar. We used the Reddit API and queried the most recent 1000 posts at the time, and the most recent 1000 comments in each of the selected subreddits. We automatically split the posts into sentences and then reviewed these manually. Markdown was removed and HTML unencoded. We removed sentences shorter than 15 characters or longer than 500 characters. We also removed sentences that did not have a source link. Table 3 shows the number of segments corresponding to each data source and the corresponding average direct assessment score.

NMT models

Transformer-based Vaswani et al. 2017 NMT models were trained for all languages using the fairseq toolkit. https://github.com/pytorch/fairseq For Et-En, Ro-En, En-De and En-Zh we trained the MT models based on the standard Transformer architecture following the implementation details described in Ott et al. 2018. We used publicly available MT datasets such as Paracrawl Esplà et al. 2019 and Europarl Koehn 2005. For Ru-En, translations were produced with the already existing Transformer-based NMT model described in Ng et al. 2019. https://github.com/pytorch/fairseq/tree/master/examples/wmt19 Si-En and Ne-En MT systems were trained based on Big-Transformer architecture as defined in Vaswani et al. 2017. For these low-resource language pairs, the models were trained following the FLORES semi-supervised setting Guzmán et al. 2019, https://github.com/facebookresearch/flores/blob/master/reproduce.sh which involves two iterations of backtranslation using the source and the target monolingual data. For Ps-En, Km-En, En-Cs and En-Ja we use multilingual MT models described in Tang et al. 2020. Instructions for training the models and generating the translations can be found at https://github.com/pytorch/fairseq/tree/master/examples/multilingual.

The data used for training the NMT models is available from http://www.statmt.org/wmt20/quality-estimation-task.html. We provide access to the information from the NMT model used to generate the translations: model score for the sentence and log probabilities for words, as well as the NMT systems themselves.

Direct assessments.

To collect human quality judgments, we followed the FLORES setup Guzmán et al. 2019 inspired by the work of Graham et al. 2013. Specifically, the annotators were asked to rate translation quality for each sentence on a 0–100 scale, where the 0–10 range represents an incorrect translation; 11–29, a translation that contains a few correct keywords, but the overall meaning is different from the source; 30–50, a translation with major mistakes; 51–69, a translation which is understandable and conveys the overall meaning of the source but contains typos or grammatical errors; 70–90, a translation that closely preserves the semantics of the source sentence; and 91–100, a perfect translation.

Each segment was evaluated independently by three professional translators from a single language service provider. To improve annotation consistency, any evaluation in which the range of scores among the raters was above 30 points was rejected, and an additional rater was requested to replace the most diverging translation rating until convergence was achieved. To further increase the reliability of the test and development partitions of the dataset, we requested an additional set of three annotations from a different group of annotators (i.e., from another language service provider) following the same annotation protocol, thus resulting in a total of six annotations per segment.

Raw human scores were converted into z-scores, that is, standardized according to each individual annotator’s overall mean and standard deviation. The scores collected for each segment were averaged to obtain the final score. Such setting allows for the fact that annotators may genuinely disagree on some aspects of quality.

Human post-editing.

For all language pairs, the translated sentences have been post-edited by human translators. For En-De and En-Zh, we used paid editors from the Unbabel community. For all other languages, we used professional translators subcontracted by Facebook. The human translators performing post-editing had no access to the direct assessments scores.

Table 2 shows average translation quality for all language pairs based on direct assessment annotation (DA) and post-editing (HTER) for the test21 partition of the dataset. Figures 1 and 2 show the distribution of the corresponding sentence-level scores, as well as the scatter plot of DA against HTER scores.

First, we note that the distribution of direct assessment scores is very different across language pairs. This illustrates the variety of the collected data in terms of MT output quality. For low-resource language pairs there are more sentences with low direct assessment scores, whereas in the case of high-resource language pairs the vast majority of translations received a high score. In particular, En-De has a very peaked distribution with very little variability in quality.

Second, we note that higher DA score often corresponds to lower translation edit rate in Table 2. Thus, on average direct assessment and post-editing effort produce consistent results as an indication of overall translation quality per language pair. However, sentence-level DA and HTER scores for the same data behave quite differently. Table 4 shows the correlation between direct assessments and HTER scores for all the language pairs on the test21 partition of the dataset. As illustrated in Table 4 and in the scatter plots on Figures 1 and 2 for most of the language pairs there is a weak negative correlation between the two types of quality scores.

Direct quality assessment and post-editing give two different perspectives on MT quality. Table 6 shows an example where direct assessment and HTER lead to a different interpretation of quality. Direct assessment score is low as the MT output contains a serious error that distorts the meaning of the sentence: “bars” (as in “metal bars”) is translated as “pub”. However the sentence is easy to post-edit as the error involves only one word to be replaced, resulting in a low HTER score. Table 6 illustrates the opposite: MT output was assigned a high direct assessment score, but the HTER score is also high, indicating that substantial changes were introduced during post-editing. The post-edited version is more fluent, whereas the MT output is a more literal rendering of the source sentence, but the meaning is preserved and, therefore, it received a high direct assessment score.

Word-level labels

In the datasets containing post-edit annotation, we also obtained word-level labels for fine-grained post-editing effort estimation. Both the source and MT sides have them.

In order to generate them, we first align source and MT outputs using SimAlign https://github.com/cisnlp/simalign. We follow the findings of Sabet et al. 2020 and use Argmax matching for high resource languages that are close to english (En-De, En-Cs) and Itermax for the rest of the language pairs. We then compute the shortest edit distances between MT and post-edited texts with Tercom http://www.cs.umd.edu/~snover/tercom/; this effectively informs us which words were deleted, inserted or replaced. Then, any word $w_{s}$ in the source aligned to a word $w_{m}$ in MT that was kept in the post-edit receives a tag ok; if $w_{s}$ is not aligned with any other word in MT or if $w_{m}$ was deleted in the post-edit, it is tagged bad. Thus, bad tags in the source side indicate which words caused MT errors.

For the MT side, we tag both words and the gaps between them, indicating whether a missing additional word should have been there. Any word $w_{m}$ aligned to another word $w_{p}$ in the post-edit receives a tag ok; words deleted or replaced are tagged bad. Any gap $g$ between words in the MT output, before the first word or after the last one receives a tag ok if no word $w_{p}$ is inserted in there, and bad otherwise The code to reproduce the word tagging and HTER calculation for the MLQE-PE data can dbe found in https://github.com/deep-spin/qe-corpus-builder.

Statistics for word-level tags are shown in Table 7. We see that most sentences in the dataset have at least one bad tag; in the case of En-Zh, it is nearly all of them. The overall amount of bad tags is also higher in the En-Zh data, especially in the source side.

Baseline performance

We report the performance of baseline systems trained on the MLQE-PE data. The baselines trained on HTER and DA scores both follow the predictor-estimator architecture Kim et al. 2017 and are implemented using the OpenKiwi framework Kepler et al. 2019. The hyper-parameters used to train the baseline models are provided in Table 8.

For the predictor (feature extraction) part, we use pre-trained, multilingual XLM-RoBERTa encoders Conneau et al. 2020. For both baselines the huggingface implementation of the XLM-RoBERTa base model is used https://huggingface.co/transformers/pretrained_models.html. The xlm-roberta-base encoder is first fine-tuned on the concatenated source and target sentences from the train and development partitions of all language pairs (see Table 1). The fine-tuning uses a masked language modeling (MLM) loss. We use a script based on: https://github.com/huggingface/transformers/blob/master/examples/legacy/run_language_modeling.py with per_machine_train_batch_size set to $16$ and block_size set to $512$. The fine-tuned model is then used to jointly encode the source and target sentences, with target first. The predictor features are generated using average pooling over the target embeddings and forwarded to the estimator module which corresponds to a feed-forward layer. The combined model parameters ($\sim 281M$ parameters) are trained on the combined training data for the DA and HTER tasks respectively ($7000$ sentence pairs for each language pair). The available combined development data ($1000$ sentence pairs for each language pair) was used to perform early stopping. Note that the configurations follow the configuration file format of OpenKiwi and any additional configurations not mentioned in Table 8 are identical to the default ones shown in the github configuration file. https://github.com/Unbabel/OpenKiwi/blob/master/config/xlmroberta.yaml

Tables 9 and 10 present the performance of our baseline systems for each label and language pair, for sentence- and word-level predictions respectively.

Conclusions

We introduced MLQE-PE, a new dataset that was mainly created to be used for the tasks of quality estimation (sentence and word-level prediction) and automatic post-editing. It contains data in seven language pairs, direct assessment and post-editing-based sentence-level labels, as well as binary good/bad word-level labels. In addition, a subset of the data contains independently created reference translations, which can be used, for example, for machine translation evaluation. The dataset is freely available and was already used for the WMT2020 and WMT2021 shared tasks on Quality Estimation and Automatic Post-Editing.

We hope that this data will foster further work on these and other tasks, such as uncertainty estimation and model calibration. We also hope it will sparkle interest from researchers who may want to contribute related resources, i.e., more data, different languages, etc.

Acknowledgments

Marina Fomicheva, Frédéric Blain and Lucia Specia were supported by funding from the Bergamot project (EU H2020 Grant No. 825303). André Martins and Erick Fonseca were funded by the P2020 programs Unbabel4EU (contract 042671) and MAIA (contract 045909), by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. We would like to thank Marina Sánchez-Torrón and Camila Pohlmann for monitoring the post-editing process. We also thank Mark Fishel from the University of Tartu for providing the Estonian reference translations.

References