Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

Introduction

While Neural Machine Translation (NTM) has obtained breakthrough improvements in standard benchmarks, it is known to be particularly sensitive to the size and quality of the training data (Koehn and Knowles, 2017; Khayrallah and Koehn, 2018). In this context, effective approaches to mine and filter parallel corpora are crucial to apply NMT in practical settings.

Traditional parallel corpus mining has relied on heavily engineered systems. Early approaches were mostly based on metadata information from web crawls (Resnik, 1999; Shi et al., 2006). More recent methods focus on the textual content instead. For instance, Zipporah learns a classifier over bag-of-word features to distinguish between ground truth translations and synthetic noisy ones (Xu and Koehn, 2017). STACC uses seed lexical translations induced from IBM alignments, which are combined with set expansion operations to score translation candidates through the Jaccard similarity coefficient (Etchegoyhen and Azpeitia, 2016; Azpeitia et al., 2017, 2018). Many of these approaches rely on cross-lingual document retrieval (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005, 2006; Abdul-Rauf and Schwenk, 2009) or machine translation (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018).

More recently, a new research line has shown promising results using multilingual sentence embeddings aloneMultilingual entence embeddings have also been used as part of a larger system, either to obtain an initial alignment that is then further filtered (Bouamor and Sajjad, 2018) or as an intermediate representation of an end-to-end classifier (Grégoire and Langlais, 2017). (Schwenk, 2018; Guo et al., 2018). These methods use an NMT inspired encoder-decoder to train sentence embeddings on existing parallel data, which are then directly applied to retrieve and filter new parallel sentences using nearest neighbor retrieval over cosine similarity with a hard threshold (España-Bonet et al., 2017; Hassan et al., 2018; Schwenk, 2018).

In this paper, we argue that this retrieval method suffers from the scale of cosine similarity not being globally consistent. As illustrated by the example in Table 1, some sentences without any correct translation have overall high cosine scores, making them rank higher than other sentences with a correct translation. This issue was also pointed out by Guo et al. (2018), who learn an encoder to score known translation pairs above synthetic negative examples and train a separate model to dynamically scale and shift the dot product on held out supervised data. In contrast, our proposed method tackles this issue by considering the margin between the cosine of a given sentence pair and that of its respective $k$ nearest neighbors.

Multilingual sentence embeddings

Figure 1 shows our encoder-decoder architecture to learn multilingual sentence embeddings, which is based on Schwenk (2018). The encoder consists of a bidirectional LSTM, and our sentence embeddings are obtained by applying a max-pooling operation over its output. These embeddings are fed into an LSTM decoder in two ways: 1) they are used to initialize its hidden and cell state after a linear transformation, and 2) they are concatenated to the input embeddings at every time step. We use a shared encoder and decoder for all languages with a joint 40k BPE vocabulary learned on the concatenation of all training corpora.Prior to BPE segmentation, we tokenize and lowercase the input text using standard Moses tools. As the only exception, we use Jieba (https://github.com/fxsjy/jieba) for Chinese word segmentation. The encoder is fully language agnostic, without any explicit signal of the input or output language, whereas the decoder receives an output language ID embedding at every time step. Training minimizes the cross-entropy loss on parallel corpora, alternating over all combinations of the languages involved. We train on 4 GPUs with a total batch size of 48,000 tokens, using Adam with a learning rate of 0.001 and dropout set to 0.1. We use a single layer for both the encoder and the decoder with a hidden size of 512 and 2048, respectively, yielding 1024 dimensional sentence embeddings. The input embeddings size is set to 512, while the language ID embeddings have 32 dimensions. After training, the decoder is discarded, and the encoder is used to map a sentence to a fixed-length vector.

Scoring and filtering parallel sentences

The multilingual encoder can be used to mine parallel sentences by taking the nearest neighbor of each source sentence in the target side according to cosine similarity, and filtering those below a fixed threshold. While this approach has been reported to be competitive (Schwenk, 2018), we argue that it suffers from the scale of cosine similarity not being globally consistent across different sentences.Note that, even if cosine similarity is normalized in the (-1, 1) range, it is still susceptible to concentrate around different values. For instance, Table 1 shows an example where an incorrectly aligned sentence pair has a larger cosine similarity than a correctly aligned one, thus making it impossible to filter it through a fixed threshold. In that case, all four nearest neighbors have equally high values. In contrast, for example B, there is a big gap between the nearest neighbor and its other candidates. As such, we argue that the margin between the similarity of a given candidate and that of its $k$ nearest neighbors is a better indicator of the strength of the alignment.As a downside, this approach will penalize sentences with many paraphrases in the corpus. While possible, we argue that such cases rarely happen in practice and, even when they do, filtering them is unlikely to cause any major harm. We next describe our scoring method inspired by this idea in Section 3.1, and discuss our candidate generation and filtering strategy in Section 3.2.

We consider the margin between the cosine of a given candidate and the average cosine of its $k$ nearest neighbors in both directions as follows:

where $\operatorname{NN}_{k}(x)$ denotes the $k$ nearest neighbors of $x$ in the other language excluding duplicates,Unless otherwise indicated, we use $k=4$. and analogously for $\operatorname{NN}_{k}(y)$. We explore the following variants of this general definition:

Absolute ($\operatorname{margin}(a,b)=a$): Ignoring the average. This is equivalent to cosine similarity and thus our baseline.

Distance ($\operatorname{margin}(a,b)=a-b$): Subtracting the average cosine similarity from that of the given candidate. This is proportional to the CSLS score (Conneau et al., 2018), which was originally motivated to mitigate the hubness problem on Bilingual Lexicon Induction (BLI) over cross-lingual word embeddings.While our work is motivated by thresholding, which is not used in BLI, this connection points out a related problem that our approach also addresses: even when the source sentence is fixed, the potentially different scales of its target candidates might also affect their relative ranking, which ultimately causes the hubness problem. Thanks to its bidirectional nature, our proposed scoring method penalizes target sentences with overall high cosine similarities, so it can learn better alignments that account for this factor.

Ratio ($\operatorname{margin}(a,b)=\frac{a}{b}$): The ratio between the candidate and the average cosine of its nearest neighbors in both directions.

2 Candidate generation and filtering

When mining parallel sentences, we explore the following strategies to generate candidates:

Forward: Each source sentence is aligned with exactly one best scoring target sentence.For efficiency, only the $k$ nearest neighbors over cosine similarity are considered, where the neighborhood size $k$ is the same as that used for the margin-based scoring. Some target sentences may be aligned with multiple source sentences or with none.

Backward: Equivalent to the forward strategy, but going in the opposite direction.

Intersection of forward and backward candidates, which discards sentences with inconsistent alignments.

Max. score: Combination of forward and backward candidates that, instead of discarding all inconsistent alignments, it selects those with the highest score.

These candidates are then sorted according to their margin scores, and a threshold is applied. This can be either optimized on the development data, or adjusted to obtain the desired corpus size.

Experiments and results

We next present our results on the BUCC mining task, UN corpus reconstruction, and machine translation over filtered ParaCrawl. All experiments use an English/French/Spanish/German multilingual encoder trained on Europarl v7 Koehn (2005) for 10 epochs. To cover all languages in BUCC, we use a separate English/French/Russian/Chinese model trained on the UN corpus (Ziemski et al., 2016) for 4 epochs.

The shared task of the workshop on Building and Using Comparable Corpora (BUCC) is a well-established evaluation framework for bitext mining (Zweigenbaum et al., 2017, 2018). The task is to mine for parallel sentences between English and four foreign languages: German, French, Russian and Chinese. There are 150K to 1.2M sentences for each language, split into a sample, training and test set. About 2–3% of the sentences are parallel.

Table 2 reports precision, recall and F1 scores on the training set.Note that the gold standard information was exclusively used to optimize the filtering threshold for each configuration, making results comparable across different variants. Our results show that multilingual sentence embeddings already achieve competitive performance using standard forward retrieval over cosine similarity, which is in line with Schwenk (2018). Both of our bidirectional retrieval strategies achieve substantial improvements over this baseline while still relying on cosine similarity, with intersection giving the best results. Moreover, our proposed margin-based scoring brings large improvements when using either the distance or the ratio functions, outperforming cosine similarity by more than 10 points in all cases. The best results are achieved by ratio, which outperforms distance by 0.3-0.5 points. Interestingly, the retrieval strategy has a very small effect in both cases, suggesting that the proposed scoring is more robust than cosine.

Table 3 reports the results on the test set for both the Europarl and the UN model in comparison to previous work.We use the ratio margin function with maximum score retrieval for our method. The filtering threshold was optimized to maximize the F1 score on the training set for each language pair and model. The gold-alignments of the test set are not publicly available – these scores on the test set are calculated by the organizers of the BUCC workshop. We have done one single submission. Our proposed system outperforms all previous methods by a large margin, obtaining improvements of 10-15 F1 points and showing very consistent performance across different languages, including distant ones.

2 UN corpus reconstruction

So as to compare our method to the similarly motivated system of Guo et al. (2018), we mimic their experiment on aligning the 11.3M sentences of the UN corpus. This task does not require any filtering, so we use forward retrieval with the ratio margin function. As shown in Table 4, our system outperforms that of Guo et al. (2018) by a large margin despite using only a fraction of the training data (2M sentences from Europarl in contrast with over 400M sentences from Google’s internal data).

3 Filtering ParaCrawl for NMT

Finally, we filter the English-German ParaCrawl corpus and evaluate NMT models trained on them. Our NMT models use fairseq’s implementation of the big transformer model (Vaswani et al., 2017), using the same configuration as Ott et al. (2018) and training for 100 epochs. Following common practice, we use newstest2013 and newstest2014 as our development and test sets, respectively, and report both tokenized and detokenized BLEU scores as computed by multi-bleu.perl and sacreBLEU. We decode with a beam size of 5 using an ensemble of the last 10 epochs. One single model is only slightly worse.

Given the large size of ParaCrawl, we first preprocess it to remove all duplicated sentence pairs, sentences for which the fastText language identification modelhttps://fasttext.cc/docs/en/language-identification.html predicts a different language, those with less than 3 or more than 80 tokens, or those with either an overlap of at least 50% or a ratio above 2 between the source and target tokens. This reduces the corpus size from 4.59 billion to 64.4 million sentence pairs, mostly due to deduplication. We then score each sentence pair with the ratio function, processing the entire corpus in batches of 5 million sentences, and take the top scoring entries up to the desired size. Figure 2 shows the development BLEU scores of the resulting system for different thresholds, which peaks at 10 million sentences. As shown in Table 5, this model clearly outperforms the two official filtered versions of ParaCrawl in the test set.

Finally, Table 6 compares our results to previous works in the literature using different training data. In addition to our ParaCrawl system, we include an additional one combining it with all parallel data from WMT18 except CommonCrawl. As it can be seen, our system outperforms all previous systems but Edunov et al. (2018), who use a large in-domain monolingual corpus through back-translation, making both works complementary. Quite remarkably, our full system outperforms Ott et al. (2018) by nearly 2 points despite using the same configuration and training data, so our improvement can be attributed to a better filtering of ParaCrawl.To confirm this, we trained a separate model on WMT data, obtaining 29.4 tokenized BLEU. This is on par with the results reported by Ott et al. (2018) for the same data (29.3 tokenized BLEU). This shows that the difference cannot be attributed to implementation details.

Conclusions and future work

In this paper, we propose a new method for parallel corpus mining based on multilingual sentence embeddings. We use a sequence-to-sequence architecture to train a multilingual sentence encoder on an initial parallel corpus, and a novel margin-based scoring method that overcomes the scale inconsistencies of cosine similarity.

Our experiments show large improvements over previous methods. Our system obtains the best published results on the BUCC mining task, outperforming previous systems by more than 10 F1 points for all the four language pairs. In addition, our method obtains up to 85% precision at reconstructing the 11.3M sentence pairs from the UN corpus, improving over the similarly motivated method of Guo et al. (2018) by more than 30 points. Finally, we show that our improvements also carry over to downstream machine translation, as we obtain 31.2 BLEU points for English-German newstest2014 training on our filtered version of ParaCrawl, an improvement of more than one point over the best performing official release.

The code of this work is freely available as part of the LASER toolkit, together with an additional single encoder which covers 93 languages.https://github.com/facebookresearch/LASER

References