Adversarial Neural Machine Translation

Introduction

Neural Machine Translation (NMT) [Cho et al., 2014, Bahdanau et al., 2014] has drawn more and more attention in both academia and industry [Luong and Manning, 2016, Jean et al., 2015, Shen et al., 2016, Tu et al., 2016b, Sennrich et al., 2016, Wu et al., 2016]. Compared with traditional Statistical Machine Translation (SMT) [Koehn et al., 2003], NMT achieves similar or even better translation results in an end-to-end framework. The sentence level maximum likelihood principle and gating units in LSTM/GRU [Hochreiter and Schmidhuber, 1997, Cho et al., 2014], together with attention mechanisms grant NMT with the ability to better translate long sentences.

Despite its success, the translation quality of latest NMT systems is still far from satisfaction and there remains large room for improvement. For example, NMT usually adopts the Maximum Likelihood Estimation (MLE) principle for training, i.e., to maximize the probability of the target ground-truth sentence conditioned on the source sentence. Such an objective does not guarantee the translation results to be natural, sufficient, and accurate compared with ground-truth translation by human. There are previous works [Ranzato et al., 2015, Shen et al., 2016, Bahdanau et al., 2016] that aim to alleviate such limitations of maximum likelihood training, by adopting sequence level objectives (e.g., directly maximizing BLEU [Papineni et al., 2002]), to reduce the objective inconsistency between NMT training and inference. Yet somewhat improved, such objectives still cannot fully bridge the gap between NMT translations and ground-truth translations.

We, in this paper, adopt a thoroughly different training objective for NMT, targeting at directly minimizing the difference between human translation and the translation given by an NMT model. To achieve this target, inspired by the recent success of Generative Adversarial Networks (GANs) [Goodfellow et al., 2014a], we design an adversarial training protocol for NMT and name it as Adversarial-NMT. In Adversarial-NMT, besides the typical NMT model, an adversary is introduced to distinguish the translation generated by NMT from that by human (i.e., ground truth). Meanwhile the NMT model tries to improve its translation results such that it can successfully cheat the adversary.

These two modules in Adversarial-NMT are co-trained, and their performances get mutually improved. In particular, the discriminative power of the adversary can be improved by learning from more and more training samples (both positive ones generated by human and negative ones sampled from NMT); and the ability of the NMT model in cheating the adversary can be improved by taking the output of the adversary as reward. In this way, the NMT translation results are professor forced [Lamb et al., 2016] to be as close as possible to ground-truth translation.

Different from previous GANs, which assume the existence of a generator in continuous space, in our proposed framework, the NMT model is in fact not a typical generative model, but instead a probabilistic transformation that maps a source language sentence to a target language sentence, both in discrete space. Such differences make it necessary to design both new network architectures and optimization methods to make adversarial training possible for NMT. We therefore on one aspect, leverage a specially designed Convolutional Neural Network (CNN) model as the adversary, which takes the (source, target) sentence pair as input; on the other aspect, we turn to a policy gradient method named REINFORCE [Williams, 1992], widely used in the reinforcement learning literature [Sutton and Barto, 1998], to guarantee both the two modules are effectively optimized in an adversarial manner. We conduct extensive experiments, which demonstrates that Adversarial-NMT can achieve significantly better translation results than traditional NMT models with even much larger vocabulary size and higher model complexity.

Related Work

End-to-end Neural Machine Translation (NMT) [Bahdanau et al., 2014, Cho et al., 2014, Sutskever et al., 2014, Jean et al., 2015, Wu et al., 2016, Zhou et al., 2016] has been the recent research focus of the community. Typical NMT system is built within the RNN based encoder-decoder framework. In such a framework the encoder RNN sequentially processes the words in a source language sentence into fixed length vectors, which act as the inputs to decoder RNN to decode the translation sentence. NMT typically adopts the principle of Maximum Likelihood Estimation (MLE) for training, i.e., maximizing the per-word likelihood of target sentence. Other training criteria, such as Minimum Risk Training (MRT) based on reinforcement learning [Ranzato et al., 2015, Shen et al., 2016] and translation reconstruction [Tu et al., 2016a], are shown to improve over such word-level MLE principle since these objectives take the translation sentence as a whole.

The training principle we propose is based on the spirit of Generative Adversarial Networks (GANs) [Goodfellow et al., 2014a, Salimans et al., 2016], or more generally, adversarial training [Goodfellow et al., 2014b]. In adversarial training, a discriminator and a generator compete with each other, forcing the generator to produce high quality outputs that are able to fool the discriminator. Adversarial training typically succeed in image generation [Goodfellow et al., 2014a, Reed et al., 2016], with limited contribution in natural language processing tasks [Yu et al., 2016, Li et al., 2017], mainly due to the difficulty of propagating the error signals from the discriminator to the generator through the discretely generated natural language tokens. ?) alleviates such a difficulty by reinforcement learning approach for sequence (e.g., music) generation. However, as far as we know, there are limited efforts on adversarial training for sequence-to-sequence task when a conditional mapping between two sequences is involved, and our work is among the first endeavors to explore the potential of acting in this way, especially for Neural Machine Translation [Yang et al., 2017].

Adversarial-NMT

The overall framework of Our Adversarial-NMT is shown in Figure 1. Let $(x=\{x_{1},x_{2},...,x_{T_{x}}\},y=\{y_{1},y_{2},...,y_{T_{y}}\})$ be a bilingual aligned sentence pair for training, where $x_{i}$ is the $i$-th word in the source sentence and $y_{j}$ is the $j$-th word in the target sentence. Let $y^{\prime}$ denote the translation sentence out from an NMT system for the source sentence $x$. As previously stated, the goal of Adversarial-NMT is to force $y^{\prime}$ to be as ‘similar’ as $y$. In the perfect case, $y^{\prime}$ is so similar to the human translation $y$ that even a human cannot tell whether $y^{\prime}$ is generated by machine or human. In order to achieve that, we introduce an extra adversary network, which acts similarly to the discriminator adopted in GANs [Goodfellow et al., 2014a]. The goal of the adversary is to differentiate human translation from machine translation, and the NMT model $G$ tries to produce a target sentence as similar as human translation so as to fool the adversary.

We adopt the Recurrent Neural Network (RNN) based encoder-decoder as the NMT model to seek a target language translation $y^{\prime}$ given source sentence $x$. In particular, a probabilistic mapping $G(y|x)$ is firstly learnt and the translation result $y^{\prime}\sim G(\cdot|x)$ is sampled from it. To be specific, given source sentence $x$ and previously generated words $y_{<t}$, the probability of generating word $y_{t}$ is:

where $r_{t}$ is the decoding state from decoder at time $t$. Here $g$ is the recurrent unit such as the Long Short Term Memory (LSTM) unit [Hochreiter and Schmidhuber, 1997] or Gated Recurrent Unit (GRU) [Cho et al., 2014], and $c_{t}$ is a distinct source representation at time $t$ calculated by an attention mechanism [Bahdanau et al., 2014]:

where $T_{x}$ is the source sentence length, $a(\cdot,\cdot)$ is a feed-forward neural network and $h_{i}$ is the hidden state from RNN encoder computed by $h_{i-1}$ and $x_{i}$:

The translation result $y^{\prime}$ can be sampled from $G(\cdot|x)$ either in a greedy way for each timestep, or using beam search [Sutskever et al., 2014] to seek globally optimized result.

2 Adversary Model

The adversary is used to differentiate translation result $y^{\prime}$ and the ground-truth translation $y$, given the source language sentence $x$. To achieve that, one needs to measure the translative matching degree of source-target sentence pair $(x,y)$. We turn to Convolution Neural Network (CNN) for this task [Yin et al., 2015, Hu et al., 2014], since with its layer-by-layer convolution and pooling strategies, CNN is able to accurately capture the hierarchical correspondence of $(x,y)$ at different abstraction levels.

The general structure is shown in Figure 2. Specifically, given a sentence pair $(x,y)$, we first construct a $2D$ image-like representation by simply concatenating the embedding vectors of words in $x$ and $y$. That is, for $i$-th word $x_{i}$ in $x$ and $j$-th word $y_{j}$ in sentence $y$, we have the following feature map:

Based on such a $2D$ image-like representation, we perform convolution on every $3\times 3$ window, with the purpose to capture the correspondence between segments in $x$ and segments in $y$ by the following feature map of type $f$:

where $\sigma(\cdot)$ is the sigmoid active function, $\sigma(x)=1/(1+\exp(-x))$.

After that we perform a max-pooling in non-overlapping $2\times 2$ windows:

We could go on for more layers of convolution and max-pooling, aiming at capturing the correspondence at different levels of abstraction. The extracted features are then fed into a multi-layer perceptron (MLP), with sigmoid activation at the last layer to give the probability that $(x,y)$ is from ground-truth data, i.e. $D(x,y)$. The optimization target of such CNN adversary is to minimize the cross entropy loss for binary classification, with ground-truth data $(x,y)$ as positive instance while sampled data (from $G$) $(x,y^{\prime})$ as negative one.

3 Policy Gradient Algorithm to Train Adversarial-NMT

With the notations for NMT model $G$ and adversary model $D$, the final training objective is:

That is, translation model $G$ tries to produce high quality translation $y^{\prime}$ to fool the adversary $D$ (the outer-loop $\min$), whose objective is to successfully classify translation results from real data (i.e., ground-truth) and from $G$ (the inner-loop $\max$).

Eqn. (6) reveals that it is straightforward to train the adversary $D$, by keeping providing $D$ with the ground-truth sentence pair $(x,y)$ and the sampled translation pair $(x,y^{\prime})$ from $G$, respectively as positive and negative training data. However, when it turns to NMT model $G$, it is non-trivial to design the training process, given that the discretely sampled $y^{\prime}$ from $G$ makes it difficult to directly back-propagate the error signals from $D$ to $G$, making $V(D,G)$ nondifferentiable w.r.t. $G$’s model parameters $\Theta_{G}$.

To tackle the above challenge, we leverage the REINFORCE algorithm [Williams, 1992], a Monte-Carlo policy gradient method in reinforcement learning literature to optimize $G$. Note that the objective of training $G$ under a fixed source language sentence $x$ and $D$ is to minimize the following loss item:

A sample $y^{\prime}$ from $G(\cdot|x)$ is used to approximate the above gradient:

in which $\nabla_{\Theta_{G}}\log G(y^{\prime}|x)$ are gradients specified with standard sequence-to-sequence NMT networks. Such a gradient approximation is used to update $\Theta_{G}$:

Using the language of reinforcement learning, in the above Eqn. (7) to (9), the NMT model $G(\cdot|x)$ is the conditional policy faced with $x$, while the term $-\log(1-D(x,y^{\prime}))$, provided by the adversary $D$, acts as a Monte-Carlo estimation of the reward. Intuitively speaking, Eqn. (9) implies, the more likely $y^{\prime}$ to successfully fool $D$ (i.e, larger $D(x,y^{\prime})$), the larger reward the NMT model will get, and the ’pseudo’ training data $(x,y^{\prime})$ will correspondingly be more favored to improve the policy $G(\cdot|x)$.

Note here we in fact use one sample $-\log(1-D(x,y^{\prime}))$ from a trajectory $y^{\prime}$ to estimate the terminal reward given by $D$. Acting in this way brings high variance, to reduce the variance, a moving average of the historical reward values is set as a reward baseline [Weaver and Tao, 2001]. One can sample multiple trajectories in each decoding step, by regarding $G$ as the roll-out policy to reduce estimation variance for immediate reward [Silver et al., 2016, Yu et al., 2016]. However, empirically we find such approach is intolerably time-consuming in our task, given that the decoding space in NMT is typically extremely large (the same as vocabulary size).

It is worth comparing our adversarial training with existing methods that directly maximize sequence level measure such as BLEU [Ranzato et al., 2015, Shen et al., 2016, Bahdanau et al., 2016] in training NMT models, using similar approaches based on reinforcement learning as ours. We argue that Adversarial-NMT makes the optimization easier compared with these methods. Firstly, the reward learned by our adversary $D$ provides rich and global information to evaluate the translation, which goes beyond the BLEU’s simple low-level n-gram matching criteria. Acting in this way provides much smoother objective compared with BLEU since the latter is highly sensitive for slight translation difference at word or phrase level. Secondly, the NMT model $G$ and the adversary $D$ in Adversarial-NMT co-evolves. The dynamics of adversary $D$ makes NMT model $G$ grows in an adaptive way rather than controlled by a fixed evaluation metric as BLEU. Given the above two reasons, Adversarial-NMT makes the optimization process towards sequence level objectives much more robust and better controlled, which is further verified by its superior performances to the aforementioned methods that will be reported in the next Section 4.

Experiments

We report the experimental results on both English$\rightarrow$French translation (En$\rightarrow$Fr for short) and German$\rightarrow$English translation (De$\rightarrow$En for short).

Dataset: For En$\rightarrow$Fr translation, for the sake of fair comparison with previous works, we use the same dataset as [Bahdanau et al., 2014, Shen et al., 2016]. The dataset is composed of a subset of WMT 2014 training corpus as training set, the combination of news-test 2012 and news-test 2013 as dev set and news-test 2014 as test set, which respectively contains roughly $12M$, $6k$ and $3k$ sentence pairs. The maximal sentence length is 50. We use top $30k$ most frequent English and French words and replace the other words as ‘UNK’ token.

For De$\rightarrow$En translation, following previous works [Ranzato et al., 2015, Bahdanau et al., 2016], the dataset is from IWSLT 2014 evaluation campaign [Cettolo et al., 2014], consisting of training/dev/test corpus with approximately $153k$, $7k$ and $6.5k$ bilingual sentence pairs respectively. The maximal sentence length is also set as 50. The dictionary for English and German corpus respectively include $22,822$ and $32,009$ most frequent words [Bahdanau et al., 2016], with other words replaced as a special token ‘UNK’.

Implementation Details: In Adversarial-NMT, the structure of the NMT model $G$ is the same as RNNSearch model [Bahdanau et al., 2014], a RNN based encoding-decoding framework with attention mechanism. Single layer GRUs act as encoder and decoder. For En$\rightarrow$Fr translation, the dimensions of word embedding and GRU hidden state are respectively set as $620$ and $1000$, and for De$\rightarrow$En translation they are both $256$.

For the adversary $D$, the CNN consists of two convolution$+$pooling layers, one MLP layer and one softmax layer, with $3\times 3$ convolution window size, $2\times 2$ pooling window size, $20$ feature map size and $20$ MLP hidden layer size.

For the training of NMT model $G$, similar as what is commonly done in previous works [Shen et al., 2016, Tu et al., 2016a], we warm start $G$ from a well-trained RNNSearch model, and optimize it using vanilla SGD with mini-batch size $80$ for En$\rightarrow$Fr translation and $32$ for De$\rightarrow$En translation. Gradient clipping is used with clipping value 1 for En$\rightarrow$Fr and 10 for De$\rightarrow$En. The initial learning rate is chosen from cross-validation on dev set ($0.02$ for En$\rightarrow$Fr and $0.001$ for De$\rightarrow$En) and we halve it every $80k$ iterations.

An important factor we find in successfully training $G$ is that the combination of adversarial objective with MLE. That is, we force $50\%$ randomly chosen mini-batch data are trained with Adversarial-NMT, while apply MLE principle to the other mini-batches. Acting in this way significantly improves stability in model training, which is also reported in other tasks such as language model [Lamb et al., 2016] and neural dialogue generation [Li et al., 2017]. We conjecture that the reason is that MLE acts as a regularizer to guarantee smooth model update, alleviating the negative effects brought by high gradient estimation variance of the one-step Monte-Carlo sample in REINFORCE.

As the first step, the CNN adversary network $D$ is initially pre-trained using the sampled data $(x,y^{\prime})$ sampled from the RNNSearch model, and the ground-truth translation $(x,y)$. After that, in joint G-D training of Adversarial-NMT, the adversary is optimized using Nesterov SGD [Nesterov, 1983] with batch size set as $32$. The initial learning rate is $0.002$ for En$\rightarrow$Fr and $0.001$ for De$\rightarrow$En, both chosen by validation on dev set. The dimension of word embedding is the same with that of $G$, and we fix the word embeddings during training. Batch normalization [Ioffe and Szegedy, 2015] is observed to significantly improve $D$’s performance. Considering efficiency, all the negative training data instances $(x,y^{\prime})$ used in $D$’s training are generated using beam search with beam size $4$.

In generating model translation for evaluation, we set beam width as $4$ and $12$ for En$\rightarrow$Fr and De$\rightarrow$En respectively according to BLEU on dev set. The translation quality is measured by tokenized case-sensitive BLEU [Papineni et al., 2002] score https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl.

2 Result on En→→\rightarrowFr translation

In Table 1 we provide the En$\rightarrow$Fr translation result of Adversarial-NMT, together with several strong NMT baselines, such as the well representative attention-based NMT model RNNSearch [Bahdanau et al., 2014]. In addition, to make our comparison comprehensive, we would like to cover several well acknowledged techniques whose effectiveness has been verified to improve En$\rightarrow$Fr translation by previously published works, including the leverage of 1) Using large vocabulary to handle rare words [Jean et al., 2015, Luong et al., 2015]; 2) Different training objectives [Shen et al., 2016, Ranzato et al., 2015, Bahdanau et al., 2016] such as Minimum Risk Training (MRT) to directly optimize evaluation measure [Shen et al., 2016], and dual learning to enhance both primal and dual tasks (e.g., En$\rightarrow$Fr and Fr$\rightarrow$En) [He et al., 2016]; 3) Improved inference process such as beam search optimization [Wiseman and Rush, 2016] and postprocessing UNK [Luong et al., 2015, Jean et al., 2015]; 4) Leveraging additional monolingual data [Sennrich et al., 2016, Zhang and Zong, 2016, He et al., 2016].

From the table, we can clearly observe that Adversarial-NMT obtains satisfactory translation quality against baseline systems. In particular, it even surpasses the performances of other models with much larger vocabularies [Jean et al., 2015], deeper layers [Luong et al., 2015], much larger monolingual training corpus [Sennrich et al., 2016], and the goal of directly maximizing BLEU [Shen et al., 2016]. In fact, as far as we know, Adversarial-NMT achieves state-of-the-art result ($34.78$) on En$\rightarrow$Fr translation for single-layer GRU sequence-to-sequence models trained with only supervised bilingual corpus on news-test 2014 test set.

Human Evaluation: Apart from the comparison based on the objective BLEU scores, to better appraise the performance of our model, we also involve human judgements as a subjective measure. To be more specific, we generate the translation results for 500 randomly selected English sentences from En$\rightarrow$Fr news-test 2014 dataset using both MRT [Shen et al., 2016] and our Adversarial-NMT. Here MRT is chosen since it is the well representative of previous NMT methods which maximize sequence level objectives, achieving satisfactory results among all single layer models (i.e., $31.3$ in Table 1). Afterwards we ask three human labelers to choose the better one from the two versions of translated sentences. The evaluation process is conducted on Amazon Mechanical Turk https://www.mturk.com with all the workers to be native English or French speakers.

Result in Table 2 shows that $59.4\%$ sentences are better translated by our Adversarial-NMT, compared with MRT [Shen et al., 2016]. Such human evaluation further demonstrates the effectiveness of our model and matches the expectation that Adversarial-NMT provides more human desired translation.

Adversarial Training: Slow or Fast: In this subsection we analyze how to set the pace for training the NMT model $G$ and adversary $D$, to make them combatting effectively. Specifically, for En$\rightarrow$Fr translation, we inspect how dev set BLEU varies along adversarial training process with different initial learning rates for $G$ (shown in 3(a)) and for $D$ (shown in 3(b)), conditioned on the other one fixed.

Overall speaking, these two figures show that Adversarial-NMT is much more robust with regard to the pace of $D$ making progress than that of $G$, since the three curves in 3(b) grow in a similar pattern while curves in 3(a) drastically differ with each other. We conjecture the reason is that in Adversarial-NMT, CNN based $D$ is powerful in classification tasks, especially when it is warm started with sampled data from RNNSearch. As a comparison, the translation model $G$ is relatively weak in providing qualified translations. Therefore, training $G$ needs carefully configurations of learning rate: small value (e.g., $0.002$) leads to slower convergence (blue line in 3(a)), while large value (e.g., $0.2$) brings un-stability (green line in 3(a)). The proper learning rate (e.g. $0.02$) induces $G$ to make fast, meanwhile stable progress along training.

3 Result on De→→\rightarrowEn translation

In Table 3 we provide the De$\rightarrow$En translation result of Adversarial-NMT, compared with some strong baselines such as RNNSearch [Bahdanau et al., 2014] and MRT [Shen et al., 2016].

Again, we can see that Adversarial-NMT performs best against other models from Table 3, achieves 27.94 BLEU scores, which is also a state-of-the-art result.

Effect of Adversarial Training: To better visualize and understand the advantages of adversarial training brought by Adversarial-NMT, we show several translation cases in Table 4. Concretely speaking, we give two German$\rightarrow$English translation examples, including the source language sentence $x$, the ground-truth translation sentence $y$, and two NMT model translation sentences, respectively out from RNNSearch and Adversarial-NMT (trained after $20$ epochs) and emphasized on their different parts by bold fonts which lead to different translation quality. For each model translation $y^{\prime}$, we also list $D(x,y^{\prime})$, i.e., the probability that the adversary $D$ regards $y^{\prime}$ as ground-truth, in the third column, and the sentence level bleu score of $y^{\prime}$ in the last column.

Since RNNSearch model acts as the warm start for training Adversarial-NMT, its translation could be viewed as the result of Adversarial-NMT at its initial phase. Therefore, from Table 4, we can observe:

With adversarial training goes on, the quality of translation sentence output by $G$ gets improved, both in terms of subjective feelings and BLEU scores as a quantitative measure.

Correspondingly, the translation quality growth makes the adversary $D$ deteriorated, as shown by $D$’s successful recognition of $y^{\prime}$ by RNNSearch as translated from model, whereas $D$ makes mistakes in classifying $y^{\prime}$ out from Adversarial-NMT as ground-truth (by human).

Conclusion

We in this paper propose a novel and intuitive training objective for NMT, that is to force the translation results be as similar as ground-truth translations generated by human. Such an objective is achieved via an adversarial training framework called Adversarial-NMT which complements the original NMT model with an adversary based on CNN. Adversarial-NMT adopts both new network architectures to reflect the mapping within (source, target) sentence, and an efficient policy gradient algorithm to tackle the optimization difficulty brought by the discrete nature of machine translation. The experiments on both English$\rightarrow$French and German$\rightarrow$English translation tasks clearly demonstrate the effectiveness of such adversarial training method for NMT.

As to future works, with the hope of achieving new state-of-the-art performance for NMT system, we plan to fully exploit the potential of Adversarial-NMT by combining it with other powerful methods listed in subsection 4.2, such as training with large vocabulary, minimum-risk principle and deep structures. We additionally would like to emphasize and explore the feasibility of adversarial training to other text processing tasks, such as image caption, dependency parsing and sentiment classification.

References