Style Transfer as Unsupervised Machine Translation

Introduction

Language style transfer is an important component of natural language generation (NLG) (?; ?; ?; ?), as it enables NLG systems to control not only the topic of produced utterance but also attributes such as sentiment and gender. As shown in Figure 1, language style transfer aims to convert a sentence with one attribute (e.g., negative sentiment) to another with a different attribute (e.g., positive sentiment), while retaining its attribute-independent content (e.g., the properties of the product being discussed).

Recently, many methods have made remarkable progress in language style transfer. One line of research (?; ?; ?) leverages the auto-encoder framework to learn an encoder and a decoder, in which the encoder constructs a latent vector by removing the style information and extracting attribute-independent content from the input sentence, and the decoder generates the output sentence with the desired style. Another line involves a delete-retrieve-generate approach (?; ?), in which attribute-related words are recognized and removed to generate a sentence containing only content information, which is used as a query to find a similar sentence with the target attribute from the corpus. Based on that, target attribute markers can be extracted and utilized to generate the final output sentence in a generation step.

Language style transfer can be regarded as a special machine translation (MT) task where the source sentence is in one style and the target sentence is in another style (as shown in Figure 1). In this paper, we leverage attention-based neural machine translation (NMT) models (?; ?; ?) to change the attribute of the input sentence by translating it from one style to another. Compared with auto-encoder methods, the attention mechanism can better make the decision on preserving the content words and transferring attribute-related words. Compared with the delete-retrieve-generate approach, our model is an end-to-end system without error propagation, and the generation of target attribute words is generated based on the context information instead of a retrieval step.

To train NMT-based systems, a large parallel corpus is required to tune the huge parameters and learn the correspondence between input and output words. However, for style transfer, sentence pairs with the same content but different attributes are difficult to acquire. Inspired by unsupervised MT approaches (?; ?; ?), we propose a two-stage joint training method to boost a forward transfer system (source style to target style) and a backward one (target style to source style) using unpaired datasets. In the first stage, we build the word-to-word transfer table based on word-level style-preference information and word embedding similarity learnt from unpaired datasets. With the inferred transfer tables and pre-trained style specific language models, bidirectional (forward and backward) statistical machine translation (SMT) (?; ?) transfer systems are built to generate a pseudo parallel corpus. In the second stage, we initialize bidirectional NMT-based transfer systems with the pseudo corpus from the first stage, which are then boosted with each other in an iterative back-translation framework. During iterative training, a style classifier is introduced to guarantee the high accuracy of style transfer result and punish the bad candidates in the generated pseudo data.

We conduct experiments on three style transfer tasks: altering sentiment of Yelp reviews, altering sentiment of Amazon reviews, and altering image captions between romantic and humorous. Both human and automatic evaluation results show that our proposed method outperforms previous state-of-the-art models in terms of both accuracy of style transfer and quality of input-output correspondence (meaning preservation and fluency). Our contributions can be summarized as follows:

Unsupervised MT methods are adapted to the style transfer tasks to tackle the lack of parallel corpus, with a three-step pipeline containing building word transfer table, constructing SMT-based transfer systems and training NMT-based transfer systems.

Our attention-based NMT models can directly model the whole style transfer process, and the attention mechanism can better make the decision of preserving the content words and transferring the attribute-related words.

A style classifier is introduced to control the noise generated during iterative back-translation training, and it is crucial to the success of our methods.

Our Approach

Given two datasets $X=\{x_{1},...,x_{n}\}$ and $Y=\{y_{1},...,y_{m}\}$ representing two different styles $s$ and $t$ respectively (e.g., for the sentiment, $s=``\text{negative}"$, $t=``\text{positive}"$), style transfer can be formalized as learning the conditional distribution $P_{s\rightarrow t}(y|x)$, which takes $(x,s)$ as inputs and generates a sentence $y$ retaining the content of $x$ while expressing in the style $t$. To model this conditional distribution, we adopt the attention-based architecture proposed by ? (?). It is implemented as an encoder-decoder framework with recurrent neural networks (RNN), in which RNN is usually implemented as Gated Recurrent Unit (GRU) (?) or Long Short-Term Memory (LSTM) networks (?). In our experiment, GRU is used as our RNN unit.

To learn style transfer using non-parallel text, we design an unsupervised sequence-to-sequence training method as illustrated in Figure 2. In general, our proposed approach can be divided into two stages: model initialization and iterative back-translation. In the first stage, given unaligned sentences $X=\{x_{1},...,x_{n}\}$ and $Y=\{y_{1},...,y_{m}\}$, we first build the transfer table to provide word-to-word transfer information, as well as two style specific language models. With the word-to-word transfer table and language models, we build two SMT-based transfer systems (source-to-target model and target-to-source model), with which we translate the unaligned sentences to construct the pseudo-parallel corpus. In the second stage, we use the pseudo data to pre-train bidirectional NMT-based transfer systems (source-to-target model $P_{s\rightarrow t}(y|x)$ and target-to-source model $P_{t\rightarrow s}(x|y)$). Based on the two initial systems, an iterative back-translation algorithm is employed to sufficiently exploit unaligned sentences $X$ and $Y$, with which bidirectional systems can achieve further improvements.

Learning to style transfer with only non-parallel data is a challenging task, since the associated style expressions cannot be learnt directly. To reduce the complexity of this task, we first learn the transfer knowledge at the word level, with which we can upgrade to the sentence level. To achieve this goal, we first construct word-level transfer table $P_{s\rightarrow t}(y_{w}|x_{w})$ in an unsupervised way. Many methods (?; ?) have been proposed to perform a similar task, but these methods rely on the homogeneity of the cross-lingual word embedding space and are only applied in the MT field. Since the two style transfer corpora are in one language, cross-lingual word embedding cannot be used to learn word-level transfer information. In order to gain proper word mapping between different attributes, we propose a new method which leverages the word embedding similarity and style preference of words to construct word-level transfer table.

The transfer probability $P_{s\rightarrow t}(y_{w}|x_{w})$ between source word $x_{w}$ in style $s$ and target word $y_{w}$ in style $t$ can be decomposed into three parts:

where $P(s|x_{w})$($P(t|y_{w})$) denotes the probability that a word $x_{w}$($y_{w}$) belongs to a style $s$($t$), $P(y_{w}|x_{w})$ represents grammatic similarity of $x_{w}$ and $y_{w}$. We observe that attribute-relevance words and their proper expressions in a target attribute typically play the same grammatic role in the sentences. In our implementation, $P(y_{w}|x_{w})$ is calculated with the normalized cosine similarity of word embedding (?), $P(s|x_{w})$ and $P(t|y_{w})$ are estimated as follows:

where $F(s,x_{w})$($F(t,y_{w})$) represents the frequency of a word $x_{w}$($y_{w}$) appearing in datasets with attribute $s$($t$).

Specifically, as shown in the model initialization part of Figure 2, we learn word embeddings of all the words using source style corpus $X$ and target style corpus $Y$, based on which, the grammatic similarity model $P(y_{w}|x_{w})$ can be learnt. Meanwhile, with style specific corpus $X$ and $Y$, we can gain the style preference models $P(s|x_{w})$ and $P(t|y_{w})$. By incorporating these three models, we can approximate the transfer probability $P_{s\rightarrow t}(y_{w}|x_{w})$, which is used to extract high-confidence word-level style mapping. For instance, both “hate” and “love” play similar grammatical roles in the sentence, so their embeddings are very similar, and cosine-based similarity is very high. Additionally, “hate” is more inclined to appear in the negative text, while “love” is more likely to occur in the positive text. So the two style preference probabilities are also high, which lead to a high translation probability $P_{s\rightarrow t}(``\text{love}"|``\text{hate}")$. The inverse translation table $P_{t\rightarrow s}(y_{w}|x_{w})$ can be generated in the same way.

To upgrade the transfer knowledge from word-level to sentence-level, we build bidirectional SMT translation systems with transfer tables and style specific language models. Our style specific language models are based on 4-gram language models and trained using the modified Kneser-Ney smoothing algorithm over the corresponding corpus. The features of SMT translation systems are designed as two word-level translation probabilities, two language model scores, and one word count penalty. For the source-to-target translation system, all the feature weights are 1, except the source style language model as -1, and similarly, the weight of target language model is set to -1 with all the remains as 1 for target-to-source translation system. With the SMT-based translation systems, we generate the translations of unaligned sentences $X$ and $Y$, and pair them to construct pseudo-parallel data.

Iterative Back-Translation

With the pseudo data generated in the first stage, we pre-train bidirectional NMT-based style transfer systems ($P^{0}_{s\rightarrow t}(y|x)$ and $P^{0}_{t\rightarrow s}(x|y)$). In this subsection, we will start with our unsupervised training objective, based on which an iterative back-translation method is designed to further improve initial NMT-based transfer models.

Given two unaligned datasets $X=\{x_{1},...,x_{n}\}$ and $Y=\{y_{1},...,y_{m}\}$ labeled with attributes $s$ and $t$ respectively, the common unsupervised training objective is to maximize the likelihood of observed data:

where $P(x_{i})$ and $P(y_{i})$ denote the language probabilities of sentences $x_{i}$ and $y_{i}$, $\theta_{s\rightarrow t}$ and $\theta_{t\rightarrow s}$ are model parameters of $P_{s\rightarrow t}(y|x)$ and $P_{t\rightarrow s}(x|y)$ respectively. Following ? (?)’s derivation, we can get the lower bound of the training objective in Equation 3 as:

This new training objective actually turns the unsupervised problem into a supervised one by generating pseudo sentence pairs via a back-translation method (?), in which the first term denotes that the pseudo sentence pairs generated by the source-to-target model $P_{s\rightarrow t}(y|x)$ are used to update the target-to-source model $P_{t\rightarrow s}(x|y)$, and the second term means use of the target-to-source model $P_{t\rightarrow s}(x|y)$ to generate pseudo data for the training of the source-to-target model $P_{s\rightarrow t}(y|x)$. In this way, two style transfer models ($P_{s\rightarrow t}(y|x)$ and $P_{t\rightarrow s}(x|y)$) can boost each other in an iterative process, as illustrated in the iterative back-translation part of Figure 2.

In practice, it is intractable to calculate Equation 4, since we need to sum over all candidates in an exponential search space for expectation computation. This problem is usually alleviated by sampling (?; ?). Following previous methods, the top-k translation candidates generated by beam search strategy are used for approximation.

In addition, with the weak supervision of the pseudo corpus, the learnt style transfer models are far from perfect, especially at the beginning of the iteration. The generated pseudo data may contains errors. Sometimes, the style of generated output is wrong, and such an error can be amplified in the iteration training. To tackle this issue, we introduce an external style classifier to provide a reward to punish poor pseudo sentence pairs. Specifically, the samples generated by $P_{s\rightarrow t}(y|x)$ or $P_{t\rightarrow s}(x|y)$ are expected to have high scores assigned by the style classifier. The objective of this reward-based training is to maximize the expected probability of pre-trained style classifier:

where $Q(t|y)$($Q(s|x)$) denotes the probability of style $t$($s$) given the generated sentence $y$($x$). This probability is assigned by a pre-trained style classifier and is subjected to $Q(t|.)=1-Q(s|.)$. For the style classifier, the input sentence is encoded into a vector by a bidirectional GRU with an average pooling layer over the hidden states, and a sigmoid output layer is used to predict the classification probability. The style classifier is trained by maximum likelihood estimation (MLE) using two datasets $X$ and $Y$.

Combining Equations 4 and 5, we get the final unsupervised training objective:

The partial derivative of $L(\theta_{s\rightarrow t},\theta_{t\rightarrow s})$ with respect to $\theta_{s\rightarrow t}$ and $\theta_{t\rightarrow s}$ can be written as follows:

where $\frac{\partial\log P_{s\rightarrow t}(y|x_{i})}{\partial\theta_{s\rightarrow t}}$ and $\frac{\partial\log P_{t\rightarrow s}(x|y_{i})}{\partial\theta_{t\rightarrow s}}$ are the gradients specified with a standard sequence-to-sequence network. Note that when maximizing the objective function $L_{1}(\theta_{s\rightarrow t},\theta_{t\rightarrow s})$, we do not back-prop through the reverse model which generates the data, following ? (?) and ? (?) . The whole iterative back-translation training is summarized in Algorithm 1.

Experiments

To examine the effectiveness of our proposed approach, we conduct experiments on three datasets, including altering sentiments of Yelp reviews, altering sentiments of Amazon reviews, and altering image captions between romantic and humorous. Following previous work (?; ?), we measure the accuracy of style transfer and the quality of content preservation with automatic and manual evaluations.

To compare our work with state-of-the-art approaches, we follow the experimental setups and datasetshttps://github.com/lijuncen/Sentiment-and-Style-Transfer in ? (?)’s work:

Yelp: This dataset consists of Yelp reviews. We consider reviews with a rating above three as positive samples and those below three as negative ones.

Amazon: This dataset consists of amounts of product reviews from Amazon (?). Similar to Yelp, we label the reviews with a rating higher than three as positive and less than three as negative.

Captions: This dataset consists of image captions (?). Each example is labeled as either romantic or humorous.

The statistics of the Yelp, Amazon and Captions datasets are shown in Table 1 and 2. Additionally, ? (?) hire crowd-workers on Amazon Mechanical Turk to write gold output for test sets of Yelp and Amazon datasets,The Captions dataset is actually an aligned corpus that contains captions for the same image in different styles, so we do not need to edit output for the test set of the Captions dataset. In our experiments, we also do not use these alignments. in which workers are required to edit a sentence to change its sentiment while preserving its content. With human reference outputs, an automatic evaluation metric, such as BLEU (?), can be used to evaluate how well meaning is preserved.

Baselines

We compare our approach with five state-of-the-art baselines: CrossAligned (?), MultiDecoder (?), StyleEmbedding (?), TemplateBased (?) and Del-Retr-Gen (Delete-Retrieve-Generate) (?). The former three methods are based on auto-encoder neural networks and leverage an adversarial framework to help systems separate style and content information. TemplateBased is a retrieve-based method that first identifies attribute-relevance words and then replaces them with target attribute expressions, which are extracted from a similar content sentence retrieved from the target style corpus. Del-Retr-Gen is a mixed model combining the TemplateBased method and an RNN-based generator, in which the RNN-based generator produces the final output sentence based on the content and the extracted target attributes.

Training Details

For the SMT model in our approach, we use Moseshttps://github.com/moses-smt/mosesdecoder with a translation table initialized as described in the Model Initialization Section. The language model is a default smoothed n-gram language model and the reordering model is disabled. The hyper-parameters of different SMT features are assigned as described in the Model Initialization Section.

RNNSearch (?) is adopted as the NMT model in our approach, which uses a single layer GRU for the encoder and decoder networks, enhanced with a feed-forward attention network. The dimension of word embedding (for both source and target words) and hidden layer are set to 300. All parameters are initialized using a normal distribution with a mean of 0 and a variance of $\sqrt{6/(d_{row}+d_{col})}$, where $d_{row}$ and $d_{col}$ are the number of rows and columns of the parameter matrix (?). Each model is optimized using the Adadelta (?) algorithm with a mini-batch size 32. All of the gradients are re-normalized if the norm exceeds 2. For the training iteration in Algorithm 1, best 4 samples generated by beam search strategy are used for training, and we run 3 epochs for Yelp and Amazon datasets, 30 epochs for the Captions dataset. At test time, beam search is employed to find the best candidate with a beam size 12.

Automatic Evaluation

In automatic evaluation, following previous work (?; ?), we measure the accuracy of style transfer for the generated sentences using a pre-trained style classifier,We train another style classifier for our iterative back-translation training process. and adopt a case-insensitive BLEU metric to evaluate the preservation of content. The BLEU score is computed using Moses multi-bleu.perl script. For each dataset, we train the style classifier on the same training data.

Table 3 shows the automatic evaluation results of different models on Yelp, Amazon and Captions datasets. We can see that CrossAligned obtains high style transfer accuracy but sacrifices content consistency. In addition, MultiDecoder and StyleEmedding can help the preservation of content but reduce the accuracy of style transfer. Compared with previous methods, TemplateBased and Del-Retr-Gen achieve a better balance between the transfer accuracy and the content preservation. Our approach achieves significant improvements over CrossAligned, MultiDecoder, StyleEmedding and Del-Retr-Gen in both transfer accuracy and quality of content preservation.

Compared with TemplateBased, our method achieves much better accuracy of style transfer, but with a lower BLEU score on the Captions dataset. The reason is that there are more different expressions to exhibit romantic and humorous compared with changing sentiment. A BLEU score based on a single human reference cannot precisely measure content consistency. In addition, as argued in ? (?), the BLEU metric, which is lack of automatic fluency evaluation, favors systems like TemplateBased, which only replaces a few words in the sentence. However, grammatical mistakes are easily made when replacing with inappropriate words. In order to reflect grammatical mistakes in the generated sentence, we conduct human evaluation with fluency as one of the criteria.

Human Evaluation

While automatic evaluation provides an indication of style transfer quality, it can not evaluate the quality of transferred text accurately. To further verify the effectiveness of our approach, we perform a human evaluation on the test set. For each dataset, we randomly select 200 samples for the human evaluation (100 for each attribute). Each sample contains the transformed sentences generated by different systems given the same source sentence. Then samples are distributed to annotators in a double-blind manner.We distribute each sample to 5 native speakers and use Fleiss’s kappa to judge agreement among them. The Fleiss’s kappa score is 0.791 for the Yelp dataset, 0.763 for the Amazon dataset and 0.721 for the Caption dataset. Annotators are asked to rate each output for three criteria on a likert scale from 1 to 5: style transfer accuracy (Att), preservation of content (Con) and fluency of sentences (Gra). Finally, the generated sentence is treated as “successful” when it is scored 4 or 5 on all three criteria.

Table 3 shows the human evaluation results. It can be clearly observed that our proposed method achieves the best performance among all systems, with 10%, 8% and 5% point improvements than Del-Retr-Gen on Yelp, Amazon and Captions respectively, which demonstrates the effectiveness of our proposed method. Compared with Del-Retr-Gen, our method is rated higher on all three criteria.

By comparing two evaluation results, we find that there is a positive correlation between human evaluation and automatic evaluation in terms of both accuracy of style transfer and preservation of content. This indicates the usefulness of automatic evaluation metrics in model development. However, since current automatic evaluation cannot evaluate the quality of transferred text accurately, the human evaluation is necessary and accurate automatic evaluation metrics are expected in the future.

Analysis

We further investigate the contribution of each component of our method during the training process. Table 5 shows automatic evaluation results of SMT-based and NMT-based transfer systems in our approach on the Yelp, Amazon and Captions datasets. We find that, with the help of word-level translations table and style specific language models, the SMT-based transfer model gains high accuracy in style transfer but fails in preserving content. Given pseudo data generated by the SMT-based model, the NMT-based model (Iteration 0) can better integrate the translation and language models, resulting in sentences with better content consistency. Using our iterative back-translation algorithm, the pre-trained NMT-based model can then be significantly improved in terms of both accuracy of style transfer and preservation of content. This result proves that the iterative back-translation algorithm can effectively leverage unaligned sentences. Besides, the style classifier plays a key role to guarantee the success of style transfer, without which, the transfer accuracy of the NMT-based model is obviously declining due to imperfect pseudo data. We also show some system outputs in Table 6.

Related Work

Language style transfer without a parallel text corpus has attracted more and more attention due to recent advances in text generation tasks. Many approaches have been proposed to build style transfer systems and achieve promising performance (?; ?; ?; ?; ?). ? (?) leverage an attribute classifier to guide the generator to produce sentences with desired attribute (e.g. sentiment, tense) in the Variational Auto-encoder (VAE) framework. ? (?) first supply a theoretical analysis of language style transfer using non-parallel text. They propose a cross-aligned auto-encoder with discriminator architecture, in which an adversarial discriminator is used to align different styles.

Instead of only considering the style transfer accuracy as in previous work, ? (?) introduce the content preservation as another evaluation metric and design two models, which encode the sentence into latent content representation and leverage the adversarial network to separate style and content information. Similarly, ? (?) attempt to use external NMT models to rephrase the sentence and weaken the effect of style attributes, based on which, multiple decoders can better preserve sentence meaning when transferring style. More recently, ? (?) design a delete-retrieve-generate system which hybrids the retrieval system and the neural-based text generation model. They first identify attribute-related words and remove them from the input sentence. Then the modified input sentence is used to retrieve a similar content sentence from the target style corpus, based on which corresponding target style expressions are extracted to produce the final output with an RNN-based generator.

Different from previous methods, we treat language style transfer as a special MT task where the source language is in one style and the target language in another style. Based on this, we adopt an attention-based sequence-to-sequence model to transform the style of a sentence. Further, following the key framework of unsupervised MT methods (?; ?; ?) to deal with the problem of lacking parallel corpus, a two-stage joint training method is proposed to leverage unpaired datasets with attribute information.

However, there are two major differences between our proposed approach and the existing unsupervised NMT method: 1) Building the word-to-word translation system in unsupervised NMT relies on the homogeneity of cross-lingual word embedding space, which is impossible for a style transfer whose input and output are in the same language. To deal with that, we propose a new method taking advantage of style-preference information and word embedding similarity to build the word-to-word transfer system; 2) We leverage the style classifier to filter the bad generated pseudo sentences, and its score is used as rewards to stabilize model training. Table 5 shows that introducing a style classifier can better guarantee the transferred style. In summary, an unsupervised NMT method cannot be directly applied in style transfer tasks, and we modify two important components to make it work.

Conclusion

In this paper, we have presented a two-stage joint training method to boost source-to-target and target-to-source style transfer systems using non-parallel text. In the first stage, we build bidirectional word-to-word style transfer systems in a SMT framework to generate pseudo sentence pairs, based on which, two initial NMT-based style transfer models are constructed. Then an iterative back-translation algorithm is employed to better leverage non-parallel text to jointly improve bidirectional NMT-based style transfer systems. Empirical evaluations are conducted on Yelp, Amazon and Captions datasets, demonstrating that our approach outperforms previous state-of-the-art models in terms of both accuracy of style transfer and quality of input-output correspondence.

In the future, we plan to further investigate the use of our method on other style transfer tasks. In addition, we are interested in designing more accurate and complete automatic evaluation for this task.

References