Training language GANs from Scratch

Introduction

Unsupervised word level text generation is a stepping stone for a plethora of applications, from dialogue generation to machine translation and summarization . While recent innovations such as architectural changes and leveraging big datasets are promising , the problem of unsupervised text generation is far from being solved.

Today, language models trained using maximum likelihood are the most successful and widespread approach to text modeling, but they are not without limitations. Since they explicitly model sequence probabilities, language models trained by maximum likelihood are often confined to an autoregressive structure, limiting applications such as one-shot language generation. Non-autoregressive maximum likelihood models have been proposed, but due to reduced model capacity they rely on distilling autoregressive models to achieve comparable performance on machine translation tasks .

When combined with maximum likelihood training, autoregressive modelling can result in poor samples due exposure bias – a distributional shift between training sequences used for learning and model data required for generation. Recently, showed that sampling from state of the art language models can lead to repetitive, degenerate output. Scheduled sampling has been proposed as a solution, but is thought to encourage sample quality by reducing sample diversity, inducing mode collapse .

Generative Adversarial Networks (GANs) are an alternative to models trained via maximum likelihood. GANs do not suffer from exposure bias since the model learns to sample during training: the learning objective is to generate samples which are indistinguishable from real data according to a discriminator. Since GANs don’t require an explicit probability model, they remove the restriction to autoregressive architectures, allowing one shot feed-forward generation .

The sequential and discrete nature of text has made the application of GANs to language challenging, with fundamental issues such as difficult gradient estimation and mode collapse yet to be addressed. Existing language GANs avoid these issues by pre-training models with maximum likelihood and limiting the amount of adversarial fine tuning by restricting the number of fine-tuning epochs and often using a small learning rate . This suggests “that the best-performing GANs tend to stay close to the solution given by maximum-likelihood training” . Even with adversarial fine-tuning playing a limited role, extensive evaluation has shown that existing language GANs do not improve over maximum likelihood-trained models .

We show that pure adversarial training is a viable approach for unsupervised word-level text generation by training a language GAN from scratch. We achieve this by tackling the fundamental limitations of training discrete GANs through a combination of existing techniques as well as carefully choosing the model and training regime. To the best of our knowledge we are the first to do so successfully; we thus call our model ScratchGAN. Compared to prior work on discrete language GANs which “barely achieve non-random results without supervised pre-training” , ScratchGAN achieves results comparable with maximum likelihood models.

Our aim is to learn models that captures both both semantic coherence and grammatical correctness of language, and to demonstrate that these properties have been captured with the use of different evaluation metrics. BLEU and Self-BLEU capture basic local consistency. The Fréchet Distance metric captures global consistency and semantic information, while being less sensitive to local syntax. We use Language and Reverse Language model scores across various softmax temperatures to capture the diversity-quality trade-off. We measure validation data perplexity, using the fact that ScratchGAN learns an explicit distribution over sentences. Nearest neighbor analysis in embedding and data space provide evidence that our model is not trivially overfitting, e.g. by copying sections of training text.

We show that GANs without any pre-training are comparable with maximum likelihood methods at unconditional text generation.

We show that large batch sizes, dense rewards and discriminator regularization are key ingredients of training language GANs from scratch.

We perform an extensive evaluation of the quality and diversity of our model. In doing so, we show that no current evaluation metric is able to capture all the desired properties of language.

The ScratchGAN code can be found at https://github.com/deepmind/deepmind-research/scratchgan.

Generative Models of Text

The generative model practitioner has two choices to make: how to model the unknown data distribution $p^{\ast}\!(\mathbf{x})$ and how to learn the parameters $\boldsymbol{\theta}$ of the model. The choice of model is where often prior information about the data is encoded, either through the factorization of the distribution, or through its parametrization. The language sequence $\mathbf{x}=[x_{1},...,x_{T}]$ naturally lends itself to autoregressive modeling:

Sampling $\hat{x}_{1},...,\hat{x}_{T}$ from an autoregressive model is an iterative process: each token $\hat{x}_{t}$ is sampled from the conditional distribution imposed by previous samples: $\hat{x}_{t}\sim p_{\boldsymbol{\theta}}(x_{t}|\hat{x}_{1},...,\hat{x}_{t-1})$. Distributions $p_{\boldsymbol{\theta}}(x_{t}|x_{1},...,x_{t-1})$ are Categorical distributions over the vocabulary size, and are often parametrized as recurrent neural networks .

The specific tokenization $x_{1},...,x_{T}$ for a given data sequence is left to the practitioner, with character level or word level splits being the most common. Throughout this work, we use word level language modeling.

Once a choice of model is made, the question of how to train the model arises. The most common approach to learn model of language is using maximum likelihood estimation (MLE):

The combination of autoregressive models and maximum likelihood learning has been very fruitful in language modeling , but it is unclear whether maximum likelihood is the optimal perceptual objective for text data . In this work we will retain the use of autoregressive models and focus on the impact of the training criterion on the quality and sample diversity of generated data, by using adversarial training instead.

2 Generative Adversarial Networks

Generative adversarial networks learn the data distribution $p^{\ast}\!(\mathbf{x})$ through a two player adversarial game between a discriminator and a generator. A discriminator $\mathcal{D}_{\phi}(\mathbf{x})$ is trained to distinguish between real data and samples from the generator distribution $p_{\boldsymbol{\theta}}(\mathbf{x})$, while the generator is trained to fool the discriminator in identifying its samples as real. The original formulation proposes a min-max optimization procedure using the objective:

Challenges of learning language GANs arise from the combination of the adversarial learning principle with the choice of an autoregressive model. Learning $p_{\boldsymbol{\theta}}(\mathbf{x})=\prod_{t=1}^{T}p_{\boldsymbol{\theta}}(x_{t}|x_{1},...,x_{t-1})$ using equation 3 requires backpropagating through a sampling operation, forcing the language GAN practitioner to choose between high variance, unbiased estimators such as REINFORCE , or lower variance, but biased estimators, such as the Gumbel-Softmax trick and other continuous relaxations . Gradient estimation issues compounded with other GAN problems such as mode collapse or training instability led prior work on language GANs to use maximum likelihood pre-training . This is the current preferred approach to train text GANs.

3 Learning Signals

To train the generator we use the REINFORCE gradient estimator :

where $R(\mathbf{x})$ is provided by the discriminator. By analogy with reinforcement learning, we call $R(\mathbf{x})$ a reward. Setting $R(\mathbf{x})=\frac{p^{*}(\mathbf{x})}{p_{\theta}(\mathbf{x})}$, recovers the MLE estimator in Eq (2) as shown by Che et al. :

The gradient updates provided by the MLE estimator can be seen as a special case of the REINFORCE updates used in language GAN training. The important difference lies in the fact that for language GANs rewards are learned. Learned discriminators have been shown to be a useful measure of model quality and correlate with human evaluation . We postulate that learned rewards provide a smoother signal to the generator than the classical MLE loss: the discriminator can learn to generalize and provide a meaningful signal over parts of the distribution not covered by the training data. As the training progresses and the signal from the discriminator improves, the generator also explores other parts of data space, providing a natural curriculum, whereas MLE models are only exposed to the dataset.

Adversarial training also enables the use of domain knowledge. Discriminator ensembles where each discriminator is biased to focus on specific aspects of the samples such as syntax, grammar, semantics, or local versus global structure are a promising approach . The research avenues opened by learned rewards and the issues with MLE pre-training motivate our search for a language GAN which does not make use of maximum likelihood pre-training.

Training Language GANs from Scratch

To achieve the goal of training a language GAN from scratch, we tried different loss functions and architectures, various reward structures and regularization methods, ensembles, and other modifications. Most of these approaches did not succeed or did not result in any significant gains. Via this extensive experimentation we found that the key ingredients to train language GANs from scratch are: a recurrent discriminator used to provide dense rewards at each time step, large batches for variance reduction, and discriminator regularization. We describe the generator architecture and reward structure we found effective in Figure 1 and provide a list of other techniques we tried but which proved unsuccessful or unnecessary in Appendix C.

Our ultimate goal is to generate entire sequences, so we could train a discriminator to distinguish between complete data sequences and complete sampled sequences, with the generator receiving a reward only after generating a full sequence. However, in this setting the generator would get no learning signal early in training, when generated sentences can easily be determined to be fake by the discriminator. We avoid this issue by instead training a recurrent discriminator which provides rewards for each generated token . The discriminator $\mathcal{D}_{\boldsymbol{\phi}}$ learns to distinguish between sentence prefixes coming from real data and sampled sentence prefixes:

While a sequential discriminator is potentially harder to learn than sentence based feed-forward discriminators, it is computationally cheaper than approaches that use Monte Carlo Tree Search to score partial sentences and has been shown to perform better empirically .

For a generated token $\hat{x}_{t}\sim p_{\boldsymbol{\theta}}(x_{t}|x_{t-1}...x_{1})$, the reward provided to the ScratchGAN generator at time step $t$ is:

Rewards scale linearly with the probability the discriminator assigns to the current prefix pertaining to a real sentence. Bounded rewards help stabilize training.

The goal of the generator at timestep $t$ is to maximize the sum of discounted future rewards using a discount factor $\gamma$:

Like ScratchGAN, SeqGAN-step uses a recurrent discriminator to provide rewards per time step to a generator trained using policy gradient for unsupervised word level text generation. Unlike SeqGAN-step, our model is trained from scratch using only the adversarial objective, without any maximum likelihood pretraining.

2 Large Batch Sizes for Variance Reduction

The ScratchGAN generator parameters $\boldsymbol{\theta}$ are updated using Monte Carlo estimates of policy gradients (Equation 4), where $N$ is the batch size:

A key component of ScratchGAN is the use of large batch sizes to reduce the variance of the gradient estimation, exploiting the ability to cheaply generate experience by sampling from the generator. To further reduce the gradient variance ScratchGAN uses a global moving-average of rewards as a baseline $b_{t}$ , as we empirically found it improves performance for certain datasets.

Providing rewards only for the sampled token as in Equation (3.2) results in a substantial training speed boost compared to methods that use $p_{\boldsymbol{\theta}}(x_{t}^{n}|\hat{x}_{t-1}^{n}...\hat{x}_{1}^{n})$ to provide rewards for each token in the vocabulary, in order to reduce variance and provide a richer learning signal. These methods score all prefixes at time $t$ and thus scale linearly with vocabulary size .

3 Architectures and Discriminator Regularization

The ScratchGAN discriminator and generator use an embedding layer followed by one or more LSTM layers . For the embedding layer, we have experimented with training the embeddings from scratch, as well as using pre-trained GloVe embeddings concatenated with learned embeddings. When GloVe embeddings are used, they are shared by the discriminator and the generator, and kept fixed during training.

Discriminator regularization in the form of layer normalization , dropout and $L_{2}$ weight decay provide a substantial performance boost to ScratchGAN. Our findings align with prior work which showed the importance of discriminator regularization on image GANs .

Despite using a recurrent discriminator, we also provide the discriminator with positional information by concatenating a fix sinusoidal signal to the word embeddings used in the discriminator . We found this necessary to ensure the sentence length distribution obtained from generator samples matches that of the training data. Ablation experiments are provided in Appendix G.

Evaluation Metrics

Evaluating text generation remains challenging, since no single metric is able to capture all desired properties: local and global consistency, diversity and quality, as well as generalization beyond the training set. We follow Semeniuta et al. and Caccia et al. in the choice of metrics. We use $n$-gram based metrics to capture local consistency, Fréchet Distance to measure distances to real data in embedding space, and language model scores to measure the quality-diversity trade-off. To show our model is not trivially overfitting we look at nearest neighbors in data and embedding space.

BLEU and Self-BLEU have been proposed as measures of quality and diversity, respectively. BLEU based metrics capture local consistency and detect relatively simple problems with syntax but do not capture semantic variation .

We highlight the limitations of BLEU metrics by training a $5$-gram model with Kneser-Ney smoothing on EMNLP2017-News and measuring its BLEU score. The results are reported in Table 1. The $5$-gram model scores close to perfect according to BLEU-5 metric although its samples are qualitatively very poor (see Table 10 in the Appendix). In the rest of the paper we report BLEU-5 and Self-BLEU-5 metrics to compare with prior work, and complement it with metrics that capture global consistency, like Fréchet Distance.

2 Fréchet Embedding Distance

Semeniuta et al. proposed the Fréchet InferSent Distance (FID), inspired by the Fréchet Inception Distance used for images . The metric computes the Fréchet distance between two Gaussian distributions fitted to data embeddings, and model sample embeddings, respectively. Semeniuta et al. showed that the metric is not sensitive to the choice of embedding model and use InferSent for model evaluation, while we use a Universal Sentence Encoder The model can be found at https://tfhub.dev/google/universal-sentence-encoder/2. We call the metric Fréchet Embedding Distance to clarify that we use a different embedding model from Semeniuta et al. .

The Fréchet Embedding Distance (FED) offers several advantages over BLEU-based metrics, as highlighted in Semeniuta et al. : it captures both quality and diversity; it captures global consistency; it is faster and simpler to compute than BLEU metrics; it correlates with human evaluation; it is less sensitive to word order than BLEU metrics; it is empirically proven useful for images.

We find that the Fréchet Embedding Distance provides a useful metric to optimize for during model development, and we use it to choose the best models. However, we notice that FED also has drawbacks: it can be sensitive to sentence length, and we avoid this bias by ensuring that all compared models match the sentence length distribution of the data (see details in Appendix E).

3 Language Model Scores

Caccia et al. proposed evaluating the quality of generated model samples using a language model (Language Model score, LM), as well as training a language model on the generated samples and scoring the original data with it (Reverse Language Model score, RLM). LM measures sample quality: bad samples score poorly under a language model trained on real data. RLM measures sample diversity: real data scores poorly under a language model trained on samples which lack diversity. While insightful, this evaluation criteria relies on training new models, and hence the results can depend on the evaluator architecture. The metric could also have an inherent bias favoring language models, since they were trained using the same criteria.

Experimental Results

We use two datasets, EMNLP2017 Newshttp://www.statmt.org/wmt17/ and Wikitext-103 . We use EMNLP2017 News to compare with prior work but note that this dataset has limitations: a small vocabulary (5.7k words), no out-of-vocabulary tokens, a sentence length limited to 50 tokens, and a size of only 300k sentences. Wikitext-103 is a large scale dataset of almost 4 million sentences that captures more of the statistical properties of natural language and is a standard benchmark in language modeling . For Wikitext-103 we use a vocabulary of 20k words. In Wikitext-103 we remove sentences with less than 7 tokens or more than 100 tokens. All our models are trained on individual sentences, using an NVIDIA P100 GPU.

In all our experiments, the baseline maximum likelihood trained language model is a dropout regularized LSTM. Model architectures, hyperparameters, regularization and experimental procedures for the results below are detailed in Appendix D. Samples from ScratchGAN can be seen in Appendix H, alongside data and MLE samples.

As suggested in Caccia et al. , we measure the diversity-quality trade-off of different models by changing the softmax temperature at sampling time. Reducing the softmax temperature below 1 results in higher quality but less diverse samples, while increasing it results in samples closer and closer to random. Reducing the temperature for a language GANs is similar to the “truncation trick” used in image GANs . We compute all metrics at different temperatures.

ScratchGAN shows improved local consistency compared to existing language GANs and significantly reduces the gap between language GANs and the maximum likelihood language models. Figure 2(a) reports negative BLEU5 versus Self-BLEU5 metrics on EMNLP2017 News for ScratchGAN and other language GANs, as reported in Caccia et al. .

ScratchGAN improves over an MLE trained model on WikiText-103 according to FED, as shown in Figure 3(a). This suggests that ScratchGAN is more globally consistent and better captures semantic information. Figure 3(b) shows the quality diversity trade-off as measured by FED as the softmax temperature changes. ScratchGAN performs slightly better than the MLE model on this metric. This contrasts with the Language Model Score-Reverse Language Model scores shown in Figure 2(b), which suggests that MLE samples are more diverse. Similar results on EMNLP2017 News are shown in Appendix A.

Unlike image GANs, ScratchGAN learns an explicit model of data, namely an autoregressive explicit model of language. This allows us to compute model perplexities on validation data by feeding the model ground truth at each step. We report ScratchGAN and MLE perplexities on EMNLP2017 News in Table 2. Evaluating perplexity favors the MLE model, which is trained to minimize perplexity and thus has an incentive to spread mass around the data distribution to avoid being penalized for not explaining training instances , unlike ScratchGAN which is penalized by the discriminator when deviating from the data manifold and thus favors quality over diversity. Improving sample diversity, together with avoiding underfitting by improving grammatical and local consistency are required in order to further decrease the perplexity of ScratchGAN to match that of MLE models.

Our diversity and quality evaluation across multiple metrics shows that compared to the MLE model, ScratchGAN trades off local consistency to achieve slightly better global consistency.

2 Nearest Neighbors

A common criticism of GAN models is that they produce realistic samples by overfitting to the training set, e.g. by copying text snippets. For a selection of ScratchGAN samples we find and present the nearest neighbors present in the training set. We consider two similarity measures, a 3-gram cosine similarity — to capture copied word sequences, and a cosine similarity from embeddings produced by the Universal Sentence Encoder —to capture semantically similar sentences. In Table 5 in Appendix B we display a selection of four random samples and the corresponding top three closest training set sentences with respect to each similarity measure, and see the training text snippets have a mild thematic correspondence but have distinct phrasing and meaning. Additionally we perform a quantitive analysis over the full set of samples; we also compare the longest matching $n$-grams between text from the training set and (a) ScratchGAN samples, (b) MLE samples, and (c) text from the validation set. In Figure 4 we see fewer ScratchGAN samples with long matching n-grams ($n\geq 5$) in comparison with MLE samples and text from the validation set. We conclude the generator is producing genuinely novel sentences, although they are not always grammatically or thematically consistent.

3 Ablation Study and SeqGAN-step comparison

We show the relative importance of individual features of ScratchGAN with an ablation study in Figure 3(c). We successively add all elements that appear important to ScratchGAN performance, namely large batch size, discriminator regularization ($L_{2}$ weight decay, dropout, and layer normalization), pre-trained embeddings, and a value baseline for REINFORCE. The increase in batch size results in the most significant performance boost, due to the reduction in gradient variance and stabilizing effect on adversarial dynamics. Discriminator regularization also leads to substantial performance gains, as it ensures the discriminator is not memorizing the training data and thus is providing a smoother learning signal for the generator.

The baseline model in Figure 3(c) is a SeqGAN-step like model without pretraining. To highlight the improvement of ScratchGAN compared to prior work, we show in Table 4 the FED difference between the two models.

4 Training Stability

Despite the high variance of REINFORCE gradients and the often unstable GAN training dynamics, our training procedure is very stable, due to the use of large batch sizes and chosen reward structure. Table 4 reports the FED scores for ScratchGAN models trained with hyperparameters from a large volume in hyper-parameter space as well as across 50 random seeds. The low variance across hyperparameters shows that ScratchGAN is not sensitive to changes in learning rate, REINFORCE discount factor, regularization or LSTM feature sizes, as long as these are kept in a reasonable range. The full hyperparameter sweep performed to obtain the variance estimates is described in Appendix F. When we fixed hyperparameters and repeated an experiment across 50 seeds, we obtained very similar FED score; no divergence or mode collapse occurred in any of the 50 runs. For WikiText-103, the results are similar (0.055 $\pm$ 0.003).

Related Work

Our work expands on the prior work of discrete language GANs, which opened up the avenues to this line of research. Methods which use discrete data have proven to be more successful than methods using continuous relaxations , but face their own challenges, such as finding the right reward structure and reducing gradient variance. Previously proposed solutions include: receiving dense rewards via Monte Carlo Search or a recurrent discriminator , leaking information from the discriminator to the generator , using actor critic methods to reduce variance , using ranking or moment matching to provide a richer learning signal and curriculum learning . Despite alleviating problems somewhat, all of the above methods require pre-training, sometimes together with teacher forcing or interleaved supervised and adversarial training .

Nie et al. recently showed that language GANs can benefit from complex architectures such as Relation Networks . Their RelGAN model can achieve better than random results without supervised pre-training, but still requires pre-training to achieve results comparable to MLE models.

Press et al. is perhaps the closest to our work: they train a character level GAN without pre-training. Unlike Press et al. , ScratchGAN is a word level model and does not require teacher helping, curriculum learning or continuous relaxations during training. Importantly, we have performed an extensive evaluation to quantify the performance of ScratchGAN, as well as measured overfitting using multiple metrics, beyond $4$-gram matching.

By learning reward signals through the use of discriminators, our work is in line with recent imitation learning work , as well as training non-differentiable generators .

Discussion

Existing language GANs use maximum likelihood pretraining to minimize adversarial training challenges, such as unstable training dynamics and high variance gradient estimation. However, they have shown little to no performance improvements over traditional language models, likely due to constraining the set of possible solutions to be close to those found by maximum likelihood. We have shown that large batch sizes, dense rewards and discriminator regularization remove the need for maximum likelihood pre-training in language GANs. To the best of our knowledge, we are the first to use Generative Adversarial Networks to train word-level language models successfully from scratch. Removing the need for maximum likelihood pretraining in language GANs opens up a new avenue of language modeling research, with future work exploring GANs with one-shot feed-forward generators and specialized discriminators which distinguish different features of language, such as semantics and syntax, local and global structure. Borrowing from the success of GANs for image generation , another promising avenue is to use powerful neural network architectures to improve ScratchGAN.

We have measured the quality and diversity of ScratchGAN samples using BLEU metrics, Frèchet distance, and language model scores. None of these metrics is sufficient to evaluate language generation: we have shown that BLEU metrics only capture local consistency; language model scores do not capture semantic similarity; and that while embedding based Frèchet distance is a promising global consistency metric it is sensitive to sentence length. Until new ways to assess language generation are developed, current metrics need to be used together to compare models.

Acknowledgments

We would like to thank Chris Dyer, Oriol Vinyals, Karen Simonyan, Ali Eslami, David Warde-Farley, Siddhant Jayakumar and William Fedus for thoughtful discussions.

References

Appendix A Fréchet Embedding Distance and Language model scores on EMNLP2017 News

On EMNLP2017 News, FED and LM/RLM results are similar to those on WikiText103, see Figure 5(a) and Figure 5(b). Here we report the FED against both the training and validation set, to assess model overfitting. On this metric, we again notice that ScratchGAN performs better than the MLE model.

Appendix B Nearest Neighbors

In Table 5 we see for a selection of four random samples and the corresponding top three closest training set sentences with respect to each similarity measure, there is not a clear pattern of overfitting or training set repetition.

Appendix C Negative results

Here we list some approaches that we tried but which proved unsuccessful or unnecessary:

Using a Wasserstein Loss on generator logits, with a straight-through gradient. This was unsuccessful.

Using ensembles of discriminators and generators. The results are on par with those obtained by a single discriminator-generator pair.

Training against past versions of generators/discriminators. Same as above.

Using bi-directional discriminators. They can work but tend to over-fit and provide less useful feedback to the generator.

Using several discriminators with different architectures, hoping to have the simple discriminators capture simple failure modes of the generators such as repeated words. It did not improve over single discriminator-generator pair.

Training on small datasets such as Penn Tree Bank. The discriminator quickly over-fit to the training data. This issue could probably be solved with stronger regularization but we favoured larger datasets.

Using a Hinge loss on the discriminator. This did not improve over the cross-entropy loss.

Using a hand-designed curriculum, where the generator is first trained against a simple discriminator, and later in training a more complex discriminator is substituted. This was unsuccessful. We suspect that adversarial training requires a difficult balance between discriminator quality and generator quality, which is difficult to reach when either component has been trained independently from the other.

Varying significantly the number of discriminator steps per generator step, say 5 discriminator steps per generator step. This was unsuccessful.

Looking at discriminator probabilities (check that $P(real)\approx 1$ and $P(fake)\approx 0$) to evaluate training. Discriminator seems to be able to provide good gradient signal even when its predictions are not close to the targets, as long as its predictions on real data are distinct from its prediction on fake data.

Using a population of discriminators to evaluate the quality of a generator, or conversely. This metric failed when the population as a whole is not making progress.

Mapping all data to GloVe embeddings, and training a one-shot feed-forward generator to generate word embeddings directly, while discriminator receives word embeddings directly. This was unsucessful.

Appendix D Experimental details

We now provide the experimental details of our work.

Generator The core of the generator is an LSTM with tanh activation function and skip connections. We use an embedding matrix which is the concatenation of a fixed pretrained GloVe embedding matrix of dimension $V\times 300$ where $V$ is the vocabulary size, and a learned embedding matrix of dimension $V\times M$ where $M$ depends on the dataset. An embedding for the token at the previous time-step is looked up in the embedding matrix, and then linearly projected using a learned matrix to the feature size of the LSTM. This is the input to the LSTM. The output of the LSTM is the concatenation of the hidden outputs of all layers. This output is linearly projected using a learned matrix to the dimension of the embedding matrix. We add a learned bias of dimension $V$ to obtain the logits over the vocabulary. We apply a softmax operation to the logits to obtain a Categorical distribution and sample from it to generate the token for the current time-step.

Discriminator The input to the discriminator is a sequence of tokens, coming either from the real data or the generator. The core of the discriminator is an LSTM. The discriminator uses its own embedding matrix, independent from the generator. It has the same structure as the generator embedding matrix. Dropout is applied to this embedding matrix. An embedding for the token at the current time-step $t$ is looked up in the embedding matrix. A fixed position embedding of dimension $8$, depending on $t$ (see G), is concatenated to the embedding. As for the generator, the embedding is linearly projected using a learned matrix to the feature size of the LSTM. This is the input to the LSTM. The output of the LSTM is itself linearly projected to dimension $1$. This scalar is passed through a sigmoid to obtain the discriminator probability $\mathcal{D}_{\boldsymbol{\phi}}(\mathbf{x}_{t})$. The discriminator LSTM is regularized with layer normalization. $L_{2}$ regularization is applied to all learned variables in the discriminator.

Losses The discriminator is trained with the usual cross-entropy loss. The generator is trained with a REINFORCE loss. The value baseline at training step $i$, denoted $b_{i}$, is computed as:

where $\bar{R}_{i}$ is the mean cumulative reward over all sequence timesteps and over the current batch at training step $i$. The generator loss at timestep $t$ and training step $i$ is then:

and the total generator loss to minimize at training step $i$ is $\sum_{t}L^{G}_{ti}$.

Optimization Both generators and discriminators are trained with Adam , with $\beta_{1}=0.5$ for both. We perform one discriminator step per generator step.

Data considerations The maximum sequence length for EMNLP2017 News is 50 timesteps. The generator vocabulary also contains a special end of sequence token. If the generator outputs the end of sequence token at any timestep the rest of the sequence is padded with spaces. At timestep the input to the generator LSTM is a space character. Generator and discriminator are both recurrent so time and space complexity of inference and training are linear in the sequence length.

D.2 Sweeps and best hyperparameters

To choose our best model, we sweep over the following hyperparameters:

The number of discriminator updates per generator update.

The LSTM feature size of the discriminator and generator.

The best hyperparameters for EMNLP2017 News are:

Discriminator learning rate: $9.38\,10^{-3}$.

The LSTM feature size of the discriminator and generator: 512 and 512.

The number of layers for the generator: 2.

Dropout rate for the discriminator embeddings: 0.1

Update frequency of baseline, $\lambda$: 0.08.

The best hyperparameters for WikiText-103 News:

Discriminator learning rate: $2.98\,10^{-3}$

The LSTM feature size of the discriminator and generator: 256 and 256.

The number of layers for the discriminator: 1.

Dropout rate for the discriminator embeddings: 0.4.

Update frequency of baseline, $\lambda$: 0.23.

D.3 Training procedure

For both datasets, we train for at least $100000$ generator training steps, saving the model every $1000$ steps, and we select the model with the best FED against the validation data. Each training run used approximately 4 Intel Skylake x86-64 CPUs at 2 GHz, 1 Nvidia Tesla V100 GPU, and 20 GB of RAM, for 1 to 5 days depending on the dataset.

D.4 Language models

The language models we compare to are LSTMs. Interestingly, we found that smaller architectures are necessary for the LM compared to the GAN model, in order to avoid overfitting. For the maximum likelihood language models, we sweep over the size of the embedding layer, the feature size of the LSTM, and the dropout rate used for the embedding layer. We choose the model with the smallest validation perplexity.

For EMNLP2017 News, the MLE model used a LSTM feature size of 512, embedding size of 512, and embedding dropout rate of 0.2.

For WikiText-103, the MLE model used a LMST feature size of 3000, embedding size of 512, and embedding dropout rate of 0.3.

D.5 Metrics

FED and BLEU/Self-BLEU metrics on EMNLP2017 News are always computed with $10000$ samples. On WikiText-103 FED is computed with $7869$ samples because this is the number of sentences in WikiText-103 validation data, after filtering outliers.

To compute the reverse language model scores at different softmax temperatures we used the same architecture as the best EMNLP2017 News. We trained a language model on 268590 model samples, and used it to score the validation data.

D.6 Datasets

Wikitext-103 is available at https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip. EMNLP2017News is available at http://www.statmt.org/wmt17/ and a preprocessed version at https://github.com/pclucas14/GansFallingShort/blob/master/real_data_experiments/data/news/.

Appendix E Fréchet Embedding Distance sensitivity to sentence length

We show that FED is slightly dependent on sentence length, highlighting a possible limitation of this metric. For each sentence length, we randomly select a subset of 10k sentences from EMNLP2017 News training set conditioned on this sentence length, and we measure the FED between this subset and the 10k validation set. We show the results in figure 6(a). We see that there is a small dependence of FED on sentence length: FED seems to be worse for sentences that are significantly shorter or longer than the mean.

Appendix F Hyperparameter variance

Here we clarify the definition of the subset of hyper-parameter space that we used to show the stability of our training procedure. All runs with hyper-parameters in the ranges defined below gave good results in our experiments as shown in Table 4 in the main text.

baseline decay ($\lambda$ in equation 8 in appendix D) is in $$.

discriminator dropout is in $\{0.1,0.2,0.3,0.4,0.5\}$

discriminator LSTM feature size is in $\{256,512,1024\}$

discriminator learning rate is in $[9.2\,10^{-5},3.7\,10^{-2}]$

discriminator $L_{2}$ weight is in $\{0,10^{-7},10^{-6},10^{-5}\}$

discriminator LSTM number of layers is in $\{1,2\}$

number of discriminator updates per training step is in $\{1,2\}$

generator LSTM feature size is in $\{256,512\}$

generator learning rate is in $[8.4\,10^{-5},3.4\,10^{-4}]$

generator LSTM number of layers is in $\{1,2\}$

number of generator updates per training step is in $\{1,2\}$

dimension of trainable embeddings is in $\{16,32,64\}$

Appendix G Positional information provided to the discriminator

Here we discuss the importance of providing positional information to the discriminator. In early experiments we noticed that the distribution of sentence length in the generator samples did not match the distribution of sentence length found in the real data. In theory, we would expect a discriminator based on a LSTM to be able to easily spot samples that are significantly too short or long, and to provide that signal to the generator. But in practice, the generator was biased towards avoiding short and long sentences.

We therefore provide the discriminator with explicit positional information, by concatenating a fix sinusoidal signal to the word embeddings used in the discriminator. We choose 8 periods log-linearly spaced $(T_{1},\dots,T_{8})$ such that $T_{1}=2$ and $T_{8}$ is 4 times the maximum sentence length. For the token $x_{t}$ at position $t$ in the sentence, the positional information is p^{i}_{t}=\sin\bigl{(}2\pi\frac{t}{T_{i}}\bigr{)}. We concatenate this positional information to the word embedding for token $x_{t}$ in the discriminator before using it as input for the discriminator LSTM.

Figure 6(b) shows distributions of sentence length in samples of two GAN models, one with and one without this positional information. We compare these distributions against the reference distribution of sentence length in the training data. Even with positional information in the discriminator, the generator still seems slightly biased towards shorter sentences, compared to the training data. But the sentence length distribution is still a much better fit with positional information than without.

Appendix H Samples

Training examples from both datasets can be found in Table 6. Samples from our model, the maximum likelihood trained language model and the $n$-gram model can be found in Tables 7 , 9 and 10.