Language GANs Falling Short

Introduction

Generating fluent natural language is a central aim in Natural Language Processing (NLP). Transformer architectures (Vaswani et al., 2017) with hundreds of millions or billions of parameters regularly reestablish state-of-the-art on held-out validation perplexities, however, the generated samples can still often lack coherence. It was hypothesized that exposure bias, differences in the training and inference procedures, is the root cause for poor sample quality. Therefore, numerous GANs (Goodfellow et al., 2014) for text generation were proposed since they do not suffer from exposure bias due a training objective that directly seeks to improve sample quality (to a learned critic).

While many of the early GAN variants demonstrated improvements in sample quality (Yu et al., 2017; Guo et al., 2017), they often ignored loss of sample diversity. This is not only a theoretical concern because it is well-documented that GANs exhibit mode collapse (Che et al., 2016; Salimans et al., 2016) where the generator produces a subset of the ”modes” in the training data, reducing diversity. Evaluating generated samples remains challenging (Liu et al., 2016; Zhao et al., 2017), but Cífka et al. (2018) recommends assessing generated samples along two dimensions: 1) the quality of each sentence; 2) the diversity across sentences. However, this too is problematic because now the assessment relies on two measures which can make it difficult to compare models. Without further information about the relative trade-off between the two dimensions it is impossible to claim which is superior (Figure 1, left subplot).

In this work, we build on the natural relationship between a model’s softmax temperature and the resultant quality-diversity trade-off: lower temperatures generate less diverse, higher-quality samples; higher temperatures increase the entropy of the distribution and produce more diverse, lower-quality samples. We propose a temperature sweep as a computationally efficient method to characterize the quality-diversity trade-off of different models. By explicitly controlling the temperature we remove a potential source of bias (models may have different ”implicit” temperatures) and obtain a more complete understanding of each model’s behavior (Figure 1 middle and right).

Additionally, we address that temperature modulation changes the entropy of the conditional distributions, as opposed to changing the entropy of the joint distribution. We explore other ways to navigate the quality-diversity space with less bias, including stochastic beam search and generator rejection. Although these methods provide small gains in the quality-diversity trade-off they have other computational limitations.

Despite the dizzying array of text-GAN variants and algorithms (Yu et al., 2017; Che et al., 2017; Lin et al., 2017; Zhang et al., 2017; Guo et al., 2017; Fedus et al., 2018; Lu et al., 2018a; Shi et al., 2018; Xu et al., 2018; Chen et al., 2018; Nie et al., 2019; d’Autume et al., 2019) our conclusions using temperature sweeps are clear: MLE models still dominate GANs. According to all metrics we studied, changing the temperature of MLE models at generation time leads to a better quality-diversity trade-off compared to GANs. This evaluation technique provides a definitive boundary for future GAN research and our hope is that it will help the NLP community accurately assess natural-language generation progress.

Adversarial Text Generation

GANs are implicit generative models learned via a competition between a generator network $G_{\theta}$ and a discriminator network $D_{\phi}$. The generator network $G_{\theta}$ produces samples from a probability distribution $p_{model}(x)$. The discriminator $D_{\phi}(x)$ attempts to distinguish whether an input value $x$ is real (training data) or generated. Mathematically, the GAN objective can be formulated as a minimax game

GANs were first proposed for continuous domains since their training procedure differentiates through the discriminator into the generator. However, modeling text, which is both discrete and typically modeled sequentially requires a challenging adaptation for GANs which, in their original formulation, were built upon “one-shot” continuous data. Discrete (sequential) data require an alternative approach. Yu et al. (2017) estimate the gradient to the generator via REINFORCE policy gradients (Williams, 1992). In their formulation, the discriminator evaluates full sequences. Therefore, to provide error attribution earlier for incomplete sequences and to reduce the variance of gradients, they perform $k$ Monte-Carlo rollouts until the sentence is completed.

Yu et al. (2017) advertise their model using two tasks which we argue (with hindsight) are flawed. First, they introduce a synthetic evaluation procedure where the underlying data distribution $P$ is known and can be queried. By representing $P$ with an LSTM (referred to as an oracle in the literature), they directly compute the likelihood of samples drawn from a generative model $G_{\theta}$. The issue is that they benchmark models against each other on this likelihood alone, i.e., the diagnostic is entirely blind to diversity. For example, a model that always outputs the same highly likely sequence would easily outperform other potentially superior models. For real data, there was no agreed-upon metric to evaluate the quality of unconditional NLG at the time. This led the authors to propose a new metric, Corpus-level BLEU, which computes the fraction of n-grams in a sample that appears in a reference corpus. Again, this metric is agnostic to diversity. Generating a single good sentence over and over will give a perfect BLEU score.

This paper sparked a stream of works that adopt their evaluation framework. Notably, RankGAN (Lin et al., 2017), MaliGAN (Che et al., 2017), TextGAN (Zhang et al., 2017), LeakGAN (Guo et al., 2017) and IRL-GAN (Shi et al., 2018) were proposed soon after.

All but one of the aforementioned papers evaluate sample quality alone. As a remedy, Zhu et al. (2018) propose a metric that compares a generated sentence with a corpus of generated sentences, called Self-BLEU. They, along with Lu et al. (2018b), provide an extensive comparison of GANs using quality (negative BLEU) and diversity (Self-Bleu). However, it is not clear which algorithm is superior, as evidenced by Figure 2, because no model simultaneously outperforms the other on both metrics. It is now standard for language GANs to evaluate simultaneously quality and diversity.

Reinforcement Learning (RL) is often difficult to optimize, unstable, and sensitive to hyperparameters. Because of this, GAN-variants have recently been proposed that eschew RL-techniques in favor of fully-differentiable objectives. Cooperative training (CoT) (Lu et al., 2018a) and feature mover GAN (FM-GAN) Chen et al. (2018) are notable approaches.

Research in both RL and non-RL GANs is still very active (e.g., Xu et al. (2018); Nie et al. (2019); d’Autume et al. (2019); Li et al. (2019); Gagnon-Marchand et al. (2019)) and so our review is not exhaustive.

Temperature Sweep: Towards Robust NLG Evaluation

During natural language generation, a single spurious sampled token can lead to an overall low-quality and incoherent sequence. In other words, a high-entropy conditional distribution may result in poor sample quality at inference. To address this problem, one can modulate the entropy of $G_{\theta}(x_{t}\mid x_{1:t-1})$ with a Boltzmann temperature parameter $\alpha$ (Ackley et al., 1988). If $o_{t}$ is the generator’s pre-logit activation and $W$ is the word embedding matrix, then the conditional distribution of the generator is given by $G_{\theta}(x_{t}\mid x_{1:t-1})=\text{softmax}(o_{t}\cdot W/\alpha).$ Decreasing $\alpha$ below $1.0$ will increase $o_{t}$ and thus decrease the entropy of $G$’s conditional probability. Temperature tuning naturally moves the model in quality/diversity space. We demonstrate this in Table 1.

The evaluation protocol for NLG explained in Section 2 is to compare models with respect to both quality and diversity metrics. Often this results in a situation where it is impossible to tell which model is superior, as shown in Figure 2 and further exemplified in Figure 1 (Left). One can then control the quality-diversity trade-off of autoregressive text generators using temperature. We can leverage this tool to design a new evaluation framework that shines a light on the real performance of each model. More precisely, we propose to generate samples at s temperatures for each model in order to compute temperature curves in quality-diversity space. This is exemplified in Figure 1. We refer to this procedure as temperature sweep. This new way of evaluating NLG models allows practitioners to answer questions such as: which model to use if interested in high quality (or diversity) samples? Does a new model improve upon others in the quality/diversity space or is it just reducing the entropy of the distribution? It could also be leveraged as a cross-validation tool e.g., early-stop once the best temperature curve is achieved according to a heuristic.

In the next section we show, using temperature sweep, that MLE models consistently outperform the new proposed GAN variants everywhere in the quality-diversity space. MLE performs equally on synthetic data to CoT and outperforms it on real data, whilst being computationally and algorithmically less complicated. Our results are further validated in independent follow-up works (d’Autume et al., 2019; Alihosseini et al., 2019)where temperature sweeps show MLE outperforming GANs.

To implement temperature sweep we have taken advantage of the fact that our model factorizes the joint distribution over an observation as a product of conditional distributions over single tokens given all previous tokens (this is a property of most autoregressive neural networks). We change the temperature of these conditionals, which is straightforward to implement. However, this is different from changing the temperature of the joint probability distribution. The samples generated in this way are biased towards having lower entropy at the beginning of sentences compared to samples obtained by changing the entropy of the joint distribution. Changing the entropy of the joint distribution quickly becomes intractable with respect to vocabulary size and/or the total number of timesteps. We show below some entropy reducing techniques are less biased than changing the temperature of the conditionals and might achieve a better quality-diversity trade-off. However, we argue that theses methods aren’t suitable for our evaluation protocol due to computational inefficiency (see Section 5.4).

Stochastic Beam Search is a popular approximate decoding technique. It is computationally expensive but leads to higher-likelihood samples than greedy decoding. Here, we focus on its stochastic version (not to be confused with the locally optimal one local beam search). In Stochastic beam search with beam size $k$, the $k$ most likely hypotheses (sampled so far) are kept at every decoding steps. In Appendix A, we detail this technique and explain how it is less biased than temperature tuning regarding the reduction of the joint distribution’s entropy. Finally, the more traditional local (deterministic) beam search removes all sample diversity and is not useful in our setting.

Generator Rejection Sampling works as follow: generate some sentences; compute their likelihood under the generator’s own distribution; accept/reject the sample given a threshold. The threshold enables the practitioner to modulate the quality-diversity trade-off. In Appendix B, we explain how it is less biased than temperature tuning, why it is computationally expensive, and how the discriminator can be used in rejection sampling.

Related Work

Concurrent with our work, Semeniuta et al. (2018) demonstrated the issues of local n-gram metrics. Their extensive empirical evaluation of GAN models and language models (LM) did not result in evidence of GAN-trained models outperforming on the new and improved global metrics from Cífka et al. (2018). Our analysis further explores this path by examining the performance of these models under a sweep of temperatures. We believe this difference to be of utmost importance, as it is the necessary ingredient towards definitively showing MLE models outperform current GAN variants on quality-diversity global metrics. Work from Ott et al. (2018) thoroughly examines the local beam search strategy for neural machine translation. In their analysis, the authors compare local beam search and generator rejection sampling and find the beam search is quite effective at finding high-likelihood regions. However, their work focuses on conditional text generation, where quality-only metrics measures performance.

Guo et al. (2017) suggest that increasing the temperature at training time leads to more diverse samples. However, we argue that this procedure leads to the opposite outcome as a model can adapt to the temperature change. This would have the net result of lowering the entropy at test time. We discuss this issue and propose a study in Appendix F.

Empirical Results

Using our evaluation approach (Section 3), we examine several recent GAN text generation models and compare against an MLE baseline. The experiments consist of two parts: synthetic data generation and long-text generation. We provide strong empirical evidence for both types of data that MLE trained models reliably outperform textual GANs in the quality-diversity space. For these experiments, we only use temperature tuning, as it is the only technique that gives a smooth control over said trade-off whilst being computationally efficient (see Section 5.4).

The ever-growing body of RL trained language GANs makes writing, running hyperparameter searches and cross-validations on all the variants prohibitively expensive.We found the majority of the official implementations to be prone to extreme mode collapse thus making it quite hard to reproduce reported results. We implemented our own language GAN, including improvements/functionalities proposed in several text GAN papers. We ensure that each functionality can be toggled during training. Then for each dataset, we ran a hyperparameter search of 300 trials encompassing all possible combinations of said functionalities. We refer to this model as RL-GAN. It is based on SeqGAN (Yu et al., 2017) with additional improvements shown to be useful including: MLE pretraining (Yu et al., 2017), leaky discriminator (Guo et al., 2017), step-level loss instead of sequence level (Fedus et al., 2018), learned baseline to reduce variance (Fedus et al., 2018), regularizing REINFORCE with a maximum-entropy loss (Williams & Peng, 1991) and alternating adversarial loss with MLE loss (Guo et al., 2017).

Through comparisons with previously reported results and results obtained via running the official repositories, we show that RL-GAN is the state-of-the-art RL trained GAN. Our MLE model is trained with the same codebase (adversarial training turned off) and only one-sixth of the trials. Moreover, we report SeqGAN and LeakGAN results in the synthetic data experiment. These were obtained with the official repositories that we modified in order to obtain samples at different temperatures. Finally, the SeqGAN, LeakGAN, RankGAN, and MaliGAN results in Figure 4(a) are taken from (Lu et al., 2018b), which was written by the authors of SeqGAN and LeakGAN.

Finally, we conducted experiments differently for non-RL GANs. The CoT (Lu et al., 2018a) results are obtained with the official repository via a careful hyperparameter search guided by a discussion with the authors. The FM-GAN (Chen et al., 2018) results were obtained with the best performing model that was provided by the authors.

We describe how we selected the range of temperatures in the temperate sweeps. We found that in practice, both the quality and the diversity axes induce bounds beyond which results are non-informative. On the diversity dimension, we found no value in increasing the entropy of the MLE model because minimizing the forward KL does not lead to an underestimation of the real data entropy. Hence, we decreased the temperatures of the other models until a similar diversity to the MLE model was reached. On the quality dimension, we decreased the temperatures to $\alpha=0$ but did not report the complete curves for the following reasons. First, with synthetic data (Figure 3) the $NLL_{test}$ explodes when diversity is decreased too much, so we reduce the temperature of the MLE model until the difference in performance compared to the GAN results was evident. Second, in the real data in Figure 4(b), we stopped decreasing the temperature when the models achieved a Reverse LM score equal to the perplexity achieved by a 1-gram (unigram) model. This means that the generated dataset is almost non-informative for the (Reverse) language model to learn on i.e., it is as informative as counting tokens. This also coincides with severe mode collapse and a temperature of $<$0.5 for the MLE model. In Figure 4(a), we once more decreased the temperature until the difference in performance of MLE with respect to the other models is unambiguous.

2 Synthetic Data Experiment

In the synthetic experiment, we learn a generative model of data produced from a fixed LSTM oracle Yu et al. (2017) with a hidden dimension of 32 with parameters drawn from a standard normal distribution. This allows us to compute a perfect quality metric, the likelihood under the Oracle NLLoracle. In Table 2, we see that artificially reducing the temperature at inference achieves state-of-the-art as evaluated by the NLLoracle. As we and others have argued, evaluating quality alone is misleading. The MLE-trained model with extremely low temperature will repeatedly output only a single sequence. It is therefore essential to evaluate the resulting sample diversity which we evaluate using the log-likelihood that a generator assigns to held-out data ($NLL_{test}$). We report the result of a temperature sweep in Figure 3.

Our RL-GAN benchmark is superior to the official SeqGAN and LeakGAN implementations. Nonetheless, MLE outperforms GANs everywhere in the quality-diversity space. Semeniuta et al. (2018) suggest that the best-performing GANs tend to stay close to the solution given by maximum-likelihood training. We find support for this conclusion, as the best performing RL-GAN models have the smallest learning rate and a considerable amount of pretraining. CoT achieves similar performance to MLE, but with dramatically increased algorithmic complexity. This is unsurprising as their objectives are somewhat similar (see Section 4). Alihosseini et al. (2019) replicated this experiment on synthetic data and arrived at the same conclusions.

3 Long-text generation

Next, we study long-text generation using EMNLP News 2017. We first compare an MLE model to the reported GAN results on the local metrics Negative BLEU and Self-BLEU. Negative BLEU5 and SBLEU-5 are used for Figure 4(a) and results for (Self-)BLEU-2 to (Self-)BLEU-4 are reported in Appendix D. Again, the conclusions are the same: MLE outperforms all RL GANs considered in the quality-diversity trade-off. Moreover, MLE outperforms FM-GAN, the state-of-the-art non-RL Language GAN. Note that the FM-GAN results were obtained with a combination of temperature tuning and noise reduction in order to achieve the best possible results. We provide additional details in Appendix H. We compare MLE to GANs using recently proposed global metrics, the Language Model score (quality) and Reverse Language Model score (diversity+quality) (Cífka et al., 2018; Semeniuta et al., 2018). See Figure 4(b) for a comparison of the training scheme in quality-diversity space. The conclusion is the same as with BLEU and Self-BLEU: MLE outperforms all GANs everywhere in the quality-diversity space. Further results on this dataset as well as on the IMDB movie reviews (Maas et al., 2011) and WikiText-103 (Merity et al., 2016) datasets are presented in independent follow-up works (d’Autume et al., 2019; Alihosseini et al., 2019) where MLE is also shown to outperform GANs using temperature sweeps.

4 Empirical Remarks

These findings are illustrative: a body of research is predicated on exposure bias as the culprit for poor sample quality. Therefore, under this hypothesis, MLE should be at its weakest on long-text generation tasks. However, our results are evidence that exposure bias is less of an issue than optimization problems arising from GAN training combined with the non-differentiability of the original objective function. There is also another way to interpret these results. It seems that MLE (pre-)training leaves the generator with a better policy according to quality-diversity. However, because GAN training removes entropy from the learned distribution (see Fig. 5 in Appendix LABEL:app:gan_entropy), which results in high-quality samples, it can lead one to believe that GAN trained models are superior.

We analyze the different decoding mechanisms as tools to move in the quality-diversity space and as an alternative to temperature sweep. We evaluate these tools by decoding from the best performing MLE model on the EMNLP2017 News dataset. We also consider the different properties of these tools, including their ability to provide smooth control over the quality-diversity trade-off and their computational efficiency. The purpose of this experiment is thus not to find the best performing strategy i.e., a model combined with a decoding method but rather to compare decoding methods. Further, a comparison between MLE and RL-GAN using these approaches is in Appendix I and it supports our prior results.

Figure 6(a) compares three methods for navigating the quality (Language Model, LM) diversity (Reverse LM) space: temperature sweep, generator rejection,For computational reasons, generator rejection sampling does not span further in the high-quality regime. and beam search. We first note that temperature tuning and generator rejection sampling yield a similar trade-off in the high-diversity regime. However, as the bias incurred by temperature tuning grows, so does the gap between both methods. An important finding is that generation rejection sampling gets exponentially slower as the threshold increases (see Figure 6(b)). It is for this reason that the generator rejection sampling curve doesn’t span further in the high-quality part of the space.

Beam search, a less biased method, also seems to offer a better trade-off compared to temperature sweep. However, beam search has a major drawback: unitary increases in beam size leads to drastic leaps in quality-diversity space. In Figure 6(a) we report beam sizes of 1 (right dot), 2 (middle dot), and 3 (left dot). This study shows that although the less biased methods seem to offer at least small gains in the quality-diversity space, temperature tuning still has important advantages to assess a model’s quality-diversity trade-off efficiently. The discreteness of beam search and the computational inefficacy of generator rejection sampling are important drawbacks. However, if one’s goal is to obtain the best samples possible, we advocate for their use.

GAN Training Reduces Entropy

Figure 5 shows the evolution of a generator’s entropy with respect to epochs. We observe that as soon as adversarial training starts, a drastic entropy reduction occurs. Future work investigates if GAN training changes the learned distribution or simply reduces the entropy in an ineffective way with respect to the quality-diversity trade-off.

Discussion

We demonstrate that well-adjusted language models are a strong baseline and that temperature sweeping can provide an unambiguous characterization of model performance in terms of quality and diversity. A well-adjusted language model outperforms the considered GAN variants as evaluated on both local, and more surprisingly, global metrics of quality and diversity. Our temperature sweeping framework shares characteristics with a Receiver Operating Curve. Analogously, if one needed a single scalar to compare NLG models, one could compute the area under the curve and seek the model with the smallest value (lower is better for our considered metrics).

GAN-based generative models have been proven effective on real-valued data. However, there exist many difficult pernicious issues involved in moving to discrete data. These issues must be overcome before they will improve over the strong MLE baselines for unconditional language generation. On the datasets and tasks considered, potential issues caused by exposure bias were less severe than the ones arising from adversarial training combined with learning the non-differentiability of the original objective function. GAN training may prove fruitful eventually, but this research lays forth clear boundaries that it must first surpass.

References

Appendix A Stochastic Beam Search.

Beam search is a popular approximate decoding technique. It is computationally expensive but usually leads to higher-likelihood samples than greedy decoding. Here, we focus on its stochastic version, not to be confused with the locally optimal one, Local Beam Search. In Stochastic Beam Search with beam size $k$, words are sampled sequentially starting from the first one. At the first timestep $k$ words are sampled, each represents a hypothesis. At subsequent timesteps, $k$ next words are sampled conditioned on each of the $k$ current hypotheses, resulting in $k^{2}$ hypotheses. The $k$ most likely are kept and so on. Because an increase in beam size results in higher-likelihood samples, beam size can be leveraged to modulate the quality-diversity trade-off. Moreover, an infinite beam size would result in sampling the most likely sentences under the generator’s distribution. This is not what we can expect from setting the temperature to 0, which is equivalent to greedy decoding. For this reason, we can hypothesize that Stochastic Beam Search is less biased regarding the reduction of the joint distribution’s entropy. However, unlike temperature tuning, one cannot smoothly trade quality for diversity, as the beam size is a discrete parameter. The more traditional local (deterministic) beam search removes all sample diversity. Thus, it is not useful in our setting.

Appendix B Generator Rejection Sampling.

Generator Rejection Sampling works as follow: generate some sentences; compute their likelihood under the Generator’s own distribution; accept/reject the sample given a threshold. The threshold enables the practitioner to modulate the quality-diversity trade-off. Similar to Stochastic Beam Search, increasing the threshold to a maximum level would result in always sampling the most likely sentence. We can again hypothesize that this method achieves a better quality-diversity trade-off compared to temperature tuning, since it is not biased towards having sentences with a lower entropy prefix. However, the running time of this procedure increases significantly as the acceptance threshold lowers, as discussed in Section 5.4. We also experimented with a rejection sampling scheme where the score is determined by a discriminator but it did not provide any additional insights.

Appendix C Qualitative Analysis of the Samples

For completeness, we show samples of SeqGAN, LeakGAN, and MLE from both datasets in Table 3. In this experiment, we used the Image COCO dataset (Lin et al., 2014) in order to generate shorter sentences as well. In this case, the samples of all models appear similar. This is because generating captions is a relatively easier task. For the EMNLP2017 News dataset, we would like to point out interesting facts from our samples. In our first sample (1), the model generates the sequence “post Brexit strategy”. This is n-gram is not present in the training set. In our second sample (2), the token “leak” and the n-gram “Freedom of Information request” never appear together in the training dataset. The extrapolations show that the model learns a certain level of generalization beyond the training set. Additional samples are presented further in Appendix J.

Appendix D Full BLEU and Self-BLEU results

Full BLEU and Self-BLEU results are shown in Table 4.

Appendix E The Limitations of BLEU

In this section, we want to highlight an important flaw in using BLEU as a proxy for quality. We tuned the temperature in order to find a MLE model with BLEU score equal to the training data’s. We show three randomly sampled sentences from the model in Table 6. Although sometimes grammatically correct, the samples lack in semantic and/or global coherence. It seems the generated text has poor information content. Surprisingly, in order to get great samples on a consistent basis, the temperature needs to be reduced to a level where BLEU-5 is twice as large as the training data’s. Thus, it seems like BLEU is not always a good proxy of sample quality. Again, we think it is of utmost importance to develop better metrics and modernize NLG’s canonical evaluation framework.

Appendix F Issues of Varying Temperature During Training

As a penultimate experiment, we analyze the effect of changing the temperature at training instead of inference. Guo et al. (2017) suggested that increasing the temperature at training time leads to more diverse samples. However, we argue that this procedure leads to the opposite outcome as a model can adapt to the temperature change. This would have the net result of lowering the entropy at test time. To examine this, we trained 30 GANs maintaining everything constant except training temperature. Negative BLEU-5 against SBLEU-5 are plotted in Figure 7. The darker the dot, the higher the $\alpha$ and consequently the temperature. As we hypothesize, models trained with increased temperature at training time adapted to the change and the net result was a colder temperature at inference (hence reduced diversity). We therefore recommend only adjusting the temperature at inference. One should consider other techniques to facilitate exploration during training.

Appendix G Discriminator Rejection Sampling

We experimented with a rejection sampling technique base on the Discriminator’s signal. For this method, we train a Discriminator on the Generator’s samples without ever updating the Generator. Next, we accept/reject samples in a similar fashion to Generator Rejection Sampling however now the score is determined by the Discriminator. The higher the threshold, the more confident the Discriminator needs to be about the realness of the data. This should increase the quality of the samples with the usual downside of reducing diversity. We show results with a Discriminator having the same architecture as the Generator (Discriminator Rejection) and the best Discriminator found with an hyper parameter search of 100 trials (Best Discriminator Rejection. Results are shown in Figure 8. Note that the domain of the figure is much smaller that in Figure 4(b). The reason is that this approach can’t move the models further in quality-diversity space. For this reason, we do not advocate for the use of this approach as a means to obtain high quality samples.

Appendix H Modulating qualitity/diversity for FM-GAN

For FM-GAN, we observed that lowering the temperature to 0 didn’t completely remove all the stochasticity in the generations. This is because FM-GAN samples noise prior to sampling the first word. This is similar to how image generation GANs are trained. Precisely, $z\sim N(0,1)$ is sampled once and concatenated to every token of a generated sentence. Because temperature tuning wasn’t covering enough of the quality/diversity space, we also reduced the variance of the noise as quality modulating tool. Results are show in Figure 9.

Appendix I RL-GAN versus MLE with Other Decoding Mechanisms

Appendix J EMNLP 2017 News Samples

We present additional samples for EMNLP 2017 News dataset.