Mix and Match: Learning-free Controllable Text Generation using Energy Language Models

Fatemehsadat Mireshghallah, Kartik Goyal, Taylor Berg-Kirkpatrick

Introduction

While large transformer-based autoregressive language models trained on massive amounts of data found on the internet exhibit exceptional capabilities to generate natural language text, effective methods for generating text that satisfy global constraints and possess holistic desired attributes remains an active area of research. These mechanisms for controlling the generation of language have the potential to mitigate undesirable biases encoded by the large language models and prevent the generation of hate speech and toxic language Xu et al. ; Gehman et al. (2020); Sap et al. (2021); Baheti et al. (2021); Mireshghallah and Berg-Kirkpatrick (2021). Much of the prior work has approached controlled generation via either training domain-conditioned neural language models Prabhumoye et al. (2020); He et al. (2020); Lample et al. (2018); Shen et al. (2017); Krishna et al. (2020); Reif et al. (2021); Ficler and Goldberg (2017); Khalifa et al. (2021) or finetuning/modifying an underlying large pre-trained base model for generation on domain-specific data for attribute sensitive generation Ziegler et al. (2019); Keskar et al. (2019); Mai et al. (2020); Gururangan et al. (2020); Chronopoulou et al. (2021). Not only do these approaches involve computational overhead and estimation errors associated with the training of language models, but they are also dependent on access to a large amount of attribute-specific language data which can be impractical in many scenarios and exacerbate privacy concerns Brown et al. (2022); Mireshghallah et al. (2021); Kandpal et al. (2022).

Our approach eschews training and focuses on generation-time control from pre-trained modules. Recent work in this space has used attribute discriminators Dathathri et al. (2020); Krause et al. (2020); Yang and Klein (2021); Holtzman et al. (2018) to steer the generation from a large autoregressive language model. These discriminators need to be separately trained on partial generations in order to be operationalized with step-wise autoregressive models. As a result, this approach also requires availability of data to train step-wise discriminators for attributes that are essentially global (at the sequence-level) in nature. Therefore, we focus on drawing samples from a test-time combination of pretrained blackbox experts that each score a desired property of output text – for example, fluency, attribute sensitivity, or faithfulness to the context. Specifically, we view the product of these black-box experts as a probabilistic energy model Hinton (2002) – i.e., a non-autoregressive, globally normalized language model – and then sample (without further training or fine-tuning) using a specialized Gibbs sampler with a Metropolis-Hastings correction step Goyal et al. (2021).

Our full framework, which we entitle Mix and Match LM (depicted in Figure 1), enables the generation of high-quality attribute-controlled samples by mixing and matching black-box models like off-the-shelf pre-trained attribute-sensitive discriminators (e.g., sentiment classifiers), large bidirectional pre-trained language models like BERT Devlin et al. (2019), and other modules specializing in capturing desirable features pertaining to faithfulness to any additional context, like hamming distance, or BertScore distance Zhang et al. (2020) between the sample and the conditioning context. We generate samples from the energy language model assembled from these component experts by using the recently proposed Gibbs-Metropolis-Hastings scheme Goyal et al. (2021) for sampling from energy models using a masked language model as a proposal distribution. In this scheme, an expressive bidirectional language model like BERT is used to make a proposal at each transition step in the Gibbs chain to jump to a sequence $\mathbf{\bar{x}}$ from the current sequence $\mathbf{x}$ . This proposal’s fitness is judged by the change in the energy language model’s score, with the sampler accepting proposals with larger energy reductions at a higher rate. While the MCMC nature of our sampler negatively impacts the runtime during decoding compared to autoregressive approaches with ancestral sampling, we find our approach to still be practical and yield high-quality diverse samples that respect the distribution induced by the product of expert black-box models.

We demonstrate the flexibility of our approach by performing a variety of controlled generation tasks, such as aspect-based text revision, style transfer, and attribute grounded generation and compare it to recently proposed controlled generation approaches that are more resource/data intensive. We observe that our approach, which does not require any gradient optimization and is able to combine arbitrary heterogeneous black-box models, outperforms other approaches according to various automated metrics of fluency, quality, and control, as well as human evaluations. We have provided code, data, and sample generations in this GitHub repository: https://github.com/mireshghallah/mixmatch (see A.1 for details on reproducing the results).

Related Work

The approaches closest in spirit to our work involve steering generation from a base language model with external attribute-sensitive control mechanisms. Plug-and-Play LM Dathathri et al. (2020) uses discriminators learned from an autoregressive LM’s top-level hidden layer to modify the LM’s states toward increasing the probability of the desired attribute via gradient ascent at each step. GeDi Krause et al. (2020) and FUDGE Yang and Klein (2021) take a similar approach but train custom step-wise attribute-sensitive discriminators that decide whether the desired attribute is likely to be satisfied by the current generation path. GeDi trains class-conditional language models for these discriminators and hence additionally relies on access to attribute sensitive language data. Kumar et al. (2021) formulate the task of controlled generation as optimizing the base LM’s likelihood subject to global differentiable attribute-based constraints by gradient descent over the position-wise simplexes over the vocabulary. DExperts Liu et al. (2021) is another decoding-time controllable generation approach that modifies the step-wise softmax logits of an autoregressive pre-trained LM with softmax logits of separately trained domain-specific expert autoregressive language models. These approaches require training of custom modules and do not readily enjoy the benefits of incorporating global attribute-based features into the generation mechanism in a simple probabilistic manner. In contrast, following the findings related to implicit energy-based models trained via non-probabilistic objectives Grathwohl et al. (2019); Goyal et al. (2021), our energy-based formulation is not only optimization-free but also fully modular and able to easily incorporate global features, allowing for heterogeneous black-box experts to be combined with each other.

Mix-and-match Language Models

In this section, we describe our approach and motivation behind our method. Specifically, we frame the problem of performing controlled generation as a problem of sampling from a specialized energy-based (or globally normalized) sequence model that defines a probability distribution that satisfies the desired constraints we wish to impose in the controlled generation setting. As described below, this energy-based model is composed of pre-trained components and does not require any further optimization. An energy-based sequence model defines the probability distribution over the space of possible sequences $\mathcal{X}$ as:For simplicity, we are concerned with a finite set of sequences limited by some maximum length. $p(X;\theta)=\frac{e^{-E(X;\theta)}}{\sum_{X^{\prime}\in\mathcal{X}}e^{-E(X^{\prime};\theta)}}$ , where $E(X;\theta)$ refers to the scalar energy of a sequence $X$ that is parametrized by $\theta$ . Lower energy corresponds to the higher likelihood of $X$ . In contrast to the common autoregressive sequence models, exact likelihood computation and efficient sampling from these models is challenging. Despite these challenges, we focus on this paradigm of sequence modeling because energy-based models offer increased flexibility via sequence-level features and constraints. As we discuss next, this capability lets us easily define expressive functions for controlled generation of sequences which is not readily offered by the autoregressive modeling paradigm.

Our approach is motivated by the perspective that the task of controlled generation requires concentrating probability mass over a small subspace of sequences in $\mathcal{X}$ that satisfies various constraints pertaining to fluency, target attributes, and other control variables. Consider the task of generating positive sentiment sentences. This requires satisfaction of two major constraints: (1) The sequence $X$ should be well-formed, (2) The sequence $X$ should express positive sentiment. If we have access to two separate probability distributions over $\mathcal{X}$ , one for modeling well-formedness ( $p_{1}(X)$ ) and another for modeling positivity ( $p_{2}(X)$ ), then a natural solution for controlled generation in this setting would be to draw samples from a probability distribution that is a product of these two distributions i.e. $p_{\texttt{desire}}(X)\propto p_{1}(X)\cdot p_{2}(X)$ . In our approach, we further relax this requirement by assuming access to expert blackboxes that yield scalar non-probabilistic energy scores $E_{1}$ and $E_{2}$ indicating fitness of a sequence w.r.t. well-formedness and positivity respectively. Under the product of experts framework above the desired probability distribution would take the form: $\log~{}p_{\texttt{desire}}(X)=-(E_{1}(X)+E_{2}(X))~{}-~{}\log Z$ . This expression shows that when working with scalar scores for the expert black-boxes, the product of expert models yields an energy model whose energy is simply the sum of the scalar energy values obtained from the expert models. Inspired by this, we propose a framework for controlled generation that involves linear combinations of various black-box experts in order to obtain a distribution whose samples satisfy the requirements of a desired controlled generation task: $E_{\texttt{M\&M}}(X)=\sum_{i=1}^{k}\alpha_{i}E_{i}(X)$ , where our proposed mix-and-match energy is composed of $k$ expert energy components, which are weighted by scalar hyperparameters $\alpha$ .

2 Expert Factors in Mix-and-Match LM

As shown in Fig. 1, we use the following black-box experts in our experiments as modules that we can add or remove to produce desired behavior:

$\mathbf{E_{\texttt{mlm}}(X)}:$ Recent work has shown that large masked language models (MLM) like BERT can discriminate between well-formed and ill-formed sentences Zhang et al. (2020) and induce an implicit energy function over the sequences Goyal et al. (2021). Hence, we use BERT-base as a black-box to model the form and fluency of sentences. Specifically, we use an energy parametrization introduced in Goyal et al. (2021) which is negative of the sum of unnormalized logits iteratively computed at each position obtained via the forward pass of the MLM after masking the corresponding position.

$\mathbf{{E_{\texttt{disc}}(X)}}:$ This particular expert module refers to the energy obtained via the discriminator for the attributes of interest. What this module returns is the raw logits of the discriminator, for the target attribute. For instance, if we have a sentiment classifier, and want to produce positive sentiment, then ${{E_{\texttt{disc}}(X)}=-\log~{}p(+|X)}$ .

$\mathbf{E_{\texttt{hamm}}(X;X^{\prime})}:$ For a given sequence $X^{\prime}$ , this quantity refers to the hamming distance between the sequence $X$ and $X^{\prime}$ . This penalizes token level deviation from $X^{\prime}$ which is useful if we are interested in only making minor edits to $X^{\prime}$ as described later.

$\mathbf{E_{\texttt{fuzzy}}(X;X^{\prime})}:$ Similar to the hamming distance, this quantity refers to the BertScore Zhang et al. (2020) computed between $X$ and $X^{\prime}$ which can be viewed as a fuzzy hamming distance that takes semantic similarity into account.

3 Sampling scheme

To sample from the energy parametrizations described in the previous section, we follow the Metropolis-Hastings Hastings (1970) MCMC scheme for sampling from the masked language models introduced by Goyal et al. (2021). While the proposal distribution we use is the same as Goyal et al. (2021) i.e. masked language model’s (BERT’s) conditionals, the energy parametrizations we use are more suitably designed for controlled generation.

We briefly explain the sampling procedure, which involves forming long Markov chains of sequences starting with a random sequence, and following the MH scheme which uses a proposal distribution to propose a new sequence at each step in a chain which is either accepted or rejected based on its fitness to the energy function. The sequences at the end of these chains correspond to samples from the desired energy-based model. Operationally, at each MCMC step, we mask out a token at a random position in the current sequence $X$ in the chain and propose a new sequence $\bar{X}$ to transition to by sampling a token from the MLM conditional softmax at the masked position. This proposed sequence is evaluated by its ability to reduce the energy from the current sequence in the chain and is accepted with the probability $p(\bar{X};X)=\min\left(1,\frac{e^{-E_{\texttt{M\&M}}(\bar{X})}~{}p_{\textrm{mlm}}(X_{i}\mid X_{\backslash i})}{e^{-E_{\texttt{M\&M}}(X)}~{}p_{\textrm{mlm}}(\bar{X}_{i}\mid X_{\backslash i})}\right)$ . $E_{M\&M}(X)$ refers to the product of experts energy, $i$ refers to the position chosen for masking, $p_{\textrm{mlm}}$ refers to the MLM’s conditional distribution at the [MASK] position. Intuitively, this acceptance probability indicates that the proposed sequence $\bar{X}$ is more acceptable if it has lower energy than the current sequence $X$ in the chain and is rare or less likely to be proposed by the proposal distribution again.

4 Controlled generation Tasks

We use the expert black-box factors and the sampling scheme described above in our framework to perform two kinds of controlled generation tasks.

Prompted generation: This task focuses on generating well-formed sentences that start with a specified prompt and also satisfy a target attribute for which we have access to a discriminator. An example task would be to generate positive sentiment sequences starting with This movie. The energy function takes the form:

$\alpha$ is a hyperparameter that controls the tradeoff between the MLM score and the discriminator’s influence. For MH-based sampling for this task, we initialize the sequence with the starting prompt and the rest of the tokens masked out, which creates a seed text of shape the movie [MASK] [MASK] ... [MASK], for the prompt example of the movie. The number of mask tokens depends on the target generation length, and we constrain the sampler to only produce proposals and revise non-prompt tokens, and mark the prompt tokens as “frozen”.

Controlled text revision: This task involves editing a source sequence $X^{\prime}$ in order to satisfy the desired target attributes exhibited by the generated sequence $X$ . The energy function for this task is: \@mathmargin0pt

This energy function in addition to valuing well-formedness and satisfying target attribute requirements also focuses on maintaining faithfulness to the source sequence $X^{\prime}$ . For sampling with this energy, we initialize the sequence with the sequence $X^{\prime}$ to be edited. This sets the length of the target sequence to be the same as the source. In this setup, the sampler can revise all tokens and is not constrained.

For both these tasks, we run a separate MCMC chain for each generated sentence for 8 to 15 epochs, depending on the task. An epoch refers to one masking cycle over all the non-frozen positions (selected randomly) of the sequence.

Experimental Setup

We provide full experimental details in appendix Section B, here we provide a brief overview of the tasks, datasets, baselines, and metrics used in the experiments.

Sentiment transfer (Yelp): We use Yelp Shen et al. (2017) dataset’s test-set for the task of sentiment transfer. The test set comprises 1000 sentences, half with positive and half with negative sentiment. We also have a reference set of handwritten sentiment transferred sentences, provided by He et al. (2020) that we use for reporting evaluation metrics.

Formality transfer (GYAFC): We use 1051 sentences from the entertainment and music domain subset of the GYAFC Rao and Tetreault (2018) dataset, which contains formal and informal sentences for the task of formality transfer (both directions of formal to informal and informal to formal).

Prompted generation: We evaluate our approach on two forms of prompted generation: 1) sentiment controlled generation and 2) topic controlled generation. For sentiment controlled generation, we set Mix and Match LM to generate text with positive or negative sentiment given prompts, by using a Yelp sentiment classifier as discriminator and compare against PPLM Dathathri et al. (2020) which is a popular sentiment controlled generation method. For topic controlled generation, we compare against FUDGE Yang and Klein (2021), and follow their experimental setup consisting of 7 distinct topics and 20 prompts.

2 Expert Component Configurations

We use a Huggingface pre-trained bert-base-uncased model as our MLM for yielding $E_{\texttt{mlm}}$ and also providing the proposal distribution in our MH MCMC sampler. For obtaining $E_{\texttt{disc}}$ , we train BERT-based classifiers on the training-set of our datasets to use as our attribute discriminators. We could have used any pre-trained attribute classifier from Huggingface for $E_{\texttt{disc}}$ , but we keep those aside to use as external attribute classifiers for fair evaluation against baselines. For experiments in which we add the BertScore Zhang et al. (2020) component to the energy, we use the pre-trained roberta-large_L17 model. Finally, for agency score, we use the lexicon provided by Sap et al. (2017) and check each generated sequence and count the number of target agency verbs that exist there. The count becomes the agency score.

3 Baselines

PowerTransformer. For the task of controllable debiasing (agency revision), we compare our work with PowerTransformer Ma et al. (2020), an approach that uses paraphrasing and self-supervision based on a reconstruction loss, building on pre-trained language models, to re-write text and control agency level of sentences.

He et al. For style transfer on sentiment an formality, we compare with He et al. (2020), a generative style transfer framework which uses a variational autoencoder (VAE) built using a sequence-to-sequence LSTM-based model to do unsupervised style transfer. This framework needs to be trained from scratch for each style transfer task.

UNMT. As a second baseline for style transfer, we use UNMT Lample et al. (2018), an unsupervised machine translation framework that demonstrates high performance for sentiment transfer.

PPLM. For the task of sentiment controlled generation, we compare to Plug-and-Play LM (PPLM) Dathathri et al. (2020), which does attribute controlled generation using the flow of gradients from discriminators trained on the last hidden layer representations of the generator, to guide generation.

FUDGE. This approach Yang and Klein (2021) trains step-wise discriminators on partial generations from GPT-2 to determine whether the constraints related to desired attributes will be satisfied by the future completion of the sequence or not. We compare against this on topic controlled generation as this approach was shown to be superior to PPLM on this task.

4 Evaluation Metrics

We use a variety of evaluation metrics to compare our approach’s performance on two major facets: (1) Quality of generated text, and (2) success on matching the target attribute used for control.

GPT-2 PPL. We feed our generated test sentences to a Huggingface Radford et al. (2019) pre-trained GPT-2 xl model, and report its perplexity (PPL), as an automatic measure of fluency. Although this measure is not a perfect indicator of fluency, we find it to be a useful metric alongside human judgements. Due to the high variance in the PPL scores generated across sentences by GPT-2, we report the median score for each system under comparison.

BLEU. For sentiment (Yelp) and formality (GYAFC) transfer where we have reference text, we report the BLEU score. For controlled debiasing, we report BLEU between generated text and source and show it as BLEU (src).

BertScore. As a measure of meaning preservation, we use the F1 BertScore metric Zhang et al. (2020) to compare the semantic similarity of the provided reference sentence with the generated output.

Hamming Distance. We also report the hamming distance between the source text and generated text, to measure the extent of the change.

4.2 Attribute Quality

Internal Classifier Accuracy. We report the accuracy of the internal classifier (the discriminator used for generation) on the generated text, assuming the target attribute is the correct label. The higher this accuracy is, the better.

External Classifier Accuracy. It is natural to get high accuracy on the internal classifier, since we are sampling from it. To have a fair comparison, we report accuracy using external classifiers from Huggingface (textattack/bert-base-uncased-yelp-polarity Morris et al. (2020) for sentiment and cointegrated/roberta-base-formality for formality).

Agency Lexicon Accuracy. For controlled debiasing, we measure the accuracy of the change in agency by comparing the target agency level with that of the generated text, extracted using the connotation frames lexicon, and following the setup from Ma et al. (2020).

Results

Tables 1 and 2 show our results for the task of text revision for controlling agency bias which is introduced by PowerTransformer Ma et al. 2020, our Baseline for this task. PowerTransformer has a vanilla (no boost) variant and a variant with vocab boosting, which up-weights the logits of verbs that belong to the target agency lexicon so as to increase their probability and incentivize generation in that direction. We also measure our metrics on the original test-set, without revision, to provide a better sense of the changes made.

We offer different variants of our framework, to provide a fair comparison and to better ablate our proposed method. “Disc” denotes our framework where we add the discriminator expert ( $E_{\texttt{disc}}$ ) which is trained to predict the agency level of a sentence, to the energy along with $E_{\texttt{mlm}}$ , and $E_{\texttt{hamm}}$ (Eq. 2). Hamming distance is computed between the generated proposals and the source sentence. The “Agency Score” variant adds an alternative term to $E_{\texttt{M\&M}}$ instead of $E_{\texttt{disc}}$ , which is the number of target agency verbs according to the connotation frames lexicon Sap et al. (2017) in the sentence. The “Disc+Agency” variant has both energy components. We also apply our method in two ways: “Verb Replace” which allows the sampler to propose revisions for only one pre-determined verb (provided in the dataset). In this setup, all tokens remain frozen, except for the given verb. The conventional mode (M&M LM), however, proposes revisions for all tokens in the sentence and is not constrained.

Table 2 shows that in the conventional setup, Mix and Match LM (Disc only) has performance similar to that of PowerTransformer, without boosting. With the Agency Score component, our method outperforms PowerTransformer in terms of accuracy of revision as per the agency lexicon accuracy metric, with negligible loss in meaning (BertScore). The reason behind this better performance in terms of applying target agency accuracy is that our method’s sampling is guided by the energy that is directly built on the metrics we care about, as opposed to trying to apply them through paraphrasing and proxies such as vocab boosting, which are employed in the PowerTransformer method.

Another important observation here is the difference between “Verb Replace” and conventional modes. This ablation shows that although our method makes few changes (the average Hamming distance between source and output sentences are between $1.37$ and $2.45$ ), it still outperforms a “static” method that has extra knowledge of the offending verb and focuses on changing only that verb, by a significant margin.

2 Style Transfer

In this section we experiment with sentiment and formality transfer, where Sentiment transfer needs fewer changes and formality transfer needs more structural change to the original sentence. We show sample sentences and transfers in Table 1 (we cannot show samples for formality as the dataset is not public).

For this task, we include two components in our energy model, the attribute discriminator ( $E_{\texttt{disc}}$ ), to induce the target style, and the hamming distance ( $E_{\texttt{hamm}}$ ), to maintain the meaning of the sentence. We don’t include the more complex semantic similarity-related component like $E_{\texttt{fuzzy}}$ , since sentiment transfer can normally be done by making only a few changes to the sentence. We report results with two different variants, one where the discriminator component has a higher coefficient in the energy (Discriminator $\uparrow$ ) and one where the hamming distance has a higher coefficient (Hamming $\uparrow$ ). In effect, these two show the trade-off between transfer quality and faithfulness to the source sentence.

We see in Table 3 that our method, with the hamming component up-weighted, outperforms both the generative baselines in terms of transfer accuracy (Ext. Clsf.) and semantic similarity (BertScore). We can also see Mix and Match LM has higher BLEU score, with respect to the provided hand-written reference sentences. We hypothesize that this superiority is due to the tendency of our model to make minimal revisions that satisfy the product of experts energy model. Therefore, our model can successfully change the style without changing the meaning of the sentence. The generative baselines, however, regenerate the sentence which imposes more change, as can be observed from the hamming distance column (Hamm.(src)) in Table 3.

2.2 Formality Transfer

For this task, we include the formality classifier ( $E_{\texttt{disc}}$ ), Hamming distance ( $E_{\texttt{hamm}}$ ), and BertScore ( $E_{\texttt{fuzzy}}$ ) components in the energy formulation, to permit the transfer of style and also maintain the meaning of the sentence. $E_{\texttt{fuzzy}}$ helps with imposing semantic similarity between the source and generated sentences, since Hamming alone isn’t sufficient for judging comparable formal and informal sentences. We show results for two setups of our framework, one where the discriminator coefficient is higher (Discriminator $\uparrow$ ) and another where the BertScore coefficient is higher (BertScore $\uparrow$ ).

In Table 4 we have broken down the external classifier accuracy for the different transfer directions of formal to informal ( $\rightarrow$ Inf.) and vice versa. We do this because the $\rightarrow$ Form. task is generally harder and therefore has lower accuracy. We observe that our method outperforms the baselines in terms of BertScore and BLEU, for similar levels of external classifier accuracy. However, we can see that the GPT-2 PPL of our method is higher than the baselines. The reason behind this is the format and noise in the data. The samples for this dataset are taken from the music and entertainment industry domain and contain some symbols and characters similar to emojis (e.g. “:)” and “***”). This is where the tendency of our approach toward minimal revisions is hurtful–our revisions of text, often do not get rid of all of these symbols, while the baselines’ generative methods successfully remove all the superfluous characters because they rewrite sentences from scratch.

3 Prompted Controlled Generation

We generate 560 sequences of different lengths ( $12$ , $20$ and $50$ tokens), given 14 prompts, 2 sentiments, and 20 sequences per sentiment, taken from Dathathri et al. (2020)’s experimental setup. The prompts and sample generations are in the appendix B.9 and A.2, and a full list of generations is in the supplementary material.

Table 6 shows our results for this experiment. Here, we have an additional metric, the MLM energy (lower is better), which, like GPT-2, indicates the quality of generated sentences Salazar et al. (2020) according to BERT. We report this extra metric here since PPLM uses a GPT model for generation, and it is natural that it would measure better on this metric. The table shows that for all lengths of generated sentences, our method is much better at inducing the target sentiment. However, we observe that PPLM performs better in terms of GPT-2 while our method performs better on the MLM energy metric. This suggests the tendency of model-based fluency metrics to be biased toward the corresponding models as the PPLM uses GPT-2 for generation and M $\&$ M LM uses BERT. To enable a more conclusive comparison of the text quality, we report results with human evaluations. For these evaluations, we randomly select 10 generated outputs for each prompt, per sentiment (240 overall), and asked three Amazon Turkers per sample pair, which sample they find more fluent. We report the majority vote of the Turkers in the table. The results show that for sequences with lengths 12 and 20, they found our generations more fluent. However, for length 50, the preference rate for M&M drops to $46.7\%$ , which shows that our method is superior to PPLM for short/medium length generation, however, PPLM does better at generating longer sequences.

3.2 Topic Controlled Generation

We follow FUDGE’s Yang and Klein (2021) experimental setup which covers $7$ topics, given $20$ prompts and generate $7\times 20$ sequences of length $20$ . To enforce topicality on our generations, we add a topic-based energy, $E_{\texttt{topic}}$ . This energy is essentially the negative count of the number of topic-related words (using the list provided by FUDGE).

Table 7 shows the results of this experiment, generations are also provided in A.2. Topic-score ( $\uparrow$ ) is the usage rate of topic-related words that were used for training and evaluation of topic controlled generation by Yang and Klein in their paper. Grammaticality ( $\uparrow$ ) is the score of grammaticality given by a Roberta-based CoLA grammaticality model averaged over all outputs Warstadt et al. (2019). The “Div” ( $\uparrow$ ) metrics show the diversity of generated text, over unigrams, bigrams and trigrams. Finally, the human evaluations show human preference, in terms of fluency of the sentences ( B.10). As shown by the table, the fluency of our method is comparable to that of FUDGE, even better in terms of human preference and grammaticality judgment. FUDGE has a slightly higher topic score, which is expected since it trains a custom step-wise discriminator for each topic that is optimized for the task. But our approach shows competitive faithfulness to the topics especially considering the fact that prompted GPT-2 generations without the FUDGE discriminators only achieve a topic-score of $0.23$ .

4 Inference Speed

Given that our model’s inference procedure involves MCMC sampling, it’s reasonable to expect its run-time to be slower than more traditional baselines. For sequences of length 20, we find that our un-optimized implementation requires 8 seconds per generation and 3 seconds per revision – while, in contrast, baseline system PPLM requires 16 seconds and FUDGE requires 0.4 seconds per generation. This is a substantial slowdown compared to FUDGE, but not one that renders the proposed approach impractical in offline settings. Further, faster sampling schemes are beyond the scope of this paper but might be explored in future work to speed up models like M&M LM.

Conclusion

We present Mix and Match Language Models (M&M LM), a training-free framework for controlled text generation that can easily mix heterogeneous expert modules. We show that our framework outperforms prior methods on a suite of text revision and attribute-controlled generation tasks. Further, our results indicate that probabilistic energy language models, typically considered intractable, can be used for practical text generation tasks when combined with an appropriate sampling scheme.

Acknowledgments

The authors would like to thank the anonymous reviewers and meta-reviewers for their helpful feedback. We also thank our colleagues at the UCSD/CMU Berg Lab for their helpful comments and feedback.

Ethical Considerations

The proposed approach takes steps towards a novel paradigm that might partially mitigate the need for energy-intensive GPU training – potentially leading to positive environmental impact down the line. The approach may also have positive impacts on accessibility as strong computational resources are not required when setting up a new controlled text generation system. We do however acknowledge that strong controlled generation methods that rely on discriminators have the potential to regurgitate sensitive training data and produce harmful outputs and toxic language Xu et al. ; Gehman et al. (2020); Wallace et al. (2020). However, if used properly and for good, we anticipate a positive impact on debiasing and safe generation.

References

Appendix A Appendix

We have provided all our code, data and our generations in https://github.com/mireshghallah/mixmatch, and our checkpoints are uploaded anonymously here https://zenodo.org/record/5855005. There is a readme file in the repo, which has instructions on how to run generation and get evaluation metrics. We have not included the data files for the formality, since the GYAFC dataset requires permission for access, so we cannot release it.

A.2 Sample Generations

Due to page limitations in the body of the paper, we include more sample generations from our method in the form of tables here. We have no samples from the formality transfer task, however, since the data used (GYAFC) is protected and needs permissions for access, so we cannot publish it. However, we have provided code needed to reproduce our results, once access to the original data is gained. Table 8 shows FUDGE generations versus Mix and Match generations.

Appendix B Experimental Setup Details

Controllable debiasing (ROC story corpus): We use the subset of the ROC story corpus Mostafazadeh et al. (2016) test-set that is used by PowerTransformer Ma et al. (2020) for their evaluations. We use this data for controllable debiasing, a text revision task which aims to correct the implicit and potentially undesirable agency biases in character portrayals. This test-set consists of 549 sentences, where 224 sentences have low agency verbs (such as wish, dream, etc.) and the rest have high agency (like pursue, achieve, etc.). The task is to revise the sentences such that the meaning is preserved, but the agency of the sentence is changed in the target direction.

Sentiment transfer (Yelp): We use Yelp Shen et al. (2017) dataset’s test-set for the task of sentiment transfer. The test set comprises of 1000 sentences, half with positive and half with negative sentiment. We also have a reference set of hand written sentiment transferred sentences, provided by He et al. (2020) that we use for reporting evaluation metrics.

Formality transfer (GYAFC): We use 1051 sentences from the test-set of the GYAFC Rao and Tetreault (2018) dataset, which contains formal and informal sentences for the task of formality transfer (both directions of formal to informal and informal to formal). Here we use the entertainment and music domain subset of this data, following the evaluation setup of He et al. (2020). This dataset also contains parallel data between formal and informal sentences, which we use as reference for reporting evaluation metrics.

Prompted generation: We evaluate our approach on two forms of prompted generation: 1) sentiment controlled generation, and 2) topic controlled generation. on prompted generation. For sentiment controlled generation, we set Mix and Match LM to generate text with positive or negative sentiment given prompts (listed in Appendix B.9) by using a Yelp sentiment classifier as discriminator and compare against PPLM Dathathri et al. (2020) which is a popular sentiment controlled generation method. For topic controlled generation, we compare against FUDGE Yang and Klein (2021), and follow their experimental setup consisting of 7 distinct topics and 20 prompts.

B.2 Expert Component Configurations

We use a Huggingface pre-trained bert-base-uncased model as our MLM for yielding $E_{\texttt{mlm}}$ and also providing the proposal distribution in our MH MCMC sampler. For obtaining $E_{\texttt{disc}}$ , we train BERT-based classifiers on the training-set of our datasets to use as our attribute discriminators. Although we could have used any pre-trained attribute classifier from a model repository like Huggingface for $E_{\texttt{disc}}$ , we train our own classifier for controlled empirical comparison. As described later, we do use pretrained Huggingface attribute classifiers as external attribute classifiers for fair evaluation against baselines. For experiments in which we add the BertScore Zhang et al. (2020) component to the energy, we download the pre-trained roberta-large_L17 models from Huggingface, respectively. We have provided implementation details and hyperparameter ablations of all the experiments in Appendix B.6, B.7, B.8 and B.9.

B.3 Baselines

He et al. For style transfer on sentiment an formality domains, we compare our work with He et al. (2020), a generative style transfer framework which uses a variational autoencoder (VAE) built using a sequence-to-sequence LSTM-based model to do unsupervised style transfer. This framework needs to be trained from scratch for each style transfer task.

UNMT. As a second baseline for style transfer, we compare our work with UNMT Lample et al. (2018), an unsupervised machine translation framework that demonstrates high performance for sentiment transfer.

PPLM. For the task of sentiment controlled generation, we compare our work to Plug-and-Play LM (PPLM) Dathathri et al. (2020), which does attribute controlled generation using the flow of gradients from discriminators trained on the last hidden layer representations of the generator, to guide generation.

B.4 Evaluation Metrics

We use a variety of evaluation metrics to compare our approach’s performance on two major facets: (1) Quality of generated text, and (2) success on matching the target attribute used for control.

BLEU. For sentiment (Yelp) and formality (GYAFC) transfer experiments, since we have reference text, we report the BLEU score. For controlled debiasing, we report BLEU between generated text and source, and show it as BLEU (src).

BertScore. As a measure of meaning preservation, we use the F1 BertScore metric Zhang et al. (2020) to compare the semantic similarity of the provided reference sentence with the generated output.

Hamming Distance. We also report the hamming distance between the source text and generated text, to measure the extent of the change induced by our framework.

B.4.2 Attribute Quality

Internal Classifier Accuracy. To evaluate the quality of applying target attributes, we report accuracy of the internal classifier (the discriminator used for generation) on the generated text, assuming the target attribute is the correct label. The higher this accuracy is, the better.

External Classifier Accuracy. Since the internal classifier is the one we are sampling from, it is natural that we would get high accuracy on it, compared to our baselines. To create a more fair comparison, we also report classification accuracy using external classifiers, downloaded from Huggingface. For sentiment classification we use textattack/bert-base-uncased-yelp-polarity Morris et al. (2020), and for formality we use cointegrated/roberta-base-formality.

Agency Lexicon Accuracy. For the controlled debiasing experiment, we measure the accuracy of the change in agency by comparing the target agency level with that of the generated text, extracted using the connotation frames lexicon, and following the setup from Ma et al. (2020).

B.5 Hyper-parameter and Component Selection

Selection of components is based on the needs of the task and is straight forward. You add each component you need, to satisfy some condition. If you want to do sentiment controlled generation, you add a sentiment classifier. Finding the hyperparameters for each component (the multiplier in energy) is also simple, since the trade-off between the different components is clear. For instance, as shown in Table 9, increasing the discriminator score results in a more successful sentiment transfer, and increasing the Hamming score results in keeping the sentence the same.

B.6 Controllable Debiasing: Hyper parameters

For the results presented in Table 2, we ran the Gibbs chain for 8 epochs (8 iterations over all the tokens) for the conventional mode of our method, and 30 iterations for verb replacement. We used the parameters $\alpha=100,\beta=50,\theta=100$ , where $\theta$ is the coefficient assigned to the agency scorer, and $\alpha$ and $\beta$ are defined in Equations 1 and 2.

B.7 Sentiment Transfer: Hyperparameters

In this section we discuss the hyperparameters used for sampling and see the effects of each one. For the results presented in Table 3, we ran the Gibbs chain for 8 epochs (8 iterations over all the tokens), and used the parameters $\alpha=100,\beta=25$ (for Discriminator $\uparrow$ ) and $\alpha=100,\beta=50$ , for Hamming $\uparrow$ . $\alpha$ and $\beta$ are defined in Equations 1 and 2.

Table 9 shows six different scenarios, with six different coefficeints for the Disciriminator ( $\alpha$ ), BERT MLM ( $\delta$ ) and Hamming distance ( $\beta$ ) components in the energy function, which helps understand the effect each expert has.

B.8 Formality Transfer: Hyperparameters

For the results presented in Table 4, we ran the Gibbs chain for 5 epochs (5 iterations over all the tokens), and used the parameters $\alpha=140,\beta=15,\gamma=100$ (for Discriminator $\uparrow$ ) and $\alpha=140,\beta=50,\gamma=300$ , for BertScore $\uparrow$ . $\alpha$ , $\beta$ and $\gamma$ are defined in Equations 1 and 2.

Table 10 shows four different scenarios, with four different coefficeints for the BLEURT and BertScore components in the energy function, which helps understand the effect each expert has. For BLEURT, we use pre-trained Elron/bleurt-base-512 from Huggingface.

B.9 Prompts and Hyperparameters Used for Controlled Generation

We have listed the prompts that we used for controlled text generation (these prompts are taken from Dathathri et al. (2020)): the country, the lake, the chicken, the movie, the pizza, the painting, the year, the city, the book, the potato, the horse, the road, the president, once upon a time. We collect these prompts from PPLMs github repo, available at this url: https://github.com/uber-research/PPLM/tree/master/human_annotation/pplm_labeled_csvs.

PPLM has multiple knobs to tune for sampling, and after running a greed search we found that gamma=1, num_iterations=10 , step_size=0.1, kl_scale=0.01 and gm_scale=0.95 yeild the best results (reported in Table 6). We generated samples by running the command python run_pplm.py -D sentiment, with the mentioned hyperparameters. For FUDGE, we tune the $\lambda$ parameter, and we find that $\lambda=10$ works best.

For our method, we ran the Gibbs chain for 15 epochs, and used hyperparameter $\alpha=40$ , from Eq. 1. We don’t use any experts other than the yelp sentiment classifier, so we don’t have any other hyperparamters.

B.10 Human Evaluations

We used Amazon Mechanical Turk for our evaluations, where each HIT was a two choice question of “which sentence is more fluent?” and the providers were paid $\$ 0.1$ per HIT. We selected Turkers from English speaking countries. We also had each each question answered 3 times (by 3 Turkers), to create redundancy and robustness.

B.11 GPU Hours and Infrastructure

One of the main purposes of this work is to introduce a paradigm in which we re-use existing models and do not retrain. As such, we did not need GPUs for training (we finetuned two classifier for demonstration purposes, which took less than two GPU hours).

However, we do use GPUs for inference (less computationally intensive), for generating samples. We used an in-house 4GPU server (NVIDIA RTX2080), and the samplings and hyperparameter tuning took an overall of around 10-14 full days on the 4 GPUs.