DIRECTOR: Generator-Classifiers For Supervised Language Modeling

Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, Jason Weston

Introduction

Language models are becoming a powerful tool in various machine learning applications due to recent advancements in large-scale transformer models Brown et al. (2020). Standard language model training relies on maximizing log-likelihood over large training corpora yielding low perplexity next-token predictions. However, the resulting model generations still suffer from a number of problems. Biases may be amplified from those already present in the large training corpora, and toxic or otherwise unsafe language can be generated Gehman et al. (2020); Welbl et al. (2021). Current models do not appear to adequately understand the deeper meaning of their generations and frequently contradict themselves Nie et al. (2020). They are also known to produce repetitive text Holtzman et al. (2019). If one has access to data labeled with such sequence generation errors, there is also no way to use it in the standard language modeling objective. Standard training can make use of “unsupervised” data only, i.e., positive examples one would like the model to generate.

In this work, we present a new model architecture, Director, that is capable of training on both standard language modeling data, and supervised data indicating desirable and undesirable sequence generations. The model consists of an otherwise standard decoder architecture with an extra classifier head for each output token, in addition to the usual language modeling head, see Figure 1. Standard unlabeled data is used to train the language model head, while labeled data trains the classifier head with the majority of the parameters of the decoder shared between the two tasks. During decoding, the outputs of the two heads are combined to decide on the left-to-right token generations. Model training can take advantage of batch and sequence-wise parallelism, and decoding speed matches that of standard language models.

Using existing labeled datasets of toxic language and contradicting sequences, we show how Director provides safer and less contradictory generations than standard training. We also show it is superior to the commonly used reranking/rejection sampling approach, and recent guided generation techniques such as FUDGE Yang and Klein (2021) and PACER Shuster et al. (2021) – with our model providing both accuracy and speed advantages. Further, we show Director has uses even when human-labeled data is not available but an automatic procedure can be constructed. In particular, we show it can be used to minimize repetitive generations — by automatically labeling repeated sequences and training on this labeled data. Overall, we find that our model is simple, performant, efficient, and a generally applicable tool with several applications where it can provide improved sequence modeling.

Related Work

Language modeling has seen a number of impressive recent improvements by scaling model and training data size Radford et al. (2019); Brown et al. (2020), with applications in dialogue Adiwardana et al. (2020); Roller et al. (2020), QA Raffel et al. (2019) and other general NLP tasks Wang et al. (2022). Despite these advances, much research is focused on resolving issues that remain, and controlling the quality of resulting generations.

A popular class of approaches is to train the language model as standard, but then control the language model at decoding time, with perhaps the most common variant being reranking (or rejection sampling). Using a separate model to rerank candidate decodings has been used to reduce toxicity Thoppilan et al. (2022), to reduce contradictions Nie et al. (2020), or to improve performance on a given task Askell et al. (2021); Nakano et al. (2021). The advantage of such an approach is that the reranker can be trained with both positive and negative examples (or stack-ranked examples) of behavior, unlike the original language model. Reranking has also been shown to outperform reinforcement learning in language tasks, e.g. in WebGPT Nakano et al. (2021).

Another class of models is the model-guiding approaches, also referred to as controllable generation models Ke et al. (2022). Reranking models can only help if there are some good candidates from the beam decoding or sampling used to generate predictions. To exert greater influence on left-to-right token decoding, several model-guiding approaches have been proposed instead.

GeDI Krause et al. (2020) proposes to use a second separate language model to “rerank” for every left-to-right token step during decoding with respect to the difference between a control code coding for the desired attribute being present or not.

Plug and play (PPLM) Dathathri et al. (2019) proposed to use a separate simple and fast attribute classifier, such as a bag-of-words classifier, to guide generation at decoding time to change e.g., topic or sentiment. This requires forward and backward passes in which gradients from the attribute model push the language model’s hidden activations and thus guide the generation.

FUDGE Yang and Klein (2021) also makes use of a second classifier, but reranks tokens rather than computing gradients with the forward and the backward passes. FUDGE was shown to outperform several other methods, including PPLM, hence we use FUDGE as one of our main baselines. However, overall, in all these methods requiring two models instead of one makes efficiency a key issue Smith et al. (2020a), in addition to requiring more memory.

PACER Shuster et al. (2021) proposes a faster and better-performing variant of FUDGE by sampling tokens, rather than reranking all of them, and then finally reranking the entire set of candidates at the end. We thus also use this as one of our baselines. In contrast, our model Director is a unified generator-classifier and makes use of parallelism to score all tokens at each step during decoding without incurring significant costs beyond the standard language model decoding scheme.

There is also related concurrent work. Jiang et al. (2022) uses a contrastive method to reduce repetition similarly to unlikelihood training Welleck et al. (2019), but as far as we can see cannot be easily adapted to general positive and negative labeled sequences. Lu et al. (2022) proposes a way to control text generation with iterative reinforcement to deal with toxic generations or negative sentiment. It only has moderate success with repetition, perhaps because it still uses the standard likelihood training (with control variables) in its main loop, which still makes it hard to penalize certain sequences. We note that sigmoid outputs have been used recently elsewhere too, e.g. for machine translation Stahlberg and Kumar (2022).

Model

In this section, we will introduce the Director model. We will start by laying out the notation and background of language modeling and then introduce our new architecture.

Standard language model (LM) training maximizes the likelihood of the training data which is expressed by the negative log-likelihood loss. Let $x_{1:T}$ be a sequence of tokens $(x_{1},...,x_{T})$ from the training data $\mathcal{D}_{\text{LM}}$ , then the loss is factorized

We thus only need an autoregressive model that predicts the next token probability conditioned on its past context. A transformer decoder achieves this by processing all tokens in parallel while masking attention maps so a token cannot see future tokens. The decoder can also be paired with a transformer encoder so the generation is conditioned on a given context, which is useful in applications such as dialogue modeling. To generate from such models, we simply compute left-to-right the probability of the next token and then sample from that distribution (e.g., greedily, via beam decoding or nucleus sampling Holtzman et al. (2019)).

2 Supervised Language Modeling

While language models can be used to generate text, they lack a mechanism for controlling their generations. In particular, standard training cannot take advantage of negative examples even if we have supervised training data with such examples.

Let $\mathcal{D}_{\text{class}}$ be supervised training data where each token sequence $x_{1:T}$ is labeled. This is either by labeling the whole sequence with a class $y=c$ or, in the fine-grained case, each token is labeled with a class, giving $y_{1:T}$ . Then the objective is to learn to generate conditioned on a given class, which means modeling $P(x_{t}|x_{1:t-1},y_{t})$ . Using Bayes’ rule, we can write

The first term can be computed by a language model, but the second term requires a classifier that optimizes the cross-entropy loss

In methods such as FUDGE, a separate classifier is trained, but it is not efficient because the classifier needs to be evaluated for each candidate token $x_{t}\in V$ in the vocabulary at every time step $t$ .

3 Director Language Model

We thus propose Director that unifies language modeling and classification into a single model. This allows the model to be efficiently trained on both unlabeled data $\mathcal{D}_{\text{LM}}$ and supervised data $\mathcal{D}_{\text{class}}$ . Then during inference time, we can generate conditioned on the desired attributes (positive class labels).

As shown in Figure 1, input tokens are first processed by a shared autoregressive core, for which we used a transformer decoder in our experiments. Then those processed token representations are fed to two separate heads. The first is a standard LM head that is comprised of a linear layer followed by a softmax to output a multinomial distribution over the vocabulary $V$ . This LM head is trained by optimizing loss $L_{\text{LM}}$ from Equation 1.

The second head is for classification and it also maps each token representation into a $|V|$ dimensional vector using a linear layer. Then, however, it applies a sigmoid to obtain an independent binomial distributionWe used sigmoid for binary classification, but softmax could potentially be used if there are more than two classes. for each word in the vocabulary $V$ . Note that while tokens $x_{1:t-1}$ are given as inputs and processed by the shared transformer core, the next token candidates for $x_{t}$ are encoded in the row vectors of the linear layer in the classifier head. This classifier head optimizes loss $L_{\text{class}}$ from Equation 3 on samples from $\mathcal{D}_{\text{class}}$ .

where $\gamma$ is a hyperparameter weighting the classification loss. In practice, we alternatively sample a batch from $\mathcal{D}_{\text{LM}}$ or $\mathcal{D}_{\text{class}}$ and optimize the corresponding loss with backpropagation through the whole model.

To generate a sequence conditioned on a certain class $c$ according to Equation 2, we combine the outputs from the two heads to compute the probability of the next token

where $Z$ normalizes the total probability to be 1. We can also adjust parameter $\gamma$ at inference time to alter the weight of the classifier compared to the language model head, where $\gamma=0$ reverts to standard language modeling. During generation, tokens are produced left-to-right in the same manner as standard language models.

The unified architecture of Director has three features that make it efficient:

The classifier is autoregressive rather than being bidirectional, thus the computations of previous token representations can be reused for future token classifications instead of needing to process the whole sequence $x_{1:t}$ at each time step $t$ .

The classification head classifies all token candidates $x_{t}\in V$ in parallel, so we only need to run it once instead of classifying each candidate separately. Even running it once has the same computational requirement as the LM head, which is often negligible in large transformers.

The classifier shares the same core with the language model, thus further reducing additional computation.

Therefore, the computational efficiency of Director is almost the same as the language model alone, both during training and inference time.

While the classifier evaluates all candidates $x_{t}\in V$ simultaneously, only one of the $|V|$ sigmoid outputs gets trained per token because $\mathcal{D}_{\text{class}}$ contains a label for only one of the candidates. Here, we propose a way to help train all sigmoid outputs. We experiment with a regularizer where we train the remaining $|V|-1$ sigmoid outputs to be close to $0.5$ , which is achieved by an additional mean squared error loss.

Experiments

In our experiments, we employ Director to generate a response to a given context such that the response exhibits certain desirable attributes and avoids certain undesirable attributes. In our experiments, we focus on three such particular undesirable attributes: (i) toxicity, (ii) contradiction; and (iii) repetition, corresponding to three different tasks in Sections 4.2, 4.3 and 4.4.

We use standard pre-trained transformers as our baseline language models in all of our experiments. In our dialogue safety and contradiction experiments, we use the BlenderBot 400M model pre-trained on pushshift.io Reddit Roller et al. (2020). In our repetition experiments, we use GPT2 Medium Radford et al. (2019). All other models use these models as a starting point.

We fine-tune a pre-trained 300M parameter transformer model (from Roller et al. (2020)) as a reranker using the same supervised data used for other models (technically, trained as a two-class classifier). This is used to rerank the beam candidates of the baseline model.

For FUDGE Yang and Klein (2021), we use the same pre-trained 300M parameter transformer as with the reranker, but train it as a “future discriminator” (i.e., left-to-right classification), and apply that to the baseline model to rerank the top 10 tokens at each step of generation by multiplying the classification probabilities with the baseline model’s token generation predictions.

PACER Shuster et al. (2021) again uses the same pre-trained 300M parameter transformer for model guiding, again reranking the top 10 tokens left-to-right during generation. The final beam candidates are then reranked by the same model similar to the reranking approach.

2 Safe Generation Task

Safe dialogue response generation is a major area of concern that needs to be addressed before the widespread deployment of dialogue agents. It is currently very easy to goad models into producing responses that are offensive or unsafe Xu et al. (2020); Gehman et al. (2020); Welbl et al. (2021). An ideal model should be able to avoid these provocations and still generate a safe yet contextual response.

Following Xu et al. (2021) we use the pushshift.io Reddit pre-trained BlenderBot 1 model Roller et al. (2020) as our baseline, and use the Wikipedia Toxic Comments (WTC) dataset Wulczyn et al. (2017) as a set of unsafe prompts. The baseline model tends to respond in a similarly toxic fashion to the prompts themselves, mimicking two toxic conversationalists speaking to each other. Our goal is to produce a model that does not have this behavior but instead generates safe responses even when the other conversationalist is toxic. We use the training set of WTC, in addition to the safety data from Dinan et al. (2019); Xu et al. (2021), as positively and negatively labeled data to train supervised models (reranker, FUDGE, PACER, Director). Final evaluations are performed using the WTC test set prompts, and evaluating those generations using an independently trained safety classifier, as well as human evaluations.

In addition to being safe, our preferred model should also perform as well as the baseline in non-toxic conversations. We thus measure generation performance on the ConvAI2 dataset, using the F1 metric, following Dinan et al. (2020). We report all the generation quality results on the validation set as the test set for ConvAI2 is hidden.

Results for Director and the various baselines on the validation set are given in Figure 2. For several of the methods there are various configurations of the hyperparameters possible (e.g., learning rate, mixing weights, etc.) which we represent as points on a scatter plot. For each method, we have selected the best configuration that trades off classifier accuracy and generation F1, represented with a black outline. For Director safe classification accuracy can be as high as 90% without losing generation quality, while the baseline has only just over 60% accuracy. Reranking and PACER fall somewhere in between 70-80%, while FUDGE only marginally improves over the baseline. Director thus has a better trade-off than competing methods.

Final results on the test set for the selected models are given in Table 1, which follow a similar pattern to the validation set. We also repeated the experiment with a larger 3-Billion parameter model. The results in Table 4 show that similar trends hold when scaling up the underlying language model.

We performed a human evaluation comparing Director and the Baseline LM on a subset of the WTC test set, asking for a given context and response pair if each model is safe or not, and which is better (or if neither is better/they are tied). Over 150 random samples, Director has 107 safe responses, while the Baseline has only 54. Director is deemed better 67 times, while the Baseline is only better 17 times, with 66 ties. Overall, we see clear wins for Director.

3 Contradiction Task

We next consider the task of generating non-contradictory dialogue. We start with a pre-trained BlenderBot 1 model Roller et al. (2020) and fine-tune it on the Blended Skill Talk (BST) tasks Smith et al. (2020b). This fine-tuned model is used for both the baselines and to initialize the Director model.

The DECODE dataset Nie et al. (2020) provides human-labeled training data of contradictions vs. non-contradictions given prompts from the BlenderBot 1 Blended Skill Talk (BST) tasks Smith et al. (2020b)). We can thus use this data to train our supervised models, and again compare them in terms of an independently trained contradiction classifier as well as generation F1 on the ConvAI2 dataset as before. Note, ConvAI2 is also one of the BST tasks, and as with safe generation tasks, we always report the generation quality results on the ConvAI2 validation set.

Results for Director and the various baselines on the validation set are given in Figure 4. Similar to subsection 4.2, we report various configurations of the supervised models. We find that the baseline has a contradiction classifier accuracy of around 75%, which is improved by all the supervised models. Reranking and FUDGE improve to around 87%, PACER to around 90% while Director performs the best with around 97%, while having a similar generation F1 to the baseline.

Final results on the test set for the selected models are given in Table 1, which again follows a similar pattern to the validation set.

4 Repetition Control

We consider the issue of repetition in language model generation. Standard language models are known to produce degenerative text, repeating tokens and sequences from their context Holtzman et al. (2019). We use GPT2-Medium Radford et al. (2019) as our baseline model, fine-tuning on the BASE data of (Lewis et al., 2021) to predict the next sentence, and using greedy decoding during generation. We then measure F1, as before, and the number of repeating $n$ -grams in the generation (either in the generated sequence itself or a repeat of the context). We measure for $n=1,\dots,5$ and a linear combination of all of those $n$ -gram sizes which we call the Repeat Score@5 (See Appendix E). We also report the average length of the generated sequences (repeated sequences tend to be longer).

Director is trained by first generating from the GPT2 baseline model, and labeling the sequences automatically at the token level according to whether they are a part of a repeating $n$ -gram or not. This labeled data is then used to train the classifier head. After training, we then generate from our model as usual. Results are given in Table 2. We find that Director maintains similar levels of F1 to the original baseline whilst having far fewer repeating $n$ -grams, and works for different levels of $n$ -gram supervision ( $n=3$ or $n=4$ ). We also find training with all $n$ -grams (weighted up to $4$ ) provides good results as well. Results on these metrics are better than token-level unlikelihood training (UL-tok) Welleck et al. (2019) and overall similar (slightly worse) compared to sequence-level unlikelihood training (UL-seq) but without the need for a computationally expensive generation step during training. They are also similar to explicit beam blocking during decoding (last row) but without having to build this specific heuristic into the inference. We also show a Director variant with fixed generation length of 52, as baseline generations are longer on average ( $\sim$ 51 vs. $\sim$ 30). The fixed-length variant still outperforms the baseline.

5 Analysis

Example generations comparing the baseline and Director are given in Table 3 for the safety task, and in the Table 5 for the contradiction task and Table 6 for the repetition control task. In the safety task, we observe several examples where the baseline is as toxic as the initial prompt, one typical pattern being the use of the same offensive words as the prompt, although new toxic words are also used. They often look like realistic responses between two toxic conversationalists. Director on the other hand tends to choose a conciliatory tone, even given quite toxic behavior, for example “I’m sorry. I didn’t mean to hurt your feelings”. In some respects, due to their safety, these responses can look less connected to the prompt itself, and more veer towards generic or less on-topic responses compared to the (toxic) baseline, but this might be a good strategy. In the repetition task, we see clear improvements over the baseline, and also, in the shown cases, over the beam blocking heuristic. The latter still tends to repeat, but using slightly different phrases, which we do not find is as much the case in Director.

We also show the classification values per token for some examples in the Appendix Figure 7 and Figure 8. We observe problematic (toxic or repetitive) tokens receive low probability, showing that our model can make explainable generation choices.

5.2 Speed and Memory

The inference speed of the various models on the safety and contradiction tasks is shown in Figure 4. Director only has an additional classifier head per token, but otherwise is the same size model as the baseline LM, and hence generates almost the same number of samples per second. The reranker, which operates on the beam candidates, does not incur much of a slowdown either despite having to encode and classify with a separate model. However, that separate model does need to be stored in memory in addition to the generation model itself. FUDGE and PACER, similarly, require a whole separate model to be stored in memory. As these two models have to be applied to each token candidate at each sampled generation step, they are quite costly in speed and end up 8-40x slower than the baseline LM. In our experiments, we used a 300M parameter classifier model for FUDGE and PACER. We note that using larger models would make them even slower; increasing the model size further quickly becomes infeasible.

5.3 Ablations and Variations

Director shares the weights of the transformer for both language modeling and classification decisions, and standard training optimizes those weights for both heads. We can also consider freezing the whole transformer core and the language model head after language model training and only then fine-tune the classifier head using the frozen representations. This would guarantee the same language model as the baseline, and predictions would only then be altered using mixing weight $\gamma>0$ . Results for our three evaluated tasks using this approach (“frozen LM”) are given in Table 1 and Table 2. We see that this approach does not work well, as the classifier is weaker without fine-tuning the whole network. We note that one could provide more (extra) layers to the classifier head, or else choose to not share some of the last layers of the transformer, again giving more capacity to the classifier. Some preliminary experiments (not shown) indicate this can indeed give better classifier accuracies at the cost of more memory (as one has a larger effective transformer) with some reduction in speed (more layers to forward through).

We also add the explicit norm described in subsection 3.3 to Director, designed to regularize classification labels that are not specified in training sequences. Results are given Table 1 and Table 2. We see improvements in most of the tasks using this approach, indicating it should be tried in further applications as well.

5.4 How good are our evaluation classifiers?

We have used independent classifiers to evaluate the safety and contradiction accuracy of the generations of our models. But the question remains: how good are these independent classifiers themselves?

Using the human-labeled Wiki Toxic Comments and DECODE datasets, we report the evaluation classifier’s classification accuracy on the validation and test splits. Results are reported in the Appendix Figure 5. We observe performance in line with classifiers from other works Xu et al. (2021); Nie et al. (2020), and similar results on both valid and test sets. For the safety classifier, we also measure performance on both the positive and negative classes separately to verify that performance is not skewed toward one class.

Discussion and Conclusion

We have presented a new architecture for training language models which takes advantage of classical supervised learning data and techniques. Unlike the standard language model architecture and training objective, our model can use both positive and negative examples of language generations by making use of a classifier head attached to the decoder layer. This allows the model to avoid undesired generations. We show the effectiveness of this approach in three setups: avoiding unsafe, contradictory, and repetitive responses. Our approach can potentially be used in any setup where examples of undesired behavior are known, feeding these in as negative examples, opening the door to the collection of more “negative class” generation datasets, which so far is a relatively unexplored area. Our code and the experimental setup are made publicly available. Future work should investigate these applications, as well as settings that consider all these kinds of undesired behavior at once, e.g. by using a multitasking approach.

References

Appendix A Limitations

While the Director model is shown to remove some toxic, repetitive, or contradictory language, the results are not perfect, and issues still remain. We have observed in some of the experiments that the perplexity of the language modeling head does increase slightly compared to the baseline, presumably because the classification head shares the same decoder weights and both tasks cannot be modeled as well without losing some performance. Our models are relatively small compared to the largest models trained in the literature, so it is possible this would no longer be a problem if one were to scale the model further. Finally, as explained in section 3 our model requires supervised data, whereas standard language model training only requires unlabeled data. This requires extra data collection or alternative/automatic labeling techniques.

Appendix B Data Preprocessing for Safe Generation Task

Most of the dialogue in our safety training data contains just a single utterance. To train an encoder-decoder model with this data, we preprocess our data by duplicating the utterances, i.e. we use the same utterance as source and target. We also experimented with other solutions such as using an empty sequence as the source and using only the multi-turn dialog for training. We found that duplicating the sequence in a single utterance dialogue resulted in a model that performs best on the validation set.

Appendix C Model and Hyperparameter Details:

In this section, we will describe the modeling details for the baselines and Director, and the hyperparameters for each of the experiments in detail.

We use a transformer-based encoder-decoder model as the baseline generator model and the Director model. The transformer model had an embedding size of 1024 and the dimension of the fully-forward layer was 4096. We use 22 encoder layers and 2 decoder layers with 16 attention heads each and a positional embedding size of 2048. We truncated the source and the target text at the maximum length of 512 tokens. This resulted in a model with approximately 400M parameters.

C.2 Safe Generation Task

In our safety experiments, we used the 400M parameter model, finetuned on the pushshift.io Reddit dataset as our baseline. This baseline model was also used as the generator model for re-ranking, PACER, and FUDGE experiments, and to initialize the encoder-decoder model and the language modeling head for the Director model.

We used a 300M parameter transformer-based classifier model trained on safety datasets from Wulczyn et al. (2017); Dinan et al. (2019); Xu et al. (2021) as our evaluation classifier. The labels from the safety classification were mapped to one of two classes: safe and unsafe. The model was trained using the Adamax Kingma and Ba (2014) with a learning rate of $5e-5$ . We used the combined weighted F1 as our validation metric for early stopping with the patience value of 200. We used this same evaluation model as the re-ranking classifier used for the re-ranking experiments.

We also used the same model architecture, optimizer, and hyperparameters to train the left-to-right (LTR) classifier or "future discriminator". We generate left-to-right or per-step classification data by propagating the sequence-level positive and negative labels to each token in the sequence.

We initialized the Director model for safety experiments using the baseline safety model. We fine-tuned the language modeling head on the pushshift.io Reddit dataset and trained the classifier head with the same safety data that was used to train the re-ranking and LTR classifier. We ensure that during training, the classifier and generation data points are equally weighted. We used the mean of classification and generation loss as our validation measure with a patience value of 50 for early stopping. We used Adam Kingma and Ba (2014) to train the model with a learning rate of 1e-5 and batch size of 8. Our best model used $\gamma(train)=0.2$ and $\gamma(infer)=5$ and explicit label normalization coefficient, $\delta=0.5$ .

C.3 Contradiction Task

We used a 400M long-context (context length: 512) transformer-based encoder-decoder model fine-tuned on BlendedSkillsTasks Smith et al. (2020b) as our baseline. This model was fine-tuned using Adam Kingma and Ba (2014) optimizer, with a learning rate of 5e-6. We used generation F1 as a validation metric, with a patience value of 50.

The evaluation, re-ranking, and LTR classifier used the same model and hyperparameters as the safety classifiers but were trained on the DECODE Nie et al. (2020) dataset.

Similar to our safety experiments, the contradiction Director model was initialized using the contradiction baseline model. The LM head of the Director model is further fine-tuned using the Blended Skill Talk (BST) tasks Smith et al. (2020b) and the classifier head is trained using the LTR version of the DECODE Nie et al. (2020) dataset. The model was trained using the Adam optimizer with a learning rate of 5e-6. The model was validated using an unweighted mean of classifier and generator loss with a validation patience value of 50. Our best model used $\gamma(train)=0.5$ and $\gamma(infer)=1.0$ , and the explicit label normalization coefficient, $\delta=1.0$ .

C.4 Repetition Control

We use GPT-2-Medium Radford et al. (2019) fine-tuned on BASE data (from Lewis et al. (2021)). The model was optimized using Adam with a learning rate of 7e-6 and batch size of 8. We used the validation perplexity as our early stopping metric with a patience value of 10.

The Director model and both the unlikelihood baselines are initialized with the baseline model. The Director model and both the sequence-level and token-level unlikelihood models are trained using the Adam optimizer with a learning rate of 7e-6. We used the validation loss as the early stopping metric with a validation patience value of 10.

The best token-level unlikelihood model was trained with $\alpha=0.25$ . The best sequence-level unlikelihood model was trained to block 3-grams from the generated sequence with unlikelihood loss optimized for 10% of the batches.

The best Director model was trained with the objective that penalized all tokens up to 4-grams weighted by their length. The $\gamma(train)$ and $\gamma(infer)$ for this run were 0.1 and 0.8 respectively. For the variant with explicit label normalization, we use the same training and inference mixing coefficients as above and use the explicit label normalization coefficient, $\delta=1.0$ .

C.5 Impact of mixing coefficient γ𝛾\gamma during training and inference

In Figure 6, we plot various values of loss mixing coefficient $\gamma$ used during the training and inference for the safety experiments. We observe that lower values of $\gamma$ during training and higher values during inference result in safer models though the model does see a monotonic decrease in generation quality with the increase in $\gamma$ during generation. For our experiments, we choose the model with $\gamma\text{(train)}=0.1$ and $\gamma\text{(infer)}=5$ as this resulted in a very safe model without compromising too much on the generation quality.

C.6 Repetition Control Generations with fixed length

We evaluate our method further on the repetition task, in order to check that Director is not better than the baseline due to generation length. We conducted experiments on GPT2-Large generating a fixed length of 60 tokens for both the baseline and Director, training in the same way as before. In this setup, we find both models have a similar F1 (both .104). However. the baseline has a 3-gram repeat of 12.1, while Director is 1.4. We thus obtain similar improvements as in the non-fixed length case.

Appendix D Safety Experiments with 3B Reddit Model

Table 4 shows the results of the safe generation task on a larger 3-Billion parameter model. We use a 3-Billion version of the BlenderBot 1 model and train the baseline and the Director model using the same hyperparameters as reported in subsection C.2. We observe similar trends as reported in Table 1,i.e., Director outperforms all the other supervised language modeling baselines on the safe generation task while maintaining the generation quality similar to the baseline model.

Appendix E Repeat Score@5

We propose a new metric, Repeat Score@5, to capture the repetition at various n-gram levels. We compute Repeat Score@5 as

where $\text{\# cuml n-grams}=\sum_{i=1}^{5}\text{\# i-grams}$ .

Intuitively, the metric captures average numbers of tokens in the sequences that are repeated, i.e., a repeat score@5 of 21 would indicate that on average a generated sequence has the equivalent of 21 tokens that were repeated.