Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity

Introduction

The exchange of stories from one’s past, or descriptions of activities in one’s present, are a fundamental part of human discourse. Trustworthy human conversationalists keep their stories roughly straight within a conversation. An interlocutor taking on your own stories and persona as theirs is especially jarring and unnatural. However, despite the improvements in state-of-the-art open-domain dialogue modeling, both in terms of distributional accuracy metrics like perplexity, and subjectively in terms of human judgements (Adiwardana et al., 2020; Roller et al., 2021), interactions with those agents reveal that they cannot keep their stories straight. In particular, they are likely to take on the role of their interlocutor; for example, if an agent’s partner says they are a software engineer, the agent is likely to say it is a software engineer too (Roller et al., 2021), or worse, appropriate their partners just told tale of a trip to NAACL as their own. Some example failure cases are given in Table 1, where models incorrectly take on the name, role or activities of their partner instead of their assigned role. These failures are related to the general problems of repetition in language models (Holtzman et al., 2020), the weak influence of word order (Sinha et al., 2021) and inability to avoid contradictions (Nie et al., 2021).

In this work we formalize and quantify this behavior, show that to some extent it can be detected automatically with a specifically trained classifier, and then study a wide variety of mitigations. These include multi-objective training, unlikelihood training (Welleck et al., 2020), classifier-assisted re-ranking based generation, and several forms modifying the attention mechanisms of the decoder in a sequence to sequence model. Our best methods can reduce mistaken identity issues by 65%, while simultaneously improving in-conversation engagingness; indeed, our models that can stick to their role in conversation are judged by humans to be significantly more engaging than their baseline counterparts. Despite these advances, we find that there is still considerable space to improve these results further in future work.

We make publicly available both our trained models and code to reproduce resultshttps://parl.ai/projects/light_whoami/.

Related Work

Much recent work has explored training open-domain dialogue models on large and small dialogue corpora, with the former imbuing raw conversational ability and the latter providing necessary conversational skills. Most crowd-sourced datasets require acting out a role to some capacity in conversation (though indeed Mazaré et al. (2018) study extraction of roles from raw data). Some involve providing persona lines that a model must assume throughout the conversation (Zhang et al., 2018; Dinan et al., 2020; Xu et al., 2021); others require more subtle "roles", such as a listener (Rashkin et al., 2019), or a teacher and student (Dinan et al., 2019; Gopalakrishnan et al., 2019; Zhou et al., 2018; Komeili et al., 2021).

A common paradigm in the state of the art of open-domain dialogue involves concatenating all relevant contextual information as input to a sequence to sequence neural model (e.g., transformers (Vaswani et al., 2017)) to obtain a conditioned response. Such models can yield human-like and engaging responses (Adiwardana et al., 2020; Roller et al., 2021). Nevertheless, various consistency issues still plague such models. Recent studies have indicated that hallucination of incorrect knowledge is still far from a solved issue (Shuster et al., 2021; Santhanam et al., 2021), with some proposing specific datasets and tools for measuring precisely the levels of this undesired attribute (Liu et al., 2021). Another clear example of failure is the short-term memory of state-of-the-art models (Xu et al., 2021), sometimes due to the lack of long-form training data or long-context models but often due to simply the modeling itself.

To address consistency issues, a variety of methods have been explored. In the context of knowledge-grounded dialogue, different ways to attend most effectively over provided contextual information have been explored (Zheng and Zhou, 2019; Ye et al., 2020; Prabhumoye et al., 2021; Wang et al., 2019). These works find that considering factual documents separately (in some capacity) improves model grounding. We explore such methods, but in the context of character identity.

Another general problem is that of contradictions. Nie et al. (2021) collect a dataset of contradictions in dialogue, and train classifiers that help re-rank model outputs at inference time; Li et al. (2020) explore unlikelihood training (Welleck et al., 2020) to reduce repetition and contradiction, among other undesired traits, in model generations. The character identity issue we study in this work can be seen as an important class of contradictions, but to the best of our knowledge, has not been explicitly focused on.

Methods

We consider a two-party chat setting. The context provided to a model includes: (i) the name of its character and the partner’s character; (ii) an extended description of its own character; (iii) and, information about the area in which the conversation takes place. The responsibility of the model is to engage its conversational partner, with no other goal prescribed; however it should stay within character and within the bounds of the defined setting.

We operate in the context of LIGHT (Urbanek et al., 2019), consisting of grounded fantasy role-playing game conversations. The LIGHT environment involves humans and models interacting with thousands of objects in hundreds of locations, all while assuming the roles of one of hundreds of characters. The dataset consists of roughly 8.5k dialogues spanning 111k utterances. It is an ideal setting for this study because of the rich and varied personas with explicit backstories.

To quantify the character identity problem, we take a state-of-the-art dialogue agent (specifically, BlenderBot (Roller et al., 2021)) fine-tuned on the LIGHT dialogue dataset and ask human annotators if the agent mistakes its identity based on its utterances in context. The agent conditions its response on the LIGHT context and prior utterances in the dialogue history. We see in Table 4 that in roughly $6.5$ percent of utterances the model mistakes its identity; this corresponds to a mistake in approximately $35$ percent of conversations.

BlenderBot uses a Byte-Level BPE tokenizer (Radford et al., 2019); an artifact from the BlenderBot pre-training is that it only considers 128 such tokens in the past, and thus has no mechanism for recovering truncated information about the LIGHT context in later conversational turns. Our second baseline lengthens the input context to 1024 BPE tokens, which allows the entire context for every example to fit into the truncation length of the model; we follow methods employed in Xu et al. (2021) to extend the positional embeddings of the model. We see in Table 4 that this actually makes the problem worse, resulting in $7.4$ percent of utterances with mistaken identity (corresponding to a failure in approximately $38$ percent of conversations).

2 Measuring Role-Playing Accuracy: RPA

We first define a metric, role-playing accuracy (RPA), to denote how often a model’s responses are “in-character”; by this, we mean how often the model’s response could feasibly be said by their character, given their assigned character identity. Measuring RPA is a non-trivial task for a variety of reasons. First, some conversations involve pairs that can reasonably say similar things (priest vs. priestess, man vs. woman, wizard vs. witch). Second, opening lines are often more generic (“hello”, “how fare your travels today”), so either character can say it in conversation. The third reason stems from the data that we study; we are relying on crowdsourced data in which humans are required to portray their characters. Some crowdworkers may be better than others, and there may be some noise in the dataset in which, e.g., a horse may proclaim its love for a queen, or a knight may discuss at length the kingdom’s tax collecting.

We thus train models specifically designed to identify whether a candidate response from a model fits the model’s role. We employ poly-encoder transformers (Humeau et al., 2020) to learn this metric, and structure the task as a ranking one; the model receives the LIGHT setting and prior utterances of dialogue as input, as well as the response currently under consideration, and the model must choose the correct character from a fixed set of candidates.

We also explore RPA classifiers trained on all partially complete sequences of labels, such that the classifiers can determine the character speaking without requiring the full utterance; we call these left-to-right (LTR) RPA classifiers. Further details about how our RPA classifiers are built is given in Appendix 10.

3 Mitigations

In this section we describe several strategies for improving the role-playing accuracy of dialogue agents, specifically ways to improve our transformer baselines.

We can employ an RPA classifier in response generation by using it to rank candidate model outputs.

Given a set of candidate responses, the RPA classifier can re-score the set and return the response yielding the highest probability of staying in character (according to the RPA score on the complete candidate generations). The dialogue models employ beam-search to generate responses, and the candidates for re-ranking are the beams within beam-search. We also try nucleus sampling (Holtzman et al., 2020) and delayed beam-search (Massarelli et al., 2020) to see whether more diverse candidates have any affect.

Re-ranking only the final beam candidates may well be suboptimal because it is well known that those candidates are not very diverse (Kulikov et al., 2019), meaning there may not be any good candidates to choose from in this final set. In order to generate utterances that agree with our classifiers, a possible improvement is to generate the utterance such that partial generations also agree with the classifier when generating left-to-right, ensuring that good candidates are surfaced. With access to LTR RPA classifiers, we can apply re-ranking to partial sequences.

Unfortunately, re-ranking at every step of beam search, for every token requires significant computation, such as in the recent FUDGE method (Yang and Klein, 2021). FUDGE re-scores tokens at each decoding step by multiplying the classifier probability with each token probability, and renormalizing, which is used for control tasks with lightweight classifiers in order to be tractable.

In our proposed approach, called PACER, we re-score candidate tokens, for each beam, according to the probability that their inclusion yields the appropriate character classification, and then finally re-rank the complete candidate beams. To make this efficient, we crucially score only a small proportion of decoding steps (e.g., 5% of token positions) as well as for only a few candidate re-scored tokens (e.g., top 10 only). We can control these hyperparameters to explore the speed vs. accuracy trade-off.

3.2 Unlikelihood

We explore utilizing an unlikelihood (UL) loss (Welleck et al., 2020) to force the model to stay in character during training. Unlikelihood training works as a counter to the standard maximum likelihood (MLE) training of language models; while MLE training pushes the model to generate the correct tokens, UL training pushes the model to not generate incorrect tokens.

While training on the LIGHT dataset with standard NLL loss, with some fixed probability we consider a candidate model generation for UL loss. The full generation is sent to the RPA classifier; if the generation is classified as coming from the incorrect character, we examine each partial generated sequence of the output, and send these sequences to the LTR RPA classifier to determine whether the candidate partial sequences match the model’s character. We apply UL loss to tokens that yield the wrong character classification.

3.3 Multi-objective Training

The RPA classifiers utilize the LIGHT setting and prior utterances of dialogue history to determine which character generates a candidate response. We hypothesize that the generation models themselves should be able to pick out and utilize these components as well. However, the RPA classifier models are trained explicitly for this task, whereas the seq2seq models are trained only to generate a plausible continuation of a dialogue history.

We thus explore a setup in which the generation models are trained to identify the speaker of an utterance as well. To do this, we use the output representations from the model (either encoder + decoder, or decoder only) as inputs to $n_{MO}$ additional transformer layers, where we vary $n_{MO}\in\{0,2\}$. The final outputs are used to compute a character score, similarly to the RPA classifier.

The model can then be trained piece-wise. After initializing the model weights with those trained on the LIGHT response generation task, we then train only the extra layers with only the character classification objective; once the classifier achieves suitable performance on the task, we can begin to back-propagate the character classification objective multi-tasking with the dialogue task itself to the generation model directly, in the hope that the model learns to update its internal representations of the context and/or the decoded response.

3.4 Expanded Decoder Attention

Maintaining identity relies on the model’s capacity to understand which inputs from the conversational history are pertinent when generating a continuation of the preceding dialogue. In a standard, open-domain chit-chat scenario, the model has free reign to decide which elements of the context it would like to condition on when generating a response, as we are dealing with a nearly unconstrained output space (so long as the output follows plausibly from the input). In LIGHT, however, we want to emphasize certain components of the context more so than others; specifically, when role-playing as a character, we want the model to always be reminded of its role, so that it can conditionally generate an optimal response while staying in character. In this lens, one can view the task as “grounding" on one’s character information when conversing.

Inspired by models demonstrating good performance in knowledge-grounded dialogue (Zheng and Zhou, 2019; Ye et al., 2020; Prabhumoye et al., 2021; Wang et al., 2019), we propose a simple extension to the transformer seq2seq architecture, specifically the decoder, to ensure the model knows to condition on the profile. The standard transformer decoder first uses self-attention over the decoded response, and then cross-attention over the encoder outputs. We add a third attention step, expanded attention, that attends again over an extracted subset of the input context (encoded separately from the normal context). We explore various subsets of the context to determine which are most important for both RPA and other automated metrics, and call this method “Profile” grounding as the subsets generally include the character and role description. We utilize the exact same (shared) parameters for both the normal cross-attention and the expanded attention; thus, model size is not affected.

Instead of directly telling the model what to re-attend to, we also explore whether the model can learn to do this automatically, based on its own (or other) representations of the context. The first method we consider is examining the decoder attention weights. Specifically, we use the attention weights from the decoder over the full context to choose $k$ tokens to re-attend to. This operation is done on a per-layer basis, and thus allows different decoder layers to re-attend to (potentially different) components of the input.

The second method we consider is a trainable mask; this involves feeding the encoded context through a “mask” layer to select various tokens to re-attend to. Specifically, we feed the context through a linear projection layer followed by a softmax to select the top-$k$ tokens. This set of tokens is then re-encoded by the encoder and fed to the decoder as the expanded attention context.

Finally, we explore using the classifier attention weights over the context from the RPA classifier itself. Intuitively, the RPA classifier has learned what components of the input are necessary for determining which character is speaking; if we look at these attention weights when considering the model’s character, we know what the classifier thinks is important to use.

We also consider combining expanded attention with re-ranking methods, or with multi-objective training, to see if the combination can improve results. For the latter we use the automated grounding trainable mask method.

Experimental Results

We first assess the quality of our RPA classifiers. We experiment with either 0, 4, or All prior context utterances, for both the standard and left-to-right (LTR) models, and either using the LIGHT context or not. We measure hits@1/427, where the model must correctly identify the character speaking out of 427 characters from the validation set, or hits@1/2 between the two speakers.

Results are given in Table 3, where each model is evaluated on the datasplit on which it is trained; Table 11 in the Appendix shows results across all splits for all models, in which we see that each model performs best on the split on which it is trained when compared to the others, and that our best models can be relatively successful on this task. Including the LIGHT context improves RPA performance nearly across the board; with no context and no utterances of dialogue history, the models can only accurately identify the character less than 20% of the time.

Results for the LTR RPA classifiers are in Table 3. We experiment with a 4-utterance LTR model; the LTR classifiers perform nearly as well as the full classifiers on the full datasplit, and outperform them quite handily on the LTR split. Given the robustness of the LTR RPA classifiers, we use this model for computing RPA throughout the remaining results, unless otherwise specified.

2 Baseline Generation Performance

We next train baseline models for the dialogue generation task itself. Performance on the LIGHT dataset test split for our baseline models can be found in Table 4, with results on the validation set in Table 17 in the Appendix. While lengthening the context from 128 to 1024 tokens yields better perplexity, the model obtains worse F1 scores and does not improve much if at all on role playing ability, both when measured by the RPA classifiers and in human evaluations (see also Table 19). Further detailed training and optimization specifications are given in Appendix A.

3 RPA Re-ranker Performance

Table 4 gives results for RPA-based re-ranking of generation models. Automated results show a slight bump in F1 on the LIGHT valid set, and indeed a bump in RPA. Including the intra-generation re-ranking with PACER yields an even higher RPA score. Table 5 contains the results of varying the candidate tokens re-ranked per intra-generation step (#Toks) and number of partial re-ranking steps (Freq), both in terms of generation metrics/RPA and relative computational cost compared to re-ranking. Increasing # of toks or increasing the frequency can lead to improved F1 and RPA, but with significant latency increase for too high values (e.g. over 11x when applying re-ranking for every partial step using the top 10 tokens each time). Applying both partial and final complete ranking helps performance.

We note that for re-ranker models the same model is both re-ranking outputs, and used to measure RPA afterwards, making that metric biased. Hence, human evaluations are required for this model, which will be detailed in Section 4.7, which will indicate that re-ranking does in fact help.

4 Unlikelihood

Results of unlikelihood (UL) training are also given in Table 4. We apply UL loss to the 128-truncation model in two different ways: (1) Top-1: apply the loss on the token that yields the most incorrect partial sequence RPA classification; (2) All: apply the loss to all tokens that yield an incorrect RPA classification on partial sequences. The RPA UL methods suffer compared to the baselines in terms of PPL and F1, yet they retain similar RPA metrics. We hypothesize that while the UL loss can adjust the model to refrain from generating out-of-character responses, there are still far too many other tokens that may yield similar outcomes that are not penalized. Table 12 in Appendix D includes similar results with the 1024-truncation model.

5 Multi-Objective Training

Multi-objective training results are in Table 6, where the base model is a 1024-truncation model. We measure generation metrics in terms of RPA (with PPL and F1 in Table 13 in Appendix E), and classification metrics in terms of Hits@1/427 as before. The model is able to predict the appropriate character using either the decoder outputs or the encoder+decoder outputs. For each case, $n_{MO}=2$ yielded better results than $n_{MO}=0$. Interestingly, despite the relatively strong performance of the model in classifying the right character (87.42 hits@1 for the best model), this does not translate to substantial RPA improvements over the baseline.

6 Expanded Attention

Expanding the decoder attention yields significant gains across all automated metrics, as seen in Table 7 for a 1024-truncate model (and in Table 15 in Appendix F for a 128-truncate model). As a baseline we explore simply re-attending to the full context again; this indeed improves metrics across the board for the short-context model, but the long-context model actually suffers. However, both models improve substantially over the baseline when including the full LIGHT context without the dialogue history, and attention over sub-components of the LIGHT context still yields strong improvements.

To see how much this expanded attention matters, we explored varying the number of rounds $r\in\{1,2,3\}$ of expanded attention, i.e., how many times the model attends to this additional context. In Table 7, we also see that a second expanded attention round yields even better results, but performance drops off after applying a third round.

We show results for the automated grounding of expanded attention in Table 8. Attempting to use the decoder attention weights to select expanded attention context yields no additional benefits, which is not surprising: if the model could identify the pertinent components of the input beforehand, it would not require a re-attention. The trainable mask does not yield any benefits either. However, using the RPA classifier attention weights to inform the model which tokens to re-attend to yields improved performance across all three metrics compared to the baseline, and PPL is nearly the same as profile grounding (12.19 vs. 12.18), while RPA trails slightly behind (91.11 vs. 91.79). We also include the usage of the bottom-$k$ tokens from the classifier weights to emphasize that there is indeed signal from the top-$k$, as using the bottom tokens does not help.

Table 6 shows that combining automated grounding with the multi-objective task yields higher hits@1 compared to not using the trainable mask, especially in the first stage of multi-objective training. However, RPA scores are only fractionally better than the baseline. Appendix E includes results across more settings (see Table 13 and Table 14).

The expanded attention and RPA re-ranker methods can also both be applied to obtain effective models. Results are in Table 4; indeed, the combination yields the highest F1 and RPA scores.

7 Human Evaluations

We performed human evaluation on our models. For each model we collected 100 human-model conversations, set up similarly to the original LIGHT dataset conversations. During the conversation, crowdworkers were asked to annotate if the model’s response: 1) contained a contradiction; 2) displayed a mistaken identity; 3) indicated a mistaken location; 4) was off-topic; 5) was repetitive; or 6) was all good. At the end of the conversation, the crowdworkers were asked to rate their partner’s engagingness on a scale from 1-5.

Results for mistaken identity, engagingness, and all-good responses are in Table 4, with full results (and comparison with a retrieval baseline) in Table 19 in the Appendix. The baseline model displays mistaken identity 6.45% of the time, and has an average engagingness score of 4.04. Longer context increases engagingness to 4.16 but also increases mistaken identity. Unlikelihood and multi-objective training similarly increase mistaken identity. The successful methods, then, are the beam re-ranking methods and the expanded attention models. The long-context beam re-ranker decreases mistaken identity to 4.81%, while the profile expanded attention model improves to 4%, and has the best engagingness of 4.34. Combining RPA Re-ranking with expanded attention yields the lowest mistaken identity (2.38%), while adding PACER leads to the highest all-good percentage (85.7%). Correlations between automatic metrics and human evaluations are measured in Appendix K, where we find that RPA and mistaken identity are indeed strongly correlated.

Qualitative Analysis

We further explored three decoding settings: standard beam-search, delayed beam search (Massarelli et al., 2020) and nucleus sampling (Holtzman et al., 2020), both in a re-ranking setting and not. When considering performance on automated metrics (provided in Table 20 in the Appendix), we see that generation settings other than beam search, when using a re-ranker, yield lower F1 scores but higher RPA scores, as the RPA re-ranker has more diversity of candidate responses from which to choose; however, these methods perform worse in human evaluations, with nucleus sampling re-ranking yielding far more problems and far lower engagingness ratings. Qualitative analysis of outputs on the test set in Appendix J.1.

2 Left-to-Right Dynamic Classification

We find that the left-to-right RPA classifiers are correctly sensitive to per-token perturbations in the input, and can accurately predict the speaker at the token level. In Table 9, we give an example where the classifier changes its character prediction, depending on the candidate utterance.

3 Classifier Failure Modes

We note that the human dialogue data is classified as being “in character” only 92.8% of the time on the validation set by the LTR RPA classifier. We examine the scenarios in which the classifier is incorrect, with example input/output pairs in Table 21 in the Appendix. First, there are instances where either character could have said the output response (row 1). Second, there are instances where there are not enough clues in the context to provide an estimation of who said the response, for example at the beginning of the conversation (row 2). And, there are still some small amount of instances that the classifier simply fails (row 3).

4 Per-Turn Character Accuracy Analysis

We consider the RPA of various models when evaluated across the turns of conversation. Intuitively, baseline models would suffer as the conversation goes on for a variety of reasons (character roles are truncated out of context, more input yields noisier outputs, etc.). In Figure 1, we display the per-turn results for a few representative models, with Figure 3 in Appendix L enumerating over a more comprehensive set. The human outputs are most often correct on the first turn, with gradual RPA decay throughout the conversation. The 128-truncate baseline, as expected, suffers a dramatic performance drop after the first couple of turns. Meanwhile, with the profile expanded attention, we see near-human performance, with better RPA in later turns. Including RPA re-ranking improves dramatically over all turns.

5 Expanded Attention Visualization

To gain some insight into what is happening with the expanded attention, we mapped out the attention between context and response tokens for both a baseline model with no expanded attention, and a model with profile expanded attention. Figure 4 and Figure 5 in the Appendix display the heat maps for an example context and response. Information about the heat map construction and the specific example used are in Appendix M.

The baseline model spreads its attention out across both the LIGHT context and the dialogue history, with the majority of the attention looking at overlapping words in the context and the response and almost no attention on the character names. The expanded attention model concentrates the first attention on the recent dialogue history heavily in the first level of attention, and then concentrates on pertinent words in the context related to the character information (i.e., the character names) in the second round of attention.

Discussion & Conclusion

In this work we explored the problem of maintaining one’s character in open dialogue, and showed that state-art-of-the-art models have a fundamental weakness in this regard. We provided a clear framing of the problem and showed one can build automatic metrics (RPA) that evaluate models using a classifier. We then explored a variety of methods throughout this paper; we offer a brief discussion of the most crucial takeaways here.

The RPA classifiers are effective in both assigning identities to unknown speakers, and measuring the role-playing effectiveness of candidate models. Those robust to partial sequences are even more effective as they can be utilized in tools requiring such partial classification, including unlikelihood training and re-ranking methods.

The alternative training methods did not yield improved results, as hoped. Unlikelihood training does not improve RPA. Multi-objective training shows that we can predict the character from the model’s internal representations, but does not seem to improve downstream generation results. It remains an open problem how to integrate improvements into training.

The straightforward application of additional re-attention to relevant contextual inputs yields substantial gains across the board, both in automated metrics and human evaluations. Intuitively this makes sense; we are telling the model that it cannot decode its response without first being reminded of its character. Automated attempts to choose this context using the RPA classifier yielded improvements in RPA compared to the baseline, re-affirming that the RPA classifiers pay attention to pertinent tokens in the input; in human evaluations the method displayed lower frequency of mistaken identity as well, though not as low as profile grounding.

The re-rankers help improve RPA measurements and indeed show stronger performance in resolving mistaken identity issues in human evaluations. As these methods are model-agnostic, we can combine with expanded attention, which yields the highest RPA metrics and lowest frequency of mistaken identity issues, demonstrating its effectiveness. PACER performs well and may be suitable for other tasks beyond the focus of this paper.

Our best methods still lag behind human (crowdworker) performance in several regards, e.g. 1.34% vs. 2.23% in terms of mistaken identity per turn, or 5% vs. 14.7% per conversation. Therefore considerable progress still has to be made on this challenging problem.

References

Appendix A Training Details

All models are trained with the ParlAI framework (Miller et al., 2017).

RPA classifier Poly-encoders are initialized with the 622M parameter models from Roller et al. (2021); we also use this architecture for dialogue response (retrieval) models which we also evaluate (see Table 19). All generative models are initialized with BlenderBot, also from Roller et al. (2021), a 2.7B parameter transformer encoder/decoder model. Each model was pre-trained on 1.5B training examples from pushshift.io Reddit (Baumgartner et al., 2020), with BlenderBot additionally fine-tuned on the BST tasks (see Roller et al. (2021) for more details), before training on LIGHT.

The RPA classifier models are trained with a cross-entropy loss over the correct label, with 99 random negatives chosen from the training set; we ensured that each character in conversation showed up in the set of candidate labels. The models were trained with a batch size of 16 on 4 32GB GPUs, with early stopping on the validation set according to valid accuracy. We used the Adam optimizer (Kingma and Ba, 2015) with weight decay (Loshchilov and Hutter, 2019), sweeping over learning rates $\{1e-5,5e-6\}$.

All variants of generative models were trained using 8 32GB GPUs, with early stopping on perplexity on the validation set. We used the Adam optimizer, sweeping over learning rates $\{1e-5,7e-6\}$, training with a batch size of 128 for the short-truncation models, and 32 for the long-truncation models. For the multi-objective models, we used the same loss (and negative-sampling) setup as the RPA classifiers for the character accuracy objective. During inference, unless otherwise specified, we generated using beam-search with beam size of 10, enforcing a minimum length of 20, and with tri-gram blocking with respect to both the context and the current generation.

Appendix B RPA Classifier Training

We build the training data for the RPA classifiers from the LIGHT dataset. The input is a concatenation of (1) the LIGHT context (set of characters, setting, etc.); (2) a fixed number of previous utterances in the conversation; and (3) a candidate utterance from any point later in the conversation (a special token separates the candidate utterance from the prior context). We experiment with either 0, 4, or $N-2$ prior utterances (dubbed “All” in relevant tables), where $N$ is the total number of utterances ($N-2$ allows the last turn for each speaker to be a candidate utterance). The left-to-right (LTR) data split is built similarly, except each example $i$ becomes $w_{i}$ examples, where $w_{i}$ is the number of tokens in the candidate utterance for example $i$. Statistics of the training dataset are given in Table 10.

Suppose we choose $n$ as the number of prior utterances to include in the input, and let us denote $D=8538$ to represent all the dialogues in the LIGHT train split, and $U=110877$ to represent all the utterances in those dialogues. For the RPA classification dataset, each dialogue is presented twice, once from each character’s POV. When $n=N-1$, where $N$ is the length of a conversation, then we have roughly $2D$ training examples. When $n=0$, we have roughly $2U$ training examples.

For any value $0<n<N-1$, we build out several examples from several slices of each conversation. Suppose we have dialogue $d_{i}$ with $N$ utterances $\{u_{0},u_{1},...,u_{N}\}$. To build the training data from dialogue $d_{i}$, we select all continuous subsets of $n$ utterances within $d_{i}$, forming contexts

Then, we look at all $N-i$ utterances following utterance $u_{i+n}$, and use these as target utterances in the task. The goal of this is to build the model to be robust to dataset artifacts; without this modification, the model could trivially pick out the character just by looking at the number of alternating utterances. These measures force the model to fully understand the task and react accordingly.

Appendix C RPA Classifier Performance: Additional Results

In Table 11, we see how each RPA classifier performs on the various datasplits, varying the number of prior utterances used during training and evaluation. Each model performs best on the split on which it was trained (the highlighted numbers).

Appendix D Unlikelihood: Additional Results

In Table 12, we compare UL models across different truncation lengths; the same story applies to the 1024-truncation models. We additionally include a third method, Random-3, where we apply the loss randomly to 3 tokens that yield incorrect RPA classifications. This method performs about the same as the Top-1 method, but the RPA is lower, indicating that the Top-1 method at least is providing some signal.

Appendix E Multi-Objective: Additional Results

Table 13 displays full PPL and F1 scores corresponding to the models in Table 6.

E.2 Multi-Objective + Automated Grounding

In Table 14, we see additionally how, when using either the encoder+decoder or just the decoder outputs, we do not require additional multi-objective layers (as we did in the non-automated-grounding case).

Appendix F Expanded Attention: Additional Results

We provide results for both the 128-truncate and 1024-truncate models with profile grounding in Table 15. Trends remain the same for both models.

Appendix G Full Valid Results

Table 17 includes results on the LIGHT validation set for models in Table 4.

Appendix H Retrieval Re-rankers

We evaluated a Poly-encoder baseline model with an RPA re-ranker as well. The Poly-encoder scores utterances from the full training set as candidates, and the candidates for re-ranking are the top-$k$ ranked utterances; results are in Table 18. Retrieval models benefit dramatically from the re-ranking, improving to almost 99% RPA as measured by the LTR classifier. As the candidate responses for retrieval models come from the set of all training utterances, and due to overlap between the set of characters appearing in the train and valid sets, we can examine how often the model output was originally spoken by its partner’s character; this can be seen as a proxy for mistaken identity. We find that the re-ranker reduces the amount of time that the model returns a message its partner said, indicating some viable and promising results.

Appendix I Full Human Evaluation Results

In Table 19, we display the full results of human evaluations across all dimensions. We note that the Poly-encoder model is best at not mistaking location or being repetitive, but this is expected given its retrieving over human-written utterances.

Appendix J Generation Settings

We provide qualitative analysis of the various generation methods below.

When examining the baseline with no re-ranking, we found that nucleus sampling can help when beam search does not work; however, both can go out of character the farther one goes in conversation.

The beam outputs in standard beam search are at times too similar, in which case re-ranking does next to nothing, unless a viable response is available.

Using nucleus setting in a re-ranking setup yields more diverse choices to choose from; however, sometimes the model simply does not address *any* character within the conversation.

This strikes a nice balance between sensible outputs from beam search and diversity from nucleus sampling.

Using mixed decoding (re-ranking several decoding schemes) can work quite well, as it is a nice blend of different generation methods.

J.1.1 Turn Annotation Analysis

Qualitative analysis of the turn annotation results are in Table 16. We generally found that beam search fails the vast majority of the time when the model thinks that it is talking to itself; i.e., it confuses its partner for its own character. The re-rankers can help shift the hallucination away from this regime.

J.2 Automated Metrics

We experiment with various generation settings, with or without re-rankers; results are in Table 20. For the baseline and re-ranker models, beam search yields the highest F1 scores; RPA can be improved with the other inference methods when combined with a re-ranker. We believe this may be due to the higher diversity of candidate responses generated from those methods.

Appendix K Human + Automatic Eval Correlation

We analyze the correlation between human annotations and the automatic metrics collected on the LIGHT validation set, as shown in Figure 2; we note some interesting trends:

perplexity appears to be positively correlated with mistaken identity, and negatively correlated with engagingness. So, perplexity is a good indicator of how fluent and engaging the model is in conversation, and can indirectly point to a better understanding of the role-playing task. An important note is that we only tested this amongst models of the same size, and only for the models we tested, so it is not clear that larger models will necessarily bring improvements.

F1 word overlap is positively correlated with engagingness as well, so F1 may be a good proxy of model performance. Correlation with mistaken identity is negative here, implying that better F1 corresponds with better role-playing ability. However, we note that F1 is not a catch-all metric (Liu et al., 2016).

RPA appears to be strongly negatively correlated with mistaken identity, indicating that it is indeed a good measure of the model’s ability to stay in character. It is weakly negatively correlated with the other issues, and is somewhat positively correlated with engagingness as well. These correlations give us confidence that our RPA classifiers are adequately measuring role-playing ability within models.

Appendix L Per-Turn Analysis, Expanded

In Figure 3, we see RPA results across turns of conversation for a wider variety of models.

The human outputs are most often correct on the first turn, with gradual decay of accuracy throughout the conversation (according to RPA).

The vanilla baseline suffers a pretty dramatic drop off after the first couple of turns; the long-context model achieves slightly higher character accuracy overall but we see similar drop offs farther down the conversation.

The unlikelihood models seem to recover somewhat in the initial turns of conversation, however later turns still yield sharp drop offs in RPA.

Similarly to the UL case, we see the most gains in initial turns compare to the vanilla baselines; however, we see even more dramatic drop offs towards the end of the conversation.

With profile grounding, we see near-human performance, with even better performance towards the end of the conversation. The automatic grounding improves over the baseline but is slightly worse than profile grounding. Combining automated grounding with multi-objective training leads to some benefits in earlier turns, but later turns still suffer.

Although we’re using the same RPA classifier to both re-ranker and score the model outputs, it is still interesting to examine on which turns the re-ranker benefits the model the most. We see in the last set of graphs that beam re-ranking seems to be most helpful in later turns, where other models generally drop off in efficacy.

Appendix M Expanded Attention Visualization

To build the heat maps in Figures 4 and 5, we look at the maximum attention applied per-head, and the maximum weight applied across the model decoder layers; other combinations were considered (mean per-head, mean over layers or last layer) and yielded similar findings.

The speaker is the mermaid, whose partner is a sea-witch. The last utterance from the sea-witch is, “What are you doing on the turquoise shore?”. The mermaid responds, “I’ve been catching waves with the dolphins all morning. What kind of victims do you expect to find in a tranquil place like this?”