Consistent Dialogue Generation with Self-supervised Feature Learning

Introduction

The notion of speaker consistency is attracting growing interest in neural response generation research Li et al. (2016b); Luan et al. (2016); Zhang et al. (2018a); Gao et al. (2018). When interacting with an open-domain neural conversation agent, users may expect the agent to develop the dialogue with consistent information, mitigating the user confusion and improving engagement. Speaker consistency presents two aspects: topic consistency and persona consistency. Topic consistency reflects the model’s ability to maintain dialogue topics such as sport, movie or music without getting sidetracked. Persona consistency envisions the agent as human-like, endowed with a relatively invariant individual personality, style of engagement (e.g., enthusiasm and casualness) or personal profile (e.g., place of residence).

Generating appropriate responses with these characteristics is a major challenge (Figure 1). Li et al. (2016b); Luan et al. (2017) and Al-Rfou et al. (2016) use persona embeddings as the additional input to train end-to-end conversational agents. Obtaining accurate persona embeddings as in Li et al. (2016b) however requires many thousands of utterances per persona, and targeted test personas may not always be found in the training data. End-to-end systems are often trained from social media data in which only a small spectrum of personas (casual speakers) is represented and professional roles (e.g. customer service) may be underrepresented, thus limiting deployment. Typically, moreover, the objective is to maintain consistency of both persona and topic throughout the dialogue, rather than inject specific personas/topics in responses. Under these scenarios, learning and leveraging persona or dialogue topic in a data-efficient and unsupervised way becomes crucial.

We present a self-supervised approach that uses the natural structure of conversational data to learn and leverage topic and persona features. Our proposals include:

1) A discriminative feature extraction mechanism that captures conversational topics and personas in a self-supervised manner, without requiring specification of speaker identity, thus allowing massive unlabeled datasets to be utilized while protecting sensitive user information.

2) Use of binary features and a disentangling loss to improve interpretability of learned features. This affords flexibility to activate or deactivate specific features when generating responses.

3) Leveraging a controllable text generation mechanism to force generated responses to adhere to high-level features such as topic and persona encoded in the controlling signal.

Related Work

Self-supervised as a subdomain of unsupervised learning, has been applied to representation learning for image, video and audio Denton and Vighnesh (2017); Doersch et al. (2015); Owens and Efros (2018). However, to the best of the authors’ knowledge, the application of self-supervision in conversational agents is rare. Borrowing definitions from other domains, self-supervised approaches in NLP make use of non-textual signals that intrinsically correlate with the text to supervise the text feature learning Denton and Vighnesh (2017).

Persona-aware response generation

Welleck et al. (2018) suggested a natural language inference (NLI) approach to improve the persona consistency, however additional labels are required. Zhang et al. (2018a); Qian et al. (2018) use explicit personal profiles as side information to guide response generation. Such information, however, may not always be available. Other work proposes injecting either emotion Zhou et al. (2018) or functional control Ke et al. (2018) into dialogue generation. As in Li et al. (2016b), learning to leverage the controlling signal in order to bias generation may require significant amounts of labelled data.

Topic-aware response generation

Leveraging topic modeling in response generation has been explored by several prior works Xing et al. (2017); Wang et al. (2017); Wu et al. (2018). Our approach differs from these methods in that we focus on learning discriminative features that help distinguish a topic or person from another. Also, our method employs a neural sentence encoder to capture richer features than the bag-of-words features that the conventional topic models opt for.

Interpretable and controllable generation

Controllable text generation Hu et al. (2017) has been employed in text style transfer and many other tasks Ficler and Goldberg (2017); Asghar et al. (2018); Ghosh et al. (2017); Dong et al. (2017). This helps disentangling high-level style information from contextual information such that the style information can be independently manipulated to produce text with different styles. Related to our work, Zhao et al. (2018) considered discrete latent actions to learn a human-interpretable representation for task-oriented dialogue systems.

Proposed Approach

The proposed approach use additional unsupervisedly learned features to generate response utterances that reflect these features. We elaborate two major components of the proposed approach: a feature extractor trained to extract topic/persona features from each utterance; and a response generator that takes the extracted features as input to generate responses accordingly.

Let $d^{(i)}\triangleq[u_{1}^{(i)},u_{2}^{(i)},\cdots,u_{T}^{(i)}]$ denote the $i$-th dialogue session in dataset $\mathcal{D}$, where $u_{j}^{(i)}$ is the $j$-th utterance and $T$ is the number of turns in this dialogue session. We assume that each dialogue $d_{i}$ only consists of the utterances between two speakers, interleaving with each other. Suppose the first $K$ ($K<T$) turns of each dialogue are revealed, our aim is to generate the remaining $T-K$ turns of the dialogue that are consistent with the observed context.

2 Discriminative feature extraction

Inspired by Denton and Vighnesh (2017), we adopt a self-supervised discriminative training scheme where we design a neural model which includes an explicit feature extraction layer as illustrated in Figure 2 and formulate a discriminative task to train the model. When training is done, the feature extraction layer yields relevant features for the associated task. In this section, we introduce two discriminative tasks to capture two types of sentence features, respectively: 1) topic-specific features ($\mathcal{T}$) that characterize conversation topics. 2) persona-specific features ($\mathcal{P}$) that reflect speaker characteristics.

In order to build a topic feature extractor with self-supervision, we rely on the assumption that utterances from the same conversation session are likely to share similar topics. Thus, we formulate a surrogate task to identify if two random sentences $s$ and $t$ from $\mathcal{D}$ belong to the same dialogue session. Specifically, when they come from the same dialogue session, i.e.$s,t\in d_{i}$, we assign 1 to target $y$ and 0 otherwise. We optimize the cross-entropy objective:

where $\mathcal{T}(\cdot)$ denotes the topic-specific feature extractor (shared among all sentences), and $\mathcal{M}(\cdot,\cdot)$ represents a matching network detecting whether the two feature vectors $\mathcal{T}(s_{i})$ and $\mathcal{T}(t_{i})$ belong to the same dialogue session. We use a 3-layer convolutional neural network (CNN) followed by a non-linear mapping for $\mathcal{T}(\cdot)$ to produce an $L$-dimensional vector. For the non-linear mapping, we explore two options: 1) We employ a sigmoid function to produce a soft-binary representation, i.e.$\mathcal{T}(x)\in(0,1)^{L}$. 2) We compute a hard-binary representation by taking 1 if it is positive and 0 otherwise, i.e.$\mathcal{T}(x)\in\{0,1\}^{L}$. This non-negative bounded representation lends itself well to interpretation and control of each component of $\mathcal{T}(x)$. For instance, we can activate or deactivate a certain topic or persona by simply turning on and off the corresponding component. For the matching function, we apply a sigmoid function to the inner-product of two feature vectors, i.e.$f(\mathcal{T}(s),\mathcal{T}(t))=\sigma(\mathcal{T}(s)\cdot\mathcal{T}(t)/\tau)$, where $\tau$ is a hyperparameter to scale the $\mathcal{T}(s)\cdot\mathcal{T}(t)$.

Persona feature extractor

Here we consider extracting persona features in a broader sense of current speaker’s status related to emotion Zhou et al. (2018), personality, tone and function control Ke et al. (2018). Note that we are only interested in maintaining consistency of any emerging persona, rather than characterizing a full spectrum of persona features. The only difference between the topic ($\mathcal{T}$) and persona ($\mathcal{P}$) feature extractors is how the positive and negative sample pairs for training are created. In the persona feature extractor, the positive pairs ($y=1$) or negative pairs ($y=0$) are the utterances from the same or different speaker within a dialogue, aiming to eliminate the topic information from the persona features. Ideally, the two speakers in a dialogue are discussing the same topic. Under this assumption, since the utterances in a negative pair are also from the same dialogue, they are like to share the same topics. Thus, the model is forced to learn the features that can capture different personas of the two speakers.

Unlike Li et al. (2016b), where each speaker is assigned a single speaker embedding vector, in our proposed method the utterances by one speaker can have different feature vectors as the manifestations of the underlying persona embedding in a different context. Nevertheless, the discriminator objective encourages these vectors to be similar since they refer to the same person. We believe that our approach is more data-efficient than Li et al. (2016b) because the former allows borrowing information from a wider range of speakers. In Li et al. (2016b), information borrowing only happens to the speakers who are similar to the current speaker in persona embedding space. As a result, persona embeddings can be poor for those not based on many dialogues. Our method, on the other hand, can leverage those speakers who share any specific features with the current speaker and is able to learn more robust representations of speakers because we aggregate personal traits across all users. However, Li et al. (2016b) complement our methods nicely in that it does not require dialogue history as the seed to initiate the first several turns.

Interpretable features

We considered two methods of making the learned features more interpretable: 1) feature vector disentanglement Cogswell et al. (2015); 2) feature vector binarization Zhao et al. (2018).

First, we employ a decorrelation (DeCorr) loss inspired by Cogswell et al. (2015), who introduced a DeCov loss to regularize deep neural networks. Specifically, we add an additional term in the objective function when training the topic and persona feature extractors:

where $||\cdot||_{F}$ represents the matrix Frobenius norm, and the $\texttt{diag}(\cdot)$ operator represents diagonalization of a matrix. $F$ denotes the feature extractor, and can be either $\mathcal{T}$ or $\mathcal{P}$. $M$ is the correlation matrix of $F$, computed from the current batch of data. Note that achieving a reasonable estimation of the correlation matrix requires a relatively large mini-batch size. The resulting final objective for the discriminator is $\mathcal{L}_{\texttt{xent}}+\lambda\mathcal{L}_{\texttt{DeCorr}}$, where $\lambda$ is a balancing hyperparameter.

Second, alternatively, we also consider binary feature vectors, where a straight-through (ST) estimator is used for the gradient calculation Bengio et al. (2013); Shen et al. (2018). Suppose the binary feature $F$ is rounded from a probability vector $p$, ST estimator back-propagate through the hard threshold by approximating the gradient $\partial F/\partial p$ as 1. We empirically found that setting $\mathcal{M}$ to use the inner product of $F(s)$ and $F(t)$ fails. We presume the reason may be that the value of the inner product between two binary vectors can only take integers from $[-L,L]$ which limits the representation power of the model. We therefore concatenate $F(s)$ and $F(t)$ and passing it through a multi-layer perceptron (MLP) to predict the matching label $y$. Interchangeability is still loosely maintained as the pair $(S,T)$ is randomly swapped when feeding into the discriminator.

Utterance pair construction

One issue in constructing the positive/negative pair for the feature extractor is that the number of positive/negative pairs need to be balanced to achieve a robust empirical result. Moreover, when constructing the positive sample pairs with $y=1$, if the $s$ and $t$ are adjacent or close to each other in a dialogue, we might end up capturing adjacency pairs Sacks and Schegloff (1973) rather than conversation topics. For example, $s=$ ’How are you?’ and $t=$ ’Fine. How are you?’. The captured similarity in feature space of this $s,t$ pair is contextual appropriateness rather than topic/persona consistency. To alleviate this, we collect only those pairs that are more than 4 turns away from each other for the positive sample pairs.

We note also that the persona features may affect the topic feature extractor because the persona features can be weak signals for predicting whether two sentences are from the same dialogue. One remedy is to select utterances from different speakers within a dialogue session when constructing the positive pairs for the topic extractor to eliminate as much as possible the effect of the persona features. However, this remedy can result in fewer positive pairs. Empirically the topic extractor works well even without this remedy, presumably because the strong signal from topic overwhelms the weak signal from persona.

3 Generator design

The conditional multi-turn generator that produces neural responses given the $K$-turn source sentences is shown in Figure 3, which is conceptually related to Serban et al. (2016). During training time (Figure 3 left panel), each source sentence is first encoded by a 3-layer CNN encoder, which shares the same architecture as the feature extractor, followed by a context aggregator ($AGG_{C}$) layer that summarizes all sentence embedding vectors $[c_{1},c_{2},\cdots,c_{K}]$ into one single context vector $C$ with the same dimension as $c_{i}$. In this paper, the $AGG_{C}$ layer is designed as first concatenating $[c_{1},c_{2},\cdots,c_{K}]$ and applying a fully-connected layer to map the resulting vector to $C$.

On the other hand, the target sentence is processed by the feature extractors to produce feature vector(s) $F$ as described in Section 3.2. The feature extractors are fixed in the response generator since we observed fine-tuning the feature extractor leads to suboptimal empirical results. The context vector $C$ and feature vector(s) $F$ are fed into an MLP to generate a fixed-length initial hidden variable $H_{0}$. This is followed by a series of long short-term memory (LSTM) units as the decoder, where $H_{0}$ is employed as input in each time-step.

Controllable objective during training

In the experiment, we applied the slope-annealing trick Chung et al. (2016), and set $\tau=0.01$ which works well in practice. The final training objective for the generation is $\mathcal{L}_{\texttt{MLE}}+\eta\mathcal{L}_{\texttt{cycl}}$.

Testing time

At test time, as shown in Figure 3 (right panel), the feature vectors from the source sentences $[u_{1},\cdots,u_{K}]$ are first collected by applying feature extractors $F(\cdot)$. We denote the feature vectors for the source sentences as $[f_{1},\cdot,f_{K}]$. We apply a feature aggregator $AGG_{F}$ layer to estimate the output feature vector $F$, which is further fed into the LSTM-RNN for the generation. Different from the context $AGG_{C}$ layer, we consider a weighted-sum aggregation function for the feature $AGG_{F}$ layer Other possibilities of such an $AGG_{F}$ layer exist, such as mean, max or concatenation (as in $AGG_{C}$). We choose weighted-sum for the $AGG_{F}$ layer due to its superior empirical performance comparing to alternatives., i.e., $F=\sum_{k=1}^{K}w_{k}f_{k},s.t.\sum_{k=1}^{K}w_{k}=1$, where $w_{k},k=[1,2,\cdots,K]$ are linear interpolation weights learned during training time, where a Euclidean distance between predicted target feature and target feature is optimized, i.e.$\text{argmin}_{w}\mathcal{L}_{p}=||f_{k+1}-\sum_{k=1}^{K}w_{k}f_{k}||^{2}$. For the persona feature, we only use the source sentences of the current speaker, thus all $w_{k}$ where $\texttt{mod}(k,2)=\texttt{mod}(K,2)$ is set as zero. Intuitively it can be perceived as the attention of each utterance. We note that more complicated attention mechanisms can further improve the model; however, we leave these for future work, since this paper focuses on the utilization of dialogue features rather than improving the multi-turn S2S structure in general.

Experimental setups

We evaluate the proposed methods on two datasets. All experiments are conducted using single Nvidia Tesla V100 GPU. The source code will be released.

We consider two datasets. For both we use a (80%, 10%, 10%) split for training, validation and test respectively.

Training data was extracted from the Twitter FireHose covering a five-year period from 2012 through 2016.Deleted tweets and closed accounts were removed. From this set, we collected total 6,658,385 8-turn dialogues where two participants chatted with each other.

Maluuba data

The Maluuba dataset consists of 40,389 dialogues with 11 turns. Each dialogue is a task-oriented conversational interaction between two real speakers regarding 51 domains and 242 tasks, collected by crowd-sourcing where one crowd worker simulates a user and another simulates a chatbot.

2 System specifications

The dimension of the LSTM hidden layer is set at 500. We use ADAM as the optimizer with learning rate 0.0001. The hyperparameters $\lambda$ and $\eta$ are set at 0.01 and 0.1, respectively. For the dimension of feature vectors we use 100. For Maluuba dataset we use a 50% dropout rate in each of the CNN layers and the $\lambda$ is set to $0.1$. The hyperparameters are selected to maintain the discrimination accuracy while reducing as much $\mathcal{L}_{\texttt{DeCorr}}$ as possible.

For evaluation, we consider three variants of our COnsistent CONversation (CoCon) models: CoCon-T: CoCon model with topic-consistency; CoCon-TP: CoCon model with topic-consistency and persona-consistency; CoCon-TP-bin: CoCon model using binary features with topic-consistency and persona-consistency. We compared our models with two baselines: a vanilla sequence-to-sequence model (S2S) and persona model (Persona) Li et al. (2016b). We implement the persona model by reusing the encoder and decoder architecture in our approach. For Twitter dataset, we map all users with fewer than 88 utterances as unknown (86% of the total training samples) and in the test set (for all compared methods) we eliminate conversation sessions with unknown users. This yields 50k total users. We use the same number of feature dimensions for all systems compared. All modules are trained until convergence.

Results

We used equal numbers of positive/negative examples to train each feature extractor. For Twitter dataset, the resulting accuracies for topic and persona feature extractor are around $0.75$ and $0.60$ (for both continuous and binary features), respectively. For Maluuba dataset, the discriminator accuracy for persona and topic feature extractors are $0.85$ and $0.67$, respectively. With the disentangling loss ($\lambda=0.01$), the correlation between features drops from 0.25 to 0.16.

Representative n-grams for some learned feature units for Twitter dataset are shown in Table 1. To calculate the feature vector for a specific n-gram, we average over the feature vector of test sentences that contain that n-gram. We then select the top-ranked n-grams with occurrences greater than 200 for each feature bit. We observe that when $\lambda=0$, i.e.without disentangling loss, the learned features exhibit heavy colinearity, which weakens the interpretability of each separate feature units.

We further visualized the topic features on both datasets using t-SNE embedding Maaten and Hinton (2008). For Maluuba dataset, Figure 4 illustrates the learned topic and persona feature embeddings on the test set. Without any label information, the learned topic and persona features separate well. For twitter dataset, we observed that the persona features of the utterances from different time zones form some clusters, indicating the features learned from our approach can partially reflect the difference in societal groups (See Appendix B).

Sampled response generation

We evaluate our approaches by generating the next response given 4 contextual seed source sentences. Some sampled results are shown in Table 2. We observed that the CoCon-T and CoCon-TP in general are able to produce informative responses which seem to be more consistent with the theme of the given context comparing with baselines. For CoCon-TP, beyond being context-aware, the responses seem to be persona-aware, i.e., mimicking the tone and personal wording preferences like mate, oh my gosh, haha, ain ’t and other words associated with them.

Feature manipulation

For Maluuba dataset, we provide sampled responses of S2S and CoCon-TP in Table 8 in Appendix 5. The context is given as 4 turns of dialogues and the task to generate all remaining 7 turns. It can be seen that during free generation, the S2S model tend to generate looping responses like thanks - you ’re welcome and is generally less informative. However, our proposed CoCon-TP approach can generate reasonably well by unsupervisedly capturing the topics of the context and role of each turn.

Automatic evaluations

In our quantitative evaluations we test both relevance and diversity metrics. For relevance, we adopt BLEU Papineni et al. (2002), METEOR Denkowski and Lavie (2014), NIST Doddington (2002) and three embedding-based metrics Greedy,Average,Extreme following Serban et al. (2017); Rus and Lintean (2012); Mitchell and Lapata (2008); Forgues et al. (2014). To evaluate diversity, we follow Li et al. (2016a) to use Dist-1 and Dist-2, which is characterized by the proportion between the number of unique n-grams and total number of n-grams of tested sentence. We also include the Entropy (Ent-n) metric Zhang et al. (2018b); Gao et al. (2019), which does not depend on the size of test data. The results of automatic evaluations are shown in Table 4 (Twitter) and Table 5 (Maluuba). For both dataset, the CoCon-TP model achieves best relevance score, while the CoCon-TP-bin outperforms other methods in diversity.

Human evaluations

We evaluated 500 randomly sampled test sources from Twitter dataset using crowd-sourcing provided by a contracting service. Systems were paired and each pair of system outputs was randomly presented to 4 judges, who ranked them for topic consistency, persona consistency, informativeness and relevance using a 5-point Likert scale. Overall judges’ preferences for the topic consistency, persona consistency, given as a percentage of total judgments are shown in Table 6. A strong overall preference can be observed for CoCon-TP over the other systems evaluated. We also evaluated for relevance and informativeness, with CoCon-TP showing similar preference gains. Further details, including the human evaluation template used, are provided in the Appendix.

Conclusion

We present a self-supervised feature learning framework to abstract high-level latent representations of topic and persona information underlying the dialogue context and leverage these representations to generate more consistent dialogue in a controllable manner. For future work, investigating the variance reduction strategies for controllable text generation would presumably improve the controllablity of the feature units. Besides, combining and aligning supervised and unsupervised features would potentially enable better feature learning and interpretability. Our approach can be adapted to facilitate style transfer and long-form text generation Guo et al. (2018); Zhang et al. (2017) to improve the generation consistency.

References

Appendix A Straight-through LSTM (ST-LSTM)

In the forward calculation, the $t$-th ST-LSTM unit takes the previously generated word $w_{t-1}$, hidden state $H_{t-1}$ and $H_{0}$ as input, and generates the next word $w_{t}$ that maximizes the probability $p_{t}$ over the vocabulary set. That is, the argmax operation is used. However, in the backward calculation, the gradient of $\partial w_{t}/\partial p_{t}$ is calculated as a constant $1/\tau$ where $\tau$ is a temperature parameter. Presumably, this approach delivers a biased but smooth gradient signal while maintaining the forward computation exact.

Appendix B Twitter additional results

Figure 5 shows the t-SNE embedding of topic features for speakers in different time zones, demonstrating our self-supervised approach learned reasonable representation to separate different users. Here we use 2000 testing utterances with speakers from 4 different time zones (500 for each).

Figure 6 shows that $\mathcal{L}_{\texttt{cycl}}$ decreases faster when $\eta$ is larger, indicating the additional controlling loss $\mathcal{L}_{\texttt{cycl}}$ can be effectively reduced using Straight-through training.

Table 7 shows some additional results on controlling the bit, in addition to Table 3.

Appendix C Maluuba results

We provide some generated samples for maluuba dataset in Table 8. All compared models use first 4 turns as seed and generate the remaining 7 turns by taking 4 previous previous turn use ground truth for first 4 and generated utterances for the rest turns as context.

Appendix D Human evaluation

Human evaluation was conducted using the form shown in Figure 7. The two response candidates were presented in random order to the judges, who used a Likert scale to indicate their preferences. To make the questionnaire less abstract to judges, persona was evaluated in terms of which response better reflected the tone and style of Person A as observable in the prior turns. The distributions of judgments for each of the questions are shown in Tables 9 through 12.