DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks

Introduction

Recently, deep neural networks have achieved state-of-the-art results in various tasks including computer vision, natural language and speech processing. Specifically, neural sequence-to-sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) have led to great progress in important downstream NLP tasks like text summarization (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; Tan et al., 2017; Yavuz et al., 2018), machine translation (Cho et al., 2014; Sutskever et al., 2014; Luong et al., 2015; Bahdanau et al., 2015), and reading comprehension (Xiong et al., 2017). However, achieving satisfactory performance on dialogue still remains an open problem. This is because dialogues can have multiple valid responses with varying semantic content. This is vastly different from the aforementioned tasks, where the generation is more conveniently and uniquely constrained by the input source.

Although neural models appear to generate meaningful responses when trained with sufficiently large datasets in the chit-chat setting, such generic chit-chat models reveal several weaknesses that were reported by previous research Serban et al. (2016); Vinyals and Le (2015). Most common problems include inconsistency in personality, dull and generic responses, and unawareness of long-term dialogue context.

To alleviate these limitations, we turn our focus on a different problem setting for dialogue response generation where the model is provided a set of relevant textual facts (speaker persona descriptions) and is allowed to harness this knowledge when generating responses in a multi-turn dialogue. To handle the personality inconsistency issue, we ground our dialogue generation model on external knowledge facts which are a list of persona descriptions in our application (Li et al., 2016a; Zhang et al., 2018). We explicitly use the dialogue history as memory for the model to condition on which potentially encourages a more natural conversation flow. Towards encouraging generation of more specific and appropriate responses while avoiding generic and dull ones, we use a hierarchical pointer network in our model such that it can copy content from two sources: current dialogue history and persona descriptions.

In this work, we propose a novel and general architecture DeepCopy that extends the attentional sequence-to-sequence model with a hierarchical pointer network that enables the decoder to jointly attend and copy tokens from any of the facts available as external knowledge in addition to the dialogue context (encoder input). This is achieved entirely in an end-to-end fashion through factoring the whole copy mechanism into following three hierarchies/components: (i) a token-level attention mechanism over the dialogue context to determine the probability of copying a token from the dialogue context, (ii) A hierarchical pointer network to determine the probability of copying a token from each fact, and (iii) An inter-source meta attention over the input sources dialogue context and external knowledge, which combines the two copying probabilities. Using these components, a single copying probability distribution over the unique tokens appearing in the model input is computed exploiting the well-defined hierarchy among them. In addition, the model is equipped with a soft switch mechanism between copying and generation modes similar to See et al. (2017), which allows us to softly combine the copying probabilities with the decoder’s generation probabilities over a fixed vocabulary into a final output probability distribution over an extended vocabulary. We empirically show the effectiveness of the proposed DeepCopy model compared to several baselines including Ghazvininejad et al. (2018); Zhang et al. (2018) on ConvAi2 challenge.

Related Work

Earlier work on data-driven, end-to-end approaches to conversational response generation treated the task as statistical machine translation, where the goal is to generate a response given the previous dialogue turn (Ritter et al., 2011; Vinyals and Le, 2015). While these studies resulted in a paradigm change compared to earlier work, they do not include mechanisms to represent conversation context. To tackle this problem and have a better representation of conversation context as input to generation, (Serban et al., 2016) proposed hierarchical recurrent encoder-decoder (HRED) networks. HRED combines two RNNs, one at the token level, modeling individual turns, and one at the dialogue level, inputting turn representations from the token-level RNNs. However, utterances generated by such neural response generation systems are often generic and contentless (Vinyals and Le, 2015). To improve the diversity and content of generated responses, HRED was later extended with a latent variable that aims to model the higher level aspects (such as topic) of the generated responses, resulting in the VHRED approach (Serban et al., 2017).

Another challenge for dialogue response generation is the integration of knowledge into the generated responses. (Liu et al., 2018) extracted facts relevant to a dialogue from knowledge using string matching, named entity recognition and linking, found additional entities from knowledge that are most relevant to the facts by a neural similarity scorer, and used these as input context features for the dialogue generation RNN. (Ghazvininejad et al., 2018) used end-to-end memory networks to base the generated responses on knowledge, where an attention over the knowledge relevant to the conversation context is estimated, and multiple knowledge representations are included as input during the decoding of responses. In this work, we use end-to-end memory networks as a baseline.

Although much research has focused on response generation in a chit-chat setting, models trained on large datasets of human-human interactions of diverse speaker characteristics often tend to generate responses which are too vague and generic (common for most speakers) or inconsistent in personality (switching between different speakers’ characteristics). Recently, (Zhang et al., 2018) presented the ConvAi2 challenge containing persona descriptions and over 10K real human chit-chats where speakers were required to converse based on their assigned persona. Li et al. (2016a) learned speaker persona embeddings from a single-speaker setting (e.g. Twitter posts) or a speaker-address style (human-human conversations) to generate personalized responses given a single utterance input. Another related work Raghu et al. (2018) applies hierarchical memory network for task oriented dialog problem. In this work, we compare our model with (Zhang et al., 2018) which uses a memory-augmented sequence-to-sequence response generator grounded on the dialogue history and persona.

Model

In this section, we first set up the problem, and then briefly revisit the baseline models using memory networks Sukhbaatar et al. (2014) and pointer-generator networks (See et al., 2017). Subsequently, we introduce the proposed DeepCopy model with a hierarchical pointer network and our training process.

Let $\mathbf{x}=(x_{1},x_{2},\ldots,x_{n})$ denote the tokens in the dialogue history. The dialogue is accompanied by a set of $K$ relevant supporting facts, where $\mathbf{f}^{(i)}=(f^{(i)}_{1},f^{(i)}_{2},\ldots,f^{(i)}_{n_{i}})$ is the list of tokens in the $i$-th fact. Our goal is to generate the response as a sequence of tokens $\mathbf{y}=(y_{1},y_{2},\ldots,y_{m})$ using the dialogue history and supporting facts. Note here that we are not interested in retrieval/ranking based models Weston et al. (2018) which rely on a set of candidate responses. Generative models are essential for this problem because we want to incorporate content from new facts during inference which may not be present in the training set. Hence, using a predefined set of candidates may not ensure high coverage.

2 Baseline Models

In this section, we describe several baseline response generation models including the ones from existing work Ghazvininejad et al. (2018); Zhang et al. (2018) and the in-house ones we propose as additional baselines.

In a sequence-to-sequence model with attention Bahdanau et al. (2015), a sequence of input tokens is encoded using an LSTM encoder. At decoder step $t$, the decoder state $h_{t}$, a context vector $c_{t}$ and the previous decoder output $y_{t-1}$ are used together to output a distribution over a fixed vocabulary of tokens obtained from the training set using a non-linear function. The context vector $c_{t}$ is an attention-weighted combination of the encoder outputs. In the following baseline models, we use different features as inputs to the encoder. The underlying model remains the same.

Seq2Seq + NoFact. Only the dialogue context tokens $\mathbf{x}$ are used as input to the encoder. Seq2Seq + BestFactContext. We select the fact $\mathbf{f}^{(c)}$ whose tokens have highest unigram tf-idf similarity to the dialogue context tokens. $[\mathbf{x}||\mathbf{f}^{(c)}]$ is then used as input to the encoder, where $||$ denotes concatenation. Seq2Seq + BestFactResponse. We select the fact $\mathbf{f}^{(r)}$ whose tokens have highest unigram tf-idf similarity to the ground truth response. $[\mathbf{x}||\mathbf{f}^{(r)}]$ is used as input to the encoder. The aim of this experiment is to have a better understanding of the effect of fact selection on response generation, since using the ground truth for fact selection is not fair.

2.2 Memory Network

MemNet. This is equivalent to the model used in Ghazvininejad et al. (2018), described above. This is essentially a sequence to sequence model without attention at every decoder step, except using the combined summary $\hat{u}$ to initialize the decoder. MemNet+ContextAttention. At each decoder step, the decoder state attends over the encoder outputs and obtains a context vector $c_{t}^{(c)}$. This is equivalent to Seq2Seq + NoFact model from Section 3.2.1, except using the fact summary $\hat{u}$ to initialize the decoder state. MemNet+FactAttention. At each decoder step, we use the decoder state to attend over the value embeddings $(m_{1},m_{2},\ldots,m_{K})$ corresponding to facts, and obtain a context vector $c_{t}^{(f)}$. This model is similar to the generative profile memory network Zhang et al. (2018), where we apply attention only on facts, and we set the decoder’s initial state to the combined summary $\hat{u}$. MemNet+FullAttention. This model employs attention over both facts and dialogue context at each decoder step. The two attention modules are combined by concatenating $c_{t}^{(c)}$ and $c_{t}^{(f)}$ Zoph and Knight (2016).

2.3 Seq2Seq with Copy Mechanism

Seq2seq models can only generate tokens present in a fixed vocabulary obtained from the training set. Pointer-generator network See et al. (2017) extends the attentional sequence-to-sequence model Bahdanau et al. (2015) by employing a pointer network Vinyals et al. (2015). It has two decoding modes, copying and generating, which are combined via a soft switch mechanism, allowing it to copy tokens from source in addition to generating from vocabulary. We report the results for the following additional baselines obtained by equipping the corresponding Seq2Seq model in Section 3.2.1 with copy mechanism: Seq2Seq + NoFact + Copy, Seq2Seq + BestFactContext + Copy, Seq2Seq + BestFactResponse + Copy.

3 DeepCopy with Hierarchical Pointer Networks

Pointer-generator network See et al. (2017) can only copy tokens from the encoder input. In this section, we present our proposed DeepCopy model that extends pointer-generator network See et al. (2017) using a novel hierarchical pointer network. Our model allows copying tokens from multiple input sources (facts $\mathbf{f}^{(i)},1\leq i\leq K$), besides the encoder input (dialogue context $x$).

Copying from Dialogue Context. Similar to our baseline models, we encode the dialogue context tokens $\mathbf{x}$ (Equation 4) and apply attention to the encoder outputs at a decoder step $t$ (Equation 5). This outputs attention weights $\alpha^{(x)}_{t}$ and a representation of the entire context $c^{(x)}_{t}$. The attention weights are aggregated to obtain the distribution over context tokens $p_{t}^{(x)}(w)$ (Equation 6),

Copying from Facts: Hierarchical Pointer Network. We introduce the hierarchical pointer network (Figure 2) as a general methodology for enabling token-level copy mechanism from multiple input sequences or facts. Each fact $\mathbf{f}^{(i)}$ is encoded (Equation 7) to obtain token level representations $\mathbf{s}^{(f)(i)}$ and overall representation $e^{(f)(i)}$. The decoder state $h_{t}$ is used to attend over token level representations (Equation 8) and the overall fact-level representations of each fact (Equation 9) by

to compute the probability of copying a word $w$ from facts as

Inter-Source Attention Fusion We now present the mechanism to fuse the two distributions $p_{t}^{(x)}(w)$ and $p_{t}^{(f)}(w)$ representing the probabilities of copying tokens from dialogue context and facts respectively. We use the decoder state $h_{t}$ to attend over dialogue context representation $c^{(x)}_{t}$ and overall fact representation $c^{(f)}_{t}$ (Equation 11). The resulting attention weight $\gamma_{t}^{\prime}=[\gamma_{t},1-\gamma_{t}]$ is used to combine the two copying distributions as shown in Equation 12.

Similar to Seq2Seq models, the decoder also outputs a distribution $p^{\text{vocab}}_{t}$ over the fixed training vocabulary at each decoder step using the overall context vector $c_{t}$ and decoder state $h_{t}$. Having defined the copy probabilities $p^{\text{copy}}_{t}$ for tokens that appear in the model input, either the dialogue context or the facts in external knowledge source, we combine $p^{\text{vocab}}_{t}$ and $p^{\text{copy}}_{t}$ using the mechanism outlined in See et al. (2017), except we use $c_{t}$ defined in Equation 11 as the context vector instead.

To better isolate the effect of copying, a key component of the proposed DeepCopy model, we also conduct experiments with MultiSeq2Seq model that incorporates the knowledge facts in the same way (by encoding each fact separately with LSTM, and attending on each by the decoder as in Zoph and Knight (2016)), but relies completely on generation probabilities without a copy mechanism.

4 Training

We train all the models described in this section using the same loss function optimization. More precisely, given a model $M$ that produces a probability $p_{t}(w|y_{<t})$ of generating token $w$ at decoding step $t$, we train the whole network end-to-end with the negative log-likelihood loss function of

for a training sample $(\mathbf{x},\mathbf{y},\{\mathbf{f}^{(i)}\}_{i=1}^{K})$ where $\mathbf{\Theta}$ denotes all the learnable model parameters.

Experiments

In this section, we describe the details of dataset, training process, evaluation metrics, and the performance results of DeepCopy model in comparison to proposed and existing baselines.

We perform experiments for our problem setup on the recently released ConvAi2 conversational AI challenge dataset, which is an extended version of PersonaChat Zhang et al. (2018). The conversations in ConvAi2 are obtained by asking a pair of crowdworkers to chat with each other naturally based on their randomly assigned personas (from a set of 1155 personas) towards getting to know each other. Personas are created by a different set of crowdworkers, and they consist of ~5 natural language sentences, each describing an aspect of a person that can range from common hobbies like "I like to play basketball" to very specific facts like "I have a pet parrot named Tasha", reflecting a wide range of different personalities. The dataset contains ~11000 dialogues with ~160000 utterances, and 2000 dialogues with non-overlapping personas are used for validation and test. For our setting, we use personas as external knowledge sources that models can ground on while generating responses.

2 Training and Implementation Details

In all the models explored in this paper, we set the dialogue context to concatenation of the last two dialogue turns separated by a special CONCAT token. The models are supplied with the persona facts of the side generating the response at the current turn, while the persona of the other side is concealed. We use a vocabulary of 18650 most frequent tokens and all the remaining tokens are replaced with a special UNK token. Embeddings of size 100 are randomly initialized and updated during training. We set the size of LSTM hidden layer to 100 for both encoder and decoder. The encoder and decoder vocabularies and embeddings are shared. A shared LSTM encoder is used for encoding both dialogue context and facts of external knowledge source. The model parameters are optimized using Adam Kingma and Ba (2015) with a batch size of 32, a fixed learning rate of 0.001. We apply gradient clipping to 5 when its norm exceeds this value. During inference, we generate responses by employing a beam search of width 4. Our models are implemented in TensorFlow Abadi et al. (2016).

3 Main Results

In this section, we present the experimental results in terms of both automatic measures and human evaluation.

In Table 1, we present our results in comparison with the existing and proposed baseline models. We report the performance of each model across several metrics commonly used for evaluation of text generation models including perplexity, corpus BLEU Papineni et al. (2002), ROUGE-L Lin and Och (2004), CIDEr Vedantam et al. (2014).

As expected, Seq2Seq + BestFactResponse model and its +COPY version outperform all the other models across all the evaluation metrics. This model pinpoints the importance of selecting the most suitable fact in the persona for the response to be generated at each turn, justifying our underlying motivation for conducting this experiment as highlighted in Section 3.2.1. However, the most suitable fact for the response is not available in the real application scenario, where the models are responsible for picking the useful pieces of information pertaining to the current dialogue turn to generate meaningful responses. Our proposed Seq2Seq + BestFactContext model and its +COPY version, on the other hand, are valid baselines for this scenario where the best fact is selected completely based on the dialogue context without relying on the ground-truth response. This model outperforms the previously proposed memory network based model MemNet Ghazvininejad et al. (2018) for knowledge grounded response generation on all the evaluation metrics, demonstrating its effectiveness despite the fact that it does not have access to all the facts unlike Ghazvininejad et al. (2018). However, this approach has the following potential weaknesses: (i) if the best persona fact selected w.r.t dialogue context is wrong (irrelevant) for the ground-truth response, the generated response might be drastically misinforming, and furthermore it is difficult for model to recover from this error because it has no access to other facts, (ii) selecting the best fact w.r.t dialogue context based on tf-idf similarity may result in poor fact selection when the lexical overlap between context and response is small which might be a common case especially for the ConvAi2 dataset as the focus of conversation may often change swiftly across the dialogue turns. The latter might be the reason why copying does not help much for this model since it might end up copying irrelevant tokens in the scenario mentioned above.

Our proposed DeepCopy model is designed to effectively address the aforementioned issues, where it has access to the entire set of persona facts per dialogue from which it is expected to include the useful pieces of information in the response. DeepCopy model outperforms all the models reported in Table 1 except for Seq2Seq + BestContextResponse models, which we already deem as kind of an upper bound because it has access to the most relevant fact to the response. This justifies the effectiveness of DeepCopy model compared to the existing works Ghazvininejad et al. (2018); Zhang et al. (2018) and the additional baselines we explored in this work. On the other hand, MultiSeq2Seq performs considerably worse than the DeepCopy model despite the fact they both have access to the entire set of facts and employ the same encoder-decoder architecture except for the copy mechanism. This further justifies the effectiveness of incorporating the proposed hierarchical pointer networks in DeepCopy because integrating the external knowledge simply by employing multi-source attention as in Zoph and Knight (2016) does not yield to a good solution with competitive results, performing even worse than Seq2Seq + NOFACT on 3 of the metrics.

3.2 Human Evaluation

Although automatic metrics provide tangible information regarding the performance of the models, we augment them with human evaluations for a more comprehensive analysis of the resulting model generated responses. Towards this end, we randomly sample 100 examples from test data and ask human raters to evaluate the candidate model generated responses in terms of appropriateness. Each example is rated by 3 raters, who are shown a dialog history along with a set of persona facts (of the person in turn), and asked to rate each response based on its appropriateness in the dialogue context with a score from 1 (worst) to 5 (best).

In Table 1, we present the results of human evaluation under the appropriateness column. Since each response is rated by $3$ different human raters, we report the average rating along with the standard deviation in parenthesis. We observe that DeepCopy outperforms both the existing memory-network baselines and the proposed sequence-to-sequence baselines on the appropriateness evaluation. It also achieves a performance that is close to the oracle model (S2SC-3), which has a leverage of having an access to the fact that is most relevant to the ground-truth response during the inference time. Overall, human evaluation of the responses in terms of appropriateness further justifies the promise and effectiveness of our proposed DeepCopy model.

Appropriateness scores also demonstrate the advantage of incorporating the soft copy mechanism. Comparing S2S (and M-S2S) models to their copy-equipped counterparts (S2SC) (and DeepCopy) in Table 1 immediately reveals a significant gain in appropriateness score. Another significant observation to note here is that ground-truth responses obtain an average appropriateness score of $4.4/5$, which reflects both the noise in ConvAI2 dataset and the difficulty of generating the perfect response even for humans.

4 Further Analysis and Discussion

Lexical Diversity Analysis. In Table 2, we report the lexical diversity results using the distinctness metric introduced in Li et al. (2016b). distinct-$n$ score corresponds to the number of distinct $n$-grams divided by total number of generated $n$-grams. We can clearly observe that DeepCopy generates the most diverse responses among all the models including the copy-augmented oracle model (S2SC-3). Hence, diversity results further show that our proposed model is promising in addressing the most commonly observed generic response problem more effectively than existing models by generating more diverse responses.

Fact Inclusion Analysis. We also conduct an analysis on the kinds of factual information included in the model-generated responses. More precisely, our goal is to understand how often the generated response includes a factual information (F.Inc), and whether this information is consistent with the persona facts (F.Per) or a hallucinated one (F.Hal). A good model can naturally include available facts from the persona and hallucinate others when the conversation context requires them. Towards this end, we ask $3$ human raters to label responses with 1 (or 0) based on whether a fact is included, and if so, whether this fact is a persona-fact or not.

In Table 2, we present an analysis for the kinds of factual information included in model generated responses. As can be seen from this analysis, models that have a copy mechanism include more facts from the persona than the ones that do not. Another important observation is that the ground-truth responses include facts from persona only in 49% of the times, which indicates that the provided persona facts remain insufficient to cover the complexity of the high entropy open-ended person-to-person conversations.

In Table 2, we present Cohen’s $\kappa$ score for each model and fact analysis metric pair using the scores from 3 raters for each example. We observe for each model and metric pair a $\kappa$ statistic of greater than 0.9, which indicates a near perfect agreement among raters. Note that the ratio of hallucinated facts (F.Hal) is derived directly from human labels for fact inclusion (F.Inc) and persona-fact (F.Per). That is why, there is no separate labelling process for hallucinated facts (F.Hal). Hence, there is no $\kappa$ statistic for F.Hal in Table 2.

Error Analysis. A deeper analysis of the examples where DeepCopy is assigned a worse appropriateness score than the best performing memory-network based baselines (M-2 and M-4) reveals the following further insights: (i) Some of these examples are corresponding to the cases where a generic response (e.g., "I’ve a dog named radar", one of the frequent generic responses, completely independent of persona facts) is rated much higher (5 to 1) than factual but slightly off (by a single word in this example) responses (e.g., "I have a dog for a living." coming from the persona fact "I walk dogs for a living."), (ii) In another subset of the analyzed examples, DeepCopy model generates a response (e.g., "yes, but I want to become a lawyer.") by incorporating a fact that has already been used in the previous turn of the dialog whereas M-2 produces a generic response (e.g., "that’s great. do you have any hobbies?", again irrelevant to facts) which is rated higher. (iii) And most of the remaining cases fall into the class of examples where incorporating knowledge facts breaks the conversation flow, which is a crucial observation specific to this dataset that can also be supported by the low persona-fact inclusion ratio (49%) of ground-truth responses.

5 Qualitative Observations

In Figure 3, we present an example dialogue where DeepCopy model generates a meaningful and fluent response by effectively mixing copy and generate modes. We can observe that it is able to attend on the right persona fact by taking the dialogue context (especially the question at the end of Person2’s turn) into consideration. Furthermore, attending to the tokens of this fact, it produces a fluent and valid answer to yes/no question by generating "yes" and copying the rest (and most) of the tokens from the fact. Although it copies most of the tokens from the fact, it is good to observe that it copies exactly the relevant pieces instead of just copying the entire fact. Seq2Seq + BestFactResponse + Copy model’s response is also meaningful and fluent although it may not be as engaging for the continuation of dialog. However, the quality of the response by Seq2Seq + BestFactResponse quickly degrades compared to its +Copy version. Although the response is still fluent and relevant to the dialogue context, it becomes rather irrelevant to the persona as the model seems to have difficulty of picking the useful information from even the best persona fact it is provided with when the copy mechanism is disabled. Lastly, the response generated by MemNet+FullAttention model seems to still suffer from repetition, semantic consistency, and relevancy problems that were observed and reported by previous work.

Conclusion and Future Work

We propose a hierarchical pointer network for knowledge grounded dialogue response generation. Our approach extends the pointer-generator network to enable the decoder to simultaneously copy tokens from the available set of relevant external knowledge in addition to dialogue context. We demonstrate the effectiveness of our approach through various automatic and human evaluations in comparison with several baselines on the ConvAi2 dataset. Furthermore, we conduct diversity, fact inclusion, and error analysis providing further insights into model behaviors. In the future, we plan to apply our model to datasets of the same fashion where the dialogue is accompanied by a much larger set of knowledge facts (e.g., Wikipedia articles) Galley et al. (2018). This could be done by adding a retrieval component which identifies a few contextually relevant facts Ghazvininejad et al. (2018) to be used as input to DeepCopy.

References