Jointly Optimizing Diversity and Relevance in Neural Response Generation

Introduction

The field of neural response generation is advancing rapidly both in terms of research and commercial applications Gao et al. (2019); Zhou et al. (2018); Yoshino et al. (2019); Zhang et al. (2019). Nevertheless, vanilla sequence-to-sequence (S2S) models often generate bland and generic responses Li et al. (2016a).

Li et al. (2016a) encourage diversity by re-ranking the beam search results according to their mutual information with the conversation context. However, as beam search itself often produces lists of nearly identical sequences, this method can require a large beam width (e.g. 200). As a result, re-ranking can be extremely time-consuming, raising difficulties for real-time applications. This highlights the need to improve the diversity of candidates before re-ranking, and the need to optimize for diversity during training rather than just at the decoding stage.

While various approaches have been explored to diversify the output of conversation models, the improvement often comes at the cost of decreased response relevance along other dimensions. For instance, Zhao et al. (2017) present an approach to enhancing diversity by mapping diverse responses to a probability distribution using a conditional variational autoencoder (CVAE). Despite the improved response diversity, this approach reduces response relevance as measured against the baseline. One possible reason for this diversity-relevance trade-off is that such probabilistic approaches are not explicitly encouraged to induce a disentangled representation in latent space for controlling diversity and relevance independently. Consider a Gaussian distribution, which is widely used for CVAE. A Gaussian distribution naturally brings frequent responses near its mean and such responses are often generic and boring. To generate diverse and interesting responses, one needs to sample a little distance from the mean. But doing so naturally leads to infrequent and thus even irrelevant responses.

In this paper, we propose a novel geometrical approach that explicitly encourages a structured latent space in which the distance and direction from a predicted response vector roughly match the relevance and diversity, respectively, as illustrated in Figure 3. To induce such a latent space, we leverage two different models: 1) a S2S model, producing the predicted response vector (the black dot at the center in Figure 3), and 2) an autoencoder (AE) model, yielding the vectors for potential responses (the colored dots). In order to make the S2S and AE share the same latent space (the cloud), we use the same decoder for both and train them jointly end-to-end with novel regularization terms. As this fuses the two latent spaces, we refer to our model as SpaceFusion.

Regularization is necessary because only sharing the decoder, as in Luan et al. (2017), does not necessarily align the latent spaces obtained by S2S and AE respectively or impose a disentangled structure onto the space. We introduce two regularization terms to tackle this issue. 1) interpolation term: we encourage a smooth semantic transition along the path between the predicted response vector and each target response vector (arrowed lines in Figure 3). This term effectively prevents semantically different responses from aligning in the same direction, essentially scattering them over different directions. 2) fusion term: we want the vectors from the two models to be distributed in a homogeneous manner, rather than forming two separate clusters (Figure 5) that can potentially make sampling non-trivial. With the resulting latent space, we can control relevance and diversity by respectively adjusting distance and direction from a predicted response vector, without sacrificing each other greatly.

Our approach also lends itself well to the intuitive visualization of latent space. Since our model allows us to geometrically find not only the predicted response vector but also the target response vector as in Figure 5, we can visually interpret the structure of latent space and identify major issues thereof. We devote Section 5.1 to show comprehensive examples for visualization-based analysis.

Automatic and human evaluations demonstrate that the proposed approach improves both the diversity and relevance of the responses, compared to strong baselines on two datasets with one-to-many context-response mapping.

Related Work

utilize extra context inputs besides conversation history, such as persona Li et al. (2016b), textual knowledge Ghazvininejad et al. (2017); Galley et al. (2019), dialog act Zhao et al. (2017) and emotion Huber et al. (2018). Our approach does not depend on such extra input and thus is complementary to this line of studies.

explicitly model the uncertainty of responses in latent space. Bowman et al. (2016) used VAE with Long-Short Term Memory (LSTM) cells to generate sentences. The basic idea of VAE is to encode the input $x$ into a probability distribution (e.g. Gaussian) $z$ instead of a point encoding. However, it suffers from the vanishing latent variable problem Bowman et al. (2016); Zhao et al. (2017) when applied to text generation tasks. Bowman et al. (2016); Fu et al. (2019) proposed to tackle this problem with word dropping and specific KL annealing methods. Zhao et al. (2017) proposed to add a bag-of-word loss, complementary to KL annealing. Applying this to a CVAE conversation model, they showed that even greedy decoding can generate diverse responses. However, as VAE/CVAE conversation models can be limited to a simple latent representations such as standard Gaussian distribution, Gu et al. (2018) proposed to enrich the latent space by leveraging a Gaussian mixture prior. Our work takes a geometrical approach that is fundamentally different from probabilistic approaches to tackle the limitations of parameteric distributions in representation and difficulties in training.

encourage diversity during the decoding stage. As “vanilla” beam search often produces lists of nearly identical sequences, Vijayakumar et al. (2016) propose to include a dissimilarity term in the objective of beam search decoding. Li et al. (2016a) re-ranked the results obtained by beam search based on mutual information with the context using a separately trained response-to-context S2S model.

is another line of studies related to the present work (see Section 3.2). Sennrich et al. (2016) use multi-task learning to improve neural machine translation by utilizing monolingual data, which usually far exceeds the amount of parallel data. A similar idea is applied by Luan et al. (2017) to conversational modeling, involving two tasks: 1) a S2S model that learns a context-to-response mapping using conversation data, and 2) an AE model that utilizes speaker-specific non-conversational data. The decoders of S2S and AE were shared, and the two tasks were trained alternately.

The SpaceFusion Model

Let $\mathcal{D}=[(x_{0},y_{0}),(x_{1},y_{1}),\cdots,(x_{n},y_{n})]$ denote a conversational dataset, where $x_{i}$ and $y_{i}$ are a context and its response, respectively. $x_{i}$ consists of one or more utterances. Our aim is to train a model on $\mathcal{D}$ to generate relevant and diverse responses given a context.

2 Fusing latent spaces

We design our model to induce a latent space where different responses for a given context are in different directions around the predicted response vector, as illustrated in Figure 3. Then we can obtain diverse responses by varying the direction and keep their relevance by sampling near the predicted response vector.

To fulfill this goal, we first produce the predicted response representation $z_{\text{S2S}}$ and target response representations $z_{\text{AE}}$ using an S2S model and an AE model, respectively, as illustrated in Figure 2. Both encoders are implemented using stacked Gated Recurrent Unit (GRU) Cho et al. (2014) cells followed by a noise layer that adds multivariate Gaussian noise $\epsilon\sim N(0,\sigma^{2}\mathbf{I})$.

We then explicitly encourage smooth semantic transition along the path from $z_{\text{S2S}}$ to $z_{\text{AE}}$ by imposing any interpolation between them to generate the same response via the following loss term:

where $z_{\text{interp}}=uz_{\text{S2S}}+(1-u)z_{\text{AE}}$ and $u\sim U(0,1)$ is a uniformly distributed random variable. $|y|$ is the number of words in $y$. Note that it is this regularization term that effectively prevents significantly different responses from aligning in the same direction, essentially scattering them over different directions. In order for this interpolation loss to work, we share the same decoder for both AE and S2S models as in Luan et al. (2017). The decoder consists of stacked GRU cells followed by a softmax layer. It is worth mentioning that $z_{\text{interp}}$ is not just randomly drawn from a single line but from a richer probabilistic region as both $z_{\text{interp}}$ and $z_{\text{S2S}}$ are stochastic due to the random component $\epsilon$.

Now, we want vectors from both the AE and S2S models to be distributed in a homogeneous manner scattered over the entire space while keeping the distance between $z_{\text{S2S}}$ and $z_{\text{AE}}$ as small as possible for any (context-response) pair in the training data. This objective is represented in the following regularization term:

where $n$ is the batch size and $d(a,b)$ is the root mean square of the difference between $a$ and $b$. For each batch, we basically disperse vectors obtained by the same model and pull the predicted response vectors to the corresponding target response vectors. In practice, we found that the performance is better if the Euclidean distance is clipped to a prescribed maximum value.This value is set as 0.3 for the present experiments

Finally, with weight parameters $\alpha$ and $\beta$, the loss function is defined as:

As $\mathcal{L}_{\text{interp}}$ and $\mathcal{L}_{\text{fuse}}$ encourage the path between $z_{\text{S2S}}$ and $z_{\text{AE}}$ to be smooth and short while scattering vectors over the entire space, they effectively fuse the $z_{\text{S2S}}$ latent space and the $z_{\text{AE}}$ latent space. Accordingly we refer this approach as SpaceFusion with path regularization.

3 Training

In contrast to previous multi-task conversation model Luan et al. (2017), where S2S and AE are trained alternately, our approach trains S2S and AE at the same time by minimizing the loss function of Equation 3.2.

4 Inference

Like Zhao et al. (2017); Bowman et al. (2016), for a given context, we sample different latent vectors to obtain multiple hypotheses. This is done by adding a random vector $r$ that is uniformly sampled from a hypersphere of radius $|r|$ to the prediction $z_{\text{S2S}}(x)$.

where $|r|$ is tuned on the validation set to optimize the trade-off between relevance and diversity. $z(x,r)$ is then fed to the decoder as the initial state of GRU cells. We then generate responses using greedy decoding.Although we use greedy decoding in this work, other decoding techniques, such as beam search, can be applied.

Experiment Setup

We used the following datasets. Some of their key features are presented in Table 1.

We use the version offered by Zhao et al. (2017), which is an extension of the original version by Godfrey and Holliman (1997). Zhao et al. (2017) collected multiple references for the test set using information retrieval (IR) techniques followed by human filtering, and randomly split the data into 2316/60/62 conversations for train/validate/test, respectively. Each conversation has multiple turns and thus multiple $(x,y)$ pairs, as listed in Table 1. As our approach does not utilize extra information except conversation history, we removed the meta data (e.g. gender, age, prompt) from this dataset.

As the Switchboard dataset is relatively small and multiple references are synthetically constructed, we have developed another multi-reference dataset by extracting posts and comments on Reddit.com during 2011 collected by a third party.http://files.pushshift.io/reddit/comments/ As each Reddit post and comment may have multiple comments, it is a natural source of multi-reference responses. We further filtered the data based on the number of replies to obtain the final conversation dataset in which each context has at least 10 different responses, and on average the number of responses is 24.1 for a given context. The size is significantly larger than Switchboard, as listed in Table 1. The conversations are randomly shuffled before being split into train/valid/test subsets.

2 Model setup

Both encoders and the shared decoder consist of two GRU cells, each with 128 hidden units. The variance of the noise layer in each decoder is $\sigma^{2}=0.1^{2}$. The word embedding dimension is 128. The weight parameters (see Equation 3.2) are set as $\alpha=1$ and $\beta=30$. For both datasets, the inference radius $|r|$ (see Equation 4) is set to 1.5 which optimizes F1 score on the validation set. All models are trained using the Adam method Kingma and Ba (2014) with a learning rate of 0.001 on both datasets until convergence (around 4 epochs for Reddit and 10 epochs for Switchboard).

3 Automatic evaluation

For a given context $x$, we have $N_{r}$ reference responses and generate the same number of hypotheses.We set the number of hypotheses equal to the number of references to encourage precision and recall have comparable impact on F1 We define the following metrics based on 4-gram BLEU Papineni et al. (2002), as suggested by Zhao et al. (2017).

We use Precision as an approximate surrogate metric for relevance and Recall for diversity. It should be noted that recall is not equivalent to other diversity metrics, e.g., distinct Li et al. (2016a) and entropy Zhang et al. (2018), which only depend on hypotheses. One potential issue of these metrics is that even randomly generated responses may yield a high diversity score. F1 is the harmonic average of these two and is used to measure the overall response quality.

4 Human evaluation

We conduct a human evaluation using crowdworkers. For each hypothesis, given its context, we ask three annotators to individually measure the quality, on a scale of 1 to 5, in terms of two aspects: relevance and interest. Interestingness is treated as an estimation of the diversity, as these two are often correlated. The hypotheses from all systems are shuffled before being provided to annotators. System names are invisible to the annotators.

5 Baselines

We compare the proposed model with the following baseline models:

We consider a vanilla version of S2S model. The dimensions are similar to our model: both encoder and decoder consist of two stacked GRU cells with 128 hidden units, and the word embedding size is 128. As in the baseline in Zhao et al. (2017), we applied softmax sampling at inference time to generate multiple hypotheses.

For the CVAE conversation model, we use the original implementation and hyperparameters of Zhao et al. (2017) with the bag-of-words (BOW) loss. The number of trainable model parameters is 15.4M, which is much larger than our model (3.2M).

Since our approach utilizes a multi-task learning scheme, we also compare it against a vanilla multi-task learning model, MTask, similar to Luan et al. (2017), to illustrate the effect of space fusion. The model architecture and hyperparameters are identical to the proposed model except that the loss function is $\mathcal{L}=-\log p(y|z_{\text{S2S}})-\log p(y|z_{\text{AE}})$.

Results and Analysis

In this section, we undertake an in-depth analysis to verify whether the latent space induced by our method manifests desirable properties, namely: 1) disentangled space structure between relevance and diversity, 2) homogeneous space distribution in which semantics changes smoothly without holes. We first provide a qualitative investigation based on real examples. Then, we present a set of corpus-level quantitative analyses focused on geometric properties.

In Table 2, we investigate three different directions from the context “Anyone want to start this game?” , which is a real example taken from Reddit. The three different directions correspond to clearly different semantics: “No I don’t”, “when?” and “Yes I do.” If we generate a response with the vector predicted by the S2S model ($u=0$), our model outputs “I would love to play this game” which is highly relevant to the context. Now as we move along each direction, we can see our model gradually transforms the response toward the corresponding responses of each direction. For instance, towards “No I don’t”, our model gradually transforms the response to “I am not interested in the game” ($u=0.18$) and then “I am not interested.” ($u=0.21$). In contrary, towards “Yes I do”, the response transforms to “I would love to play it.” ($u=0.15$). Besides the positive or negative directions, the same transition applies to other directions such as “When?”. This example clearly shows that there is a rough correspondence between geometric properties and semantic properties in the latent space induced by our method as shown in Figure 3– the relevance of the response decreases as we move away from the predicted response vector and different directions are associated with semantically different responses.

1.2 Direction vs. diversity

In order to quantitatively verify the correspondence between direction and diversity, we visualize the distribution of cosine similarities among multiple references for each context for a set of 1000 random samples drawn from the test dataset. Specifically, for a context $x_{k}$ and its associated reference responses $[y_{k,0},y_{k,1},\cdots]$, we compute the cosine similarity between $z_{\text{AE}}(y_{k,i})-z_{\text{S2S}}(x_{k})$ and $z_{\text{AE}}(y_{k,j})-z_{\text{S2S}}(x_{k})$. In Figure 3, we compare the distribution of our model with that of MTask, which does not employ our regularization terms. While our method yields a bell shaped curve with average cosine similarity being close to zero (0.38), the distribution of MTask is extremely skewed with average cosine similarity being close to 1 (0.95). This indicates that the directions of the reference responses are more evenly distributed in our latent space whereas everything is packed in a narrow band in the MTask’s space. This essentially makes the inference process simple and robust in that one can choose arbitrary directions to generate diverse responses.

1.3 Distance vs. relevance

In order to quantitatively verify the correspondence between distance and relevance, we visualize the perplexity of reference responses along the path from the associated $z_{\text{S2S}}$ ($u=0$) to the $z_{\text{AE}}$ ($u=1$) corresponding to the predicted response. In Figure 4, we compare our model with MTask, which as already noted, does not employ our regularization terms. While our model shows a gradual increase in perplexity, there is a huge bump for MTask’s line. This clearly indicates that there is a rough correspondence between distance and relevance in our latent space whereas even a slight change can lead to an irrelevant response in the MTask’s space.

We further illustrate the smooth change in relevance according to distance for a specific example in Table 3. Given the context “Anyone want to start this game?”, our model is able to transition from the predicted response “I would love to play this game” to a one of reference responses “Yes I do”. The relevance smoothly descreases, generating intermediate responses such as “I would love to play it.” In contrary, the MTask model tends to produce irrelevant or ungrammatical responses as it moves away from the predicted response.

1.4 Homogeneity and Convexity

Other desirable properties, with which we want to equip our latent space are homogeneity and convexity. If the space is not homogeneous, we have to sample differently depending on the regional traits. If the space is not convex, we have to worry about running into the holes that are not properly associated with valid semantic meanings. In order to verify homogeneity and convexity, we visualize our latent space in a 2D space produced by the multidimensional scaling (MDS) algorithm Borg and Groenen (2003), which approximately preserves pairwise distance. For comparison, we also provide a visualization for MTask. As shown in Figure 5, our latent space offers great homogeneity and convexity regardless of which model is used to produce a dot (i.e. $z_{S2S}$ or $z_{AE}$). In contrary, MTask’s latent space forms two separate clusters for $z_{S2S}$ and $z_{AE}$ with a large gap in-between where no training samples were mapped to.

2 Automatic evaluation

We let each system generate 100 hypotheses $\{h_{j}\}$ for each context $x_{i}$ in the test dataset. Assuming $x_{i}$ has $N_{r,i}$ references, we pick the top $N_{r,i}$ distinct hypotheses ranked by $\log p(h_{j}|x_{i})+\lambda|h_{j}|$. Similar to Li et al. (2016a); Wu et al. (2016), we takes $|h_{j}|$ into consideration, as BLEU is sensitive to length. For fair comparison, $\lambda$ is tuned such that the average hypothesis length becomes roughly the same for all systems and approaches the average length of the references.Approximately 10 words/tokens for Switchboard and 12 for Reddit

The automatic evaluation results are reported in Table 4. On both datasets, the proposed system consistently outperforms the baselines by a large margin in Precision, Recall, and F1.

Examples of system outputs and human references can be found in Table 5 and Table 6 for Reddit and Switchboard, respectively. As shown in the examples, CVAE+BOW and other baseline models may generate diverse but not-so-relevant responses.

3 Human evaluation

We randomly sampled 500 contexts from the Reddit test dataset and picked the top 1 hypothesis generated for each context ranked by $\log p(h_{j}|x_{i})+\lambda|h_{j}|$. As in the automatic evaluation, we tuned $\lambda$ such that all systems have roughly the same average hypothesis length. We also randomly select one reference for each context and compare them with the systems (labeled ”human” in Table 7)

As illustrated in Table 7, the proposed model outperforms all systems except human, consistent with our automatic evaluation results.

Conclusion

We propose a SpaceFusion model to jointly optimize diversity and relevance that leverages novel regularization terms to essentially fuse the latent space of a S2S model with that of an autoencoder model. This fused latent space exhibits desirable properties such as smooth semantic interpolation between two points. The distance and direction from the predicted response vector roughly match relevance and diversity, respectively. These properties also enable intuitive visualization of the latent space. Both automatic and human evaluation results demonstrate that the proposed approach brings significant improvement compared to strong baselines in terms of both diversity and relevance. In future work, we will provide theoretical justification of the effectiveness of the proposed regularization terms. We expect that this technique will find application as an efficient ”mixing board” for conversation that draws on multiple sources of information.

References