Challenges in Building Intelligent Open-domain Dialog Systems

Introduction

Building intelligent open-domain dialog systems that can converse with humans coherently and engagingly has been a long-standing goal of artificial intelligence (AI). Early dialog systems such as Eliza (Weizenbaum, 1966), Parry (Colby et al., 1971), and Alice (Wallace, 2009), despite being instrumental to significantly advancing machine intelligence, worked well only in constrained environments. An open-domain social bot remains an elusive goal until recently. The Microsoft XiaoIce (‘Little Ice’ literally in Chinese) system, since its release in May, 2014, has attracted millions of users and can converse with users on a wide variety of topics for hours (Zhou et al., 2018a; Shum et al., 2018). In 2016, the Alexa Prize challenge was proposed to advance the research and development of social bots that are able to converse coherently and engagingly with humans on popular topics such as sports, politics, and entertainment, for at least 20 minutes (Chen et al., 2018; Ram et al., 2018) Even though the dialog systems in this challenge are very complicated, they are more informational systems where user emotion need is less considered. . The evaluation metric, inspired by the Turing Test (Turing, 1950), is designed to test the social bots’ capacity of delivering coherent, relevant, interesting, free-form conversations and keeping users engaged as long as possible. However, the general intelligence demonstrated by these systems is still far behind humans. Building open-domain dialog systems that can converse on various topics like humans remains extremely challenging (Gao et al., 2019a).

In this paper we focus our discussion on three challenges in developing neural-based open-domain dialog systems, namely semantics, consistency and interactiveness. The rest of the paper is structured as follows. In the rest of Section 1, we compare open-domain dialog bots with traditional task-oriented bots and elaborate the three challenges. In Section 2, we survey three typical approaches to building neural-based open-domain dialog systems, namely, retrieval-based, generation-based, and hybrid methods. In Sections 3, 4, and 5, we review the approaches that have been proposed to address the three challenges, respectively. In Section 6, we discuss recent work on open-domain dialog evaluation. In Section 7, we present an incomplete survey of frequently-used or recently-proposed benchmarks for open-domain conversation modeling. We conclude the paper by presenting several future research trends in Section 8.

Generally speaking, there are two types of dialog systems: task-oriented and open-domain dialog. Task-oriented dialog systems are designed for specific domains or tasks, such as flight booking, hotel reservation, customer service, and technical support, and have been successfully applied in some real-world applications. Open-domain dialog systems, however, are much more challenging to develop due to its open-ended goal.

As outlined by Gao et al. (2019a), although both task-oriented dialog and open-domain dialog can be formulated as an optimal decision making process with the goal of maximizing expected reward, the reward in the former is better-defined and much easier to optimize than the latter. Consider a ticket-booking bot. It is straightforward to optimize the bot to get all necessary information to have the ticket booked in minimal dialog turns. The goal of an open-domain dialog agent is to maximize the long-term user engagement. This is difficult to optimize mathematically because there are many different ways (known as dialog skills) to improve the engagement (e.g., providing entertainment, giving recommendations, chatting on an interesting topic, providing emotional comforting) and it requires the systems to have a deep understanding of dialog context and user’s emotional needs to select the right skill at the right time, and generate interpersonal responses with a consistent personality.

Open-domain dialog systems also differ from task-oriented bots in system architecture. A task-oriented bot is typically developed based on a pre-defined task-specific schemaA task schema typically defines a set of user intents, and for each intent defines a set of dialog acts, slot-value pairs. and is designed as a modular system which consists of domain-specific components like language understanding, dialog managementDialog management performs both dialog state tracking (Henderson et al., 2013; Mrksic et al., 2017) and response selection via policy (Zhao and Eskénazi, 2016; Peng et al., 2017; Su et al., 2016; Lipton et al., 2018). , and language generationRecently, there are end-to-end methods (Rojas-Barahona et al., 2017; Bordes et al., 2017; Zhang et al., 2019c) that output a response given the previous dialog history. But in general, domain knowledge about the task needs to be explicitly considered, which differs significantly from open-domain dialog systems.. These components can be either hand-crafted based on domain knowledge or trained on task-specific labeled data. On the other hand, due to the open-ended nature, open-domain dialog systems need to deal with open-domain knowledge without any pre-defined task-specific schemas or labels. In recent years, there has been a trend towards developing fully data-driven, end-to-end systems that map user’s input to system’s response using neural networks. Since the primary goal of open-domain dialog bots is to be AI companions to humans with an emotional connection rather than completing specific tasks, they are often developed to mimic human conversations by training neural response generation models on large amounts of (Sordoni et al., 2015; Vinyals and Le, 2015; Shang et al., 2015).

Unlike task-oriented bots, most neural response generation models developed for open-domain dialog systems are not grounded in real world, which prevents these systems from effectively conversing about anything that relates to the user’s environment. Only recently have researchers begun to explore how to ground open-domain dialog systems in real-world entities and knowledge (Ghazvininejad et al., 2018; Mostafazadeh et al., 2017; Qin et al., 2019). Knowledge grounding is also crucial for the system to provide interpersonal responses. For instance, the conversations between friends are quite different from those between strangers. So the system needs to be grounded in the personas of the speaker and addressee, respectively (Li et al., 2016b). The tone of system responses needs to be adjusted according to user’s emotional states and affect by grounding in affect or emotion of the user (Huber et al., 2018; Winata et al., 2017; Xu et al., 2018a).

2. Three Challenges

Thanks to the recent success of neural approaches to natural language processing and conversational AI, there have been substantial advancements on open-domain dialog systems.

Most neural approaches formulate conversation as a task of generating output response given user input and dialog context. Almost all neural response generation models follow the neural text generation frameworks, such as sequence-to-sequence (Seq2Seq) (Sutskever et al., 2014; Vinyals and Le, 2015; Shang et al., 2015), conditional variational autoencoder (CVAE) (Sohn et al., 2015), and generative adversarial network (GAN) (Li et al., 2017a). These models are trained on large-scale conversational data, and can deliver plausible responses to any topic that users are interested in. However, these neural-based systems often fail to establish long-term emotional connections with users due to many issues as illustrated by the examples in Table 1 and summarized in Figure 1. In what follows, we elaborate three key issues which we will discuss in the rest of the paper Note that the challenges discussed in this section are also fundamental to traditional, non-neural dialog systems..

Semantics is the heart of any dialog system because conversation is a semantic activity (Zhou et al., 2018f). It requires not only to understand the content of the conversation, which is often in multi-modality including text, image and video, but also to understand users by identifying information beyond the dialog content such as a user’s personality and personaPersonality is someone’s character or nature while a persona is a superficial identity of the character or nature., emotion, sentiment, and the user’s profile and background. From the technical perspective, semantics mainly involves the key techniques of natural language understanding and user understanding, including named entity recognition, entity linking, domain detection, topic and intent detection, user sentiment/emotion/opinion detection, and knowledge/ commonsense reasoning.

Consistency

In order to gain user’s long-term confidence and trust, it is crucial for a dialog system to present consistent behaviors and respond consistently given user’s input and dialog history (Li et al., 2016b; Qian et al., 2018; Zheng et al., 2019; Zhou et al., 2018a). For instance, a social bot should not deliver a response that conflicts with her pre-set persona, or her previous responses in temporal dependency, causality, or logic. Specifically, the system’s response needs to be consistent in three dimensions. First is persona consistency where the response needs to fit the pre-defined personality of the dialog system. Second is stylistic consistency where a consistent speaking style is presented. Third is contextual consistency in which the response needs to be coherent and consistent with respect to the dialog context. From the technical perspective, consistency mainly involves personalization, stylistic generation, and multi-turn context modeling.

Interactiveness

As mentioned above, meeting user’s social needs, such as emotional affection and social belonging, is the primary design goal of an open-domain dialog system. Interactiveness refers to the system’s ability to achieve complex social goals such as entertainment and conforming by optimizing its behaviors and dialog strategies in multi-turn conversation. To improve interactiveness, it is important to understand the user’s emotion state or affect (Zhou et al., 2018b, a), to respond not only reactively but also proactively (Yu et al., 2016; Wang et al., 2018b; Rao and Daumé III, 2018), to control the topic maintenance or transition (Wang et al., 2018a), and to optimize the interaction strategy (i.e., dialog policy) in multi-turn conversations to maximize long-term user engagement. From the technical perspective, interactiveness mainly involves sentiment and emotion detection, dialog state tracking, topic detection and recommendation, dialog policy learning, and controllable response generation.

We summarize the techniques required to address the three issues in Figure 1, including named entity recognition, entity linking, domain/topic/intent detection, and sentiment/emotion detection. As demonstrated in the Alexa Prize challenge which targets at developing dialog systems for conversing coherently and engagingly with humans on various popular topics, the winning dialog systems (Fang et al., 2017; Chen et al., 2018) are composed of different modules that are developed based on these techniques, including language understanding, dialog management, and natural language generation. In such modular designs, the semantic issue is mainly related to the understanding module which is intended to understand the dialog (e.g., content, entity, topic, etc.) and user (e.g., opinion, personality, emotional needs). The other two issues are mainly related to the dialog management and generation modules, aiming to generate responses that are not only consistent in content and personality, but also interactive so as to increase the long-term user engagement. These issues are highly interleaved. For example, understanding dialog and user (semantics) is fundamental to generating consistent and interactive responses.

Frameworks for Building Open-domain Dialog Systems

As discussed in Section 1.1, open-domain dialog systems are typically implemented using an end-to-end architecture, rather than a modular architecture used by task-oriented bots for which task-specific schemas and labels are available for the development of these dialog modules. At the heart of an open-domain dialog system is a response generation engine, which takes user input at $t$-th dialog turn $X_{t}=x_{1}^{t}x_{2}^{t}\cdots x_{n}^{t}$ and dialog context $C_{t}$, which will be explained in a minute, and generates response $Y_{t}=y_{1}^{t}y_{2}^{t}\cdots y_{m}^{t}$ as

where $\Omega$ denotes the set of all candidate responses, $\mathcal{P}_{\theta}$ is a learned model of scoring candidate responses, parameterized by $\theta$, and argmax the search algorithm to find among all candidates the best one with the highest score.

This formulation unifies three typical methods of building open-domain dialog systems: retrieval-based, generation-based, and hybrid. In retrieval-based methods, the search space $\Omega$ is obtained by retrieving candidate responses from a pre-collected human conversational dataset consisting of input-context-response pairs. $\mathcal{P}_{\theta}(Y|X_{t},C_{t})$ is implemented as a matching or ranking function which scores the relevance of each candidate given $X_{t}$ and $C_{t}$. In generation-based methods, the search space $\Omega$ is very large, namely $Y\in V^{m}$ where $V$ is the vocabulary size and $m$ is the response length, and $\mathcal{P}_{\theta}(Y|X_{t},C_{t})$ is typically implemented as an auto-regressive model that generates a sentence word by word. In the hybrid methods, it is typical to first retrieve prototype responses from a dataset and then generates a response by utilizing prototype responses.

Note that the introduction of context $C_{t}$ offers a lot of flexibility to model various aspects of dialog. For instance, when $C_{t}=\varnothing$, it models single-turn dialog; Setting $C_{t}=X_{1}Y_{1}X_{2}Y_{2}\cdots X_{t-1}$ models multi-turn dialogs. $C_{t}$ can also encode other (non-content) contexts such as persona (Qian et al., 2018; Zhang et al., 2018b; Zheng et al., 2019) for personalized dialog generation, emotion labels (Zhou et al., 2018b; Asghar et al., 2018) for emotional response generation, and knowledge graphs (Zhou et al., 2018f; Ghazvininejad et al., 2018) for knowledge-aware response generation.

Given a dialog corpus and the user’s post, IR-based systems can use any retrieval algorithm to choose an appropriate response from the corpus (Jafarpour et al., 2010; Carpenter, 2011; Leuski and Traum, 2011). In such a setting, the system retrieves the most similar post to the given user post, and the response to the retrieved post is returned as the response to the user’s post. Traditional learning-to-rank methods were introduced by Ji et al. (2014) for response selection from a large-scale post-response repository. Afterwards, many neural models have been proposed. Figure 2 illustrates the process of retrieval-based response generation methods. Using input $X\oplus C$ Hereafter, we will use $X\oplus C$ to denote the input query that combines the current user input $X$ and the dialog context $C$. as a query, such methods first retrieve a list of candidates from a large repository which consists of input-context-output pairs, and choose the top-scored candidate as output response $Y$ using the matching function $\mathcal{P}_{\theta}(Y|X,C)$, which can be implemented using either traditional learning-to-rank algorithms (Liu, 2010), or modern neural matching models (Lu and Li, 2013; Huang et al., 2013; Fan et al., 2017). The model parameters $\theta$ is commonly learned by minimizing the margin-based pair-wise ranking loss as follows Note that the method of pair-wise ranking is widely used in the literature, but other ways such as point-wise and list-wise ranking methods (Liu, 2010) are also feasible.:

where $\gamma$ is a margin (a hyper-parameter), $Y_{+}$ is a ground-truth (positive) response, $Y_{-}$ is a negative response which can be randomly sampled from the dataset or generated by corrupting $Y_{+}$, and $match_{\theta}(Y,X\oplus C)$ is the matching function to be learned.

Alternatively, we can also use a likelihood loss defined as:

Modern neural models of $match(Y,X\oplus C)$ can be roughly grouped into two categories, shallow and deep interaction networksShallow or deep is regarding interaction, namely whether the learned representations are obtained by early-stage interactions (deep), or late-stage (sometimes no) interactions (shallow). The two words are not referring to whether the model structure is deep or not., as illustrated in Figure 3. In shallow interaction networks, candidate $Y$ and input $X\oplus C$ are first encoded independently into the two vectors which then have some shallow interactions such as subtraction or element-wise multiplication before being fed to the classification layer. In deep interaction networks, $Y$ and $X\oplus C$ interact via an interaction network to form a fused representation, which is then fed to the classification layer.

For shallow interaction networks, many efforts have been devoted to learning good representations for query and candidate independently. Huang et al. (2013) proposed to use deep structured similarity models (DSSMs) to extract semantic features from query and document independently before computing their relevance. DSSM is further augmented by introducing Convolutional layers (Shen et al., 2014; Gao et al., 2014; Hu et al., 2014; Severyn and Moschitti, 2015) and recurrent layers with Long Short-Term Memory (LSTM) units (Palangi et al., 2016). To effectively incorporate dialog history, Yan et al. (2016) reformulated input query $X$, and combined matching scores computed based on the reformulated and original queries, and retrieved queries and responses, respectively. Zhou et al. (2016) used a hierarchical Recurrent Neural Network (RNN) to encode a candidate and the utterance sequence in context, respectively, before computing their matching score. These shallow models are simple to implement and efficient to execute.

For deep interaction networks, query $X\oplus C$ and response $Y$ interact via a neural network to generate a single feature vector that preserves all query-response interaction information at different levels of abstraction. The matching score is then derived from the vector using another neural network. Hu et al. (2014) extracted matching features from all $n$-gram combinations of input $X$ and response $Y$ to obtain low-level feature maps with a Convolutional Neural Network (CNN). Afterwards, the feature maps are transformed with multiple CNN layers to form the final representation for classification. Wu et al. (2017) proposed a sequential matching network (SMN) for multi-turn dialog where each contextual utterance in $X\oplus C$ is encoded conditioned on $Y$, and these utterances are connected sequentially by GRUs. The matching score is computed on top of the weighted sum of the GRUs’ states. Zhou et al. (2018c) proposed a deep attention matching network. The query and its candidate response are firstly represented with self-attention inspired by the transformer network (Vaswani et al., 2017), and then the interactions between them were made with cross-attention to obtain word-by-word matching matrices, and finally the matching score is computed by aggregating all the matching information with a 3D matching tensor. Yang et al. (2018) extended SMN with external knowledge in information-seeking conversation systems. The method first expands response candidates using pseudo-relevance feedback, and then makes the candidates interact with the query to obtain word-by-word matching matrices. The subsequent operations are very similar to SMN. Zhang et al. (2018e) proposed a deep utterance aggregation model which shares a similar structure with SMN. The difference lies in that gated self-attention was used to obtain the representations of the query and a response candidate, and the subsequent operations are almost the same to SMN. Wu et al. (2018) proposed to consider topic clues for query-response matching. The authors first extracted topical words for the query and response respectively using LDA. Then, a query representation is conditioned not only on the response representation but also on the attentive read of the topical words of the response. A response representation is computed similarly conditioned on the message’s topical words and the query representation. Other matching models that were proposed originally for non-dialog tasks such as paraphrase detection, language inference, and reading comprehension (Wang et al., 2017a; Pang et al., 2016), have also been adapted and applied to dialog response ranking.

One of the most notable deep interaction networks for learning the matching function (as defined by Eq. 2) is BERT (Devlin et al., 2019), which achieves state-of-the-art performance on many NLP tasks, including response selection. $X_{t}\oplus C_{t}$ and a candidate response $y$, normally separated by a special token [SEP], form the input of a multi-layer Transformer (Vaswani et al., 2017) blocks (12-48 blocks). Each block consists of multi-head a self-attention module, layer normalization, a feed forward layer, and residual connections. The vectors at the output layer are fed to a fine-tuned classifier to determine whether the response $y$ is appropriate for the input. This structure has been widely adopted in retrieval-based methods (Henderson et al., 2019).

There is a short review on deep retrieval-based dialogue systems (Boussaha et al., 2019) where the authors discussed existing work with respect to single-turn matching models, multi-turn matching models, and ensemble models. In comparison, we summarize existing work from the interaction perspective: whether a candidate response makes deep matching with the input (post, or along with the context) at early or late stage. In general, deep interaction networks usually work better than shallow interaction networks (Tao et al., 2019).

2. Generation-based Methods

Neural generative models have been widely applied to open-domain dialog generation. Inspired by the early template-based generation method (Higashinaka et al., 2014) and statistical machine translation (SMT) (Ritter et al., 2011), sequence-to-sequence (Seq2seq) models (Sutskever et al., 2014; Vinyals and Le, 2015; Shang et al., 2015; Sordoni et al., 2015) have become the most popular choice for dialog generation. Other frameworks, including conditional variational autoencoder (CVAE) (Serban et al., 2017; Zhao et al., 2017; Ke et al., 2018; Shen et al., 2018; Zhao et al., 2018; Du et al., 2018) and generative adversarial network (GAN) (Li et al., 2017a; Xu et al., 2018b), are also applied to dialog generation. Very recently, Transformer-based language models pretrained with large-scale corpora are another popular choice (Radford et al., 2018; Wolf et al., 2018; Golovanov et al., 2019; Zhang et al., 2019f), which obtains strong performance in dialog generation (Wolf et al., 2018).

Generation-based models usually formulate $\mathcal{P}(Y|X_{t}\oplus C_{t})$ as:

where $y_{<i}=y_{1}y_{2}\cdots y_{i-1}$. Typically, the output response is generated word by word, e.g., at each time step a word is sampled according to $P(y|y_{<i};X_{t}\oplus C_{t})$. Using RNNs, during the course of generation, the generated prefix is autoregressively encoded into the input to generate the next word.

Most neural generation models adopt an encoder-decoder framework. The encoder transforms the input $X_{t}\oplus C_{t}$ into semantic vectors as

Then, at each $i$-th step of generation, the decoder updates its state vector $\textbf{s}_{i}$ and samples a word from distribution $\mathbf{o}_{i}$ as follows:

where $\mathbf{W}_{o}$ is the weight matrix of the decoder. The decoder’s state is updated by

where $\mathbf{Att}(\mathbf{X}_{t}\oplus\mathbf{C}_{t};\mathbf{s}_{i-1})$ is an attentive read of the encoded input conditioned on state $\mathbf{s}_{i-1}$, typically using attention mechanism (Bahdanau et al., 2015); and $\mathbf{y}_{i-1}$ is the vector representation of the previously generated word $y_{i-1}$.

The formulation of generation-based models mentioned above is auto-regressive in that these models generate a target sequence word by word, each word conditioned on the words that are previously generated. To make the decoding parallelizable, non-autoregressive models based on Transformer have been proposed to generate all the tokens simultaneously (Kaiser et al., 2018; Lee et al., 2018). Non-autoregressive modeling factorizes the distribution over a target sequence given a query into a product of conditionally independent per-step distributions, as follows:

Though the performance of such non-autoregressive models is still not as good as their autoregressive counterparts, it opens new opportunities for fast training using very large scale datasets (Gu et al., 2017; Lee et al., 2018).

Noticeably, the large-scale pre-trained models, such as BERT and GPT-2 (Devlin et al., 2019; Radford et al., 2018), can be easily applied in the above encoder-decoder framework. The encoder can be a pre-trained BERT model or a GPT-2 model, the decoder a GPT-2 model. Both the parameters of the encoder and the decoder are initialized using the pre-trained models and then fine-tuned on a dialog corpus (Radford et al., 2018; Wolf et al., 2018; Golovanov et al., 2019; Zhang et al., 2019f; Adiwardana et al., 2020). The fine-tuning process is often tailored to the dialog scenario via encoding with dialog state embeddings(Wolf et al., 2018), classifying golden and negatively sampled responses given the same dialog context(Golovanov et al., 2019), designing dialog-specific pre-training tasks (Mehri et al., 2019; Budzianowski and Vulić, 2019), and so on. These models have shown strong performance in the NeurIPS Conversational Intelligence Challenge 2 (ConvAI 2)http://convai.io/ and were used in the TREC Conversational Assistance Track (Conversational Information Seeking)http://www.treccast.ai/. Notably, Zhang et al. (2019f) released the DialoGPT model that was trained on 147M conversation-like exchanges extracted from on Reddit comment threads, providing a good starting point for future research.

3. Hybrid Methods

Retrieval-based methods retrieve an output response from a repository of human-human conversations. Such human-produced conversations are fluent, grammatical, and of high quality. However, the scale of the repository is critical to the success of the methods, which unfortunately is never large enough for open-domain dialog systems. Moreover, retrieval-based methods cannot generate unseen responses. On the other hand, generation-based methods can produce novel responses. But they often generate undesirable responses that are either ungrammatical or irrelevant. Hybrid methods combine the strengths of both and usually adopt a two-stage procedure (Yang et al., 2019; Weston et al., 2018; Zhou et al., 2018a). In the first stage, some relevant conversations, known as prototype responses in (Wu et al., 2019b), are retrieved from a dataset using input $X\oplus C$ as a query. Then, prototype responses are used to help generate new responses in the second stage.

Based on the Seq2Seq architecture, Song et al. (2018) used additional encoders to represent the set of retrieved responses, and applied the attention (Bahdanau et al., 2015) and copy (Gu et al., 2016) mechanism in decoding to generate new responses. Pandey et al. (2018) first retrieved similar conversations from training data using a TF-IDF model. The retrieved responses were used to create exemplar vectors that were used by the decoder to generate a new response. Wu et al. (2019b) first retrieved a prototype response from training data and then edited the prototype response according to the differences between the prototype context and current context. The motivation is that the retrieved prototype provides a good start-point for generation because it is grammatical and informative, and the post-editing process further improves the relevance and coherence of the prototype. Zhang et al. (2019g) proposed an adversarial learning framework to enhance a retrieval-generation ensemble model. Their model consists of a language-model-like generator, a ranker generator, and a ranker discriminator. This model encourages the two generators to generate responses that are scored higher by the discriminative ranker, while the discriminator down-weighs adversarial samples and selects those responses that are favored by the two generators.

Semantics

A typical symptom of a dialog system that suffers from the semantics issue is that it often generates bland and generic responses, such as “I don’t know”, “thank you”, “OK” , or simply repeats whatever a user says (Sordoni et al., 2015; Vinyals and Le, 2015; Serban et al., 2016; Gao et al., 2019a). We observe similar phenomena in human conversations. When we don’t understand what the other party is talking about but have to respond, we often pick those safe but bland responses.

To make an engaging conversation, the dialog system needs to produce contentful, interesting, and interpersonal responses based on its understanding of the dialog content, user’s sentiment and emotion, and real-world knowledge that is related to the dialog. In this section, we review some of the most prominent neural approaches that have been proposed recently to address the semantics issue. We first describe the ways of improving the encoder-decoder framework to generate diverse and informative responses by improving the understanding (embedding) of dialog context and users. Then, we describe the methods of grounding dialog in real-world knowledge to make system responses more contentful.

Most state of the art neural response generation models are based on the encoder-decoder framework which consists of four components: (1) an encoder that encodes user input and dialog context, (2) an intermediate representation, (3) an decoder that generates candidate responses, and (4) a ranker that picks the best candidate as the response. In what follows, we review the proposed methods in four categories, each focusing on improving one of the four components.

Encoding richer information from query $X\oplus C$, such as longer dialog history (Sordoni et al., 2015), persona (Li et al., 2016b), hidden topics (Serban et al., 2017), has proved to be helpful for generating more informative responses. Xing et al. (2017) extracted topic words, rather than hidden topic vectors, using LDA, and encoded such words in a topic-aware model. The model generates a response by jointly attending to query $X\oplus C$ and the topic words. Topic words are also used to model topic transition in multi-turn conversations (Wang et al., 2018a). The hybrid methods described in Section 2.3 (Pandey et al., 2018; Song et al., 2018; Wu et al., 2019b) encode the retrieved prototype responses to help generate more informative responses.

Intermediate Representation

Instead of encoding $X\oplus C$ using a fixed-size vector as in (Sutskever et al., 2014), methods have been proposed to use more flexible intermediate representations (e.g., additional latent variables) to enhance the representation capability to address the one-to-many issue in dialog, and to improve the interpretability of the representation in order to better control the response generation. Zhao et al. (2017) introduced CVAE for dialogue generation and adopted a Gaussian distribution, rather than a fixed-size vector, as the intermediate representation, thus obtaining more diverse responses via sampling the latent variable. Du et al. (2018) introduced a sequence of continuous latent variables to model response diversity, and demonstrated empirically that it is more effective than using a single latent variable. Zhao et al. (2018) proposed an unsupervised representation learning method to use discrete latent variables, instead of dense continuous ones, which improves the interpretability of representation. Zhou et al. (2017, 2018d) assumed that there exist some latent responding mechanisms, each of which can generates different responses for a single input post. These responding mechanisms are modeled as latent embeddings, and can be used to encode the input into mechanism-aware context to generate responses with the controlled generation styles and topics. Gao et al. (2019b) proposed a SpaceFusion model which induces a latent space that fuses the two latent spaces generated by Seq2Seq and auto-encoder, respectively, in such a way that after encoding $X\oplus C$ into a vector in the space, the distance and direction from the predicted response vector given the context roughly match the relevance and diversity, respectively.

Decoder

Assigning additional probability mass to desirable words in decoder is a commonly used method to gain some control of what to generate. Mathematically, this can be implemented by adjusting the output word distribution as follows:

where $y_{<i}=y_{1}y_{2}\cdots y_{i-1}$ is the generated prefix; $P_{bias}$ assigns additional probability mass to the desirable words to be generated; and $\textbf{Normalize}(\cdot)$ is a normalization function to ensure a probability distribution. Many existing controllable decoding methods essentially fall into this formulation. The most notable example is CopyNet (Gu et al., 2016), which copies desirable but infrequent words from the input to the output, thus assigning higher probabilities to those words. In (Zhang et al., 2018c), $P_{bias}$ is formulated as a Gaussian distribution, which assigns higher probabilities to rare words to control the specificity of a response, where the specificity score of a word is proportional to its IDF (inverse document frequency) score.

Candidate Ranker

To obtain more diverse responses, beam search is commonly used to generate multiple candidates, which (together with retrieved candidates in hybrid dialog systems) are then ranked by another model, which uses information that is not available in decoding (e.g., mutual information between input and response) or is too expensive to use in decoding (e.g., a large pre-trained language model such as BERT (Devlin et al., 2019)) to select the final response. Li et al. (2016a) proposed to use Maximum Mutual Information (MMI) as the objective to rank candidates to promote the diversity of generated responses. As the standard beam search often produces near-identical results, recent work addresses it by encouraging the diversity among (partial) hypotheses in the beam. For example, Li et al. (2016c) penalized lower-ranked siblings extended from the same parents, so that the N-best hypotheses in the beam at each time step are more likely to expand from different parents, and thus more diverse. Vijayakumar et al. (2018) divided the hypotheses into several groups and applied beam search group-by-group. The model favours the hypotheses that are dissimilar to the ones in the previous groups. Constrained beam search (Balakrishnan et al., 2019) was also proposed to generate desirable responses by constraining a generated response to obey the input structure.

2. Knowledge Grounded Dialog Models

Knowledge is crucial for language understanding and generation. To build effective human-machine interactions, it is indispensable to ground the concepts, entities, and relations in text in commonsense knowledge or real-world facts such as those stored in Freebase and Wikipedia. An knowledge-grounded open-domain dialog system should be able to identify the entities and topics mentioned in user input, link them into real-world facts, retrieve related background information, and thereby respond users in a proactive way e.g., by recommending new, related topics to discuss.

Knowledge has been shown useful in both retrieval-based and generation-based dialog systems. A well-known example of the former is Microsoft XiaoIce (Zhou et al., 2018a). XiaoIce relies on a large knowledge graph (KG) to identify the topics and knowledge related to user input for both response generation and topic management. In (Young et al., 2018), a Tri-LSTM model is proposed to use commonsense knowledge as external memories to facilitate the model to encode commonsense assertions for response selection. An early example of using knowledge for generating responses is (Han et al., 2015), where manually crafted templates are used to generate responses which are filled with relevant knowledge triples. In (Ghazvininejad et al., 2018), a knowledge-grounded model is proposed to generate a response by incorporating some retrieved posts that are relevant to the input. However, the quality of these unstructured posts is mixed. Pre-compiled structured knowledge, which is in the form of fact triples, is believed to be of higher quality and has been shown to more helpful in conversation generation (Zhu et al., 2017; Liu et al., 2018a). Zhu et al. (2017) dealt with a scenario where two speakers are conversing based on each other’s private knowledge base in the music domain. The generation model can generate a word in response from the context or the knowledge base. In (Liu et al., 2018a), a knowledge diffusion model is proposed to not only answer factoid questions based on a knowledge base, but also generate an appropriate response containing knowledge base entities that are relevant to the input. Zhou et al. (2018f) exploited the use of large-scale commonsense knowledge for conversation generation. First, a one-hop subgraph is retrieved from ConceptNet (Speer et al., 2017) for each word in an input post. Then, the word vectors, along with the graph vectors which extend the meaning of the word via its neighboring entities and relations, are used to encode the input post. During decoding, a graph attention mechanism is applied in which the model first attends to a knowledge graph and then to a triple within each graph, and the decoder chooses a word to generate from either the graph or the common vocabulary. Qin et al. (2019) presented a new end-to-end approach that jointly models response generation and on-demand machine reading for generating contentful conversations. The key idea is to provide the model with relevant long-form text on the fly as a source of external knowledge. The model performs QA-style reading comprehension on this text in response to each conversational turn, thereby allowing for more focused integration of external knowledge than prior approaches.

We summarize the aforementioned knowledge-grounded dialog systems in Table 2. Most these studies focus on two problems: (1) knowledge selection – selecting appropriate knowledge to be incorporated in the next response given the dialog context and previously-selected knowledge (Lian et al., 2019; Ren et al., 2019; Zhang et al., 2019e; Liu et al., 2019), and (2) knowledge-aware generation – injecting the required knowledge into a generated response (Ghazvininejad et al., 2018; Zhou et al., 2018f; Li et al., 2019; Qin et al., 2019). In addition, zero-shot adaptation to updated, unseen knowledge graphs without conversational data (Chen and Lee, 2019) is worth more comprehensive exploration in the future. solving the problem would allow dialog systems to generate proper responses with selected knowledge even though the knowledge has never been used.

Recently, there is a significant burst in constructing document or knowledge grounded dialog corpora (Zhou et al., 2018e; Moghe et al., 2018; Dinan et al., 2018; Moon et al., 2019; Chen and Lee, 2019; Wu et al., 2019a; Gopalakrishnan et al., 2019; Qin et al., 2019), which will be described in Section 7 in details.

Consistency

A human-like dialog system needs to embody consistent behaviors, so that it can gain the user’s confidence and trust (Shum et al., 2018; Zhou et al., 2018a). The consistency issue refers to generating responses that are consistent in persona, style, and context (with respect to topic, logic, causality, etc.). We group existing studies into three lines: (1) persona consistency modeling including implicit and explicit methods, (2) stylistic response generation, and (3) contextual consistency.

Existing dialog models that address persona consistency can be roughly grouped into two catetories: implicit personalization and explicit personalization. In the former, the persona is implicitly represented by a persona vector. For instance, Kim et al. (2014) proposed a ranking-based approach to integrate a personal knowledge base and user interests in dialogue system. Bang et al. (2015) extended the user input by exploiting examples retrieved from her personal knowledge base to help identify the candidate responses that fit her persona. Li et al. (2016b); Zhang et al. (2017) used an embedding vector to represent a user (speaker) persona and fed the user embedding into each decoding position of the decoder. Such models need to be trained using conversational data labeled by user identifiers, which is expensive to collect for large quantities. Thus, Wang et al. (2017c) proposed to train personalized models with only group attributes (e.g., male or female). The group attributes are embedded to vectors and then fed into the decoder for response generation. Zhang et al. (2019b) proposed a neural conversation model that generates consistent responses by maintaining certain features related to topics and personas throughout the conversation. Unlike other work that requires external supervision such as user identities, which are often unavailable, this approach trains topic and persona feature extractors in a self-supervised way by utilizing the natural structure of dialogue data. Although Zhang et al. (2018d); Ouchi and Tsuboi (2016) showed that user embedding is an effective technique to distinguish roles of speakers and addressees in multi-party conversation, personalization in these models are handled in an implicit way and thus not easy to interpret and control in generating desired responses.

In (Qian et al., 2018), an explicit persona model is proposed to generate personality-coherent responses given a pre-specified user profile. The chatbot’s persona is defined by a key-value table (i.e., profile) which consists of name, gender, age, hobbies, and so on. During generation, the model first chooses a key-value from the profile and then decodes a response from the chosen key-value pair forward and backward. This model can be trained on generic dialogue data without user identifier. XiaoIce also uses an explicit persona model (Zhou et al., 2018a).

We have discussed two categories of methods for modeling persona consistency: implicit modeling (Li et al., 2016b; Zhang et al., 2019b) which utilizes learned user persona features to capture user-level consistency implicitly, and explicit modeling (Qian et al., 2018; Zhou et al., 2018a) which controls the conversation generation using explicitly-defined user profile. However, most existing methods are insufficient in modeling the user’s psychological personality. For instance, we do not yet have a dialog system that can exhibit extrovert or introvert personality. Building such an intelligent dialog system requires breakthroughs in multi-disciplined research on psychology, cognitive, and social science.

2. Stylistic Response Generation

Stylistic response generation (Wang et al., 2017b; Oraby et al., 2018) can be viewed as a form of personalization in conversation. There are two main challenges: how to disentangle content and style in representation, and how to construct training data containing pairs of responses that are of the same content but in different styles. Wang et al. (2017b) utilized a small-scale stylistic data and proposed a topic embedding model to generate responses in specific styles and topics simultaneously. Oraby et al. (2018) demonstrated that it is possible to automate the construction of a parallel corpus where each meaning representation can be realized in different styles with controllable stylistic parameters.

Stylistic conversation generation is closely related to domain adaptation and transfer learning (Zhang et al., 2017; Mo et al., 2018; Casanueva et al., 2015; Wang et al., 2017b). The idea is to first train a general conversation model on a large corpus in source domain and then to transfer the model to a new speaker or target domain using small amounts of personalized (or stylistic) data in target domain. Casanueva et al. (2015) proposed to automatically gather conversations from similar speakers to improve the performance of policy learning of personalized dialogue systems. Zhang et al. (2017) proposed a two-phase transfer learning approach, namely initialization then adaptation, to generate personalized responses. They also proposed a quasi-Turing test method to evaluate the performance of the generated responses. Yang et al. (2017) presented a transfer learning framework similar to Zhang et al. (2017), but proposed to use a new adaptation mechanism based on reinforcement learning. Luan et al. (2017) proposed a multi-task learning approach where the response generation and utterance representation are treated as two sub-tasks for speaker role adaptation.

3. Contextual Consistency

Unlike the studies on persona consistency, the work on modeling contextual consistency is yet to be explored. Early work has focused on better representing dialog contexts (Serban et al., 2015b, 2017) using hierarchical models, which can be viewed as implicit modeling of contextual consistency. Recently, Welleck et al. (2018) and Dziri et al. (2019) characterized the contextual consistency as a natural language inference (NLI) problem (Dagan et al., 2005). In this setting, a response is considered consistent if it can be inferred from the dialog context or the given persona. Welleck et al. (2018) constructed a dialog NLI dataset based on Persona-Chat(Zhang et al., 2018b). Zhang et al. (2019b) proposed to learn topic features from dialog context on-the-fly and utilize controllable response generation techniques to generate topic-consistent responses.

Interactiveness

This issue is mainly about how to optimize the behaviors and strategies of a dialog system to maximize long-term user engagement and accomplish long-term, complex goals such as providing emotional comfort, or even psychological counseling (Althoff et al., 2016; Pérez-Rosas et al., 2017; Zhang et al., 2019a; Cao et al., 2019). To improve interactiveness, it is important to understand user’s emotion and affect, in addition to dialog content, and to optimize the system’s behavior and interaction strategy in multi-turn conversations.

Emotion perception and expression is vital for building a human-like dialog system. Earlier attempts to building emotional dialog systems are mostly inspired by psychology findings. Those systems are either rule-based or trained on small-scale data, and work well only in a controlled environment. Thanks to the availability of large-scale data and the recent progress on neural conversational AI, many neural response generation models have been proposed to perceive and express emotions in an open-domain dialog setting. Zhou et al. (2018b) proposed Emotional Chatting Machine (ECM) to generate emotional responses given a pre-specified emotion. ECM consists of three components: (1) an emotion category embedding which is fed into each decoding position, (2) an internal emotion state which assumes that the emotion state decays gradually and finally to zero during decoding, and (3) an external memory which allows the model to choose emotional (e.g., lovely) or generic (e.g., person) words explicitly at each decoding step. The authors also presented some typical emotion interaction patterns in human-human conversations such as empathy and comfort, which would inspire more fine-grained design of emotion interaction between human and machine. Asghar et al. (2018) developed a method of affective response generation that consists of three components: (1) the affective vectors based on Valence/Arousal/Dominance dimensions (Warriner et al., 2013; Colombo et al., 2019), which serve as a supplement to word vectors; (2) the affective loss functions which maximize or minimize the affective consistency between a post and a response; and (3) the affective beam search algorithm for seeking affective responses. In (Zhou and Wang, 2018), a conditional variational autoencoder is proposed to generate more emotional responses conditioned on an input post and some pre-specified emojis. Huber et al. (2018) studied how emotion can be grounded in an image to generate more affective conversations. In addition to text, the decoder takes as input the scene, sentiment, and facial coding features extracted from a given image. Recently, an empathetic dialog corpus is developed to facilitate the research on modeling empathetic interactions in conversation (Rashkin et al., 2019). We will present dialog datasets in Section 7.

Controlling the emotion or sentiment has become a popular topic in language generation (Hu et al., 2017; Radford et al., 2017; Ghosh et al., 2017). In (Radford et al., 2017), an RNN-based language model is trained on large-scale review data where some neurons are reported to be highly correlated with sentiment expression. Ghosh et al. (2017) proposed an affective language model which generates an affective sequence from a leading context. At each decoding position, the model estimates an affective vector of the already generated prefix by keyword spotting using the Linguistic Inquiry and Word Count (LIWC) dictionary (Pennebaker et al., 2001). The vector is then used to generate the next word. In (Wang and Wan, 2018), to generate the reviews of a particular polarity, the authors proposed a multi-class generative adversarial network which consists of multiple generators for multi-class polarities and a multi-class discriminator.

Despite the research effort reviewed, it is still challenging for a dialog system to express complex emotions in natural language. One difficulty is emotion representation. A simple approach is to project an emotion label to a vector (Zhou et al., 2018b), which is implicit, unexplainable, and subtle. A more sophisticated method is to use Valence/Arousal/Dominance representations: the emotion of each word, sentence, and user state can be represented as V-A-D vectors (Warriner et al., 2013; Colombo et al., 2019), which is intended to capture psychological and linguistic clues beyond the emotion vector. Another issue of most existing work is that the user’s emotion transition during a conversation (Majumder et al., 2019) is not explicitly modeled. This is crucial for a dialog system to establish a long-term connection with a user because the user is more willing to engage with the system if the system can always detect negative change of her emotion during the conversation and cheer her up through e.g., shifting to new topics that are more comfortable for both parties.

2. Modeling Conversation Behavior and Strategy

As pointed out in (Zhou et al., 2018a), an open-domain dialog system needs to have enough social skills to have engaging conversations with users and eventually establish long-term emotional connections with users. These social skills include topic planning and dialog policy which can determine whether to drive the conversation to a new topic when e.g., the conversation has stalled, or whether or not to be actively listening when the user herself is engaged in the conversation. Nothdurft et al. (2015) elucidated the challenges of proactiveness in dialogue systems and how they influence the effectiveness of turn-taking behaviour in multimodal and unimodal dialogue systems. Yu et al. (2016) proposed several generic conversational strategies (including grounding on entities and OOV words, topic switch, activity initiation, and joke telling) to handle possible system breakdowns in non-task-oriented dialog systems, and designed policies to select these strategies according to dialog context. Zhang et al. (2018a) addressed the problem of predicting from the very beginning of a conversation whether it will get out of hand. The authors developed a framework for capturing pragmatic devices, such as politeness strategies and rhetorical prompts, used to start a conversation, and analyzed their relation to its future trajectory. Applying this framework in a controlled setting, it is possible to detect early warning signs of antisocial behavior in online discussions.

The above studies inspire researchers to devise new methods of incorporating social skills into an open-domain dialog system. In (Li et al., 2016d), a retrieval-based method is proposed to first detect the sign of stalemate using rules, and then retrieve responses that contain the entities that are relevant to the input, assuming that a proactive reply should contain the entities that can be triggered from the ones in the input. Yan and Zhao (2018) proposed a proactive suggestion method where a look-ahead post for a user is decoded in addition to the system response, conditioned on the context and the previously generated response. The user can use the generated post directly, or type a new one during conversation. Wang et al. (2018b) argued that asking good questions in conversation is shown to be an important proactive behavior. A typed decoder is proposed to generate meaningful questions by predicting a type distribution over topic words, interrogatives, and ordinary words at each decoding position. The final output distribution is modeled by the type distribution, leading to a strong control over the question to be generated. Rao and Daumé III (2018) also argued that question asking is fundamental to communication, and that a good question is the one whose expected answer will be useful. They built a neural network model for ranking clarification questions, evaluated on a dataset of clarification questions (post-question pairs) extracted from StackExchange. Ke et al. (2018) conducted a systematic study of generating responses with different sentence functions, such as interrogative, imperative, and declarative sentences. These sentence functions play different roles in conversations. For instance, imperative responses are used to make requests, give directions and instructions, or elicit further interactions while declarative responses make statements or explanations. Tang et al. (2019) proposed a new dialog planning task in which the conversation should eventually reach a target (defined by a topical keyword) from any initial topics. In such a task, it is required to plan proactively the topic path to the final target.

There are two important directions for future research. First is the comprehensive investigation of conversation behaviors in human-human dialog. This is still largely ignored, possibly due to the lack of real-world conversations. The dialog data in online forums (Zhang et al., 2018a) and psychological counseling (Althoff et al., 2016; Pérez-Rosas et al., 2017; Zhang et al., 2019a; Cao et al., 2019) are of high value for this research. But the data in a wide variety of scenarios are still in significant shortage. Second is to create a more sophisticated real-world dialog setting for system development and evaluation. Existing work largely targets at modeling atomic strategy in dialog systems, namely, single strategy for emotion interaction (Zhou et al., 2018b), topic control (Wang et al., 2018a), question asking (Wang et al., 2018b; Ke et al., 2018), and so on. Most of the studies are merely evaluated with the single-turn setting. However, to accomplish more complex social goals such as emotional comfort or counseling, it is necessary to design composite strategies that consider emotion, topic, and proactivity comprehensively in multi-turn conversation. Therefore, there is increasing demand for collecting or constructing more complex dialog data with well-designed task goals, and for developing more sophisticated dialog policy models.

Open-domain Dialog Evaluation

Evaluating the quality of an open-domain dialog system is challenging because open-domain conversations are inherently open-ended (Ram et al., 2018). For example, if a user asks the question ”what do you think of Michael Jackson?”, there are hundreds of distinct but plausible responses. Evaluation of a dialog system can be performed manually or in an automatic way. In manual evaluation, human judges are hired to assess the generated results in terms of predefined metrics, with well-documented guidelines and exemplars. Evaluation is conducted by either scoring each individual result (point-wise) or comparing two competing results (pair-wise). In some dialog evaluation challenges, manual evaluation is commonly adopted in the final-stage competition (Dinan et al., 2019; Ram et al., 2018). For instance, the second conversational intelligence challenge (Dinan et al., 2019) adopted manual evaluation by paid workers from Amazon Mechanical Turk and unpaid volunteers, and the organizers reported the rating difference between the two user groups: the volunteers’ evaluation had relatively fewer good (i.e. long and consistent) dialogues, while paid workers tended to rate the models higher than the volunteers.

Since manual evaluation is expensive, time-consuming, and not always reproducible, automatic evaluation is more frequently used, especially at the early stage of development. For retrieval-based methods, traditional information retrieval evaluation metrics such as precision@k, mean average precision (MAP), and normalized Discounted Cumulative Gain (nDCG) (Manning et al., 2008) are applicable. For generation-based models, metrics such as perplexity, BLEU (Papineni et al., 2002), and distinct-$n$ (Li et al., 2016a), are widely used. Perplexity measures how well a probabilistic model fits the data, and is a strong indicator whether the generated text is grammatical. BLEU, adopted from machine translation, measures the lexical overlap between the generated responses and the reference ones. Distinct-$n$ measures the diversity by computing the proportion of unique $n$-grams in a generated set. However, (Liu et al., 2016) argued that automatic metrics such as BLEU, ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) all have low correlation with manual evaluation. But as pointed out in (Gao et al., 2019a), the correlation analysis in (Liu et al., 2016) is performed at the sentence level while BLEU is designed from the outset to be used as a corpus-level metric. (Galley et al., 2015) showed that the correlation of string-based metrics (BLEU and deltaBLEU) significantly increases with the units of measurement bigger than a sentence. Nevertheless, in open-domain dialog systems, the same input may have many plausible responses that differ in topics or contents significantly. Therefore, low BLEU (or other metrics) scores do not necessarily indicate low quality as the number of reference responses is always limited in test set. Therefore, there has been significant debate as to whether such automatic metrics are appropriate for evaluating open-domain dialog systems (Gao et al., 2019a).

Recently, trainable metrics for open-domain dialog evaluation have attracted some research efforts. Lowe et al. (2017) proposed a machine-learned metric, called ADEM, for open-domain dialog evaluation. They presented a variant of the VHRED model (Serban et al., 2017) that takes context, user input, gold and system responses as input, and produces a qualitative score between 1 and 5. The authors claimed that the learned metric correlates better with human evaluation than BLEU and ROUGE. (Tao et al., 2018) proposed an evaluation model, called RUBER, which does not rely on human judged scores. RUBER consists of a referenced component to measure the overlap between a system response and a reference response, and an unreferenced component to measure the correlation between the system response and the input utterance. However, as pointed out in (Sai et al., 2019), ADEM can be easily fooled with a variation as simple as reversing the word order in the text. Their experiments on several such adversarial scenarios draw out counter-intuitive scores on the dialogue responses. In fact, any trainable metrics lead to potential problems such as overfitting and “gaming of the metric” In discussing the potential pitfalls of machine-learned evaluation metrics, Albrecht and Hwa (2007) argued for example that it would be “prudent to defend against the potential of a system gaming a subset of the features.” In the case of deep learning, this gaming would be reminiscent of making non-random perturbations to an input to drastically change the network’s predictions, as it was done, e.g., with images in (Szegedy et al., 2013) to show how easily deep learning models can be fooled. Readers refer to Chapter 5 in Gao et al. (2019a) for a detailed discussion. (Albrecht and Hwa, 2007), which might explain why none of the previously proposed machine-learned evaluation metrics (Corston-Oliver et al., 2001; Kulesza and Shieber, 2004; Lita et al., 2005; Albrecht and Hwa, 2007; Giménez and Màrquez, 2008; Pado et al., 2009; Stanojević and Sima’an, 2014, etc.) is used in official machine translation benchmarks. Readers refer to (Gao et al., 2019a) for a detailed discussion.

These research attempts indicate that what makes a good conversation is a challenging question to answer. See et al. (2019) discussed four attributes that are associated with the control of open-domain dialog generation: repetition, specificity, response-relatedness, and question-asking. They argued that existing work has ignored the importance of the conversational flow, because existing models repeat or contradict previous statements, fail to balance specificity with genericness, and are unable to balance asking questions with other dialogue acts. Experiments on Persona-Chat (Zhang et al., 2018b) show that higher engagingness scores in human judgement can be obtained by optimizing the control of the four attributes in multi-turn conversations. Therefore, considering these attributes in automatic evaluation, implicitly or explicitly, is expected to lead to new evaluation metrics that correlate well with human evaluation.

Recently, there are research attempts to combine human evaluation and automatic evaluations for natural language generation systems. Hashimoto et al. (2019) argued that human evaluation captures quality but not diversity while statistical evaluation (i.e., perplexity) captures diversity but not quality. They proposed a unified framework which evaluates both in terms of the optimal error rate of predicting whether a sentence is human- or machine-generated. As mentioned above, automatic metrics such as sentence-level BLEU correlates poorly with human judgement, thereby easily leading to systematic bias against model improvements. On the other hand, the average of human judgements is unbiased but is very expensive to collect. Therefore, Chaganty et al. (2018) combined automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than using solely human evaluation.

All of the above research suggests that automatic evaluation of dialog systems is by no means a solved problem. We argue that, for open-domain dialog evaluation, the major difficulty derives from in the one-to-many essence: in any given dataset, the number of observable responses for the same input post is limited, yet there are many appropriate responses not presented in the dataset. Therefore, automatic metrics that are trained on a dataset will be inherently questionable because the topic coverage and the number of observable outputs are largely limited by the dataset. Thus, uncovering those underlying outputs for an input post is an interesting area for future research.

Open-domain Dialog Corpora

Recently, the availability of dialog corpora has largely advanced the development of neural models for open-domain conversation generation. An incomplete survey on these dialog datasets is presented in Table 3Readers may refer to an old survey published in 2015, which covers datasets for both open-domain and task-oriented dialog models (Serban et al., 2015a). We only list the corpora that are frequently used or recently proposed in the literature, most of which are not covered by (Serban et al., 2015a). . These corpora differ in topic, source (where or how the data is collected), language, data scale, and the design features.

Short Text Conversation (STC) (Shang et al., 2015): This corpus is collected from a Chinese social media, Weibo. There are 219,905 posts and 4,308,211 responses in the training data. It can be used for studying the one-to-many problem in dialog modeling since each post has multiple responses. On top of this corpus, Zhou et al. (2018b) proposed an emotional STC dataset (ESTC) in which each utterance is tagged in terms of six emotion classes by an emotion classifier with an accuracy of 62.3%. ESTC is frequently used in building empathetic dialog systems (Zhou et al., 2018b).

Twitter Triple Corpus (Sordoni et al., 2015): This corpus contains 29M context-message-response triples from the Twitter FireHose, covering the 3-month period from June 2012 through August 2012. Additionally, the validation and test sets have 4,232 triples which are scored no less than 4 in 5-point scale by human annotators. However, this corpus is not publicly available.

PersonalDialog (Zheng et al., 2019): This corpus is constructed toward building personalized conversation models. The data is collected from a Chinese social media, Weibo. Each dialogue is composed of a post and its following replies from different users. The personal profile of each user is collected, which includes five personality traits: Gender, Age, Location, Interest Tags, and Self Description. This dataset contains 20.83M conversations and 8.47M user profiles. The total number of utterances are 56.25M and each utterance contains 9.35 tokens. A considerable amount of dialogues (3.43M sessions) in this dataset have multiple turns (more than 4 utterances). This corpus is the first dialogue corpus that contains real social conversations and diversified personality traits for each user.

DailyDialog (Li et al., 2017b): This corpus contains multi-turn dialogs on daily life topics. The raw data were crawled from several websites which serve for English learner to practice English. The dataset contains 13,118 dialogs, with an average of 7.9 turns per dialog and 14.6 words per turn. The appealing feature of this corpus is that it provides manual annotation on intent (Inform, Questions, Directives, and Commissive) and emotion (Anger, Disgust, Fear, Happiness, Sadness, and Surprise), which may support the research on emotion interaction and dialog act modeling.

Ubuntu Dialog Corpus (Lowe et al., 2015): This corpus contains two-party conversations that solve technical issues with Ubuntu. The data were extracted from online conversation logs in Ubuntu-related chat rooms on the Freenode Internet Relay Chat (IRC) network. In each log, a user may ask a technical question to be solved and other users can respond to the question. The log session will terminate until the problem is solved. A two-party conversation will be extracted from the chat logEach chat log is a multi-party conversation, but only two-party sub-conversations which involve the same two users are retained.. The corpus contains 930,000 human-human dialogs and 7,100,000 utterances, with an average of 7.71 turns per dialog and 10.34 words per utterance. Strictly speaking, this dataset is task-specific instead of open-domain conversation. This corpus is commonly used to evaluate retrieval-related models.

Persona-Chat (Zhang et al., 2018b): This crowdsourced corpus is designed for personalized dialog modeling. In each conversation, each worker is given a persona which is defined by up to 5 sentences describing personal hobby or state (e.g., I like swimming, or I need to lose weight). Two workers are instructed to know each other through interaction. During the conversation, each worker should follow her own persona and try to know the partner’s information. The dataset consists of 10,981 dialogs with 164,356 utterances.

CMU Document-grounded conversation (CMU DOG) (Zhou et al., 2018e): This corpus, designed for document or knowledge grounded dialog modeling, contains crowd-sourced conversations that are talking about 30 movies. The information about each movie is given through a correspondent Wikipedia article. There are two modes for data collection: only one worker has the movie document and both workers have the movie document during conversation. The dataset consists of 4,112 conversations with an average of 31.6 utterances per dialog and 10.8 words per utterance.

(Moghe et al., 2018): The corpus can be viewed as an expanded version of CMU DOG. The conversations discuss about 921 movies, and the knowledge about each movie is composed of a fact table which contains box office collection, similar movies (for recommendation), awards, and tag-lines., the plot description, and reviews and comments on the movie. The corpus contains 9,071 conversations and 90,810 utterances with an average of 10 utterances per dialog and 15.3 words per utterance. The corpus is useful for studying the use of heterogeneous knowledge in conversation generation.

Wizard of Wikipedia (Dinan et al., 2018): This corpus contains conversations that are grounded with knowledge retrieved from Wikipedia. The dataset covers 1,365 topics, each linked to a Wikipedia article. These topics include commuting, Gouda cheese, music festivals, podcasts, bowling, and Arnold Schwarzenegger. Each conversation is made between a knowledge expert and a curious learner, and the expert has full access to the Wikipedia article of a topic but the learner does not. The corpus consists of 22,311 dialogues and 201,999 utterances, with an average of 9 utterances per dialog. Each utterance is grounded to a selected knowledge sentence or indicated by that no knowledge is used.

Grounded Response Generation at DSTC7 (Qin et al., 2019): The dataset, which is first released for the ”sentence generation” task at the 7th Dialog System Technology Challenges (DSTC7) (Galley et al., 2019), is developed for grounded conversation modeling. It consists of conversation threads extracted from Reddit data. Each conversation contains exactly one URL to a web page (grounding) that defines the topic of the conversation. The dataset contains 2.8M conversation instances respectively divided into train, validation, and test based on date ranges: years 2011-2016 for train, Jan-Mar 2017 for validation, and the rest of 2017 for test, which consists of 2,208 conversational turns, each with 6 human responses. To access the human performance using the test set, one of the 6 human responses is set aside, and the remaining 5 responses serve as ground truths for evaluating different systems.

Topical-Chat (Gopalakrishnan et al., 2019): This corpus is designed towards building dialog systems that can converse with humans on various topics. It covers 300 popular topic entities spanning 8 domains including fashion, politics, books, and sports. For each entity, the authors fetched the Wikipedia lead section, and crowdsourced 8-10 fun facts. Furthermore, they fetched Washington Post articles in 2018 that each referenced 3 or more of the 300 entities. The authors then created a set of reading sets, each containing the wiki-information, several fun facts, and a Washington Post article. Workers were partnered up to converse, with symmetric or asymmetric settings where symmetric means two workers have the same reading set, and asymmetric with different sets. The dataset contains 11,319 conversations and 248,014 utterances with an average of 22 turns per dialog and 19.8 words per turn.

OpenDialKG (Moon et al., 2019): In this corpus, each dialog is paired with its corresponding ”knowledge graph (KG) paths” that weave together the KG entities and relations. It was collected with a Wizard-of-Oz setting by connecting two crowd-workers to engage in a chat session. The first worker is given a seed entity and asked to initiate a conversation about that entity. The second worker is provided with a list of facts relevant to that entity, and asked to choose the most natural and relevant facts and use them to frame a free-form conversational response. After the second worker sends her response, new multi-hop facts from KG are surfaced to include paths initiating from new entities introduced in the latest message. The circle continues for several rounds, which simulates a random walk over the knowledge graph. The dataset covers four domains (movies, books, sports and music), with a KG of total 1,190,658 fact triples. It contains 15,673 dialogs and 91,209 turns with an average of 5.8 turns per dialog. This corpus is useful in studying conversational reasoning, while it is not yet publicly available.

DuConv (Wu et al., 2019a): This corpus covers topics on movies and film stars whose related knowledge was crawled from the Web. Then, two linked entities were randomly sampled to construct a conversation goal like ”[start]$\rightarrow entity_{a}\rightarrow entity_{b}$” where $entity_{b}$ is the final target of the conversation. Two annotators were asked to conduct knowledge-driven conversations with a leader-follower mode. The leader needs to change the conversation topics following the conversation goal and meanwhile keeps the conversation as engaging as possible. The dataset contains 29,858 dialogs and 270,399 utterances with an average of 9.1 turns per dialog and 10.6 words per turn. This corpus is useful in constructing knowledge-driven proactive dialogue systems.

DyKgChat (Chen and Lee, 2019): This corpus is collected for knowledge-grounded conversation modeling. The conversations are from the scripts of a Chinese palace drama (Hou Gong Zhen Huan Zhuang, with 76 episodes and hundreds of characters), and an English sitcom ”Friends” (with 236 episodes and six main characters). The paired knowledge graphs are manually constructed. The corpus contains 1,247/3,092 dialogs, with 13.76/18.68 turns per dialog and 27.0/16.5 words per turn for the Chinese and English TV series, respectively. The most interesting feature of this corpus is that it contains evolving knowlege graphs.

EmpatheticDialogues (Rashkin et al., 2019): This corpus is constructed toward building empathetic open-domain conversation models. The data is collected by crowd workers with a speaker-listener mode. The speaker starts the conversation from a pre-set emotion state (e.g., afraid) and a personal situation description (e.g., Speaker felt afraid when she has been hearing noises around the house at night), and the listener becomes aware of the underlying situation through what the Speaker says and responds. The corpus contains 24,850 conversations, and the average number of utterances per conversation and words per turn is 4.31/15.2 respectively. The corpus is useful in modeling emotion interactions in multi-turn conversation.

Target-Guided Conversation (Tang et al., 2019): This corpus is constructed towards building target-guided open-domain conversation models. It’s derived from Persona-Chat (Zhang et al., 2018b) without the persona information. The keywords of each utterance, which indicate the targets in this task, are automatically extracted by a rule-based keyword extractor. The corpus contains 8,939/500/500 dialogs, 101,935/5,602/5,317 utterances and 2,678/2,080/1,571 keywords in the training/validation/test set, respectively. The average number of keywords in each utterance is about 2.0. This corpus is expected to model the turn-level keyword transition and the discourse-level target-guided dialogue strategy.

PERSUASION-FOR-GOOD (Wang et al., 2019): This corpus contains persuasion conversations for charity donation where each speaker’s psychological profile attributes and sociodemographic backgrounds such as age and income were also collected. The data is collected with a persuader-persuadee mode in four steps. First, workers were asked to complete a pre-task survey to assess their psychological profile variables. Second, two workers were randomly assigned the roles of persuader and persuadee where the persuader needed to persuade the persuadee to donate part of his/her task earning to the charity, and the persuader could also choose to donate. Third, both the persuader and the persuadee were asked to input the intended donation amount privately though a text box when the conversation was ended. Last, workers were asked to complete a post-survey to assess their sociodemographic backgrounds. The corpus contains 1,017 dialogs, with an averge of 10.43 turns per dialog and 19.36 words per utterances. It also provides manual annotation in terms of persuasion strategy and dialog act for each sentence. This dataset is interesting for studying personalized dialog and complex strategy modeling.

Discussions and Future Trends

In this paper, we review the recent progress in developing open-domain dialog systems. We focus the discussion on neural approaches that have been proposed to deal with three key challenges: semantics, consistency, and interactiveness. We review open-domain dialog evaluation metrics for both manual and automatic evaluation, and share our thoughts on how to develop better automatic evaluation metrics. We survey frequently-used and recently-proposed corpora for the development of evaluation of open-domain dialog systems.

Differing from early generations of dialog assistants which are designed for simple tasks that require only short, domain-specific conversations, such as making reservation or asking for information, open-domain dialog systems are design to be AI companions that are able to have long, free-form social chats with human users. (Ram et al., 2018; Zhou et al., 2018a). Despite the recent progress as reviewed in this paper, achieving sustained, coherent, and engaging open-domain conversations remains very challenging. We conclude this paper by discussing some future research trends

To deliver contentful conversations, it is important to ground conversations in real-world topics and entities (e.g., in knowledge bases). This is part of the semantics challenge we have discussed in Section 3. Since natural language understanding in open-domain dialog systems is extremely challenging, knowledge grounding provides to some degree the ability of understanding language in dialog context, as shown in several preliminary studies (Zhou et al., 2018f; Liu et al., 2018a; Zhu et al., 2017). Even though an open-domain dialog system has no access to annotated dialog acts (which are available only for task-oriented dialog) to learn to explicitly detect an user’s intents (labeled by dialog acts), the system can still play a proactive role of leading the conversation by for example suggesting new topics, if the key concepts and entities are correctly recognized and linked to a knowledge base (Fang et al., 2018; Pichl et al., 2018; Wang et al., 2018b; Zhou et al., 2018a). Several recently proposed corpora, as described in Section 7, provide new test beds for this research.

Empathetic Computing

Sentiment and emotion form a key factor for making effective social interactions, and is crucial for building an empathetic social bot. Existing studies (Zhou et al., 2018b; Asghar et al., 2018; Zhou et al., 2018a; Zhou and Wang, 2018; Rashkin et al., 2019) in this direction are still in the infant stage, as they only deal with superficial expression of emotion. A future empathetic machine should be able to perceive a user’s emotion state and change, deliver emotionally influential conversations, and evaluate the emotional impact of its action, much of which should be tightly aligned with psychological studies. These become more important in more complicated scenarios such as psychological treatment, mental health, and emotional comforting. Moreover, it is insufficient for an empathetic machine to use only text information. The signals from other modalities such as facial expression and speech prosody should also be leveraged (Liao et al., 2018; Zhang et al., 2019d; Cui et al., 2019). To foster the research, Saha et al. (2017) developed a conversational dataset consisting of multi-modal dialog sessions in a fashion domain where each turn contains a textual utterance, one or more images, or a mix of text and images.

Personality of a Social Bot

A coherent personality is important for a social bot to gain human trust, thereby improving the consistency and interactiveness of human-machine conversations. Personality (e.g., Big five traits) has been well-defined in psychology (Norman, 1963; Gosling et al., 2003). However, existing studies (Li et al., 2016b; Qian et al., 2018; Zhang et al., 2018b; Zhou et al., 2018a) are yet to be significantly extended by incorporating the results of multidiscipline research covering psychology, cognitive science, computer science, etc. The central problem is how to ensure personality-coherent behaviors in conversations and evaluate such behaviors from the perspectives of multidisciplines, particularly via psychological studies.

Controllability of dialog generation

Most existing open-domain dialog systems are based on neural response generation models. Due to the essence of probabilistic sampling used in language generation, controllability is a challenging issue as repetitive, bland, illogical or even unethical responses are frequently observed. Controllability is closely related to the interpretability and robustness of neural network models. Achieving controllability requires new breakthroughs in modeling, such as the hybrid approaches that combine the strengths of both neural and symbolic methods.

Acknowledgement

This work was supported by the National Science Foundation of China (Grant No. 61936010/61876096), and the National Key R&D Program of China (Grant No. 2018YFC0830200). We would like to thank THUNUS NExT Joint-Lab for the support.

We would like to thank Pei Ke, Qi Zhu, Chujie Zheng, Yaoqin Zhang, Hao Zhou, Chris Brockett, Bill Dolan, and Michel Galley for their discussions and contributions to this paper. We truly thank anonymous reviewers for their valuable reviews and comments.

References