End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager

Introduction

Recent progress of designing conversational agents for commercial purposes, such as Microsoft’s Cortana, Apple’s Siri, and Amazon’s Echo, has attracted more attention from both academia and industry. Two essential components of these conversational agents are natural language understanding (NLU) and dialog manager (DM). NLU typically detects dialog domains by parsing user utterances followed by user intent classification and filling associated slots according to a domain-specific semantic template ; DM keeps monitoring the belief distribution over all possible user states underlying current user behaviors, and predicts responsive system actions . For example, given a user utterance “any action movies recommended this weekend?”, NLU predicts intent $\mathtt{request\_movie}$ and slots $\mathtt{genre}$ and $\mathtt{date}$; thereafter, DM predicts system action $\mathtt{request\_location}$.

Traditional approaches for NLU usually model tasks of domain/intent classification and slot filling separately. Sequential labeling methods, such as hidden Markov models (HMMs) and conditional random field (CRF) are widely used in slot tagging tasks ; maximum entropy and support vector machines with linear kernel (LinearSVM) are applied to user intent prediction . These models highly rely on careful feature engineering that is laborious and time-consuming. Deep learning techniques making incredible progress on learning expressive feature representations have achieved better solutions to NLU modeling in ATIS domain . The performance was improved significantly by incorporating recurrent neural networks (RNN) and CRF model . Convolutional neural networks are also used for domain/intent classification .

Slot tags and intents, as semantics representations of user behaviors, may share knowledge with each other such that separate modeling of these two tasks is constrained to take full advantage of all supervised signals. Flexible architectures of neural networks provide a way of jointly training with intent classification and slot filling . Contextual information of previous queries and domain/intent prediction was also incorporated into RNN structures .

Information flows from NLU to DM, and noisy outputs of NLU are apt to transfer errors to the following DM, so that it brings in challenges for monitoring the belief distribution and predicting system actions. Most successful approaches cast the dialog manager as a partially observable Markov decision process , which uses hand-crafted features to represent the state and action space, and requires a large number of annotated conversations or human interactions . Converting these methods into practice is far from trivial, and exact policy learning is computational intractable. Therefore, they are constrained to narrow domains.

In order to address the above problems, we propose an end-to-end deep RNN with limited contextual dialog memory that can be jointly trained by three supervised signals—user slot tagging, intent prediction and system action prediction (SAP). Our model expresses superb advantages in natural language understanding and dialog manager. Highly expressive feature representations beyond conventional aggregation of slot tags and intents are expected to be captured in our joint model, so that the affects of noisy output from NLU can be mitigated. Extra supervised signal from system actions is capable of refining NLU model by backpropagating the associated error gradients.

End-to-End Joint Model

The joint model can be considered as a SAP model stacked on top of a history of NLU models (see Fig. 1). NLU model is designed as a multi-tasking framework by sharing bi-directional long short-term memory (biLSTM) layers with slot tagging and intent prediction.

Given a sequence of input vectors $\mathbf{x}$=$\set{x_{t}}_{1}^{T}$, a recurrent unit $\mathcal{H}$ computes a sequence of hidden vectors $\mathbf{h}$=$\set{h_{t}}_{1}^{T}$ and a sequence of output symbols $\mathbf{\hat{y}}$=$\set{\hat{y}_{t}}_{1}^{T}$ by iterating the following equations,

where $\mathtt{softmax}\left(z_{m}\right)=e^{z_{m}}/\sum_{i}e^{z_{i}}$, $\sigma$ is an activation function, and $W_{xh}$, $U_{hh}$ and $W_{hy}$ are weight matrices. The goal of sequence to sequence model (Seq2Seq) is to estimate a conditional probability $p\left(\mathbf{\hat{y}}|\mathbf{x}\right)=\prod_{t=1}^{T}p\left(\hat{y}_{t}|\mathbf{x}\right)$ such that the distance (loss) between predicted distribution $p\left(\hat{y_{t}}|\mathbf{x}\right)$ and target distribution $q\left(y_{t}|\mathbf{x}\right)$ is minimized, namely,

where $M$ is the number of unique output labels. The loss of this Seq2Seq model can be optimized using backpropagation. LSTM cells are chosen as recurrent units since LSTM can mitigate problems of vanishing or exploding gradients in long-term dependencies via self-regularization . The LSTM recurrent unit $\mathcal{H}$ can be further expanded as,

where the sigmoid functions $\mathtt{sigm}$ and $\mathtt{tanh}$ are applied element-wise, and $\odot$ denotes element-wise product. Since preceding and following lexical contexts are important in analysis of user utterances, bi-directional LSTM cells are used. Therefore sequence $\mathbf{x}$ and its reverse go through LSTM layers separately, followed by the concatination of the corresponding forward output $\overrightarrow{\mathbf{h}}$ and backward output $\overleftarrow{\mathbf{h}}$,

where $\overrightarrow{W}_{hy}$ and $\overleftarrow{W}_{hy}$ are bi-directional weight matrices.

2 Joint Modeling

The proposed joint model is a RNN classifier that utilizes bi-directional LSTM cells $\mathcal{H}$, which takes as inputs $I$-$1$ history of current hidden outputs $\mathbf{h}_{j}^{(nlu)}=\set{h_{i}^{(nlu)}}_{j-I+1}^{j}$ from NLU units and performs one-vs-all binary classifications for SAP at the output layer (see Fig. 1), in other word,

where $k\in\left[1,K\right]$ denotes the index of system action labels. NLU model at the $i$-th history is considered as a multi-task joint model with shared biLSTM layers for two tasks, where it takes as inputs a sequence of word vectors $\mathbf{w}=\set{w_{t}}_{1}^{T}$, and performs Seq2Seq for slot tagging and one-vs-all binary classifications for intent prediction (see Fig. 2). The biLSTM architecture mentioned in Section 2.1 can be directly applied to slot tagging task with $M$ unique user slot tags,

where $\overrightarrow{h}_{t}^{1(i)}$ and $\overleftarrow{h}_{t}^{1(i)}$ denotes hidden outputs of the shared forward and backward layers, respectively.

As for intent prediction task in NLU, we add one more recurrent LSTM layer on top of biLSTM layers, and only consider the last hidden vector $h_{T}^{2(int_{i})}$ as the output of this second recurrent layer. Real human-human dialogs encode various number of intents in a single user utterance, and therefore, we design a set of one-vs-all binary classifiers at the output layer where each neuron is activated using a sigmoid function. The positive label of each classifier is predicted if its probability is no less than the threshold,

where $n\in\left[1,N\right]$ is the index of intent labels. We choose the same two-layer recurrent architecture as the intent model to calculate the hidden vector $h_{i}^{(nlu)}$ out from the $i$-th NLU component with the size of $M$+$N$, where $M$ and $N$ are the number of unique slot tags and unique intents, respectively.

End-to-end joint training estimates the conditional probability given a history of word vectors $\mathbf{w}_{h}=\set{\mathbf{w}^{(i)}}_{1}^{I}$ such that $\mathtt{loss}=l^{(act)}+l^{(tag)}+l^{(int)}$ is minimized, where

Experiments

We choose a mini-batch stochastic gradient descent method Adam with the batch size of 32 examples. The size of each hidden recurrent layer is 256, and the size of hidden output vector of NLU units is $M$+$N$, where $M$ and $N$ are the size of unique slot tags and intents, respectively. We assume the joint model can only get access to previous history with $I$=$5$. The dimension of word embeddings is 512. Dropout rate is 0.5. We apply 300 training epochs without using any early stop strategy. Best models for three tasks are selected separately upon decision thresholds well tuned on dev set under different metrics. Token-level micro-average F1 score is used for slot filling; frame-level accuracy (it counts only when the whole frame parse is correct) is used for user intent prediction and system action prediction. The code is releasedhttps://github.com/XuesongYang/end2end_dialog.git.

2 Corpus

DSTC4 corpushttp://www.colips.org/workshop/dstc4/data.html is selected, which collected human-human dialogs of tourist information in Singapore from Skype calls that spanned five domains—accommodation, attraction, food, shopping, and transportation. Each tourist and guide tend to be expressed in a series of multiple turns. The guide is defined as the system in this paper. We transform raw data into examples that fit our experiments. Each example includes an user utterance and its associated slot tags in IOB format , user intents, and responsive system actions. Labels of system actions are defined as the concatenation of categories and attributes of speech acts, e.g. $\mathtt{QST\_WHEN}$. $\mathtt{NULL}$ is added as a waiting response from guides when they are expressed in multiple turns. The consecutive guide actions in response to a single tourist utterance is merged as multiple labels. The whole corpus is split into train/dev/test (see Table 1). Unseen tokens such as words, user intents, slot tags, and system actions in the dev/test set are categorized as $\mathtt{UNK}$.

3 Evaluation Results

We compare our proposed joint model with following models in three tasks: slot filling, intent prediction and SAP.

Baseline (CRF+SVMs): NLU and SAP are trained separately, followed by being pipelined for testing. CRF is used to train slot filling model with lexical feature of words; one-vs-all SVMs with linear kernel (LinearSVMs) is used to train intent model with bag-of-words features of user utterances; SAP utilizes LinearSVMs with features of one-hot vectors of aggregated user slot tags and intents. Decision thresholds for intent model and SAP are 0.225 and 0.162.

Pipeline (biLSTMs): NLU in Fig. 2 and SAP in Fig. 3 are separately trained, followed by being pipelined for testing. Best decision thresholds for intent model and SAP model are 0.391 and 0.064.

Oracle-SAP (SVMs): The inputs of SAP are clean slot tags and intents annotated by human experts; LinearSVMs is used for training and testing SAP. Best decision threshold is 0.162.

Oracle-SAP (biLSTM): SAP takes as inputs the same to Oracle-SAP but uses biLSTM for training and testing (see Fig. 3). Best decision threshold is 0.064.

Evaluation results of end-to-end models are illustrated in Table 2. Our proposed joint model outperforms all other end-to-end models in frame-level accuracy by a large margin. The joint model and biLSTMs pipeline achieved absolute increase over baseline with 15.03% and 4.25%, respectively. Both models beat the SVMs oracle scores. The biLSTMs pipeline model get worse than biLSTM oracle as expected since it transfer the errors from NLU to the SAP model. Nevertheless, the joint model obtains 10.88% increase than pipeline model and 3.17% than biLSTM oracle. These promising improvements indicate that joint training can mitigate the downside of pipeline model in that the hidden outputs from a history of NLU units capture highly more expressive feature representations than the conventional aggregation of user intents and slot tags. In comparison of these two oracle models, the large improvement (12.02%) for biLSTM model indicates that the contextual user turns make significant contribution to system action prediction. In real human interaction scenarios, frame-level metrics are far more important than token-level ones especially for these multi-label classification tasks since predicting precise number of labels is more challenging.

Evaluation results of NLU models that are frozen as independent models are illustrated in Table 3. Baseline using CRF and SVMs still maintains a strong frame-level accuracy with 33.13%, however, biLSTM models taken from pipeline and joint model achieve better increase 3.25% and 4.25%, respectively. This observation indicates that joint training with two tasks of slot filling and intent prediction captures implicit knowledge underlying the shared user utterances, while another supervised signal from system actions is capable of refining the biLSTM based model by backpropagating the associated error gradients. Best accuracy at frame-level for slot filling task is obtained by traditional CRF baseline with only lexical features of words, and our biLSTM models fall behind with absolute decrease 0.47% and 0.82%. Best frame accuracy for intent prediction task is achieved by our proposed model with 5.21% improvement.

Conclusion

We proposed an end-to-end deep recurrent neural network with limited contextual dialog memory that can be jointly trained by three supervised signals of user slot filling, intent prediction and system action prediction. Experiments on multi-domain human-human dialogs demonstrated that our proposed model expressed superb advantages in natural language understanding and dialog manager. It achieved better frame-level accuracy significantly than the state of the art that pipelines separate models of NLU and SAP together. The promising performance illustrated that contextual dialog memory made significant contribution to dialog manager, and highly expressive feature representations beyond conventional aggregation of slot tags and intents could be captured in our joint model such that the affects of noisy output from NLU were mitigated. Extra supervised signal from system actions is capable of refining NLU model by backpropagating.

References