Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models

Introduction

Task-oriented dialog system is playing an increasingly important role in enabling human-computer interactions via natural spoken language. Different from chatbot type of conversational agents , task-oriented dialog systems assist users to complete everyday tasks, which usually involves aggregating information from external resources and planning over multiple dialog turns. Conventional task-oriented dialog systems are designed with complex pipelines , and there are usually separately developed modules for natural language understanding (NLU) , dialog state tracking (DST) , and dialog management (DM) . Such pipeline approach inherently make it hard to scale a dialog system to new domains, as each of these modules has to be redesigned separately with domain expertise. Moreover, credit assignment in such pipeline systems can be challenging, as errors made in upper stream modules may propagate and be amplified in downstream components.

Recent efforts have been made in designing end-to-end frameworks for task-oriented dialogs. Wen et al. and Liu et al. proposed supervised learning (SL) based end-to-end trainable neural network models. Zhao and Eskenazi and Li et al. introduced end-to-end trainable systems using deep reinforcement learning (RL) for dialog policy optimization. Comparing to SL based models, systems trained with RL by exploring the space of possible strategies showed improved model robustness against diverse dialog situations.

In RL based dialog policy learning, ideally the agent should be deployed to interact with users and get rewards from real user feedback. However, many dialog samples may be required for RL policy shaping, making it impractical to learn from real users directly from the beginning. Therefore, a user simulator is usually used to train the dialog agent to a sufficient starting level before deploying it to the real environment. Quality of such user simulator has a direct impact on the effectiveness of dialog policy learning. Designing a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. User simulators used in most of the recent RL based dialog models are designed with expert knowledge and complex rules.

To address the challenge of lacking a reliable user simulator for dialog agent policy learning, we propose a method in jointly optimizing the dialog agent policy and the user simulator policy with deep RL. We first bootstrap a basic neural dialog agent and a basic neural user simulator by learning directly from dialog corpora with supervised training. We then improve them further by simulating task-oriented dialogs between the two agents and iteratively optimizing their dialog policies with deep RL. The intuition is that we model task-oriented dialog as a goal fulfilling task, in which we let the dialog agent and the user simulator to positively collaborate to achieve the goal. The user simulator is given a goal to complete, and it is expected to demonstrate coherent but diverse user behavior. The dialog agent attempts to estimate the user’s goal and fulfill his request by conducting meaningful conversations. Both the two agents aim to learn to collaborate with each other to complete the task but without exploiting the game.

Our contribution in this work is two-fold. Firstly, we propose an iterative dialog policy learning method that jointly optimizes the dialog agent and the user simulator in end-to-end trainable neural dialog systems. Secondly, we design a novel neural network based user simulator for task-oriented dialogs that can be trained in a data-driven manner without requiring the design of complex rules.

The remainder of the paper is organized as follows. In section 2, we discuss related work on end-to-end trainable task-oriented dialog systems and RL policy learning methods. In section 3, we describe the proposed framework and learning methods in detail. In Section 4, we discuss the experiment setup and analyze the results. Section 5 gives the conclusions.

Related Work

Popular approaches for developing task-oriented dialog systems include treating the problem as a partially observable Markov Decision Process (POMDP) . RL methods can be applied to optimize the dialog policy online with the feedback collected via interacting with users. In order to make the RL policy learning tractable, dialog state and system actions have to be carefully designed.

Recently, people have proposed neural network based methods for task-oriented dialogs, motivated by their superior performance in modeling chit-chat type of conversations . Bordes and Weston proposed modeling task-oriented dialogs with a reasoning approach using end-to-end memory networks. Their model skips the belief tracking stage and selects the final system response directly from a list of response candidates. Comparing to this approach, our model explicitly tracks dialog belief state over the sequence of turns, as robust dialog state tracking has been shown to boost the success rate in task completion. Wen et al. proposed an end-to-end trainable neural network model with modularity connected system components. This system is trained in supervised manner, and thus may not be robust enough to handle diverse dialog situations due to the limited varieties in dialog corpus. Our system is trained by a combination of SL and deep RL methods, as it is shown that RL training may effectively improved the system robustness and dialog success rate . Moreover, other than having separated dialog components as in , we use a unified network for belief tracking, knowledge base (KB) operation, and dialog management, to fully explore the knowledge that can be shared among different tasks.

In many of the recent work on using RL for dialog policy learning , hand-designed user simulators are used to interact with the dialog agent. Designing a good performing user simulator is not easy. A too basic user simulator as in may only be able to produce short and simple utterances with limited variety, making the final system lack of robustness against noise in real world user inputs. Advanced user simulators may demonstrate coherent user behavior, but they typically require designing complex rules with domain expertise. We address this challenge using a hybrid learning method, where we firstly bootstrapping a basic functioning user simulator with SL on human annotated corpora, and continuously improving it together with the dialog agent during dialog simulations with deep RL.

Jointly optimizing policies for dialog agent and user simulator with RL has also been studied in literature. Chandramohan et al. proposed a co-adaptation framework for dialog systems by jointly optimizing the policies for multiple agents. Georgila et al. discussed applying multi-agent RL for policy learning in a resource allocation negotiation scenario. Barlier et al. modeled non-cooperative task dialog as a stochastic game and learned jointly the strategies of both agents. Comparing to these previous work, our proposed framework focuses on task-oriented dialogs where the user and the agent positively collaborate to achieve the user’s goal. More importantly, we work towards building end-to-end models for task-oriented dialogs that can handle noises and ambiguities in natural language understanding and belief tracking, which is not taken into account in previous work.

Proposed Framework

In this section, we first provide a high level description of our proposed framework. We then discuss each module component and the training methods in detail.

In the supervised pre-training stage, we train the dialog agent and the user simulator separately using task-oriented dialog corpora. In the RL training stage, we simulate dialogs between the two agents. The user simulator starts the conversation based on a sampled user goal. The dialog agent attempts to estimate the user’s goal and complete the task with the user simulator by conducting multi-turn conversation. At the end of each simulated dialog, a reward is generated based on the level of task completion. This reward is used to further optimize the dialog policies of the two agents with RL.

Figure 1 illustrates the design of the dialog agent. The dialog agent is capable of tracking dialog state, issuing API calls to knowledge bases (KB), and producing corresponding system actions and responses by incorporating the query results, which are key skill sets in conducting task-oriented dialogs. State of the dialog agent is maintained in the LSTM state and being updated after the processing of each turn. At the $k$th turn of a dialog, the dialog agent takes in (1) the previous agent output encoding $o_{k-1}^{A}$, (2) the user input encoding $o_{k-1}^{U}$, (3) the retrieved KB result encoding $o_{k}^{KB}$, and updates its internal state conditioning on the previous agent state $s_{k-1}^{A}$. With the updated agent state $s_{k}^{A}$, the dialog agent emits (1) a system action $a_{k}^{A}$, (2) an estimation of the belief state, and (3) a pointer to an entity in the retrieved query results. These outputs are then passed to an NLG module to generate the final agent response.

Utterance Encoding For natural language format inputs at turn $k$, we use bi-directional LSTM to encode the utterance to a continuous vector $o_{k}$. With $U_{k}$ representing the utterance at the $k$th turn with $T_{k}$ words, the utterance vector $o_{k}$ is produced by concatenating the last forward and backward LSTM states: $o_{k}=[\overrightarrow{h^{U_{k}}_{T_{k}}},\overleftarrow{h^{U_{k}}_{1}}]$.

Action Modeling We use dialog acts as system actions, which can be seen as shallow representations of the utterance semantics. We treat system action modeling as a multi-class classification problem, where the agent select an appropriate action from a predefined list of system actions based on current dialog state $s_{k}^{A}$:

where $\operatorname{ActDist_{A}}$ in the agent’s network is a multilayer perceptron (MLP) with a single hidden layer and a $\operatorname{softmax}$ activation function over all possible system actions.

Belief Tracking Belief tracker, or dialog state tracker , continuously tracks the user’s goal by accumulating evidence in the conversation. We represent the user’s goal using a list of slot values. The belief tracker maintains and updates a probability distribution $P(l^{A}_{m,k})$ over candidate values for each slot type $m\in M$ at each turn $k$:

where $\operatorname{SlotDist}_{m}$ is an MLP with a single hidden layer and a $\operatorname{softmax}$ activation function for slot type $m\in M$.

KB Operation The proposed dialog agent is able to access external information by interfacing with a KB or a database by issuing API calls. Making API call is one of the dialog acts that can be emitted by the agent, conditioning on the state of the conversation. An API call command template is firstly generated with slot type tokens. The final API call command is produced by replacing slot type tokens with the corresponding slot values from the belief tracker outputs.

At the $k$th turn of a dialog, the KB input encoding $o_{k}^{KB}$ is a binary value informing the availability of the entities that match the KB query. Corresponding output is the probability distribution of the entity pointer. We treat the KB results as a list of structured entities and let the model to maintain an entity pointer. The agent learns to adjust the entity pointer when user requests for alternative options.

Response Generation We use a template-based NLG module to convert the agent outputs (system action, slot values, and KB entity values) to natural language format.

2 User Simulator

Figure 2 shows the design of the user simulator. User simulator is given a randomly sampled goal at the beginning of the conversation. Similar to the design of the dialog agent, state of the user simulator is maintained in the state of an LSTM. At the $k$th turn of a dialog, the user simulator takes in (1) the goal encoding $g_{k}^{U}$, (2) the previous user output encoding $o_{k-1}^{U}$, (3) the current turn agent input encoding $o_{k}^{A}$, and updates its internal state conditioning on the previous user state $s_{k-1}^{U}$. On the output side, the user simulator firstly emits a user action $a_{k}^{U}$ based on the updated state $s_{k}^{U}$. Conditioning on this emitted user action and the user dialog state $s_{k}^{U}$, a set of slot values are emitted. The user action and slot values are then passed to an NLG module to generate the final user utterance.

User Goal We define a user’s goal $g^{U}$ using a list of informable and requestable slots . Informable slots are the slots that users can provide a value for to describe their goal (e.g. slots for food type, area, etc.). Requestable slots are the slots that users want to request the value for, such as requesting the address of a restaurant. We treat informable slots as discrete type of inputs that can take multiple values, and treat requestable slots as inputs that take binary values (i.e. a slot is either requested or not). In this work, once the a goal is sampled at the beginning of the conversation, we fix the user goal and do not change it during the conversation.

Action Selection Similar to the action modeling in dialog agent, we treat user action modeling as a multi-class classification problem conditioning on the dialog context encoded in the dialog-level LSTM state $s_{k}^{U}$ on the user simulator side:

Once user action is generated at turn $k$, it is used together with the current user dialog state $s_{k}^{U}$ to generate value for each informable slot:

Similar to the design of the dialog agent, $\operatorname{ActDist_{U}}$ and $\operatorname{SlotDist_{U,m}}$ are MLPs with a single hidden layer and use $\operatorname{softmax}$ activation over their corresponding outputs.

Utterance Generation We use a template-based NLG module to convert the user simulator’s outputs (action and slot values) to the natural language surface form.

3 Deep RL Policy Optimization

RL policy optimization is performed on top of the supervised pre-trained networks. The system architecture is shown in Figure 3. We defines the state, action, and reward in our RL training setting and present the training details.

State For RL policy learning, states of the dialog agent and the user simulator at the $k$th turn are the dialog-level LSTM state $s_{k}^{A}$ and $s_{k}^{U}$ respectively. Both $s_{k}^{A}$ and $s_{k}^{U}$ captures the dialog history up till current turn. The user state $s_{k}^{U}$ also encodes the user’s goal.

Action Actions of the dialog agent and user simulator are the system action outputs $a_{k}^{A}$ and $a_{k}^{U}$. An action is sampled by the agent based on a stochastic representation of the policy, which produces a probability distribution over actions given a dialog state. Action space is finite and discrete for both the dialog agent and the user simulator.

Reward Reward is calculated based on the level of task completion. A turn level reward $r_{k}$ is applied based on the progress that the agent and user made in completing the predefined task over the past turn. At the end of each turn, a score $score_{k}$ is calculated indicating to what extend the agent has fulfilled the user’s request so far. The turn level reward $r_{k}$ is then calculated by the difference of the scores received in two consecutive turns:

where $\mathcal{D}$ is the scoring function. $g_{k}^{U}$ and $g_{k}^{A}$ are the true user’s goal and the agent’s estimation of the user’s goal, both are represented by slot-value pairs. Alternatively, the turn level reward $r_{k}$ can be obtained by using the discounted reward received at the end of the dialog (positive reward for task success, and negative reward for task failure).

This last expression above gives us an unbiased gradient estimator. We sample agent action and user action at each dialog turn and compute the policy gradient. Similarly, gradient on the user simulator side can be derived as:

A potential drawback of using REINFORCE is that the policy gradient might have high variance, since the agent may take many steps over the course of a dialog episode. We also explore using Advantage Actor-Critic (A2C) in our study, in which we approximate a state-value function using a feed forward neural network.

During model training, we use softmax policy for both the dialog agent and the user simulator to encourage exploration. Softmax policy samples action from the action probability distribution calculated by the $\operatorname{softmax}$ in the system action output. During evaluation, we apply greedy policy to the dialog agent, and still apply softmax policy to the user simulator. This is to increase randomness and diversity in the user simulator behavior, which is closer to the realistic dialog system evaluation settings with human users. This also prevents the two agents from fully cooperating with each other and exploiting the game.

Experiments

We prepare the data in our study based the corpus from the second Dialog State Tracking Challenge (DSTC2) . We converte this corpus to our required format by adding API call commands and the corresponding KB query results. The dialogs simulation is based on real KB search results, which makes the dialog agent evaluation closer to real cases. Different from DSTC2, agent and user actions in our system are generated by concatenating the act and slot names in the original dialog act output (e.g. “$confirm(food=italian)$” maps to “$confirm\_food$”). Slot values are captured in the belief tracking outputs. Table 1 shows the statistics of the dataset used in our experiments.

2 Training Procedure

In supervised pre-training, the dialog agent and the user simulator are trained separately against dialog corpus. We use the same set of neural network model configurations for both agents. Hidden layer sizes of the dialog-level LSTM for dialog modeling and utterance-level LSTM for utterance encoding are both set as 150. We perform mini-batch training using Adam optimization method . Initial learning rate is set as 1e-3. Dropout ($p=0.5$) is applied during model training to prevent to model from over-fitting.

In deep RL training stage, the policy network parameters are initialized with $\operatorname{ActDist}$ parameters from the SL training. State-value function network parameters in A2C are initialized randomly. To ameliorate the non-stationarity problem when jointly training the two agents, we update the two agents iteratively during RL training. We take 100 episodes as a RL training cycle, in which we fix one agent and only update the other, and switch the training agent in the next cycle until convergence. In dialog simulation, we end the dialog if the dialog length exceeds the maximum turn size (20 in our experiment) or the user simulator emits the end of dialog action.

3 Results and Analysis

We evaluate the system on task success rate, average task reward, and average dialog length on simulated dialogs. A dialog is considered successful if the agent’s belief tracking outputs match the informable user goal slot values completely, and all user requested slots are fulfilled. Note that the results on task success rate in this work should not be directly compared to the numbers in , as both the dialog agent and the user simulator in our study are end-to-end models that take noisy natural language utterance as input and directly generate the final dialog act output. Moreover, instead of using greedy policy on user simulator, we sample user actions based on the action probability distribution from the user policy network to encourage diversity and variety in user behavior.

Table 2 shows the evaluation results. The baseline model uses the SL trained agents. REINFORCE-agent and A2C-agent apply RL training on the dialog agent only, without updating the user simulator. REINFORCE-joint and A2C-joint apply RL on both the dialog agent and user simulator over the SL pre-trained models. Figure 4, 5, and 6 show the learning curves of these five models during RL training on dialog success rate, average reward, and average success dialog length.

Success Rate As shown in Table 2, the SL model achieves the lowest task success rate. Model trained with SL on dialog corpus has limited capabilities in capturing the change in state, and thus may not be able to generalize well to unseen dialog situations during simulation. RL efficiently improves the dialog task success rate, as it enables the dialog agent to explore strategies that are not in the training corpus. The agent-update-only models using REINFORCE and A2C achieve similar results, outperforming the baseline model by 14.9% and 15.3% respectively. The jointly optimized models improved the performance further over the agent-update-only models. Model using A2C for joint policy optimization achieves the best task success rate.

Average Reward RL curves on average dialog reward show similar trends as above. One difference is that the joint training model using REINFORCE achieves the highest average reward, outperforming that using A2C by a small margin. This is likely due to the better performance of our REINFORCE models in earning reward in the failed dialogs. We find that our user simulator trained with A2C tends to have sharper action distribution from the softmax policy, making it easier to get stuck when it falls into an unfavorable state. We are interested in exploring fine-grained control strategies in joint RL policy optimization framework in our future work.

Average Success Turn Size The average turn size of success dialogs tends to decrease along the episode of RL policy learning. This is in line with our expectation as both the dialog agent and the user simulator improve their policies for more efficient and coherent strategies with the RL training.

Conclusions

In this work, we propose a reinforcement learning framework for dialog policy optimization in end-to-end task-oriented dialog systems. The proposed method addresses the challenge of lacking a reliable user simulator for policy learning in task-oriented dialog systems. We present an iterative policy learning method that jointly optimizes the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Experiment results show that our proposed method leads to promising improvements on task success rate and task reward over the supervised training and single-agent RL training baseline models.

References