Deep Reinforcement Learning For Sequence to Sequence Models

I Introduction

Sequence-to-sequence (seq2seq) models constitute a common framework for solving sequential problems . In seq2seq models, the input is a sequence of certain data units and the output is also a sequence of data units. Traditionally, these models are trained using a ground-truth sequence via a mechanism known as teacher forcing , where the teacher is the ground-truth sequence. However, due to some of the drawbacks of this training approach, there has been significant line of research connecting inference of these models with reinforcement learning (RL) techniques. In this paper, we aim to summarize such research in seq2seq training utilizing RL methods to enhance the performance of these models and discuss various challenges that arise when applying RL methods to train a seq2seq model. We intend for this paper to provide a broad overview on the strength and complexity of combining seq2seq training with RL training and to guide researchers in choosing the right RL algorithm for solving their problem. In this section, we will briefly introduce the working of a simple seq2seq model and outline some of the problems that are inherent to seq2seq models. We will then provide an introduction to RL models and explain how these models could solve the problems of seq2seq models.

Seq2seq models are common in various applications ranging from machine translation , news headline generation , text summarization , speech-to-text applications , and image captioning .

In recent years, the general framework for solving these problems uses deep neural networks that comprise two main components: an encoder which reads the sequence of input data and a decoder which uses the output generated by the encoder to produce the sequence of final outputs. Fig 1 gives a schematic of this simple yet effective framework. The encoder and decoder are usually implemented by recurrent neural networks (RNN) such as Long Short-Term Memory (LSTM) . The encoder takes a sequence of length $T_{e}$ inputsIn this paper, we use input/output and action interchangeably since choosing the next input is akin to choosing the next action and generating the next output is akin to generating the next action., $X=\{x_{1},x_{2},\cdots,x_{T_{e}}\}$, where $x_{t}\in\mathcal{A}=\{1,\cdots,|\mathcal{A}|\}$ is a single input coming from a range of possible inputs ($\mathcal{A}$), and generates the output state $h_{t}$. In addition, each encoder receives the the previous encoder’s hidden state, $h_{t-1}$, and if the encoder is a bidirectional LSTM, it will also receive the state from the next encoder’s hidden state, $h_{t+1}$, to generate its current hidden state $h_{t}$. The decoder, on the other hand, takes the last state from the encoder, i.e., $h_{T_{e}}$ and starts generating an output of size $T<T_{e}$, $\hat{Y}=\{\hat{y}_{1},\hat{y}_{2},\cdots,\hat{y}_{T}\}$, based on the current state of the decoder $s_{t}$ and the ground-truth output $y_{t}$. The decoder could also take as input an additional context vector $c_{t}$, which encodes the context to be used while generating the output . The RNN learns a recursive function to compute $s_{t}$ and outputs the distribution over the next output:

where $t^{{}^{\prime}}=t+1$, $\theta$ denotes the parameters of the model, and the function for $\pi_{\theta}$ and $\Phi_{\theta}$ depends on the type of RNN. A simple Elman RNN would use a sigmoid function for $\Phi$ and a softmax function for $\pi$ :

where $o_{t}$ is the output distribution of size $|\mathcal{A}|$ and the output $\hat{y}_{t}$ is selected from this distribution. $W_{1}$, $W_{2}$, $W_{3}$, $W_{4}$, and $W_{5}$ are matrices of learnable parameters of sizes $W_{1,2,3}\in R^{d\times d}$ and $W_{4,5}\in R^{d\times|\mathcal{A}|}$, where $d$ is the size of the input representation (e.g., size of the word embedding in text summarization). The input to the first decoder is a special input indicating the beginning of a sequence, denoted by $y_{0}=\emptyset$ and the first forward hidden state $h_{0}$ and the last backward hidden state $h_{T_{e}+1}$ for the encoder are set to a zero vector. Moreover, the first hidden state for decoder $s_{0}$ is set to the output that is received from the last encoding state, i.e., $h_{T_{e}}$.

The most widely used method to train the decoder for sequence generation is called the teacher forcing algorithm , which minimizes the maximum-likelihood loss at each decoding step. Let us define $y=\{y_{1},y_{2},\cdots,y_{T}\}$ as the ground-truth output sequence for a given input sequence $X$. The maximum-likelihood training objective is the minimization of the following cross-entropy (CE) loss:

Once the model is trained with the above objective, the model generates an entire sequence as follows: Let $\hat{y}_{t}$ denotes the action (output) taken by the model at time $t$. Then, the next action is generated by:

This process could be improved by using beam search to find a reasonable good output sequence . Now, given the ground-truth output $Y$ and the model generated output $\hat{Y}$, the performance of the model is evaluated with a specific measure. In seq2seq problems, discrete measures such as ROUGE , BLEU , METEOR , and CIDEr are used to evaluate the model. For instance, $\textrm{ROUGE}_{l}$, an evaluation measure for textual seq2seq tasks, uses the largest common substring between $Y$ and $\hat{Y}$ to evaluate the goodness of the generated output. Algorithm 1 shows these steps.

I-B Problems with Seq2seq Models

One of the main issues with the current seq2seq models is that minimizing $\mathcal{L}_{CE}$ does not always produce the best results for the above discrete evaluation measures. Therefore, using cross-entropy loss for training a seq2seq model creates a mismatch in generating the next action during training and testing. As shown in Fig 1 and also according to Eq. (3), during training, the decoder uses the two inputs, the previous output state $s_{t-1}$ and the ground-truth input $y_{t}$, to calculate its current output state $s_{t}$ and uses it to generate the next action, i.e., $\hat{y}_{t}$. However, at the test time, as given in Eq. (4), the decoder completely relies on the previously generated action from the model distribution to predict the next action, since the ground-truth data is not available anymore. Therefore, in summary, the input to the decoder is from the ground-truth during training, but the input comes from the model distribution during model testing. This exposure bias results in error accumulation during the output generation at test time, since the model has never been exclusively exposed to its own predictions during training. To avoid the exposure bias problem, we need to remove the ground-truth dependency during training and use only the model distribution to minimize Eq. (3). One way to handle this situation is through the scheduled sampling method or Gibbs sampling . In scheduled sampling, the model is first pre-trained using cross-entropy loss and will subsequently and slowly replace the ground-truth with a sampled action from the model. Therefore, a decision is randomly taken to whether use the ground-truth action with probability $\epsilon$, or an action coming from the model itself with probability $(1-\epsilon)$. When $\epsilon=1$, the model is trained using Eq. (3), and when $\epsilon=0$ the model is trained based on the following loss:

Note the difference between this equation and CE loss; in CE the ground-truth output $y_{t}$ is used to calculate the loss, while in Eq. (5), the output of the model $\hat{y}_{t}$ is used to calculate the loss.

Although scheduled sampling is a simple way to avoid the exposure bias, due to its random selection between choosing ground-truth output or model output, it does not provide a clear solution for the back-propagation of error and therefore it is statistically inconsistent . Recently, Goyal et al. proposed a solution for this problem by creating a continuous relaxation over the argmax operation to create a differentiable approximation of the greedy search during the decoding steps.

As yet another line of research on avoiding the exposure bias problem, adversarial generative models are also proposed for various seq2seq models . In general, adversarial models are comprised of a discriminator and a generator . The generator tries to generate data similar to the ground-truth data while the discriminator’s job is to discern whether the generated data is close to real data or it is a fake. Finally, the generator takes the feedback from the discriminator and optimizes its actions towards generating higher quality data. Since generator will only rely on its own output in generating the data, similar to the scheduled sampling, it is avoiding on reliance to the ground-truth data and hence avoids the exposure bias problem. However, adversarial generative models, in general, suffer from the reward sparsity and mode collapse problems. Although, there are ways to avoid these two problems , studying these solutions is outside the scope of this work.

The second problem with seq2seq models is that, while the model training is done using the $\mathcal{L}_{CE}$, the model is typically evaluated during the test time using discrete and non-differentiable measures such as BLEU and ROUGE. This will create a mismatch between the training objective and the test objective and therefore could yield inconsistent results. Thus, a solution that could use these measures during training of the model will inherently solve this mismatch problem. Recently, it has been shown that both the exposure bias and non-differentiability of evaluation measures can be addressed by incorporating techniques from reinforcement learning .

I-C Reinforcement Learning

Note that the value function $V_{\pi}$ is defined over only the states whereas $Q_{\pi}$ is defined over (state, action) pairs. The $Q_{\pi}$ formulation is advantageous in model-free contexts since it can be applied to current states without having access to a model of the environment. In contrast, the $V_{\pi}$ formulation must, by necessity, be applied to future states and thus requires a model of the environment (i.e., which states and actions lead to which other future states). The preceding state-action function ($Q$ function for short) can be computed recursively with dynamic programming:

Given the above definitions, we can define a function called advantage, denoted by $A_{\pi}$ relating the value function $V$ and $Q$ function as follows:

I-D Paper Organization

In general, we define the following problem statement that we are trying to solve by combining these two different models of learning.

Problem Statement: Given a series of input data and a series of ground-truth outputs, train a model that:

Only relies on its own output, rather than the ground-truth, to generate the results (avoiding exposure bias).

Directly optimize the model using the evaluation measure (avoiding mismatch between training and test measures).

Although recently there had been a couple of survey articles on the topic of deep reinforcement learning , these works heavily focused on the reinforcement learning methods and their applications in robotics and vision, while giving less emphasis to how these models could be used in a variety of other tasks. In this paper, we will summarize some of the most recent frameworks that attempted to find a solution for the above problem statement in a broad range of applications and explain how RL and seq2seq learning could benefit from each other in solving complex tasks. To this end, we will provide insights on some of the challenges and issues with the current models and how one can improve them with better RL models. The goal of this paper is to provide information about how we can broaden the power of seq2seq models with RL methods and understand challenges that exist in applying these methods to deep learning contexts. In addition, currently, there does not exist a good open-source framework for implementing these ideas. Along with this paper, we provide a library that combines state-of-the-art methods for the complex task of abstractive text summarization with recent techniques used in deep RL. The library provides a variety of different options and hyperparameters for training the seq2seq model using different RL models. Moreover, We provide experimental results on some of the most common techniques that are explained in this paper and we encourage researchers to experiment with other hyperparameters and explore how they can use this framework to gain better performance on different seq2seq tasks. The contributions of this paper are summarized as follows:

We provide a comprehensive summary of RL methods that are used in deep learning and specifically in the context of training seq2seq models.

We summarize the challenges, advantages, and disadvantages of using different RL methods for seq2seq training.

We provide guidelines on how one could improve a specific RL method to obtain a better and smoother training for seq2seq models.

We provide an open-source library for implementing a complex seq2seq model using different RL techniques www.github.com/yaserkl/RLSeq2Seq/ along with experiments that aim for identifying an accurate estimate on the amount of improvement that RL algorithm provide for current seq2seq models.

This paper is organized as follows: Section III provides details of some of the common RL techniques used in training seq2seq models. We provide a brief introduction to different seq2seq models in Section IV and later explain various RL models that can be used along with the seq2seq model training process. We provide a summary of recent real-world applications that combine RL training with seq2seq training and in Section V we present the framework that we implemented and discuss the details about how this framework can applied to different seq2seq problems and provide experimental results for some of the well-known RL algorithm. Finally, in Section VI, we discuss conclusions of our work.

II Seq2seq Models and their Applications

Sequence to Sequence (seq2seq) models have been an integral part of many real-world problems. From Google Machine Translation to Apple’s Siri speech to text , seq2seq models provide a clear framework to process information that is in the form of sequences. In a seq2seq model, the input and output are in the form of sequences of single units like sequence of words, images, or speech units. Table II provides a brief summary of various seq2seq models and their corresponding inputs and outputs. We also cite some of the important research papers for each application domain.

In recent years, different models and frameworks were proposed by researchers to achieve better and robust results on these tasks. For instance, attention-based models have been successfully applied to problems such as machine translation , text summarization , question answering , image captioning , speech recognition , and object detection . In an attention-based model, at each decoding step, the previous decoder output is combined with the information from the encoder’s output at a specific position to select the best decoder output.

Although attention-based models can significantly improve the performance of seq2seq models in various tasks, in applications with large output space, it is challenging for the model to reach a desirable outcome.

On the other hand, there are more advanced models in seq2seq training like pointer-generator model and the transformers model which uses self-attention layers , but discussing these models is outside the scope of this paper.

Aside from these well-defined seq2seq problems, there are other related problems that partially work on the sequence of inputs but the output is not in the form of a sequence. Here are a few prominent applications that fall into this category.

Sentiment Analysis : The input is a sequence of words and the output is a single sentiment (positive, negative, or neutral).

Natural Language Inference : Given two sentences, one as a premise and the other as a hypothesis, the goal is to classify the relationship between these two sentences into one of the entailment, neutrality, and contradiction classes.

Sentiment Role Labeling : Given a sentence and a predicate, the goal is to answer questions like “who did what to whom and when and where”.

Relation Extraction : Given a sentence, the goal is to identify whether a specific relationship exists in that sentence or not. For instance, based on the sentence “Barack Obama is married to Michelle Obama”, we can extract the “spouse” relationship.

Pronoun Resolution : Given a sentence and a question about a pronoun in the sentence, the goal is to identify who that pronoun is referring to. For instance, in the sentence “Susan cleaned Alice’s bedroom for the help she had given”, the goal is to find who the word “she” is referring to.

Note that, although in these applications, only the input data is represented in terms of sequences, we still consider them to be seq2seq problems.

Seq2seq models are usually trained with cross-entropy loss, i.e., Eq. (3). However, the performance of these models is evaluated using discrete measures. There are various discrete measures that are used for evaluating these models and each application requires its own evaluation measure. We briefly provide a summary of these measures according to their application context:

ROUGE https://github.com/andersjo/pyrouge/ , BLEU https://www.nltk.org/_modules/nltk/translate/bleu_score.html , METEOR http://www.cs.cmu.edu/~alavie/METEOR/ : These are three of the most commonly used measures in applications such as machine translation, headline generation, text summarization, question answering, dialog generation, and other applications that require evaluation of text data. $ROUGE$ measure finds the common unigram ($ROUGE$-1), bigram ($ROUGE$-2), and largest common substring (LCS) ($ROUGE$-L) between the ground-truth text and the output generated by the model and calculate respective precision, recall, and F-score for each measure. $BLEU$ works similar to $ROUGE$ but through a modified precision calculation, it inclines to provide higher scores to outputs that are closer to human judgement. In a similar manner, $METEOR$ uses the harmonic mean of unigram precision and recall and it gives higher importance to recall than the precision. Although these methods are designed to work for all text-based applications, $METEOR$ is more often used in machine translation tasks, while $ROUGE$ and $BLEU$ are mostly used in text summarization, question answering, and dialog generation.

CIDEr https://github.com/vrama91/cider , SPICE http://www.panderson.me/spice/ : $CIDEr$ is frequently used in image and video captioning tasks in which having captions that have higher human judgement scores is more important. Using sentence similarity, the notions of grammaticality, saliency, importance, accuracy, precision, and recall are inherently captured by these metrics.

$SPICE$ is a recent evaluation metric proposed for image captioning that tries to solve some of the problems of $CIDEr$ and $METEOR$ by mapping the dependency parse trees of the caption to the semantic scene graph (contains objects, attributes of objects, and relations) extracted from the image. Finally, it uses the F-score that is calculated using the tuples of the generated and ground-truth scene graphs to provide the caption quality score.

Word Error Rate (WER): This measure, which is mostly used in speech recognition, finds the number of substitutions, deletions, insertions, and corrections required to change the generated output to the ground-truth and combines them to calculate the $WER$.

II-B Datasets

In this section, we briefly describe some of the datasets that are commonly used in various seq2seq models. We provide a short list of some of the most common datasets that are used in various seq2seq applications as follows:

Machine Translation: The most common dataset used for Machine Translation task is the WMT’14 http://www.statmt.org/wmt14/translation-task.html dataset which contains 850M words from English-French parallel corpora of UN (421M words), Europarl (61M words), news commentary (5.5M words), and two crawled corpora of 90M and 272.5M words. The data pre-processing for this dataset is usually done following the code http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/ provided by Axelrod et al. .

Text Summarization: One of the main datasets used in text summarization is the CNN-Daily Mail dataset which is part of the DeepMind Q&A Dataset https://cs.nyu.edu/~kcho/DMQA/ and contains around 287K news articles along with 2 to 4 highlights (summary) for each news article For downloading and pre-processing please refer to: https://github.com/abisee/cnn-dailymail. Recently, another dataset, called Newsroom, was released by Connected Experiences Lab https://summari.es/ which contains 1.3M news articles and various metadata information such as the title and summary of the news. The document summarization challenge https://duc.nist.gov/data.html also provides some datasets for text summarization. More specifically, in this dataset, DUC-2003 and DUC-2004 which contain 500 news articles (each paired with 4 different human-generated reference summaries) from the New York Times and Associated Press Wire services, respectively. Due to the small size of this dataset, researchers usually use this dataset only for evaluation purposes.

Headline Generation: Headline generation is similar to the task of text summarization and typically all the datasets that are used in text summarization will be useful in headline generation, too. There is a big dataset which is called Gigaword and contains more than 8M news articles from multiple news agencies like New York Times and Associate Press. However, this dataset is not freely available and researchers are required to buy the license to be able to use it though one can still find pre-trained models on different tasks using this dataset http://opennmt.net/Models/.

Question Answering and Question Generation: The CNN-Daily Mail dataset was originally designed for question answering and is one of the earliest datasets that is available for tackling this problem. However, recently two large-scale datasets that are solely designed for this problem were released. Stanford Question Answering Dataset (SQuAD) https://rajpurkar.github.io/SQuAD-explorer/(1.0 and 2.0) is a dataset for reading comprehension and contains more than 100K pairs of questions and answers collected by crowd-sourcing over a set of Wikipedia articles. The answer to each question is a segment which identifies the start and end indices of the answer within the article. The second dataset is called TriviaQA http://nlp.cs.washington.edu/triviaqa/ , and similar to SQuAD, it is designed for reading comprehension and question answering task. This dataset contains 650K triples of questions, answers, and evidences (which help to find the answer).

Dialogue Generation: The dataset for this problem usually comprises of dialogues between different people. The OpenSubtitles dataset http://opus.nlpl.eu/OpenSubtitles.php , Movie Dialog dataset http://fb.ai/babi , and Cornell Movie Dialogues Corpus http://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html are three examples of these types of datasets. OpenSubtitles contains conversations between movie characters for more than 20K movies in 20 languages. The Cornell Movie Dialogues corpus contains more than 220K dialogues between more than 10K movie characters.

Semantic Parsing: Recently Zhong et al. released a dataset called WikiSQL https://github.com/salesforce/WikiSQL for this problem which contains 80654 hand-annotated questions and SQL queries distributed across 24241 tables from Wikipedia. Although this is not the only dataset for this problem but it offers a larger set of examples from other datasets such as WikiTableQuestion https://nlp.stanford.edu/software/sempre/wikitable/ and Overnight .

Sentiment Analysis: For this application, Amazon product review http://jmcauley.ucsd.edu/data/amazon/ dataset is one of the largest dataset which contains more than 82 million product reviews from May 1996 to July 2014 in its de-duplicated version. Another big dataset for this task is the Stanford Sentiment Treebank (SSTb) https://nlp.stanford.edu/sentiment/ , which includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences.

Natural Language Inference: Stanford Natural Language Inference (SNLI) https://nlp.stanford.edu/projects/snli/ is the standard dataset for this task which contains 570K human-written English sentence pairs manually labeled for the three classes entailment, contradiction, and neutral. The Multi-Genre Natural Language Inference (MultiNLI) https://www.nyu.edu/projects/bowman/multinli/ corpus is another new dataset which is collected through crowd-sourcing and contains 433K sentence pairs annotated with textual entailment information.

Semantic Role Labeling: Proposition Bank (PropBank) http://propbank.github.io/ is the standard dataset for this task which contains a corpus of text annotation with information about basic semantic propositions in seven different languages.

Relation Extraction: Freebase https://old.datahub.io/dataset/freebase is a huge dataset containing billions of triples: the entity pair and the specific relationship between them which are selected from the New York Times corpus (NYT).

Pronoun Resolution: The OntoNotes 5.0 dataset https://catalog.ldc.upenn.edu/LDC2013T19 is the standard dataset for this task. Specifically, researchers use the Chinese portion of this dataset to do the pronoun resolution in Chinese .

Image Captioning: There are two datasets that are mainly used in image captioning. The first one is the COCO dataset http://cocodataset.org/ which is designed for object detection, segmentation, and image captioning. This dataset contains around 330K images amongst which 82K images are used for training and 40K used for validation in image captioning. Each image has five ground-truth captions. SBU is another dataset which consists of 1M images from Flickr and contains descriptions provided by image owners when they uploaded the images to Flickr.

Video Captioning: For this problem, MSR-VTT http://ms-multimedia-challenge.com/2017/challenge and YouTube2Text/MSVD http://www.cs.utexas.edu/users/ml/clamp/videoDescription/ are two of the widely used datasets. MSR-VTT consists 10K videos from a commercial video search engine each containing 20 human annotated captions and YouTube2Text/MSVD which has 1970 videos each containing on an average 40 human annotated captions.

Image Classification: The most popular dataset in computer vision is the MNIST dataset http://yann.lecun.com/exdb/mnist/ . This dataset consists of handwritten digits and contains a training set of 60K examples and a test set of 10K examples. Aside from this dataset, there is a huge list of datasets that are used for various computer vision problems and explaining each of them is beyond the scope of this paper Please refer to this link for a comprehensive list of datasets that are used in computer vision: http://riemenschneider.hayko.at/vision/dataset/.

Speech Recognition: LibriSpeech ASR Corpus http://www.openslr.org/12/ is one of the main datasets used for the speech recognition task. This dataset is free and is composed of 1000 hours of segmented and aligned 16kHz English speech which is derived from audiobooks. Wall Street Journal (WSJ) also has two Continuous Speech Recognition corpora containing 70 hours of speech and text from a corpus of Wall Street Journal news text. However, unlike the LibriSpeech dataset, this dataset is not freely available and researchers have to buy a license to use it. Similar to the WSJ dataset, TIMIT https://catalog.ldc.upenn.edu/ldc93s1 is another dataset containing the read speech data. It contains time-aligned orthographic, phonetic, and word transcriptions of recordings for 630 speakers of eight major dialects of American English in which each of them are reading ten phonetically sentences.

III Reinforcement Learning Methods

In reinforcement learning, the goal of an agent interacting with an environment is to maximize the expectation of the reward that it receives from the actions. Therefore, the focus is on maximizing one of the following objectives:

In all reinforcement algorithms, an agent takes some action according to a specific policy $\pi$. The definition of a policy varies according to the specific application that is being considered. For instance, in text summarization, the policy is a language model $p(y|X)$ that, given input $X$, tries to generate the output $y$. Now, let us assume that our agent is represented by an RNN and takes actions from a policy $\pi_{\theta}$In seq2seq model, this represents $\pi_{\theta}(y_{t}|\hat{y}_{t-1},s_{t},c_{t-1})$ in Eq. (1). In a deterministic environment, where the agent takes discrete actions, the output layer of the RNN is usually a softmax function and it generates the output from this layer. In Teacher Forcing, a set of ground-truth sequences are given and the actions are chosen according to the current policy during training and the reward is observed only at the end of the sequence or when an End-Of-Sequence (EOS) signal is seen. Once the agent reaches the end of sequence, it compares the sequence of actions from the current policy ($\hat{y}_{t}$) against the ground-truth action sequence ($y_{t}$) and calculate a reward based on any specific evaluation metric. The goal of training is to find the parameters of the agent in order to maximize the expected reward. This loss is defined as the negative expected reward of the full sequence:

where $\hat{y}_{t}$ is the action chosen by the model at time $t$ and $r(\hat{y}_{1},\cdots,\hat{y}_{T})$ is the reward associated with the actions $\hat{y}_{1},\cdots,\hat{y}_{T}$. Usually, in practice, one will approximate this expectation with a single sample from the distribution of actions acquired by the RNN model. Hence, the derivative for the above loss function is given as follows:

Using the chain rule, this equation can be re-written as follows :

where $o_{t}$ is the input to the softmax function. The gradient of the loss $\mathcal{L}_{\theta}$ with respect to $o_{t}$ is given by :

where $\textbf{1}(\hat{y}_{t})$ is the 1-of-$|\mathcal{A}|$ representation of the ground-truth output and $r_{b}$ is a baseline reward and could be any value as long as it is not dependent on the parameters of the RNN network. This equation is very similar to the gradient of a multi-class logistic regression. In logistic regression, the cross-entropy gradient is the difference between the prediction and the actual 1-of-$|\mathcal{A}|$ representation of the ground-truth output:

Note that, in Eq. (15), the generated output from the model is used as a surrogate ground-truth for the output distribution, while, in Eq. (16), the ground-truth is used to calculate the gradient.

The goal of the baseline reward is to force the model to select actions that yield a reward $r>r_{b}$ and discourage those that have reward $r<r_{b}$. Since only one sample is being used to calculate the gradient of the loss function, it is shown that, having this baseline would reduce the variance of the gradient estimator . If the baseline is not dependent on the parameters of the model $\theta$, Eq. (15) is an unbiased estimator. To prove this, we simply need to show that adding the baseline reward $r_{b}$ does not have any effect on the expectation of loss:

This algorithm is called $REINFORCE$ and is a simple yet elegant policy gradient algorithm for seq2seq problems. One of the challenges with this method is that the model suffers from high variance since only one sample is used for training at each time step. To alleviate this problem, at each training step, one can sample $N$ sequences of actions and update the gradient by averaging over all these $N$ sequences as follows:

Having this, the baseline reward could be set to be the mean of the $N$ rewards that are sampled, i.e., $r_{b}=1/N\sum_{i=1}^{N}r(\hat{y}_{i,1},\cdots,\hat{y}_{i,T})$. Algorithm 2 shows how this method works.

Fig 2 shows how an attention-based pointer-generator seq2seq model can be used to extract the reward and its baseline in Self-Critic model.

The second problem with this method is that the reward is only observed after the full sequence of actions is sampled. This might not be a pleasing feature for most of the seq2seq models. If we see the partial reward of a given action at time $t$, and the reward is bad, the model needs to select a better action for the future to maximize the reward. However, in the REINFORCE algorithm, the model is forced to wait till the end of the sequence to observe its performance. Therefore, the model often generates poor results or takes longer to converge. This problem magnifies especially in the beginning of the training phase where the model is initialized randomly and thus selects arbitrary actions. To alleviate this problem to a certain extent, Ranzato et al. suggested to pre-train the model for a few epochs using the cross-entropy loss and then slowly switch to the REINFORCE loss. Finally, as another way to solve the high variance problem of the REINFORCE algorithm, importance sampling can also be used. The basic underlying idea of using the importance sampling with REINFORCE algorithm is that rather than sampling sequences from the current model, one can sample them from an old model and use them to calculate the loss.

III-B Actor-Critic Model

As mentioned in Section III-A, adding a baseline reward is a necessary component of the PG algorithm in order to reduce the variance of the model. In PG, the average reward from multiple samples in the batch was used as the baseline reward for the model. In Actor-Critic (AC) model, the goal is to train an estimator for calculating the baseline reward. For computing this quantity, AC models try to maximize the advantage function through Eq. (7) extension. Therefore, these methods are also called Advantage Actor-Critic (A2C) models.

In these models, the goal is to solve this problem using the following objective:

Similar to the PG algorithm, to avoid the expensive inner expectation calculation, we can only sample once and approximate advantage function as follows:

Now, in order to estimate $V_{\pi}(s)$, a function approximator can be used to approximate the value function. In AC, neural networks is typically used as the function approximator for the value function. Therefore, we fit a neural network $V_{\pi}(s;{\Psi})$ with parameters $\Psi$ to approximate the value function. Now, if we consider $r_{t}+\gamma V_{\pi}(s_{t^{\prime}})$ as the expectation of reward-to-go at time $t$, $V_{\pi}(s_{t})$ could play as a surrogate for the baseline reward. Similar to the PG, the variance of the model would be high since only one sample is used to train the model. Therefore, the variance can be reduced using multiple samples. In the AC model, the Actor (our policy, $\theta$) provides samples (policy states at time $t$ and $t+1$) for the Critic (neural network estimating value function, $V_{\pi}(s;{\Psi})$) and the Critic returns the estimation to the Actor, and finally, the Actor uses these estimations to calculate the advantage approximation and update the loss according to the following equation:

Therefore, in the AC models, the inference at each time $t$ would be as follows:

Fig. 3 provides an illustration of how this model works at one of the decoding steps.

As mentioned in the previous section, the Critic is a function estimator which tries to estimate the expected reward-to-go for the model at time $t$. Therefore, training the Critic is basically a regression problem. Usually, in AC models, a neural network is used as the function approximator and the value function is trained using the mean square error:

where $v_{i}=\sum_{t^{\prime}=t}^{T}r(s_{i,t^{\prime}},y_{i,t^{\prime}})$ is the true reward-to-go at time $t$. During training the Actor model, we collect $(s_{i},v_{i})$ pairs and pass them to the Critic model to train the estimator. This model is called on-policy AC, which refers to the fact that the samples are collected at the current time to train the Critic model. However, the samples that are passed to the Critic will be correlated to each other which causes poor generalization for the estimator. These methods could be turn to off-policy by collecting training samples into a memory buffer and select mini-batches from this memory buffer and train the Critic network. Off-policy AC provides a better training due to avoiding the correlation of samples that exists in the on-policy methods. Therefore, most the models that we discuss in this paper are primarily off-policy and use a memory buffer for training the Critic model.

Algorithm 3 shows the batch Actor-Critic algorithm since for training the Critic network, we use a batch of state-rewards pair. In the online AC algorithm, the Critic network is simply updated using just one sample and, as expected, online AC algorithm has a higher variance due to reliance on one sample for training the network. To alleviate this problem for online AC, we can use Synchronous Advantage AC learning or Asynchronous Advantage AC (A3C) learning . In the synchronous approach, $N$ different threads are used to train the model and each thread performs online AC for one sample and at the end of the algorithm, the gradient of these $N$ threads is used to update the gradient of the Actor model. In the more widely used A3C algorithm, as soon as a thread calculates $\theta$, it will send the update to other threads and other threads use the updated $\theta$ to train the model. A3C is an on-policy method with multi-step returns while there are other methods like Retrace , UNREAL , and Reactor which provide the off-policy variations of this model by using the memory buffer. Also, ACER mixes on-policy (from current run) and off-policy (from memory) to train the Critic network.

In general, AC models usually have low variance due to the batch training and the use of critic as the baseline reward, but they are not unbiased if the critic is erroneous and makes a lot of mistakes. As mentioned in Section III-A, PG algorithm has high variance but it provides an unbiased estimator. Now, if the PG and AC model are combined, we will likely be ending up with a model that has no bias and low variance. This idea comes from the fact that, for deterministic policies (like seq2seq models), a partially observable loss could be driven by using the $Q$-function as follows :

However, this model requires training two different networks for $Q_{\Psi}$ function and $V_{\Psi^{\prime}}$ function as the baseline. Note that the same model cannot be used to estimate both $Q$ function and value function since the estimator will not be an unbiased estimator anymore . As yet another solution to create a trade-off between the bias and variance in AC, Schulman et al. proposed the Generalized Advantage Estimation ($GAE$) model as follows:

where $\lambda$ controls the trade-off between the bias and variance such that large values of $\lambda$ yield to larger variance and lower bias, while small values of $\lambda$ do the opposite.

III-C Actor-Critic with Q-Learning

As mentioned in the previous section, the value function is used to maximize the advantage function. As an alternative to solve the maximization of advantage estimates, we can try to solve the following objective function:

This is true since we are trying to find the actions that maximize the advantage estimate and since value function does not rely on the actions, we can simply remove them from the maximization objective. Therefore, the advantage maximization problem is simplified to $Q$ function estimation problem. This method is called $Q$-learning and it is one of the most commonly used algorithms for RL problems. The $Q$-learning is called to a family of off-policy algorithm used to learn a $Q$-function. Similar to this method, SARSA algorithm is an on-policy algorithm for calculating the $Q$-function. The major difference between SARSA and $Q$-Learning, is that the maximum reward for the next state is not necessarily used for updating the $Q$-values. In $Q$-learning, the Critic tries to provide an estimation for the $Q$-function. Therefore, given that the policy $\pi_{\theta}$ is being used, our goal is to maximize the following loss at each training step:

Similar to the value network training, the $Q$-function estimation is a regression problem and the Mean Squared Error (MSE) is used for training it. However, one of the differences between the $Q$-function training and value function training is the way in which the true estimates are chosen. In value function estimation, the ground-truth data is used to calculate the true reward-to-go as $v_{i}=\sum_{t^{\prime}=t}^{T}r(s_{i,t^{\prime}},y_{i,t^{\prime}})$, however, in $Q$-learning, the estimation from the network approximator itself is used to train the regression model:

where $s^{\prime}_{i}$ and $y^{\prime}_{i}$ are the state and action at the next time, respectively. Although the $Q$-value estimation has no direct relation to the true $Q$-values calculated using ground-truth data, in practice, it is known to provide good estimation and faster training due to not collecting ground-truth reward at each step of the training. However, there are no rigorous studies that analyze how far are these estimates from the true $Q$-values. As shown in Eq. (29), the true $Q$ estimations is calculated using the estimation from network approximator at time $t+1$, i.e. $max_{y}^{\prime}Q_{\Psi}(s^{\prime}_{i},y^{\prime}_{i})$. Although, not relying on the true ground-truth estimation and explicitly using the reward function might seems to be a bad idea, however in practice it is shown that these models provide better and more robust estimators. Therefore, the training process in $Q$-learning consists of first collecting a dataset of experiences $e_{t}=(s_{t},y_{t},s_{t^{\prime}},r_{t})$ during training our Actor model and then use them to train the network approximator. This is the standard way of training the $Q$-network and was frequently used in earlier temporal-difference learning models. But, there is a problem with this method. Generally, the Actor-Critic models with neural network as function estimator are tricky to train and unless there are guarantees that the estimator is good, the model does not converge. Although the original $Q$-learning method is proven to converge , when a neural networks is used to approximate the estimator, the convergence guarantee no longer holds. Usually, since samples are coming from a specific sets of sequences, there is a correlation between the samples that are chosen to train the model. Thus, this may cause any small updates to Q-network to significantly change the data distribution, and ultimately affects the correlations between $Q$ and the target values. Recently, Mnih et al. proposed using an experience buffer In some literatures, it is called a replay buffer to store the experiences from different sequences and then randomly select a batch from this dataset and train the $Q$-network. Similar to the off-policy AC model, one benefit of using this buffer is the potential to increase efficiency of the model by re-using the experiences in multiple updates and reducing the variance of the model. Since, by sampling uniformly from the buffer, the correlation of samples used in the updates is reduced. As another improvement to the experience buffer, a prioritized version of this buffer is used in which, to select the mini-batches during training, only samples that have low temporal difference error are selected . Algorithm 4 provides the pseudo-code for a $Q$-learning algorithm called Deep $Q$-Network or DQN.

III-D Advanced Q-Learning

One of the problems with the Deep $Q$-Network (DQN) is the overestimation of $Q$-values as discussed in . Specifically, the problem lies in the fact that the ground-truth reward is not used to train these models and the same network is used to calculate both the estimation of network $Q_{\Psi}(s_{i},y_{i})$ and true values for regression training, $q_{i}$. To alleviate this problem, one could use two different networks in which the first one chooses the best action when calculating $max_{y^{\prime}}Q_{\Psi}(s^{\prime}_{n},y^{\prime}_{n})$ and the other calculate the estimation of $Q$ value, i.e., $Q_{\Psi}(s_{i},y_{i})$. In practice, a modified version of the current DQN network is used as the second network in which the current network freezes its parameters for a certain period of time and updates the second network, periodically. Let us call the second network as the target network with parameters $\Psi^{\prime}$. We know that $max_{y^{\prime}}Q_{\Psi}(s^{\prime}_{n},y^{\prime}_{n})$ is the same as choosing the best action according to the network $Q_{\Psi}$. Therefore, this equation can be re-written as $Q_{\Psi}(s^{\prime}_{t},\operatorname*{arg\,max}_{y^{\prime}_{t}}Q_{\Psi}(s^{\prime}_{t},y^{\prime}_{t}))$. As shown in this equation, $Q_{\Psi}$ is used for both calculating the $Q$-value and finding the best action. Given a target network, the best action is chosen using our target network and the $Q$-value is estimated using the current network. Hence, using the target network, $Q_{\Psi^{\prime}}$, the $Q$-estimation will be given as follows:

where EOS stands for the End-Of-Sequence action. This method is called Double DQN (DDQN) and is shown to resolve the problem of overestimation in DQN and provides more realistic estimations. However, even this model suffers from the fact that there is no relation between the true $Q$-values and the estimation provided by the network. Algorithm 5 shows the pseudocode for this model.

III-D2 Dueling Networks

DDQN tried to solve one of the problems with DQN model by using two networks in which the target network selects the next best action while the current network estimates the $Q$-values given the action selected by the target. However, in most applications, it is unnecessary to estimate the value of each action choice. This is especially important in discrete problems with a large sets of possible actions where only a small portion of actions are suitable. For instance, in text summarization the output of the model is a vector of the distribution over the vocabulary and therefore, the output has the same dimension as the vocabulary size which is usually selected to be between 50K to 150K. In most of the applications that use DDQN, the action space is limited to less than a few hundred. For instance, in an Atari game, the possible actions could be to move left, right, up, down, and shoot. Therefore, using DDQN would be easy for these types of applications. Recently, Wang et al. proposed the idea of using a dueling net to overcome this problem. In their proposed method, rather than estimating the $Q$-values directly from the $Q$-net, two different values are estimated for the value function and advantage function as follows:

In order to be able to calculate the $V_{\Psi}(s_{t})$, the value estimates is replicated $|\mathcal{A}|$ times. However, as discussed in , using Eq. (33) to calculate the $Q$ is bad and can potentially yield poor performance since Eq. (33) is unidentifiable in the sense that a constant can be added to $V_{\Psi}(s_{t})$ and subtracted the same constant from $A_{\Psi}(s_{t},y_{t})$. To solve this problem, the authors suggested to force the advantage estimator to have a zero at the selected action:

This way, for the action $y^{*}=\operatorname*{arg\,max}_{y}Q_{\Psi}(s_{t},y)=\operatorname*{arg\,max}_{y}A_{\Psi}(s_{t},y)$, $Q_{\Psi}(s_{t},y^{*})=V_{\Psi}(s_{t})$ is obtained. As an alternative to Eq. (32) and to make the model more stable, the author suggested to replace the max operator with average:

Note that the dueling net will not decrease the number of actions but will provide a better normalization over the target distribution. Similar to DQN and DDQN, this model also suffers from the fact that there is no relation between the true values of $Q$-function and the estimation provided by the network. In Section V, we propose a simple and effective solution to overcome this problem by doing schedule sampling between the $Q$-value estimations and true $Q$-values to pre-train our function approximator. Fig. 4 summarizes some of the strengths and weaknesses of these different RL methods.

IV Combining RL with Seq2seq Models

In this section, we will provide some of the recent models that combined the seq2seq training with reinforcement learning techniques. For most of these models, the main goal is to solve the train/test evaluation mismatch problem which exists in all previously described seq2seq models. This is usually done by adding a reward function to the training objective. There are a growing number of research works that used the REINFORCE algorithm to improve the current state-of-the-art seq2seq models. However, more advanced techniques such as Actor-Critic models, DQN and DDQN, have not been used often for these tasks. As mentioned earlier, one the main difficulties of using $Q$-Learning and its derivatives is the large action space for seq2seq models. For instance, in text summarization, the model should provide estimates for each word in the vocabulary and therefore the estimation could be inferior even with a well trained model. Due to these reasons, researchers mostly focused on the easier yet problematic approaches such as REINFORCE algorithm to train the seq2seq model. Therefore, combining the power of $Q$-Learning training with seq2seq model is still considered to be an open area of research. Table III shows the policy, action, and reward function for each seq2seq task and Table IV summarizes these models along with the respective seq2seq application and specific RL algorithm they used to improve that application.

As mentioned in Section III-A, in Policy Gradient (PG), the reward of the sampled sequence is observed at the end of the sequence generation and back-propagate that error equally to all the decoding steps according to Eq. (15). Also, we talked about the exposure bias problem that exists in seq2seq models during training the decoder because of using the Cross-Entropy (CE) error. The idea of improving generation by letting the model use its own predictions at training time was first proposed by Daume III et al. . Based on their proposed method, SEARN, the structured prediction problems can be cast as a particular instance of reinforcement learning. The basic idea is to let the model use its own predictions at training time to produce a sequence of actions (e.g., the choice of the next word). Then, a greedy search algorithm is run to determine the optimal action at each time step, and the policy is trained to predict that action. An imitation learning framework was proposed by Ross et al. in a method called DAGGER, where an oracle of the target word given the current predicted word is required. However, for tasks such as text summarization, computing the oracle is infeasible due to the large action space. This problem was later addressed by the ‘Data As Demonstrator (DAD)’ model where the target action at step $k$ is the $k^{th}$ action taken by the optimal policy. One drawback of DAD is that at every time step the target label is always selected from the ground-truth data and if the generated summaries are shorter than the ground-truth summaries, the model still forces to generate outputs that could already exist in the model. One way to avoid this problem in DAD is to use a method called End2EndBackProp in which, at each step $t$, the top-$k$ actions are retrieved from the model and the normalized probabilities of these actions are used as weights (of importance) and the normalized combination of their representation is fed to the next decoding step.

Finally, REINFORCE algorithm tries to overcome all these problems by using the PG rewarding function and avoiding the CE loss by using the sampled sequence as the ground-truth to train the seq2seq model, Eq. (18). In real-world applications, the training is usually started with the CE loss and a pre-trained model is acquired. Then, the REINFORCE algorithm is used to train the model. Some of the earliest adoptions of REINFORCE algorithm for training seq2seq models are in computer vision , image captioning , and speech recognition . Recently, other researchers showed that using a combination of CE loss and REINFORCE loss could yield a better result than just simply performing the pre-training. In these models, the training is started by using the CE loss and is slowly switched from CE loss to REINFORCE loss to train the model. There are various ways in which one can do the transition from CE loss to REINFORCE loss. Ranzato et al. used an incremental scheduling algorithm called ‘MIXER’ which combines DAGGER with DAD methods. In this method, the RNN is trained with the cross-entropy loss for $N_{CE}$ epochs using the ground-truth sequences. This ensures that the model starts off with a much better policy than random because now the model can focus on promising regions of the search space. Then, they use an annealing schedule in order to gradually teach the model to produce stable sequences. Therefore, after the initial $N_{CE}$ epochs, they continue training the model for $N_{CE}+N_{R}$ epochs, such that, for every sequence, they use the $\mathcal{L}_{CE}$ for the first $(T-\delta)$ steps, and the REINFORCE algorithm for the remaining $\delta$ steps. The MIXER model was successfully used on a variety of tasks such as text summarization, image captioning, and machine translation.

Another way to handle the transition from using CE loss to REINFORCE loss is to use the following combined loss:

where $\eta\in(0,1)$ is the parameter that controls the transition from CE to REINFORCE loss. In the beginning of the training, $\eta=0$ and the model completely relies on CE loss, while as the training progresses, the $\eta$ value is increased in order to slowly reduce the effect of CE loss. By the end of the training process (where $\eta=1$), the model completely uses the REINFORCE loss for training. This mixed training loss was used in many of the recent works on text summarization , paraphrase generation , image captioning , video captioning , speech recognition , dialogue generation , question answering , and question generation .

IV-B Actor-Critic Models

One of the problems with the PG model is that we need to sample the full sequences of actions and observe the reward at the end of the generation. This, in general, will be problematic since the error of generation accumulates over time and usually for long sequences of actions, the final sequence is so far away from the ground-truth sequence. Thus, the reward of the final sequence would be small and model would take a lot of time to converge. To avoid this problem, Actor-Critic models observe the reward at each decoding step using the Critic model and fix the sequence of future actions that the Actor is going to take. The Critic model usually tries to maximize the advantage function through the estimation of value function or $Q$-function. As one of the early attempts of using AC models, Bahdanau et al. and He et al. used this model for the problem of machine translation. In , the author used temporal-difference (TD) learning for advantage function estimation by considering the $Q$-value for the next action, i.e., $Q(s_{t},y_{t+1})$, as a surrogate for the its true value at time $t$, i.e., $V_{\Psi}(s_{t})$. We mentioned that for a deterministic policy, $y^{*}=\operatorname*{arg\,max}_{y}Q(s,y)$, it follows that $Q(s,y^{*})=V(s)$. Therefore, the $Q$-value for the next action could be used as the true estimates of the value function at current time. To accommodate for the large action space, they also use the shrinking estimation trick that was used in dueling net to push the estimate to be closer to their means. Additionally, the Critic training is done through the following mixed objective function:

where $q_{i}$ is the true estimation of $Q$ from a delayed Actor. The idea of using delayed Actor is similar to the idea used in Double $Q$-Learning where a delayed target network is used to get the estimation of the best action. Later, Zhang et al. used a similar model on image captioning task.

He et al. proposed a value network that uses a semantic matching and a context-coverage module and passed them through a dense layer to estimate the value function. However, their model requires a fully-trained seq2seq model to train the value network. Once the value network is trained, they use the trained seq2seq model and trained value estimation model to do the beam search during translation. Therefore, the value network is not used during the training of the seq2seq model. During inference, however, similar to the AlphaGo model , rather multiplying the advantage estimates (value or $Q$ estimates) to the policy probabilities (like in Eq. (23)), they combine the output of the seq2seq model and the value network as follows:

where $V_{\Psi}(\hat{y}_{1\cdots T})$ is the output of the value network and $\eta$ controls the effect of each score.

In a different model, Li et al. proposed a model that controls the length of seq2seq model using RL-based ideas. They train a $Q$-value function approximator which estimates the future outcome of taking an action $y_{t}$ in the present and then incorporate it into a score $S(y_{t})$ at each decoding step as follows:

Specifically, the $Q$ function, in this work, takes only the hidden state at time $t$ and estimates the length of the remaining sequence. While decoding, they suggest an inference method that controls the length of the generated sequence as follows:

Recently, Li et al. proposed an AC model which uses a binary classifier as the Critic. In this specific model, the Critic tries to distinguish between the generated summary and the human-written summary via a neural network binary classifier. Once they pre-trained the Actor using CE loss, they start training the AC model alternatively using PG and the classifier score is considered as a surrogate for the value function. AC and PG were used also in the work of Liu et al. where they combined AC and PG learning along with importance sampling to train a seq2seq model for image captioning. In this method, they used two different neural networks for $Q$-function estimation, i.e., $Q_{\Psi}$, and value estimation, i.e., $V_{\Psi^{\prime}}$. They also used a mixed reward function that combines a weighted sum of $ROUGE$, $BLEU$, $METEOR$, and $CIDEr$ measures to achieve a higher performance on this task.

IV-C Current RL-Based Model Issues

Throughout this paper, we discussed about various situations where using RL provides a better solution than traditional methods. However, utilizing RL methods creates its own training challenges and in most of the cases the improvement received from these models are not significant. In this section, we will discuss some of the issues that exist in current RL techniques used for seq2seq problems. As discussed in Section III, sample efficiency and high variance in RL models is one the of the main issues in applying them to seq2seq problems. Therefore, models such as RAML and SPG are proposed to provide a middle ground between the MLE and RL training. In RAML , a reward-aware perturbation is added to MLE while in SPG , the reward distribution is utilized for effective sampling of policy gradient. Recently, Tan et al. provided a general formulation that connects the MLE and RL training through Entropy Regularized Policy Optimization (ERPO). However, even these solutions suffer from their own problems. RAML arguably suffers from the exposure bias while SPG requires a lot of engineering to work on a specific problem and, as shown in Table III, that is why REINFORCE-based models such as MIXER are preferred in most of the current seq2seq problems.

Although REINFORCE-based models are simple to implement and provide better results, training these models is time-consuming and the improvement over baselines is usually marginal. This is why in most of the current works, these models are only used for fine-tuning purposes.

Aside from these issues, there are problems inherent to specific applications that make it hard for researchers to combine RL techniques with current seq2seq models. For instance, in most of the NLP problems, the output or action space is massive comparing to the size of actions in a robotic or game-playing problems. This is mostly due to the fact that in applications such as machine translation, text summarization, and image captioning the size of the output is equal the size of the vocabulary used during training. Now, compare this to an agent that plays a simple Atari game which requires deciding on usually less than 20 actions . This will show the severity of this problem and the reward sparsity issue that exist in these applications.

Moreover, most of the current seq2seq models which use RL training, rely on well-defined reward function such as BLEU or ROUGE for providing feedback for the model. Although these are the standard metrics for evaluating various seq2seq models, relying on them creates a different set of problems. For instance, in abstractive text summarization, ROUGE and BLEU scores are being used as the standard metric for evaluation of summarization models. However, a good abstractive summary will definitely have a low ROUGE and BLEU score. This problem could be further investigated and possibly improved by Inverse Reinforcement Learning (IRL) by forcing the model to learn its own rewarding function. However, to the best of our knowledge, no work has been done in this area.

Recently, new methods are introduced for game playing using RL algorithm which combine the best performing models in this area and apply some of the best practices used in previous models to achieve state-of-the-art results. Rainbow and Quantile Nets are among such frameworks. In Rainbow , the authors combine DDQN, prioritized experience buffer, dueling net, multi-step learning (using step-based reward rather than general reward), and distributional RL to achieve state-of-the-art in 57 games in the Atari 2600 framework. A similar ensembling method could also be useful to be applied for seq2seq tasks but this is also left for future works.

V RLSeq2Seq: An Open-Source Library for Training Seq2seq Models with RL Methods

As part of this comprehensive study, we developed an open-source library which implements various RL techniques for the problem of abstractive text summarization. This library is made available at www.github.com/yaserkl/RLSeq2Seq/. Since experimenting each specific configuration of these models, even requires few days of training on GPUs, we encourage researchers, who use this library to build and enhance their own models, to also share their trained model at this website. In this section, we explain some of the important features of our library. As mentioned before, this library provides modules for abstractive text summarization. The core of our library is based on a model called pointer-generator https://github.com/abisee/pointer-generator which itself is based on Google TextSum model https://github.com/tensorflow/models/tree/master/research/textsum. We also provide a similar imitation learning used in training REINFORCE algorithm to train the function approximator. This way, we propose training our DQN (DDQN, Dueling Net) using a schedule sampling in which we start training the model in the beginning based on ground-truth $Q$-values while as we move on with the training process, we completely rely on the function estimator to train the network. This could be seen as a pre-training step for the function approximator. Therefore, the model is guaranteed to start by using better ground-truth data since it is exposed to the true ground-truth values versus the random estimation it receives from the model itself. In summary, our library implements the following features:

Adding temporal attention and intra-decoder attention that was proposed in .

Adding scheduled sampling along with its differentiable relaxation proposed in and E2EBackProb for solving exposure bias problem.

Adding adaptive training of REINFORCE algorithm by minimizing the mixed objective loss in Eq. (34).

Providing Self-Critic training by adding the greedy reward as the baseline.

Providing Actor-Critic training options for training the model using asynchronous training of Value Network, DQN, DDQN, and Dueling Net.

Providing options for scheduled sampling for training of the $Q$-Function in DQN, DDQN, and Dueling Net.

To test the power of some of the studied models in this paper, we have done a range of various experiments using our open-source library. As mentioned in Section IV, most of the RL-based models play as a fine-tuning technique in seq2seq applications. Thus, we first pre-train our model for 15 epochs using only cross-entropy loss and then add the RL training for another 10 epochs. Our experiments follows the same setup as to the pointer-generator model and we only show the results after activating the coverage mechanism. We activate the coverage mechanism only for the last epoch and select the best model using the evaluation data. We use a linear scheduling probability as $\epsilon=step/totalsteps$ and we use $\epsilon=1$ after activating the coverage so that the model completely relies on its own output for the rest of training and for E2EBackPropagation model, $K$ is set to 4. All experiments are done using two NVIDIA P100 GPUs one used for training the model and the other for select the best trained model based on the evaluation data.

Table V shows the results of our experiments based on ROUGE score on this dataset. All our ROUGE scores have a 95% confidence interval of at most $\pm 0.25$ as reported by the official ROUGE script. In this table, PG stands for Pointer-Generation and SS stands for Scheduled Sampling. As shown in this table, both scheduled sampling model and E2E model are superior to the pointer-generator. We have also used our framework to train the Self-Critic Policy Gradient (SCPG) based model proposed by Paulus et al. . However, as shown in this table, although the SCPG improves the performance of the pointer-generator model, this improvement is very marginal. This result is totally in contrast with the result in the original paper and shows that SCPG, as claimed by the authors, will not greatly improve the performance of the pointer-generator model. One of the main reason for this difference in the result of our experiment with the Paulus et al. paper is that they use a completely different set of hyperparameters for training their model. For instance, the input for their encoder is 800 words while in our default setting, for all our experiments, it is set to 400. Also, the vocabulary size is set to 150K and 50K for input and output while our default is set to 50K for both input and output. Moreover, the size of hidden layers for encoder and decoder in their work is larger that our default values and they also use a pre-trained GloVe word-embedding for training their model. Finally, we are comparing all these policy-gradient based models with an Actor-Critic model proposed by Chen et al. which holds the state-of-the-art result in text summarization in CNN/DM dataset. As shown in Table V, this model is superior to any of the policy-gradient based models according to the ROUGE scores.

V-A2 Analysis of the training time

In general, the pointer-generator framework requires more than 3 days of training for an effective results, while this time will also be expanded after adding the self-critic policy gradient. On average, each batch of training during MLE training will take 2-3 seconds while once we add the SCPG loss this time will be increased to 5-6 seconds which means the whole training time will be double after activation of the RL loss. On the other hand, the whole training time for the Actor-Critic model before and after RL activation is only a few hours which shows that not only it has superiority in the ROUGE score results but also the training process converges much faster than the other models.

VI Conclusion

In this paper, we provided a general overview of a specific type of deep learning models called sequence-to-sequence (seq2seq) models and discussed some of the recent advances in combining training of these models with Reinforcement Learning (RL) techniques. Seq2seq models are common in a wide range of applications from machine translation to speech recognition. However, traditional models in this field usually suffer from various problems during model training, such as inconsistency between the training objective and testing objective and exposure bias. Recently, with advances in deep reinforcement learning, researchers offered various types of solutions to combine the RL training with seq2seq training for alleviating the problems and challenges of training seq2seq models. In this paper, we summarized some of the most important works that tried to combine these two different techniques and provided an open-source library for the problem of abstractive text summarization that shows how one could train a seq2seq model using different RL techniques.

Acknowledgments

This work was supported in part by the US National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1838730.

References