Data Noising as Smoothing in Neural Network Language Models

Introduction

Language models are a crucial component in many domains, such as autocompletion, machine translation, and speech recognition. A key challenge when performing estimation in language modeling is the data sparsity problem: due to large vocabulary sizes and the exponential number of possible contexts, the majority of possible sequences are rarely or never observed, even for very short subsequences.

In other application domains, data augmentation has been key to improving the performance of neural network models in the face of insufficient data. In computer vision, for example, there exist well-established primitives for synthesizing additional image data, such as by rescaling or applying affine distortions to images (LeCun et al., 1998; Krizhevsky et al., 2012). Similarly, in speech recognition adding a background audio track or applying small shifts along the time dimension has been shown to yield significant gains, especially in noisy settings (Deng et al., 2000; Hannun et al., 2014). However, widely-adopted noising primitives have not yet been developed for neural network language models.

Classic $n$-gram models of language cope with rare and unseen sequences by using smoothing methods, such as interpolation or absolute discounting (Chen & Goodman, 1996). Neural network models, however, have no notion of discrete counts, and instead use distributed representations to combat the curse of dimensionality (Bengio et al., 2003). Despite the effectiveness of distributed representations, overfitting due to data sparsity remains an issue. Existing regularization methods, however, are typically applied to weights or hidden units within the network (Srivastava et al., 2014; Le et al., 2015) instead of directly considering the input data.

In this work, we consider noising primitives as a form of data augmentation for recurrent neural network-based language models. By examining the expected pseudocounts from applying the noising schemes, we draw connections between noising and linear interpolation smoothing. Using this connection, we then derive noising schemes that are analogues of more advanced smoothing methods. We demonstrate the effectiveness of these schemes for regularization through experiments on language modeling and machine translation. Finally, we validate our theoretical claims by examining the empirical effects of noising.

Related Work

Our work can be viewed as a form of data augmentation, for which to the best of our knowledge there exists no widely adopted schemes in language modeling with neural networks. Classical regularization methods such as $L_{2}$-regularization are typically applied to the model parameters, while dropout is applied to activations which can be along the forward as well as the recurrent directions (Zaremba et al., 2014; Semeniuta et al., 2016; Gal, 2015). Others have introduced methods for recurrent neural networks encouraging the hidden activations to remain stable in norm, or constraining the recurrent weight matrix to have eigenvalues close to one (Krueger & Memisevic, 2015; Arjovsky et al., 2015; Le et al., 2015). These methods, however, all consider weights and hidden units instead of the input data, and are motivated by the vanishing and exploding gradient problem.

Feature noising has been demonstrated to be effective for structured prediction tasks, and has been interpreted as an explicit regularizer (Wang et al., 2013). Additionally, Wager et al. (2014) show that noising can inject appropriate generative assumptions into discriminative models to reduce their generalization error, but do not consider sequence models (Wager et al., 2016).

The technique of randomly zero-masking input word embeddings for learning sentence representations has been proposed by Iyyer et al. (2015), Kumar et al. (2015), and Dai & Le (2015), and adopted by others such as Bowman et al. (2015). However, to the best of our knowledge, no analysis has been provided besides reasoning that zeroing embeddings may result in a model ensembling effect similar to that in standard dropout. This analysis is applicable to classification tasks involving sum-of-embeddings or bag-of-words models, but does not capture sequence-level effects. Bengio et al. (2015) also make an empirical observation that the method of randomly replacing words with fixed probability with a draw from the uniform distribution improved performance slightly for an image captioning task; however, they do not examine why performance improved.

Method

We consider language models where given a sequence of indices $X=(x_{1},x_{2},\cdots,x_{T})$, over the vocabulary $V$, we model

In $n$-gram models, it is not feasible to model the full context $x_{<t}$ for large $t$ due to the exponential number of possible histories. Recurrent neural network (RNN) language models can (in theory) model longer dependencies, since they operate over distributed hidden states instead of modeling an exponential number of discrete counts (Bengio et al., 2003; Mikolov, 2012).

As an extension, we also consider encoder-decoder or sequence-to-sequence (Cho et al., 2014; Sutskever et al., 2014) models where given an input sequence $X$ and output sequence $Y$ of length $T_{Y}$, we model

2 Smoothing and Noising

Recall that for a given context length $l$, an $n$-gram model of order $l+1$ is optimal under the log-likelihood criterion. Hence in the case where an RNN with finite context achieves near the lowest possible cross-entropy loss, it behaves like an $n$-gram model.

Like $n$-gram models, RNNs are trained using maximum likelihood, and can easily overfit (Zaremba et al., 2014). While generic regularization methods such $L_{2}$-regularization and dropout are effective, they do not take advantage of specific properties of sequence modeling. In order to understand sequence-specific regularization, it is helpful to examine $n$-gram language models, whose properties are well-understood.

Noising for RNN models

We would like to apply well-understood smoothing methods such as interpolation to RNNs, which are also trained using maximum likelihood. Unfortunately, RNN models have no notion of counts, and we cannot directly apply one of the usual smoothing methods. In this section, we consider two simple noising schemes which we proceed to show correspond to smoothing methods. Since we can noise the data while training an RNN, we can then incorporate well-understood generative assumptions that are known to be helpful in the domain. First consider the following two noising schemes:

unigram noising For each $x_{i}$ in $x_{<t}$, with probability $\gamma$ replace $x_{i}$ with a sample from the unigram frequency distribution.

blank noising For each $x_{i}$ in $x_{<t}$, with probability $\gamma$ replace $x_{i}$ with a placeholder token “ ”.

While blank noising can be seen as a way to avoid overfitting on specific contexts, we will see that both schemes are related to smoothing, and that unigram noising provides a path to analogues of more advanced smoothing methods.

3 Noising as Smoothing

We now consider the maximum likelihood estimate of $n$-gram probabilities estimated using the pseudocounts of the noised data. By examining these estimates, we draw a connection between linear interpolation smoothing and noising.

where $c_{\gamma}(x)=c(x)$ since our proposal distribution $q(x)$ is the unigram distribution, and the last line follows since $c(x_{t-1})/p(x_{t-1})=c(x_{t})/p(x_{t})$ is equal to the total number of tokens in the training set. Thus we see that the noised data has pseudocounts corresponding to interpolation or a mixture of different order $n$-gram models with fixed weighting.

where the mixture coefficients are $\pi(|{J}|)=(1-\gamma)^{|{J}|}\gamma^{t-1-|{J}|}$ with $\sum_{{J}}\pi(|{J|})=1$. ${J}\subseteq\{1,2,\ldots,t-1\}$ denotes the set of indices whose corresponding tokens are left unchanged, and ${K}$ the set of indices that were replaced.

Blank noising as interpolation

where ${J}\subseteq\{1,2,\ldots,t-1\}$, which is also a mixture of the unnoised probabilities over subsequences of the current context. For example, in the case of trigrams, we have

where the mixture coefficient $\pi(i)=(1-\gamma)^{i}\gamma^{2-i}$.

4 Borrowing Techniques

With the connection between noising and smoothing in place, we now consider how we can improve the two components of the noising scheme by considering:

Adaptively computing noising probability $\gamma$ to reflect our confidence about a particular input subsequence.

Selecting a proposal distribution $q(x)$ that is less naive than the unigram distribution by leveraging higher order $n$-gram statistics.

Although it simplifies analysis, there is no reason why we should choose fixed $\gamma$; we now consider defining an adaptive $\gamma(x_{1:t})$ which depends on the input sequence. Consider the following bigrams:

The first bigram is one of the most common in English corpora; its probability is hence well estimated and should not be interpolated with lower order distributions. In expectation, however, using fixed $\gamma_{0}$ when noising results in the same lower order interpolation weight $\pi_{\gamma_{0}}$ for common as well as rare bigrams. Intuitively, we should define $\gamma(x_{1:t})$ such that commonly seen bigrams are less likely to be noised.

The second bigram, “Humpty Dumpty,” is relatively uncommon, as are its constituent unigrams. However, it forms what Brown et al. (1992) term a “sticky pair”: the unigram “Dumpty” almost always follows the unigram “Humpty”, and similarly, “Humpty” almost always precedes “Dumpty”. For pairs with high mutual information, we wish to avoid backing off from the bigram to the unigram distribution.

Let $N_{1+}(x_{1},\bullet)\stackrel{{\scriptstyle\rm def}}{{=}}|\{x_{2}:c(x_{1},x_{2})>0\}|$ be the number of distinct continutions following $x_{1}$, or equivalently the number of bigram types beginning with $x_{1}$ (Chen & Goodman, 1996). From the above intuitions, we arrive at the absolute discounting noising probability

Proposal Distribution

While choosing the unigram distribution as the proposal distribution $q(x)$ preserves unigram frequencies, by borrowing from the smoothing literature we find another distribution performs better. We again begin with two motivating examples:

Both bigrams appear frequently in text corpora. As a direct consequence, the unigrams “Francisco” and “York” also appear frequently. However, since “Francisco” and “York” typically follow “San” and “New”, respectively, they should not have high probability in the proposal distribution as they might if we use unigram frequencies (Chen & Goodman, 1996). Instead, it would be better to increase the proposal probability of unigrams with diverse histories, or more precisely unigrams that complete a large number of bigram types. Thus instead of drawing from the unigram distribution, we consider drawing from

Table 1 summarizes the discussed noising schemes.

5 Training and Testing

During training, noising is performed per batch and is done online such that each epoch of training sees a different noised version of the training data. At test time, to match the training objective we should sample multiple corrupted versions of the test data, then average the predictions (Srivastava et al., 2014). In practice, however, we find that simply using the maximum likelihood (uncorrupted) input sequence works well; evaluation runtime remains unchanged.

6 Extensions

The schemes described are for the language model setting. To extend them to the sequence-to-sequence or encoder-decoder setting, we noise both $x_{<t}$ as well as $y_{<t}$. While in the decoder we have $y_{<t}$ and $y_{t}$ as analogues to language model context and target prediction, it is unclear whether noising $x_{<t}$ should be beneficial. Empirically, however, we find this to be the case (Table 4).

Experiments

We train networks for word-level language modeling on the Penn Treebank dataset, using the standard preprocessed splits with a 10K size vocabulary (Mikolov, 2012). The PTB dataset contains 929k training tokens, 73k validation tokens, and 82k test tokens. Following Zaremba et al. (2014), we use minibatches of size 20 and unroll for 35 time steps when performing backpropagation through time. All models have two hidden layers and use LSTM units. Weights are initialized uniformly in the range $[-0.1,0.1]$. We consider models with hidden sizes of $512$ and $1500$.

We train using stochastic gradient descent with an initial learning rate of 1.0, clipping the gradient if its norm exceeds 5.0. When the validation cross entropy does not decrease after a training epoch, we halve the learning rate. We anneal the learning rate 8 times before stopping training, and pick the model with the lowest perplexity on the validation set.

For regularization, we apply feed-forward dropout (Pham et al., 2014) in combination with our noising schemes. We report results in Table 2 for the best setting of the dropout rate (which we find to match the settings reported in Zaremba et al. (2014)) as well as the best setting of noising probability $\gamma_{0}$ on the validation set.Code will be made available at: http://deeplearning.stanford.edu/noising Figure 1 shows the training and validation perplexity curves for a noised versus an unnoised run.

Text8

In order to determine whether noising remains effective with a larger dataset, we perform experiments on the Text8 corpushttp://mattmahoney.net/dc/text8.zip. The first 90M characters are used for training, the next 5M for validation, and the final 5M for testing, resulting in 15.3M training tokens, 848K validation tokens, and 855K test tokens. We preprocess the data by mapping all words which appear 10 or fewer times to the unknown token, resulting in a 42K size vocabulary. Other parameter settings are the same as described in the Penn Treebank experiments, besides that only models with hidden size 512 are considered, and noising is not combined with feed-forward dropout. Results are given in Table 3.

2 Machine Translation

For our machine translation experiments we consider the English-German machine translation track of IWSLT 2015http://workshop2015.iwslt.org/. The IWSLT 2015 corpus consists of sentence-aligned subtitles of TED and TEDx talks. The training set contains roughly 190K sentence pairs with 5.4M tokens. Following Luong & Manning (2015), we use TED tst2012 as a validation set and report BLEU score results (Papineni et al., 2002) on tst2014. We limit the vocabulary to the top 50K most frequent words for each language.

We train a two-layer LSTM encoder-decoder network (Sutskever et al., 2014; Cho et al., 2014) with $512$ hidden units in each layer. The decoder uses an attention mechanism (Bahdanau et al., 2014) with the dot alignment function (Luong et al., 2015). The initial learning rate is 1.0 and we start halving the learning rate when the relative difference in perplexity on the validation set between two consecutive epochs is less than $1\%$. We follow training protocols as described in Sutskever et al. (2014): (a) LSTM parameters and word embeddings are initialized from a uniform distribution between $[-0.1,0.1]$, (b) inputs are reversed, (c) batch size is set to 128, (d) gradient clipping is performed when the norm exceeds a threshold of 5. We set hidden unit dropout rate to 0.2 across all settings as suggested in Luong et al. (2015). We compare unigram, blank, and bigram Kneser-Ney noising. Noising rate $\gamma$ is selected on the validation set.

Results are shown in Table 4. We observe performance gains for both blank noising and unigram noising, giving roughly $+0.7$ BLEU score on the test set. The proposed bigram Kneser-Ney noising scheme gives an additional performance boost of $+0.5$-$0.7$ on top of the blank noising and unigram noising models, yielding a total gain of $+1.4$ BLEU.

Discussion

We now examine whether discounting has the desired effect of noising subsequences according to their uncertainty. If we consider the discounting

We compare the performance of models trained with a fixed $\gamma_{0}$ versus a $\gamma_{0}$ rescaled using discounting. As shown in Figure 3, bigram discounting leads to gains in perplexity for a much broader range of $\gamma_{0}$. Thus the discounting ratio seems to effectively capture the “right” tokens to noise.

2 Noised versus Unnoised Models

Recall that in interpolation smoothing, a weighted combination of higher and lower order $n$-gram models is used. As seen in Figure 3, the softmax distributions of noised models are significantly closer to the lower order frequency distributions than unnoised models, in particular in the case of the unigram distribution, thus validating our analysis in Section 3.3.

Unseen n𝑛n-grams

Smoothing is most beneficial for increasing the probability of unobserved sequences. To measure whether noising has a similar effect, we consider bigrams and trigrams in the validation set that do not appear in the training set. For these unseen bigrams (15062 occurrences) and trigrams (43051 occurrences), we measure the perplexity for noised and unnoised models with near-identical perplexity on the full set. As expected, noising yields lower perplexity for these unseen instances.

Conclusion

In this work, we show that data noising is effective for regularizing neural network-based sequence models. By deriving a correspondence between noising and smoothing, we are able to adapt advanced smoothing methods for $n$-gram models to the neural network setting, thereby incorporating well-understood generative assumptions of language. Possible applications include exploring noising for improving performance in low resource settings, or examining how these techniques generalize to sequence modeling in other domains.

Acknowledgments

We thank Will Monroe for feedback on a draft of this paper, Anand Avati for help running experiments, and Jimmy Wu for computing support. We also thank the developers of Theano (Theano Development Team, 2016) and Tensorflow (Abadi et al., 2016). Some GPUs used in this work were donated by NVIDIA Corporation. ZX, SW, and JL were supported by an NDSEG Fellowship, NSERC PGS-D Fellowship, and Facebook Fellowship, respectively. This project was funded in part by DARPA MUSE award FA8750-15-C-0242 AFRL/RIKF.

References

Appendix A Sketch of Noising Algorithm

We provide pseudocode of the noising algorithm corresponding to bigram Kneser-Ney smoothing for $n$-grams (In the case of sequence-to-sequence tasks, we estimate the count-based parameters separately for source and target). To simplify, we assume a batch size of one. The noising algorithm is applied to each data batch during training. No noising is applied at test time.