Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Yikang Shen, Shawn Tan, Alessandro Sordoni, Aaron Courville

Introduction

Natural language has a sequential overt form as spoken and written, but the underlying structure of language is not strictly sequential. This structure is usually tree-like. Linguists agree on a set of rules, or syntax, that determine this structure (Chomsky, 1956; 1965; Sandra & Taft, 2014) and dictate how single words compose to form meaningful larger units, also called “constituents” (Koopman et al., 2013). The human brain can also implicitly acquire the latent structure of language (Dehaene et al., 2015): during language acquisition, children are not given annotated parse trees. This observation brings more interest in latent structure induction with artificial neural network approaches, which are inspired by information processing and communication patterns in biological nervous systems. From a practical point of view, integrating a tree structure into a neural network language model may be important for multiple reasons:

to obtain a hierarchical representation with increasing levels of abstraction, a key feature of deep neural networks (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015);

to model the compositional effects of language (Koopman et al., 2013; Socher et al., 2013) and help with the long-term dependency problem (Bengio et al., 2009; Tai et al., 2015) by providing shortcuts for gradient backpropagation (Chung et al., 2016);

to improve generalization via a better inductive bias and at the same time potentially reducing the need of a large amount of training data.

The study of deep neural network techniques that can infer and use tree structures to form better representations of natural language sentences has received a great deal of attention in recent years (Bowman et al., 2016; Yogatama et al., 2016; Shen et al., 2017; Jacob et al., 2018; Choi et al., 2018; Williams et al., 2018; Shi et al., 2018).

Given a sentence, one straightforward way of predicting the corresponding latent tree structure is through a supervised syntactic parser. Trees produced by these parsers have been used to guide the composition of word semantics into sentence semantics (Socher et al., 2013; Bowman et al., 2015), or even to help next word prediction given previous words (Wu et al., 2017). However, supervised parsers are limiting for several reasons: i) few languages have comprehensive annotated data for supervised parser training; ii) in some domains, syntax rules tend to be broken (e.g. in tweets); and iii) languages change over time with use, so syntax rules may evolve.

On the other hand, grammar induction, defined as the task of learning the syntactic structure from raw corpora without access to expert-labeled data, remains an open problem. Many such recent attempts suffer from inducing a trivial structure (e.g., a left-branching or right-branching tree (Williams et al., 2018)), or encounter difficulties in training caused by learning branching policies with Reinforcement Learning (RL) (Yogatama et al., 2016). Furthermore, some methods are relatively complex to implement and train, like the PRPN model proposed in Shen et al. (2017).

Recurrent neural networks (RNNs) have proven highly effective at the task of language modeling (Merity et al., 2017; Melis et al., 2017). RNNs explicitly impose a chain structure on the data. This assumption may seem at odds with the latent non-sequential structure of language and may pose several difficulties for the processing of natural language data with deep learning methods, giving rise to problems such as capturing long-term dependencies (Bengio et al., 2009), achieving good generalization (Bowman et al., 2015), handling negation (Socher et al., 2013), etc. Meanwhile, some evidence exists that LSTMs with sufficient capacity potentially implement syntactic processing mechanisms by encoding the tree structure implicitly, as shown by Gulordava et al. (2018); Kuncoro et al. (2018) and very recently by Lakretz et al. (2019). We believe that the following question remains: Can better models of language be obtained by architectures equipped with an inductive bias towards learning such latent tree structures?

Related Work

There has been prior work leveraging tree structures for natural language tasks in the literature. Socher et al. (2010); Alvarez-Melis & Jaakkola (2016); Zhou et al. (2017); Zhang et al. (2015) use supervised learning on expert-labeled treebanks for predicting parse trees. Socher et al. (2013) and Tai et al. (2015) explicitly model the tree-structure using parsing information from an external parser. Later, Bowman et al. (2016) exploited guidance from a supervised parser (Klein & Manning, 2003) in order to train a stack-augmented neural network.

Theoretically, RNNs and LSTMs can model data produced by context-free grammars and context-sensitive grammars (Gers & Schmidhuber, 2001). However, recent results suggest that introducing structure information into LSTMs is beneficial. Kuncoro et al. (2018) showed that RNNGs (Dyer et al., 2016), which have an explicit bias to model the syntactic structures, outperform LSTMs on the subject-verb agreement task (Linzen et al., 2016). In our paper, we run a more extensive suite of grammatical tests recently provided by Marvin & Linzen (2018). Bowman et al. (2014; 2015) also demonstrate that tree-structured models are more effective for downstream tasks whose data was generated by recursive programs. Interestingly, Shi et al. (2018) suggests that while the prescribed grammar tree may not be ideal, some sort of hierarchical structure, perhaps task dependent, might help. However, the problem of efficiently inferring such structures from observed data remains an open question.

The task of learning the underlying grammar from data is known as grammar induction (Chen, 1995; Cohen et al., 2011). Early work incorporated syntactic structure in the context of language modeling (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000). More recently, there have been attempts at incorporating some structure for downstream tasks using neural models (Grefenstette et al., 2015; Sun et al., 2017; Joulin & Mikolov, 2015). Generally, these works augment a main recurrent model with a stack and focus on solving algorithmic tasks. Yogatama et al. (2018) focus on language modeling and syntactic evaluation tasks (Linzen et al., 2016) but they do not show the extent to which the structure learnt by the model align with gold-standard parse trees. Shen et al. (2017) introduced the Parsing-Reading-Predict Networks (PRPN) model, which attempts to perform parsing by solving a language modeling task. The model uses self-attention to compose previous states, where the range of attention is controlled by a learnt “syntactic distance”. The authors show that this value corresponds to the depth of the parse tree. However, the added complexity in using the PRPN model makes it unwieldy in practice.

Ordered Neurons

Given these requirements, we introduce ordered neurons, an inductive bias that forces neurons to represent information at different time-scales. In our model, high-ranking neurons contain long-term or global information that will last anywhere from several time steps to the entire sentence, representing nodes near the root of the tree. Low-ranking neurons encode short-term or local information that only last one or a few time steps, representing smaller constituents, as shown in Figure 2(b). The differentiation between high-ranking and low-ranking neurons is learnt in a completely data-driven fashion by controlling the update frequency of single neurons: to erase (or update) high-ranking neurons, the model should first erase (or update) all lower-ranking neurons. In other words, some neurons always update more (or less) frequently than the others, and that order is pre-determined as part of the model architecture.

ON-LSTM

In this section, we present a new RNN unit, ON-LSTM (“ordered neurons LSTM”). The new model uses an architecture similar to the standard LSTM, reported below:

The difference with the LSTM is that we replace the update function for the cell state $c_{t}$ with a new function that will be explained in the following sections. The forget gates $f_{t}$ and input gates $i_{t}$ are used to control the erasing and writing operation on cell states $c_{t}$ , as before. Since the gates in the LSTM act independently on each neuron, it may be difficult in general to discern a hierarchy of information between the neurons. To this end, we propose to make the gate for each neuron dependent on the others by enforcing the order in which neurons should be updated.

To enforce an order to the update frequency, we introduce a new activation function:

The variable $d$ represents the split point between the two segments. We can compute the probability of the $k$ -th value in $g$ being 1 by evaluating the probability of the disjunction of any of the values before the $k$ -th being the split point, that is $d\leq k=(d=0)\vee(d=1)\vee\cdots\vee(d=k)$ . Since the categories are mutually exclusive, we can do this by computing the cumulative distribution function:

2 Structured Gating Mechanism

In order to explain the intuition behind the new update rule, we assume that the master gates are binary:

Experiments

We evaluate the proposed model on four tasks: language modeling, unsupervised constituency parsing, targeted syntactic evaluation (Marvin & Linzen, 2018), and logical inference (Bowman et al., 2015).

Word-level language modeling is a macroscopic evaluation of the model’s ability to deal with various linguistic phenomena (e.g. co-occurence, syntactic structure, verb-subject agreement, etc). We evaluate our model by measuring perplexity on the Penn TreeBank (PTB) (Marcus et al., 1993; Mikolov, 2012) task.

For fair comparison, we closely follow the model hyper-parameters, regularization and optimization techniques introduced in AWD-LSTM (Merity et al., 2017). Our model uses a three-layer ON-LSTM model with 1150 units in the hidden layer and an embedding of size 400. For master gates, the downsize factor $C=10$ . The total number of parameters was slightly increased from 24 millions to 25 millions with additional matrices for computing master gates. We manually searched some of the dropout values for ON-LSTM based on the validation performance. The values used for dropout on the word vectors, the output between LSTM layers, the output of the final LSTM layer, and embedding dropout where (0.5, 0.3, 0.45, 0.1) respectively. A weight-dropout of 0.45 was applied to the recurrent weight matrices.

As shown in Table 1, our model performs better than the standard LSTM while sharing the same number of layers, embedding dimensions, and hidden states units. Recall that the master gates only control how information is stored in different neurons. It is interesting to note that we can improve the performance of a strong LSTM model without adding skip connections or a significant increase in the number of parameters.

2 Unsupervised Constituency Parsing

The unsupervised constituency parsing task compares the latent stree structure induced by the model with those annotated by human experts. Following the experiment settings proposed in Htut et al. (2018), we take our best model for the language modeling task, and test it on WSJ10 dataset and WSJ test set. WSJ10 has 7422 sentences, filtered from the WSJ dataset with the constraint of 10 words or less, after the removal of punctuation and null elements (Klein & Manning, 2002). The WSJ test set contains 2416 sentences with various lengths. It is worth noting that the WSJ10 test set contains sentences from the training, validation, and test set of the PTB dataset, while WSJ test uses the same set of sentences as the PTB test set.

To infer the tree structure of a sentence from a pre-trained model, we initialize the hidden states with the zero vector, then feed the sentence into the model as done in the language modeling task. At each time step, we compute an estimate of $d^{f}_{t}$ :

where $p_{f}$ is the probability distribution over split points associated to the master forget gate and $D_{m}$ is the size of the hidden state. Given $\hat{d}^{f}_{t}$ , we can use the top-down greedy parsing algorithm proposed in Shen et al. (2017) for unsupervised constituency parsing. We first sort the $\{\hat{d}^{f}_{t}\}$ in decreasing order. For the first $\hat{d}^{f}_{i}$ in the sorted sequence, we split the sentence into constituents $((x_{<i}),(x_{i},(x_{>i})))$ . Then, we recursively repeat this operation for constituents $(x_{<i})$ and $(x_{>i})$ , until each constituent contains only one word.

The performance is shown in Table 2. The second layer of ON-LSTM achieves state-of-the-art unsupervised constituency parsing results on the WSJ test set, while the first and third layers do not perform as well. One possible interpretation is that the first and last layers may be too focused on capturing local information useful for the language modeling task as they are directly exposed to input tokens and output predictions respectively, thus may not be encouraged to learn the more abstract tree structure. Since the WSJ test set contains sentences of various lengths which are unobserved during training, we find that ON-LSTM provides better generalization and robustness toward longer sentences than previous models. We also see that ON-LSTM model can provide strong results for phrase detection, including ADJP (adjective phrases), PP (prepositional phrases), and NP (noun phrases). This feature could benefit many downstream tasks, like question answering, named entity recognition, co-reference resolution, etc.

3 Targeted Syntactic Evaluation

Targeted syntactic evaluation tasks have been proposed in Marvin & Linzen (2018). It is a collection of tasks that evaluate language models along three different structure-sensitive linguistic phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. Given a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence, a language model should assign a higher probability to a grammatical sentence than an ungrammatical one.

Using the released codebasehttps://github.com/BeckyMarvin/LM_syneval. We notice that the test set generated from the code is different from the one used in the original paper Marvin & Linzen (2018). Therefore, our results are not strictly comparable with the results in Marvin & Linzen (2018). and the same settings proposed in Marvin & Linzen (2018), we train both our ON-LSTM model and a baseline LSTM language model on a 90 million word subset of Wikipedia. Both language models have two layers of 650 units, a batch size of 128, a dropout rate of 0.2, a learning rate of 20.0, and were trained for 40 epochs. The input embeddings have 200 dimensions and the output embeddings have 650 dimesions.

Table 3 shows that the ON-LSTM performs better on the long-term dependency cases, while the baseline LSTM fares better on the short-term ones. This is possibly due to the relatively small number of units in the hidden states, which is insufficient to take into account both long and short-term information. We also notice that the results for NPI test cases have unusually high variance across different hyper-parameters. This result maybe due to the non-syntactic cues discussed in Marvin & Linzen (2018). Despite this, ON-LSTM actually achieves better perplexity on the validation set.

4 Logical Inference

We also analyze the model’s performance on the logical inference task described in Bowman et al. (2015). This task is based on a language that has a vocabulary of six words and three logical operations, ${or,and,not}$ . There are seven mutually exclusive logical relations that describe the relationship between two sentences: two types of entailment, equivalence, exhaustive and non-exhaustive contradiction, and two types of semantic independence. Similar to the natural language inference task, this logical inference task requires the model to predict the correct label given a pair of sentences. The train/test split is as described in the original codebasehttps://github.com/sleepinyourhat/vector-entailment, and 10% of training set is set aside as the validation set.

Figure 3 shows the performance of ON-LSTM and standard LSTM on the logical inference task. While both models achieve nearly 100% accuracy on short sequences ( $\leq 3$ ), ON-LSTM attains better performance on sequences longer then 3. The performance gap continues to increase on longer sequences ( $\geq 7$ ) that were not present during training. Hence, the ON-LSTM model shows better generalization while facing structured data with various lengths and comparing to the standard LSTM. A tree-structured model can achieve strong performance on this dataset (Bowman et al., 2015), since it is provided with the ground truth structure as input. The recursive application of the same composition function is well suited for this task. We also include the result of RRNet (Jacob et al., 2018), which can induce the latent tree structure from downstream tasks. Note that the results may not be comparable, because the hyper-parameters for training were not provided.