Compound Probabilistic Context-Free Grammars for Grammar Induction

Introduction

Grammar induction is the task of inducing hierarchical syntactic structure from data. Statistical approaches to grammar induction require specifying a probabilistic grammar (e.g. formalism, number and shape of rules), and fitting its parameters through optimization. Early work found that it was difficult to induce probabilistic context-free grammars (PCFG) from natural language data through direct methods, such as optimizing the log likelihood with the EM algorithm Lari and Young (1990); Carroll and Charniak (1992). While the reasons for the failure are manifold and not completely understood, two major potential causes are the ill-behaved optimization landscape and the overly strict independence assumptions of PCFGs. More successful approaches to grammar induction have thus resorted to carefully-crafted auxiliary objectives Klein and Manning (2002), priors or non-parametric models Kurihara and Sato (2006); Johnson et al. (2007); Liang et al. (2007); Wang and Blunsom (2013), and manually-engineered features Huang et al. (2012); Golland et al. (2012) to encourage the desired structures to emerge.

We revisit these aforementioned issues in light of advances in model parameterization and inference. First, contrary to common wisdom, we find that parameterizing a PCFG’s rule probabilities with neural networks over distributed representations makes it possible to induce linguistically meaningful grammars by simply optimizing log likelihood. While the optimization problem remains non-convex, recent work suggests that there are optimization benefits afforded by over-parameterized models Arora et al. (2018); Xu et al. (2018); Du et al. (2019), and we indeed find that this neural PCFG is significantly easier to optimize than the traditional PCFG. Second, this factored parameterization makes it straightforward to incorporate side information into rule probabilities through a sentence-level continuous latent vector, which effectively allows different contexts in a derivation to coordinate. In this compound PCFG—continuous mixture of PCFGs—the context-free assumptions hold conditioned on the latent vector but not unconditionally, thereby obtaining longer-range dependencies within a tree-based generative process.

To utilize this approach, we need to efficiently optimize the log marginal likelihood of observed sentences. While compound PCFGs break efficient inference, if the latent vector is known the distribution over trees reduces to a standard PCFG. This property allows us to perform grammar induction using a collapsed approach where the latent trees are marginalized out exactly with dynamic programming. To handle the latent vector, we employ standard amortized inference using reparameterized samples from a variational posterior approximated from an inference network Kingma and Welling (2014); Rezende et al. (2014).

On standard benchmarks for English and Chinese, the proposed approach is found to perform favorably against recent neural approaches to unsupervised parsing Shen et al. (2018, 2019); Drozdov et al. (2019); Kim et al. (2019).

Probabilistic Context-Free Grammars

We consider context-free grammars (CFG) consisting of a 5-tuple $\mathcal{G}=(S,\mathcal{N},\mathcal{P},\Sigma,\mathcal{R})$ where $S$ is the distinguished start symbol, $\mathcal{N}$ is a finite set of nonterminals, $\mathcal{P}$ is a finite set of preterminals,Since we will be inducing a grammar directly from words, $\mathcal{P}$ is roughly the set of part-of-speech tags and $\mathcal{N}$ is the set of constituent labels. However, to avoid issues of label alignment, evaluation is only on the tree topology. $\Sigma$ is a finite set of terminal symbols, and $\mathcal{R}$ is a finite set of rules of the form,

A probabilistic context-free grammar (PCFG) consists of a grammar $\mathcal{G}$ and rule probabilities $\boldsymbol{\pi}=\{\pi_{r}\}_{r\in\mathcal{R}}$ such that $\pi_{r}$ is the probability of the rule $r$. Letting $\mathcal{T}_{\mathcal{G}}$ be the set of all parse trees of $\mathcal{G}$, a PCFG defines a probability distribution over $\boldsymbol{t}\in\mathcal{T}_{\mathcal{G}}$ via $p_{\boldsymbol{\pi}}(\boldsymbol{t})=\prod_{r\in\boldsymbol{t}_{\mathcal{R}}}\pi_{r}$ where $\boldsymbol{t}_{\mathcal{R}}$ is the set of rules used in the derivation of $\boldsymbol{t}$. It also defines a distribution over string of terminals $\boldsymbol{x}\in\Sigma^{\ast}$ via

where $\mathcal{T}_{\mathcal{G}}(\boldsymbol{x})=\{\boldsymbol{t}\,|\,\textsf{yield}(\boldsymbol{t})=\boldsymbol{x}\}$, i.e. the set of trees $\boldsymbol{t}$ such that $\boldsymbol{t}$’s leaves are $\boldsymbol{x}$. We will slightly abuse notation and use

to denote the posterior distribution over the unobserved latent trees given the observed sentence $\boldsymbol{x}$, where $\mathds{1}[\cdot]$ is the indicator function.Therefore when used in the context of a posterior distribution conditioned on a sentence $\boldsymbol{x}$, the variable $\boldsymbol{t}$ does not include the leaves $\boldsymbol{x}$ and only refers to the unobserved nonterminal/preterminal symbols.

The standard way to parameterize a PCFG is to simply associate a scalar to each rule $\pi_{r}$ with the constraint that they form valid probability distributions, i.e. each nonterminal is associated with a fully-parameterized categorical distribution over its rules. This direct parameterization is algorithmically convenient since the M-step in the EM algorithm Dempster et al. (1977) has a closed form. However, there is a long history of work showing that it is difficult to learn meaningful grammars from natural language data with this parameterization Carroll and Charniak (1992).In preliminary experiments we were indeed unable to learn linguistically meaningful grammars with this PCFG. Successful approaches to unsupervised parsing have therefore modified the model/learning objective by guiding potentially unrelated rules to behave similarly.

Recognizing that sharing among rule types is beneficial, we propose a neural parameterization where rule probabilities are based on distributed representations. We associate embeddings with each symbol, introducing input embeddings $\mathbf{w}_{N}$ for each symbol $N$ on the left side of a rule (i.e. $N\in\{S\}\cup\mathcal{N}\cup\mathcal{P}$). For each rule type $r$, $\pi_{r}$ is parameterized as follows,

where $\mathcal{M}$ is the product space $(\mathcal{N}\cup\mathcal{P})\times(\mathcal{N}\cup\mathcal{P})$, and $f_{1},f_{2}$ are MLPs with two residual layers. Note that we do not use an MLP for rules of the type $\pi_{A\to BC}$, as it did not empirically improve results. See section A.1 for the full parameterization. Going forward, we will use $\mathbf{E}_{\mathcal{G}}=\{\mathbf{w}_{U}\,|\,U\in\{S\}\cup\mathcal{N}\cup\mathcal{P}\}\cup\{\mathbf{u}_{V}\,|\,V\in\mathcal{N}\cup\mathcal{M}\cup\Sigma\}$ to denote the set of input/output symbol embeddings for grammar $\mathcal{G}$, and $\lambda$ to refer to the parameters of the neural network $f_{1},f_{2}$ used to obtain the rule probabilities. A graphical model-like illustration of the neural PCFG is shown in Figure 1 (left).

It is clear that the neural parameterization does not change the underlying probabilistic assumptions. The difference between the two is analogous to the difference between count-based vs. feed-forward neural language models, where feed-forward neural language models make the same Markov assumptions as the count-based models but are able to take advantage of shared, distributed representations.

Compound PCFGs

A compound probability distribution Robbins (1951) is a distribution whose parameters are themselves random variables. These distributions generalize mixture models to the continuous case, for example in factor analysis which assumes the following generative process,

Compound distributions provide the ability to model rich generative processes, but marginalizing over the latent parameter can be computationally intractable unless conjugacy can be exploited.

In this work, we study compound probabilistic context-free grammars whose distribution over trees arises from the following generative process: we first obtain rule probabilities via

where $p_{\gamma}(\mathbf{z})$ is a prior with parameters $\gamma$ (spherical Gaussian in this paper), and $f_{\lambda}$ is a neural network that concatenates the input symbol embeddings with $\mathbf{z}$ and outputs the sentence-level rule probabilities $\boldsymbol{\pi}_{\mathbf{z}}$,

where $[\mathbf{w};\mathbf{z}]$ denotes vector concatenation. Then a tree/sentence is sampled from a PCFG with rule probabilities given by $\boldsymbol{\pi}_{\mathbf{z}}$,

This can be viewed as a continuous mixture of PCFGs, or alternatively, a Bayesian PCFG with a prior on sentence-level rule probabilities parameterized by $\mathbf{z},\lambda,\mathbf{E}_{\mathcal{G}}$.Under the Bayesian PCFG view, $p_{\gamma}(\mathbf{z})$ is a distribution over $\mathbf{z}$ (a subset of the prior), and is thus a hyperprior. Importantly, under this generative model the context-free assumptions hold conditioned on $\mathbf{z}$, but they do not hold unconditionally. This is shown in Figure 1 (right) where there is a dependence path through $\mathbf{z}$ if it is not conditioned upon. Compound PCFGs give rise to a marginal distribution over parse trees $\boldsymbol{t}$ via

One motivation for the compound PCFG is that simple, unlexicalized grammars (such as the PCFG we have been working with) are unlikely to represent an adequate model of natural language, although they do facilitate efficient learning and inference.A piece of evidence for the misspecification of unlexicalized first-order PCFGs as a statistical model of natural language is that if one pretrains such a PCFG on supervised data and continues training with the unsupervised objective (i.e. log marginal likelihood), the resulting grammar deviates significantly from the supervised initial grammar while the log marginal likelihood improves Johnson et al. (2007). Similar observations have been made for part-of-speech induction with Hidden Markov Models Merialdo (1994). We can in principle model richer dependencies through vertical/horizontal Markovization Johnson (1998); Klein and Manning (2003) and lexicalization Collins (1997). However such dependencies complicate training due to the rapid increase in the number of rules. Under this view, we can interpret the compound PCFG as a restricted version of some lexicalized, higher-order PCFG where a child can depend on structural and lexical context through a shared latent vector.Note that the compound “PCFG” is a slight misnomer because the model is no longer context-free in the usual sense. Another interpretation of the model is to view it as a vectorized version of indexed grammars Aho (1968), which extend CFGs by augmenting nonterminals with additional index strings that may be inherited or modified during derivation. Compound PCFGs instead equip nonterminals with a continuous vector that is always inherited. We hypothesize that this dependence among siblings is especially useful in grammar induction from words, where (for example) if we know that watched is used as a verb then the noun phrase is likely to be a movie.

In contrast to the usual Bayesian treatment of PCFGs which places priors on global rule probabilities Kurihara and Sato (2006); Johnson et al. (2007); Wang and Blunsom (2013), the compound PCFG assumes a prior on local, sentence-level rule probabilities. It is therefore closely related to the Bayesian grammars studied by Cohen et al. (2009) and Cohen and Smith (2009), who also sample local rule probabilities from a logistic normal prior for training dependency models with valence (DMV) Klein and Manning (2004).

The expressivity of compound PCFGs comes at a significant challenge in learning and inference. Letting $\theta=\{\mathbf{E}_{\mathcal{G}},\lambda\}$ be the parameters of the generative model, we would like to maximize the log marginal likelihood of the observed sentence $\log p_{\theta}(\boldsymbol{x})$. In the neural PCFG the log marginal likelihood

Notice that while the integral over $\mathbf{z}$ makes this quantity intractable, when we condition on $\mathbf{z}$, we can tractably perform the inner summation to obtain $p_{\theta}(\boldsymbol{x}\,|\,\mathbf{z})$ using the inside algorithm. We therefore resort to collapsed amortized variational inference. We first obtain a sample $\mathbf{z}$ from a variational posterior distribution (given by an amortized inference network), then perform the inner marginalization conditioned on this sample. The evidence lower bound $\operatorname*{ELBO}(\theta,\phi;\boldsymbol{x})$ is then,

and we can calculate $p_{\theta}(\boldsymbol{x}\,|\,\mathbf{z})$ given a sample $\mathbf{z}$ from a variational posterior $q_{\phi}(\mathbf{z}\,|\,\boldsymbol{x})$. For the variational family we use a diagonal Gaussian where the mean/log-variance vectors are given by an affine layer over max-pooled hidden states from an LSTM over $\boldsymbol{x}$. We can obtain low-variance estimators for $\nabla_{\theta,\phi}\operatorname*{ELBO}(\theta,\phi;\boldsymbol{x})$ by using the reparameterization trick for the expected reconstruction likelihood and the analytical expression for the KL term Kingma and Welling (2014).

We remark that under the Bayesian PCFG view, since the parameters of the prior (i.e. $\theta$) are estimated from the data, our approach can be seen as an instance of empirical Bayes Robbins (1956).See Berger (1985) (chapter 4), Zhang (2003), and Cohen (2016) (chapter 3) for further discussion on compound models and empirical Bayes.

2 MAP Inference

After training, we are interested in comparing the learned trees against an annotated treebank. This requires inferring the most likely tree given a sentence, i.e. $\operatorname*{argmax}_{\boldsymbol{t}}\,\,p_{\theta}(\boldsymbol{t}\,|\,\boldsymbol{x})$. For the neural PCFG we can obtain the most likely tree by using the Viterbi version of the inside algorithm (CKY algorithm). For the compound PCFG, the $\operatorname*{argmax}$ is intractable to obtain exactly, and hence we estimate it with the following approximation,

Experimental Setup

We test our approach on the Penn Treebank (PTB) Marcus et al. (1993) with the standard splits (2-21 for training, 22 for validation, 23 for test) and the same preprocessing as in recent works Shen et al. (2018, 2019), where we discard punctuation, lowercase all tokens, and take the top 10K most frequent words as the vocabulary. This setup is more challenging than traditional setups, which usually experiment on shorter sentences and use gold part-of-speech tags.

We further experiment on Chinese with version 5.1 of the Chinese Penn Treebank (CTB) Xue et al. (2005), with the same splits as in Chen and Manning (2014). On CTB we also remove punctuation and keep the top 10K word types.

2 Hyperparameters

Our PCFG uses 30 nonterminals and 60 preterminals, with 256-dimensional symbol embeddings. The compound PCFG uses 64-dimensional latent vectors. The bidirectional LSTM inference network has a single layer with 512 dimensions, and the mean and the log variance vector for $q_{\phi}(\mathbf{z}\,|\,\boldsymbol{x})$ are given by max-pooling the hidden states of the LSTM and passing it through an affine layer. Model parameters are initialized with Xavier uniform initialization. For training we use Adam Kingma and Ba (2015) with $\beta_{1}$ = 0.75, $\beta_{2}=0.999$ and learning rate of 0.001, with a maximum gradient norm limit of 3. We train for 10 epochs with batch size equal to 4. We employ a curriculum learning strategy Bengio et al. (2009) where we train only on sentences of length up to 30 in the first epoch, and increase this length limit by 1 each epoch. Similar curriculum-based strategies have used in the past for grammar induction Spitkovsky et al. (2012). During training we perform early stopping based on validation perplexity.However, we used $F_{1}$ against validation trees on PTB to select some hyperparameters (e.g. grammar size), as is sometimes done in grammar induction. Hence our PTB results are arguably not fully unsupervised in the strictest sense of the term. The hyperparameters of the PRPN/ON baselines are also tuned using validation $F_{1}$ for fair comparison. Finally, to mitigate against overfitting to PTB, experiments on CTB utilize the same hyperparameters from PTB.

3 Baselines and Evaluation

While we induce a full stochastic grammar (i.e. a distribution over symbolic rewrite rules) in this work, directly assessing the learned grammar is itself nontrivial. As a proxy, we adopt the usual approach and instead evaluate the induced grammar as an unsupervised parsing system. However, even in this setting we observe that there is enough variation across prior work on to render a meaningful comparison difficult.

In particular, some important dimensions along which prior works vary include, (1) input data: earlier work on generally assumed gold (or induced) part-of-speech tags Klein and Manning (2004); Smith and Eisner (2004); Bod (2006); Snyder et al. (2009), while more recent works induce grammar directly from words Spitkovsky et al. (2013); Shen et al. (2018); (2) use of punctuation: even within papers that induce parse trees directly from words, some papers employ heuristics based on punctuation as punctuation is usually a strong signal for start/end of constituents Seginer (2007); Ponvert et al. (2011); Spitkovsky et al. (2013), some train with punctuation Jin et al. (2018); Drozdov et al. (2019); Kim et al. (2019), while others discard punctuation altogether for training Shen et al. (2018, 2019); (3) train/test data: some works do not explicitly separate out train/test sets Reichart and Rappoport (2010); Golland et al. (2012) while some do Huang et al. (2012); Parikh et al. (2014); Htut et al. (2018). Maintaining train/test splits is less of an issue for unsupervised structure learning, however in this work we follow the latter and separate train/test data. (4) evaluation: for unlabeled $F_{1}$, almost all works ignore punctuation (even approaches that use punctuation during training typically ignore them during evaluation), but there is some variance in discarding trivial spans (width-one and sentence-level spans) and using corpus-level versus sentence-level $F_{1}$.Corpus-level $F_{1}$ calculates precision/recall at the corpus level to obtain $F_{1}$, while sentence-level $F_{1}$ calculates $F_{1}$ for each sentence and averages across the corpus. In this paper we discard trivial spans and evaluate on sentence-level $F_{1}$ per recent work Shen et al. (2018, 2019).

Given the above, we mainly compare our approach against two recent, strong baselines with open source code: Parsing Predict Reading Network (PRPN)https://github.com/yikangshen/PRPN Shen et al. (2018) and Ordered Neurons (ON)https://github.com/yikangshen/Ordered-Neurons Shen et al. (2019). These approaches train a neural language model with gated attention-like mechanisms to induce binary trees, and achieve strong unsupervised parsing performance even when trained on corpora where punctuation is removed. Since the original results were on both language modeling and unsupervised parsing, their hyperparameters were presumably tuned to do well on both and thus may not be optimal for just unsupervised parsing. We therefore tune the hyperparameters of these baselines for unsupervised parsing only (i.e. on validation $F_{1}$).

Results and Discussion

Table 1 shows the unlabeled $F_{1}$ scores for our models and various baselines. All models soundly outperform right branching baselines, and we find that the neural PCFG/compound PCFG are strong models for grammar induction. In particular the compound PCFG outperforms other models by an appreciable margin on both English and Chinese. We again note that we were unable to induce meaningful grammars through a traditional PCFG with the scalar parameterization despite a thorough hyperparameter search.Training perplexity was much higher than in the neural case, indicating significant optimization issues. However we did not experiment with online EM Liang and Klein (2009), and it is possible that such methods would yield better results. See section A.2 for the full results broken down by sentence length for sentence- and corpus-level $F_{1}$.

Table 2 analyzes the learned tree structures. We compare similarity as measured by $F_{1}$ against gold, left, right, and “self” trees (top), where self $F_{1}$ score is calculated by averaging over all 6 pairs obtained from 4 different runs. We find that PRPN is particularly consistent across multiple runs. We also observe that different models are better at identifying different constituent labels, as measured by label recall (Table 2, bottom). While left as future work, this naturally suggests an ensemble approach wherein the empirical probabilities of constituents (obtained by averaging the predicted binary constituent labels from the different models) are used either to supervise another model or directly as potentials in a CRF constituency parser. Finally, all models seemed to have some difficulty in identifying SBAR/VP constituents which typically span more words than NP constituents, indicating further opportunities for improvement on unsupervised parsing.

While the compound PCFG has fewer independence assumptions than the neural PCFG, it is still a more constrained model of language than standard neural language models (NLM) and thus not competitive in terms of perplexity: the compound PCFG obtains a perplexity of 196.3 while an LSTM language model (LM) obtains 86.2 (Table 3).We did manage to almost match the perplexity of an NLM by additionally conditioning the terminal probabilities on previous history, i.e. $\displaystyle\pi_{\mathbf{z},T\to w_{t}}\propto\exp(\mathbf{u}^{\top}_{w}\,f_{2}([\mathbf{w}_{T};\mathbf{z};\mathbf{h}_{t}])+b_{w}),$ where $\mathbf{h}_{t}$ is the hidden state from an LSTM over $\boldsymbol{x}_{<t}$. However the unsupervised parsing performance was far worse ($\approx$ 25 $F_{1}$ on the PTB). In contrast, both PRPN and ON perform as well as an LSTM LM while maintaining good unsupervised parsing performance.

We thus experiment to see if it is possible to use the induced trees to supervise a more flexible generative model that can make use of tree structures—namely, recurrent neural network grammars (RNNG) Dyer et al. (2016). RNNGs are generative models of language that jointly model syntax and surface structure by incrementally generating a syntax tree and sentence. As with NLMs, RNNGs make no independence assumptions, and have been shown to outperform NLMs in terms of perplexity and grammaticality judgment when trained on gold trees Kuncoro et al. (2018); Wilcox et al. (2019).

We take the best run from each model and parse the training set,The train/test $F_{1}$ was similar for all models. and use the induced trees to supervise an RNNG for each model using the parameterization from Kim et al. (2019).https://github.com/harvardnlp/urnng We are also interested in syntactic evaluation of our models, and for this we utilize the framework and dataset from Marvin and Linzen (2018), where a model is presented two minimally different sentences such as:

and must assign higher probability to grammatical sentence.

Additionally, Kim et al. (2019) report perplexity improvements by fine-tuning an RNNG trained on gold trees with the unsupervised RNNG (URNNG)—whereas the RNNG is is trained to maximize the joint log likelihood $\log p(\boldsymbol{t})$, the URNNG maximizes a lower bound on the log marginal likelihood $\log\sum_{\boldsymbol{t}\in\mathcal{T}_{\mathcal{G}}(\boldsymbol{x})}p(\boldsymbol{t})$ with a structured inference network that approximates the true posterior. We experiment with a similar approach where we fine-tune RNNGs trained on induced trees with URNNGs. We perform early stopping for both RNNG and URNNG based on validation perplexity. See section A.3 for the full experimental setup.

The results are shown in Table 3. For perplexity, RNNGs trained on induced trees (Induced RNNG in Table 3) are unable to improve upon an LSTM LM, in contrast to the supervised RNNG which does outperform the LSTM language model (Table 3, bottom). For grammaticality judgment however, the RNNG trained with compound PCFG trees outperforms the LSTM LM despite obtaining worse perplexity,Kuncoro et al. (2018, 2019) also observe that models that achieve lower perplexity do not necessarily perform better on syntactic evaluation tasks. and performs on par with the RNNG trained on binarized gold trees. Fine-tuning with the URNNG results in improvements in perplexity and grammaticality judgment across the board (Induced URNNG in Table 3). We also obtain large improvements on unsupervised parsing as measured by $F_{1}$, with the fine-tuned URNNGs outperforming the respective original models.Li et al. (2019) similarly obtain improvements by refining a model trained on induced trees on classification tasks. This is potentially due to an ensembling effect between the original model and the URNNG’s structured inference network, which is parameterized as a neural CRF constituency parser Durrett and Klein (2015); Liu et al. (2018).While left as future work, it is possible to use the compound PCFG itself as an inference network. Also note that the $F_{1}$ scores for the URNNGs in Table 3 are optimistic since we selected the best-performing runs of the original models based on validation $F_{1}$ to parse the training set. Finally, as noted by Kim et al. (2019), a URNNG trained from scratch fails to outperform a right-branching baseline on this version of PTB where punctuation is removed.

2 Model Analysis

We analyze our best compound PCFG model in more detail. Since we induce a full set of nonterminals in our grammar, we can analyze the learned nonterminals to see if they can be aligned with linguistic constituent labels. Figure 2 visualizes the alignment between induced and gold labels, where for each nonterminal we show the empirical probability that a predicted constituent of this type will correspond to a particular linguistic constituent in the test set, conditioned on its being a correct constituent (for reference we also show the precision). We observe that some of the induced nonterminals clearly align to linguistic nonterminals. Further results, including preterminal alignments to part-of-speech tags,As a POS induction system, the many-to-one performance of the compound PCFG using the preterminals is 68.0. A similarly-parameterized compound HMM with 60 hidden states (an HMM is a particularly type of PCFG) obtains 63.2. This is still quite a bit lower than the state-of-the-art Tran et al. (2016); He et al. (2018); Stratos (2019), though comparison is confounded by various factors such as preprocessing. A neural PCFG/HMM obtains 68.2 and 63.4 respectively. are shown in section A.4.

We next analyze the continuous latent space. Table 4 shows nearest neighbors of some sentences using the mean of the variational posterior as the continuous representation of each sentence. We qualitatively observe that the latent space seems to capture topical information.

We are also interested in the variation in the leaves due to $\mathbf{z}$ when the variation due to the tree structure is held constant. To investigate this, we use the parsed dataset to obtain pairs of the form $(\boldsymbol{\mu}_{\phi}(\boldsymbol{x}^{(n)}),\boldsymbol{t}^{(n)}_{j})$, where $\boldsymbol{t}^{(n)}_{j}$ is the $j$-th subtree of the (approximate) MAP tree $\boldsymbol{t}^{(n)}$ for the $n$-th sentence. Therefore each mean vector $\boldsymbol{\mu}_{\phi}(\boldsymbol{x}^{(n)})$ is associated with $|\boldsymbol{x}^{(n)}|-1$ subtrees, where $|\boldsymbol{x}^{(n)}|$ is the sentence length. Our definition of subtree here ignores terminals, and thus each subtree is associated with many mean vectors. For a frequently occurring subtree, we perform PCA on the set of mean vectors that are associated with the subtree to obtain the top principal component. We then show the constituents that had the 5 most positive/negative values for this top principal component in Table 5. For example, a particularly common subtree—associated with 180 unique constituents—is given by

The top 5 constituents with the most negative/positive values are shown in the top left part of Table 5. We find that the leaves $[w_{1},\dots,w_{6}]$, which form a 6-word constituent, vary in a regular manner as $\mathbf{z}$ is varied. We also observe that root of this subtree (NT-04) aligns to prepositional phrases (PP) in Figure 2, and the leaves in Table 5 (top left) are indeed mostly PP. However, the model fails to identify ((T-40 $w_{5}$) (T-22 $w_{6}$)) as a constituent in this case (as well as well in the bottom right example). See appendix A.5 for more examples. It is possible that the model is utilizing the subtrees to capture broad template-like structures and then using $\mathbf{z}$ to fill them in, similar to recent works that also train models to separate “what to say” from “how to say it” Wiseman et al. (2018); Peng et al. (2019); Chen et al. (2019a, b).

3 Limitations

We report on some negative results as well as important limitations of our work. While distributed representations promote parameter sharing, we were unable to obtain improvements through more factorized parameterizations that promote even greater parameter sharing. In particular, for rules of the type $A\to BC$, we tried having the output embeddings be a function of the input embeddings (e.g. $\mathbf{u}_{BC}=g([\mathbf{w}_{B};\mathbf{w}_{C}])$ where $g$ is an MLP), but obtained worse results. For rules of the type $T\to w$, we tried using a character-level CNN dos Santos and Zadrozny (2014); Kim et al. (2016) to obtain the output word embeddings $\mathbf{u}_{w}$ Jozefowicz et al. (2016); Tran et al. (2016), but found the performance to be similar to the word-level case.It is also possible to take advantage of pretrained word embeddings by using them to initialize output word embeddings or directly working with continuous emission distributions Lin et al. (2015); He et al. (2018) We were also unable to obtain improvements by making the variational family more flexible through normalizing flows Rezende and Mohamed (2015); Kingma et al. (2016). However, given that we did not exhaustively explore the full space of possible parameterizations, the above modifications could eventually lead to improvements with the right setup.

Relatedly, the models were quite sensitive to parameterization (e.g. it was important to use residual layers for $f_{1},f_{2}$), grammar size, and optimization method. We also noticed some variance in results across random seeds, as shown in Table 2. Finally, despite vectorized GPU implementations, training was significantly more expensive (both in terms of time and memory) than NLM-based unsupervised parsing systems due to the $O(|\mathcal{R}||\boldsymbol{x}|^{3})$ dynamic program, which makes our approach potentially difficult to scale.

Related Work

Grammar induction and unsupervised parsing has a long and rich history in natural language processing. Early work on with pure unsupervised learning was mostly negative Lari and Young (1990); Carroll and Charniak (1992); Charniak (1993), though Pereira and Schabes (1992) reported some success on partially bracketed data. Clark (2001) and Klein and Manning (2002) were some of the first successful statistical approaches. In particular, the constituent-context model (CCM) of Klein and Manning (2002), which explicitly models both constituents and distituents, was the basis for much subsequent work Klein and Manning (2004); Huang et al. (2012); Golland et al. (2012). Other works have explored imposing inductive biases through Bayesian priors Johnson et al. (2007); Liang et al. (2007); Wang and Blunsom (2013), modified objectives Smith and Eisner (2004), and additional constraints on recursion depth Noji et al. (2016); Jin et al. (2018).

While the framework of specifying the structure of a grammar and learning the parameters is common, other methods exist. Bod (2006) consider a nonparametric-style approach to unsupervised parsing by using random subsets of training subtrees to parse new sentences. Seginer (2007) utilize an incremental algorithm to unsupervised parsing which makes local decisions to create constituents based on a complex set of heuristics. Ponvert et al. (2011) induce parse trees through cascaded applications of finite state models.

More recently, neural network-based approaches have shown promising results on inducing parse trees directly from words. Shen et al. (2018, 2019) learn tree structures through soft gating layers within neural language models, while Drozdov et al. (2019) combine recursive autoencoders with the inside-outside algorithm. Kim et al. (2019) train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and Shi et al. (2019) utilize image captions to identify and ground constituents.

Our work is also related to latent variable PCFGs Matsuzaki et al. (2005); Petrov et al. (2006); Cohen et al. (2012), which extend PCFGs to the latent variable setting by splitting nonterminal symbols into latent subsymbols. In particular, latent vector grammars Zhao et al. (2018) and compositional vector grammars Socher et al. (2013) also employ continuous vectors within their grammars. However these approaches have been employed for learning supervised parsers on annotated treebanks, in contrast to the unsupervised setting of the current work.

Conclusion

This work studies a neural network-based approach grammar induction with PCFGs. We first propose to parameterize a PCFG’s rule probabilities with neural networks over distributed representations of latent symbols, and find that this neural PCFG makes it possible to induce linguistically meaningful grammars with simple maximum likelihood learning. We then extend the neural PCFG through a sentence-level continuous latent vector, which induces marginal dependencies beyond the traditional first-order context-free assumptions. We show that this compound PCFG learns richer grammars and leads to improved performance when evaluated as an unsupervised parser. The collapsed amortized variational inference approach is general and can be used for generative models which admit tractable inference through partial conditioning. Learning deep generative models which exhibit such conditional Markov properties is an interesting direction for future work.

Acknowledgments

We thank Phil Blunsom for initial discussions which seeded many of the core ideas in the present work. We also thank Yonatan Belinkov and Shay Cohen for helpful feedback, and Andrew Drozdov for providing the parsed dataset from their DIORA model. YK is supported by a Google Fellowship. AMR acknowledges the support of NSF 1704834, 1845664, AWS, and Oracle.

References

Appendix A Appendix

We associate an input embedding $\mathbf{w}_{N}$ for each symbol $N$ on the left side of a rule (i.e. $N\in\{S\}\cup\mathcal{N}\cup\mathcal{P}$) and run a neural network over $\mathbf{w}_{N}$ to obtain the rule probabilities. Concretely, each rule type $\pi_{r}$ is parameterized as follows,

where $\mathcal{M}$ is the product space $(\mathcal{N}\cup\mathcal{P})\times(\mathcal{N}\cup\mathcal{P})$, and $f_{1},f_{2}$ are MLPs with two residual layers,

In the compound PCFG the rule probabilities $\boldsymbol{\pi}_{\mathbf{z}}$ given a latent vector $\mathbf{z}$,

Again $f_{1},f_{2}$ are as before where the first layer’s input dimensions are appropriately changed to account for concatenation with $\mathbf{z}$.

For completeness we show the corpus-level and sentence-level $F_{1}$ broken down by sentence length in Table 6, averaged across 4 different runs of each model. In Figure 1 we use the following to refer to rule probabilities of different rule types for the neural PCFG (left),

where $L(A)$ denotes the set of rules with $A$ on the left hand side. The set of rule probabilities for the compound PCFG (right) is similarly defined,

A.3 Experiments with RNNGs

For experiments on supervising RNNGs with induced trees, we use the parameterization and hyperparameters from Kim et al. (2019), which uses a 2-layer 650-dimensional stack LSTM (with dropout of 0.5) and a 650-dimensional tree LSTM Tai et al. (2015); Zhu et al. (2015) as the composition function.

where the MLP has a single hidden layer with $\operatorname*{ReLU}$ nonlinearity followed by layer normalization Ba et al. (2016).

For experiments on fine-tuning the RNNG with the unsupervised RNNG, we take the discriminative parser (which is also pretrained alongside the RNNG on induced trees) to be the structured inference network for optimizing the evidence lower bound. We refer the reader to Kim et al. (2019) and their open source implementationhttps://github.com/harvardnlp/urnng for additional details. We also observe that as noted by Kim et al. (2019), a URNNG trained from scratch on this version of PTB without punctuation failed to outperform a right-branching baseline.

The LSTM language model baseline is the same size as the stack LSTM (i.e. 2 layers, 650 hidden units, dropout of 0.5), and is therefore equivalent to an RNNG with completely right branching trees. The PRPN/ON baselines for perplexity/syntactic evaluation in Table 3 also have 2 layers with 650 hidden units and 0.5 dropout. Therefore all models considered in Table 3 have roughly the same capacity. For all models we share input/output word embeddings Press and Wolf (2016). Perplexity estimation for the RNNGs and the compound PCFG uses 1000 importance-weighted samples.

For grammaticality judgment, we modify the publicly available dataset from Marvin and Linzen (2018)https://github.com/BeckyMarvin/LM_syneval to only keep sentence pairs that did not have any unknown words with respect to our PTB vocabulary of 10K words. This results in 33K sentence pairs for evaluation.

A.4 Nonterminal/Preterminal Alignments

Figure 3 shows the part-of-speech alignments and Table 7 shows the nonterminal label alignments for the compound PCFG/neural PCFG.

A.5 Subtree Analysis

Table 8 lists more examples of constituents within each subtree as the top principical component is varied. Due to data sparsity, the subtree analysis is performed on the full dataset. See section 5.2 for more details.