Filtering Variational Objectives

Introduction

Learning in statistical models via gradient descent is straightforward when the objective function and its gradients are tractable. In the presence of latent variables, however, many objectives become intractable. For neural generative models with latent variables, there are currently a few dominant approaches: optimizing lower bounds on the marginal log-likelihood , restricting to a class of invertible models , or using likelihood-free methods . In this work, we focus on the first approach and introduce filtering variational objectives (FIVOs), a tractable family of objectives for maximum likelihood estimation (MLE) in latent variable models with sequential structure.

Specifically, let $x$ denote an observation of an $\mathcal{X}$-valued random variable. We assume that the process generating $x$ involves an unobserved $\mathcal{Z}$-valued random variable $z$ with joint density $p(x,z)$ in some family $\mathcal{P}$. The goal of MLE is to recover $p\in\mathcal{P}$ that maximizes the marginal log-likelihood, $\log p(x)=\log\left(\int p(x,z)\ dz\right)$1. The difficulty in carrying out this optimization is that the log-likelihood function is defined via a generally intractable integral. To circumvent marginalization, a common approach is to optimize a variational lower bound on the marginal log-likelihood . The evidence lower bound $\mathcal{L}(x,p,q)$ (ELBO) is the most common such bound and is defined by a variational posterior distribution $q(z|x)$ whose support includes $p$’s,

$\mathcal{L}(x,p,q)$ lower-bounds the marginal log-likelihood for any choice of $q$, and the bound is tight when $q$ is the true posterior $p(z|x)$. Thus, the joint optimum of $\mathcal{L}(x,p,q)$ in $p$ and $q$ is the MLE. In practice, it is common to restrict $q$ to a tractable family of distributions (e.g., a factored distribution) and to jointly optimize the ELBO over $p$ and $q$ with stochastic gradient ascent . Because of the KL penalty from $q$ to $p$, optimizing (1) under these assumptions tends to force $p$’s posterior to satisfy the factorizing assumptions of the variational family which reduces the capacity of the model $p$. One strategy for addressing this is to decouple the tightness of the bound from the quality of $q$. For example, observed that Eq. (1) can be interpreted as the log of an unnormalized importance weight with the proposal given by $q$, and that using $N$ samples from the same proposal produces a tighter bound, known as the importance weighted auto-encoder bound, or IWAE.

Indeed, it follows from Jensen’s inequality that the log of any unbiased positive Monte Carlo estimator of the marginal likelihood results in a lower bound that can be optimized for MLE. The filtering variational objectives (FIVOs) build on this idea by treating the log of a particle filter’s likelihood estimator as an objective function. Following , we call objectives defined as log-transformed likelihood estimators Monte Carlo objectives (MCOs). In this work, we show that the tightness of an MCO scales like the relative variance of the estimator from which it is constructed. It is well-known that the variance of a particle filter’s likelihood estimator scales more favourably than simple importance sampling for models with sequential structure . Thus, FIVO can potentially form a much tighter bound on the marginal log-likelihood than IWAE.

The main contributions of this work are introducing filtering variational objectives and a more careful study of Monte Carlo objectives. In Section 2, we review maximum likelihood estimation via maximizing the ELBO. In Section 3, we study Monte Carlo objectives and provide some of their basic properties. We define filtering variational objectives in Section 4, discuss details of their optimization, and present a sharpness result. Finally, we cover related work and present experiments showing that sequential models trained with FIVO outperform models trained with ELBO or IWAE in practice.

Background

We briefly review techniques for optimizing the ELBO as a surrogate MLE objective. We restrict our focus to latent variable models in which the model $p_{\theta}(x,z)$ factors into tractable conditionals $p_{\theta}(z)$ and $p_{\theta}(x|z)$ that are parameterized differentiably by parameters $\theta$. MLE in these models is then the problem of optimizing $\log p_{\theta}(x)$ in $\theta$. The expectation-maximization (EM) algorithm is an approach to this problem which can be seen as coordinate ascent, fully maximizing $\mathcal{L}(x,p_{\theta},q)$ alternately in $q$ and $\theta$ at each iteration . Yet, EM rarely applies in general, because maximizing over $q$ for a fixed $\theta$ corresponds to a generally intractable inference problem.

Instead, an approach with mild assumptions on the model is to perform gradient ascent following a Monte Carlo estimator of the ELBO’s gradient . We assume that $q$ is taken from a family of distributions parameterized differentiably by parameters $\phi$. We can follow an unbiased estimator of the ELBO’s gradient by sampling $z\sim q_{\phi}(z|x)$ and updating the parameters by $\theta^{\prime}=\theta+\eta\nabla_{\theta}\log p_{\theta}(x,z)$ and $\phi^{\prime}=\phi+\eta(\log p_{\theta}(x,z)-\log q_{\phi}(z|x))\nabla_{\phi}\log q_{\phi}(z|x)$, where the gradients are computed conditional on the sample $z$ and $\eta$ is a learning rate. Such estimators follow the ELBO’s gradient in expectation, but variance reduction techniques are usually necessary .

A lower variance gradient estimator can be derived if $q_{\phi}$ is a reparameterizable distribution . Reparameterizable distributions are those that can be simulated by sampling from a distribution $\epsilon\sim d(\epsilon)$, which does not depend on $\phi$, and then applying a deterministic transformation $z=f_{\phi}(x,\epsilon)$. When $p_{\theta}$, $q_{\phi}$, and $f_{\phi}$ are differentiable, an unbiased estimator of the ELBO gradient consists of sampling $\epsilon$ and updating the parameter by $(\theta^{\prime},\phi^{\prime})=(\theta,\phi)+\eta\nabla_{(\theta,\phi)}(\log p_{\theta}(x,f_{\phi}(x,\epsilon))-\log q_{\phi}(f_{\phi}(x,\epsilon)|x))$. Given $\epsilon$, the gradients of the sampling process can flow through $z=f_{\phi}(x,\epsilon)$.

Monte Carlo Objectives (MCOs)

Monte Carlo objectives (MCOs) generalize the ELBO to objectives defined by taking the $\log$ of a positive, unbiased estimator of the marginal likelihood. The key property of MCOs is that they are lower bounds on the marginal log-likelihood, and thus can be used for MLE. Motivated by the previous section, we present results on the convergence of generic MCOs to the marginal log-likelihood and show that the tightness of an MCO is closely related to the variance of the estimator that defines it.

One can verify that the ELBO is a lower bound by using the concavity of log and Jensen’s inequality,

For example, the ELBO is constructed from a single unnormalized importance weight $\hat{p}(x)=p(x,z)/q(z|x)$. The IWAE bound takes $\hat{p}_{N}(x)$ to be $N$ averaged i.i.d. importance weights,

We consider additional examples in the Appendix. To avoid notational clutter, we omit the arguments to an MCO, e.g., the observations $x$ or model $p$, when the default arguments are clear from context. Whether we can compute stochastic gradients of $\operatorname{\mathcal{L}_{N}}$ efficiently depends on the specific form of the estimator and the underlying random variables that define it.

Many likelihood estimators $\hat{p}_{N}(x)$ converge to $p(x)$ almost surely as $N\to\infty$ (known as strong consistency). The advantage of a consistent estimator is that its MCO can be driven towards $\log p(x)$ by increasing $N$. We present sufficient conditions for this convergence and a description of the rate:

Properties of Monte Carlo Objectives. Let $\operatorname{\mathcal{L}_{N}}(x,p)$ be a Monte Carlo objective defined by an unbiased positive estimator $\hat{p}_{N}(x)$ of $p(x)$. Then,

(Bound) $\operatorname{\mathcal{L}_{N}}(x,p)\leq\log p(x)$.

(Consistency) If $\log\hat{p}_{N}(x)$ is uniformly integrable (see Appendix for definition) and $\hat{p}_{N}(x)$ is strongly consistent, then $\operatorname{\mathcal{L}_{N}}(x,p)\to\log p(x)$ as $N\to\infty$.

See the Appendix for the proof and a sufficient condition for controlling the first inverse moment when $\hat{p}_{N}(x)$ is the average of i.i.d. random variables. ∎

In some cases, convergence of the bound to $\log p(x)$ is monotonic, e.g., IWAE , but this is not true in general. The relative variance of estimators, $\operatorname*{var}(\hat{p}_{N}(x)/p(x))$, tends to be well studied, so property (c) gives us a tool for comparing the convergence rate of distinct MCOs. For example, study marginal likelihood estimators defined by particle filters and find that the relative variance of these estimators scales favorably in comparison to naive importance sampling. This suggests that a particle filter’s MCO, introduced in the next section, will generally be a tighter bound than IWAE.

Filtering Variational Objectives (FIVOs)

The filtering variational objectives (FIVOs) are a family of MCOs defined by the marginal likelihood estimator of a particle filter. For models with sequential structure, e.g., latent variable models of audio and text, the relative variance of a naive importance sampling estimator tends to scale exponentially in the number of steps. In contrast, the relative variance of particle filter estimators can scale more favorably with the number of steps—linearly in some cases . Thus, the results of Section 3 suggest that FIVOs can serve as tighter objectives than IWAE for MLE in sequential models.

Let our observations be sequences of $T$ $\mathcal{X}$-valued random variables denoted $x_{1:T}$, where $x_{i:j}\equiv(x_{i},\ldots,x_{j})$. We also assume that the data generation process relies on a sequence of $T$ unobserved $\mathcal{Z}$-valued latent variables denoted $z_{1:T}$. We focus on sequential latent variable models that factor as a series of tractable conditionals, $p(x_{1:T},z_{1:T})=p_{1}(x_{1},z_{1})\prod_{t=2}^{T}p_{t}(x_{t},z_{t}|x_{1:t-1},z_{1:t-1})$.

A particle filter is a sequential Monte Carlo algorithm, which propagates a population of $N$ weighted particles for $T$ steps using a combination of importance sampling and resampling steps, see Alg. 1. In detail, the particle filter takes as arguments an observation $x_{1:T}$, the number of particles $N$, the model distribution $p$, and a variational posterior $q(z_{1:T}|x_{1:T})$ factored over $t$,

The particle filter maintains a population $\{{w}_{t-1}^{i},z_{1:t-1}^{i}\}_{i=1}^{N}$ of particles $z_{1:t-1}^{i}$ with weights $w_{t-1}^{i}$. At step $t$, the filter independently proposes an extension $z_{t}^{i}\sim q_{t}(z_{t}|x_{1:t},z_{1:t-1}^{i})$ to each particle’s trajectory $z_{1:t-1}^{i}$. The weights $w_{t-1}^{i}$ are multiplied by the incremental importance weights,

and renormalized. If the current weights $w_{t}^{i}$ satisfy a resampling criteria, then a resampling step is performed and $N$ particles $z_{1:t}^{i}$ are sampled in proportion to their weights from the current population with replacement. Common resampling schemes include resampling at every step and resampling if the effective sample size (ESS) of the population $(\sum_{i=1}^{N}(w_{t}^{i})^{2})^{-1}$ drops below $N/2$ . After resampling the weights are reset to $1$. Otherwise, the particles $z_{1:t}^{i}$ are copied to the next step along with the accumulated weights. See Fig. 1 for a visualization.

Given a single forward pass of Alg. 1 with reparameterized $z_{t}^{i}$, the terms inside the expectation form a Monte Carlo estimator of Eq. (8). However, the terms from resampling events contribute to the majority of the variance of the estimator. Thus, the gradient estimator that we found most effective in practice consists only of the gradient $\nabla_{(\theta,\phi)}\log\hat{p}_{N}(x_{1:T})$, the solid red arrows of Figure 1. We explore this experimentally in Section 6.3.

2 Sharpness

As with the ELBO, FIVO is a variational objective taking a variational posterior $q$ as an argument. An important question is whether FIVO achieves the marginal log-likelihood at its optimal $q$. We can only guarantee this for models in which $z_{1:t-1}$ and $x_{t}$ are independent given $x_{1:t-1}$.

Most models do not satisfy this assumption, and deriving the optimal $q$ in general is complicated by the resampling dynamics. For the restricted the model class in Proposition 2, the optimal $q_{t}$ does not condition on future observations $x_{t+1:T}$. We explored this experimentally with richer models in Section 6.4, and found that allowing $q_{t}$ to condition on $x_{t+1:T}$ does not reliably improve FIVO. This is consistent with the view of resampling as a greedy process that responds to each intermediate distribution as if it were the final. Still, we found that the impact of this effect was outweighed by the advantage of optimizing a tighter bound.

Related Work

The marginal log-likelihood is a central quantity in statistics and probability, and there has long been an interest in bounding it . The literature relating to the bounds we call Monte Carlo objectives has typically focused on the problem of estimating the marginal likelihood itself. use Jensen’s inequality in a forward and reverse estimator to detect the failure of inference methods. IWAE is a clear influence on this work, and FIVO can be seen as an extension of this bound. The ELBO enjoys a long history and there have been efforts to improve the ELBO itself. generalize the ELBO by considering arbitrary operators of the model and variational posterior. More closely related to this work is a body of work improving the ELBO by increasing the expressiveness of the variational posterior. For example, augment the variational posterior with deterministic transformations with fixed Jacobians, and extend the variational posterior to admit a Markov chain.

Other approaches to learning in neural latent variable models include , who use importance sampling to approximate gradients under the posterior, and , who use sequential Monte Carlo to approximate gradients under the posterior. These are distinct from our contribution in the sense that for them inference for the sake of estimation is the ultimate goal. To our knowledge the idea of treating the output of inference as an objective in and of itself, while not completely novel, has not been fully appreciated in the literature. Although, this idea shares inspiration with methods that optimize the convergence of Markov chains .

We note that the idea to optimize the log estimator of a particle filter was independently and concurrently considered in . In the bound we call FIVO is cast as a tractable lower bound on the ELBO defined by the particle filter’s non-parameteric approximation to the posterior. additionally derive an expression for FIVO’s bias as the KL between the filter’s distribution and a certain target process. Our work is distinguished by our study of the convergence of MCOs in $N$, which includes FIVO, our investigation of FIVO sharpness, and our experimental results on stochastic RNNs.

Experiments

In our experiments, we sought to: (a) compare models trained with ELBO, IWAE, and FIVO bounds in terms of final test log-likelihoods, (b) explore the effect of the resampling gradient terms on FIVO, (c) investigate how the lack of sharpness affects FIVO, and (d) consider how models trained with FIVO use the stochastic state. To explore these questions, we trained variational recurrent neural networks (VRNN) with the ELBO, IWAE, and FIVO bounds using TensorFlow on two benchmark sequential modeling tasks: natural speech waveforms and polyphonic music. These datasets are known to be difficult to model without stochastic latent states .

The VRNN is a sequential latent variable model that combines a deterministic recurrent neural network (RNN) with stochastic latent states $z_{t}$ at each step. The observation distribution over $x_{t}$ is conditioned directly on $z_{t}$ and indirectly on $z_{1:t-1}$ via the RNN’s state $h_{t}(z_{t-1},x_{t-1},h_{t-1})$. For a length $T$ sequence, the model’s posterior factors into the conditionals $\prod_{t=1}^{T}p_{t}(z_{t}|h_{t}(z_{t-1},x_{t-1},h_{t-1}))g_{t}(x_{t}|z_{t},h_{t}(z_{t-1},x_{t-1},h_{t-1}))$, and the variational posterior factors as $\prod_{t=1}^{T}q_{t}(z_{t}|h_{t}(z_{t-1},x_{t-1},h_{t-1}),x_{t})$. All distributions over latent variables are factorized Gaussians, and the output distributions $g_{t}$ depend on the dataset. The RNN is a single-layer LSTM and the conditionals are parameterized by fully connected neural networks with one hidden layer of the same size as the LSTM hidden layer. We used the residual parameterization for the variational posterior.

We evaluated VRNNs trained with the ELBO, IWAE, and FIVO bounds on 4 polyphonic music datasets: the Nottingham folk tunes, the JSB chorales, the MuseData library of classical piano and orchestral music, and the Piano-midi.de MIDI archive . Each dataset is split into standard train, valid, and test sets and is represented as a sequence of 88-dimensional binary vectors denoting the notes active at the current timestep. We mean-centered the input data and modeled the output as a set of 88 factorized Bernoulli variables. We used 64 units for the RNN hidden state and latent state size for all polyphonic music models except for JSB chorales models, which used 32 units. We report bounds on average log-likelihood per timestep in Table 1. Models trained with the FIVO bound significantly outperformed models trained with either the ELBO or the IWAE bounds on all four datasets. In some cases, the improvements exceeded $1$ nat per timestep, and in all cases optimizing FIVO with $N=4$ outperformed optimizing IWAE or ELBO for $N=\{4,8,16\}$.

2 Speech

The TIMIT dataset is a standard benchmark for sequential models that contains 6300 utterances with an average duration of 3.1 seconds spoken by 630 different speakers. The 6300 utterances are divided into a training set of size 4620 and a test set of size 1680. We further divided the training set into a validation set of size 231 and a training set of size 4389, with the splits exactly as in . Each TIMIT utterance is represented as a sequence of real-valued amplitudes which we split into a sequence of 200-dimensional frames, as in . Data preprocessing was limited to mean centering and variance normalization as in . For TIMIT, the output distribution was a factorized Gaussian, and we report the average log-likelihood bound per sequence relative to models trained with ELBO. Again, models trained with FIVO significantly outperformed models trained with IWAE or ELBO, see Table 1.

3 Resampling Gradients

All models in this work (except those in this section) were trained with gradients that did not include the term in Eq. (8) that comes from resampling steps. We omitted this term because it has an outsized effect on gradient variance, often increasing it by 6 orders of magnitude. To explore the effects of this term experimentally, we trained VRNNs with and without the resampling gradient term on the TIMIT and polyphonic music datasets. When using the resampling term, we attempted to control its variance using a moving-average baseline linear in the number of timesteps. For all datasets, models trained without the resampling gradient term outperformed models trained with the term by a large margin on both the training set and held-out data. Many runs with resampling gradients failed to improve beyond random initialization. A representative pair of train log-likelihood curves is shown in Figure 2 — gradients without the resampling term led to earlier convergence and a better solution. We stress that this is an empirical result — in principle biased gradients can lead to divergent behaviour. We leave exploring strategies to reduce the variance of the unbiased estimator to future work.

4 Sharpness

FIVO does not achieve the marginal log-likelihood at its optimal variational posterior $q^{*}$, because the optimal $q^{*}$ does not condition on future observations (see Section 4.2). In contrast, ELBO and IWAE are sharp, and their $q^{*}$s depend on future observations. To investigate the effects of this, we defined a smoothing variant of the VRNN in which $q$ takes as additional input the hidden state of a deterministic RNN run backwards over the observations, allowing $q$ to condition on future observations. We trained smoothing VRNNs using ELBO, IWAE, and FIVO, and report evaluation on the training set (to isolate the effect on optimization performance) in Table 2 . Smoothing helped models trained with IWAE, but not enough to outperform models trained with FIVO. As expected, smoothing did not reliably improve models trained with FIVO. Test set performance was similar, see the Appendix for details.

5 Use of Stochastic State

A known pathology when training stochastic latent variable models with the ELBO is that stochastic states can go unused. Empirically, this is associated with the collapse of variational posterior $q(z|x)$ network to the model prior $p(z)$ . To investigate this, we plot the KL divergence from $q(z_{1:T}|x_{1:T})$ to $p(z_{1:T})$ averaged over the dataset (Figure 2). Indeed, the KL of models trained with ELBO collapsed during training, whereas the KL of models trained with FIVO remained high, even while achieving a higher log-likelihood bound.

Conclusions

We introduced the family of filtering variational objectives, a class of lower bounds on the log marginal likelihood that extend the evidence lower bound. FIVOs are suited for MLE in neural latent variable models. We trained models with the ELBO, IWAE, and FIVO bounds and found that the models trained with FIVO significantly outperformed other models across four polyphonic music modeling tasks and a speech waveform modeling task. Future work will include exploring control variates for the resampling gradients, FIVOs defined by more sophisticated filtering algorithms, and new MCOs based on differentiable operators like leapfrog operators with deterministically annealed temperatures. In general, we hope that this paper inspires the machine learning community to take a fresh look at the literature of marginal likelihood estimators—seeing them as objectives instead of algorithms for inference.

We thank Matt Hoffman, Matt Johnson, Danilo J. Rezende, Jascha Sohl-Dickstein, and Theophane Weber for helpful discussions and support in this project. A. Doucet was partially supported by the EPSRC grant EP/K000276/1. Y. W. Teh’s research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013) ERC grant agreement no. 617071.

References

Appendix to Filtering Variational Objectives

There is an extensive literature on marginal likelihood estimators . Each defines an MCO, and we consider two in more detail, annealed importance sampling and multiple importance sampling . Let $x$ denote an observation of an $\mathcal{X}$-valued random variable generated in a process with an unobserved $\mathcal{Z}$-valued random variable $z$. Let $p(x,z)$ be the joint density.

Annealed importance sampling (AIS) is a generalization of importance sampling . We present an MCO derived from a special case of the AIS algorithm. Let $q(z|x)$ be a variational posterior distribution and let $\beta_{i}$ be a sequence of real numbers for $i\in\{1,\ldots,N+1\}$ such that $0\leq\beta_{i}\leq 1$ and $\beta_{1}=0$ and $\beta_{N+1}=1$. Let $T_{i}(z^{\prime}|z,x)$ be a Markov transition distribution whose stationary distribution is proportional to $q(z|x)^{1-\beta_{i}}p(x,z)^{\beta_{i}}$. Then for $z_{1}\sim q(z|x)$ and $z_{i}\sim T_{i}(z^{\prime}|z_{i-1},x)$ for $i\in\{2,\dots,N\}$ we have the following unbiased estimator,

Notice two things. First, there is no assumption that the states $z_{i}$ are at equilibrium, and second, we did not require a transition operator keeping $p(x,z)$ as an invariant distribution. All together, we can define the AIS MCO,

This is a sharp objective, if we take $q$ as the true posterior, $q(z|x)=p(z|x)$, and $T_{i}(z^{\prime}|z,x)=\delta(z^{\prime}-z)$ to be the Dirac delta copy operator. The difficulty in applying this MCO is finding $T_{i}$, which are scalable and easy to optimize. Generalizations of the AIS procedure have been proposed in . The resulting Sequential Monte Carlo samplers procedures also provide an unbiased estimator of the marginal likelihood and are structurally identical to the particle algorithm presented in this paper.

Multiple importance sampling (MIS) is another generalization of importance sampling. Let $q_{i}(z|x)$ be $N$ possibly distinct variational posterior distributions and $w_{i}(x)\geq 0$ be such that $\sum_{i=1}^{N}w_{i}(x)=1$. There are a variety of distinct estimators that could be formed from the $q_{i}$ . We present just one. Let $z_{i}\sim q_{i}(z|x)$, then we have the following unbiased estimator

Notice that the latent sample $z_{i}\sim q_{i}(z|x)$ is evaluated under all $q_{i}$’s. One can view this as a Rao-Blackwellized estimator corresponding to the mixture distribution $\sum_{i=1}^{N}w_{i}(x)q_{i}(z|x)$. All together,

Again, this objective is sharp, if we take any $q_{i}(z|x)=p(z|x)$ and $w_{i}(x)=1$. The difficulty in making this objective more useful is optimizing it in a way that distinguishes the $q_{i}$ and assigns the appropriate $w_{i}$.

Proof of Proposition 1.

By the concavity of $\log$ and Jensen’s inequality,

$\hat{p}_{N}(x)$ is strongly consistent, i.e. $\hat{p}_{N}(x)\overset{a.s.}{\longrightarrow}p(x)$ as $N\to\infty$.

Controlling the first inverse moment.

We provide a sufficient condition that guarantees that the inverse moment of the average of i.i.d. random variables is bounded, a condition used in Proposition 1 (c). Intuitively, this is a fairly weak condition, because it only requires that the mass in an arbitrarily small neighbourhood of zero is bounded.

We sketch an argument that the random variable $\hat{p}_{N}(x_{1:T})$ defined by Algorithm 1 is an unbiased estimator of the marginal likelihood $p(x_{1:T})$. This is a well-known fact , and our sketch is based on . The strategy is to cast the particle filter’s estimator $\hat{p}_{N}(x_{1:T})$ as a single importance weight over an extended space. The lack of bias in the particle filter therefore reduces to the unbiasedness of importance sampling. Key to this is identifying the target and proposal distributions in the extended space. The target distribution is called “conditional sequential Monte Carlo”, Algorithm 2. The proposal distribution is the particle filter itself, Algorithm 1.

We argue that it is enough to consider just an arbitrary fixed (non-adaptive) resampling schedule that always resamples at step $T$. First, consider adaptive resampling criteria, i.e. criteria that are deterministic functions of the weights $w_{t}^{i}$. For such criteria the joint density of random variables in Algorithm 1 will be piecewise continuous, composed of $2^{T}$ regions corresponding to a sequence of resample/no-resample decisions. This density has a form on each piece that is exactly the same as the density for some fixed resampling schedule. Moreover, it is globally normalized, because of the sequential structure of the filter. Because Algorithm 2 makes the same decisions, it also is partitioned along the same sets and each piece has the same fixed resampling schedule. Thus, it is enough to consider only a fixed resampling schedule. Second, notice that in the final step, step $T$, of Algorithms 1 and 2 resampling has no effect on $\hat{p}_{N}(x_{1:T})$. Thus, we assume that the resampling criteria of Algorithms 1 and 2 at step $T$ is set to always resample. All together it is safe to assume a fixed resampling schedule with $R$ resampling events, $1\leq R\leq T$, at steps $k_{r}\in\{1,\dots,T\}$ for $r\in\{0,\ldots,R\}$ with $k_{R}=T$ and $k_{0}=0$.

Now we derive the joint density of Algorithm 1 and 2 taken at each iteration after possibly resampling. To avoid notational clutter we let $g,f$ (omitting their arguments) represent the densities of the variables in Algorithms 1 and 2. Technically, we should also be keeping track of the indices that indicate the inheritance of the resampling step. So, let the random variables $\{\{w_{t}^{i},z_{1:t}^{i}\}_{i=1}^{N}\}_{t=1}^{T}$ be the particles before resampling and $s(i)\in\{1,\ldots,N\}$ be the index that is selected for inheritance of the $i$th particle after resampling. Then the density corresponding to Algorithm 1 is

Now, letting $\alpha_{k}(z_{1:k}^{i})=\frac{p(z_{k}^{i},x_{k}|x_{1:k-1},z_{1:k-1}^{i})}{q_{k}(z_{k}^{i}|x_{1:k},z_{1:k-1}^{i})}$ one can show that for this sequence of resampling times the weight $w_{k_{r}}^{j}$ telescopes into

and the estimator $\hat{p}_{N}(x_{1:T})=\prod_{t=1}^{T}\hat{p}_{t}$ telescopes into

The result follows. An intuitive way to understand this result is the following: Algorithm 2 matches the distribution of every random variable in the particle filter except it interleaves a true posterior sample into the set of particles with uniform probability. The only mismatch in the densities are the normalization terms of the resampling probabilities of that privileged posterior sample, with terms $\sum_{i=1}^{N}\prod_{k=k_{r-1}+1}^{k_{r}}\alpha_{k}(z_{1:k}^{i})$ coming from the filter’s resampling and terms $N$ from conditional SMC’s resampling. Of course, we never run Algorithm 2, it just serves to define the target density.

First, we assume that the decision to resample is not adaptive (i.e., depends in some way on the random variables already produced until that point in Algorithm 1), and are fixed ahead of time. When the sampling $z_{t}^{i}$ is not reparameterized there are three terms to the gradient: (1) the gradients of $\log\hat{p}_{N}(x_{1:T})$ with respect to the parameters conditional on the latent states, (2) gradients of the densities $q_{t}$ with respect to their parameters, and (3) gradients of the resampling probabilities with respect to the parameters. All together, the following is a gradient of FIVO,

In this work we only considered reparameterized $q_{t}$s, and we dropped the terms of the gradient that arise from resampling.

Proof of Proposition 2.

Thus, all particles have the same weight and $\hat{p}_{1}=p_{1}(x_{1})$. Now for any $t$ we have that the weights must be $1/N$ since the particles all have the same weight and

Implementation details

We initialized weights using the Xavier initialization and used the Adam optimizer with a batch size of 4. During training, we did not truncate sequences and performed full backpropagation through time for all datasets. For the results presented in Sections 6.1 and 6.2 we performed a grid search over learning rates $\{3\times 10^{-4},1\times 10^{-4},3\times 10^{-5},1\times 10^{-5}\}$ and picked the run and early stopping step by the validation performance.

Evaluation and Comparison of Bounds

Comparing models trained with different log-likelihood lower bounds is challenging because calculating the actual log-likelihood is intractable. Burda et al. showed that the IWAE bound is at least as tight as the ELBO and monotonically increases with $N$. This suggests comparing models based on the IWAE bound evaluated with a large $N$. However, we found that IWAE and ELBO bounds tended to diverge for models trained with FIVO.

Although FIVO is not provably a tighter bound than the ELBO or IWAE, our experiments suggest that this tends to be the case in practice. In Figure 3, we plotted all three bounds over training for a representative experiment. All plots use the same model architecture, but the training objective changes in each panel. For the model trained with IWAE, the FIVO and IWAE bounds are tighter than their counterparts on the model trained with ELBO, suggesting that the model trained with IWAE is superior. The ELBO bound evaluated on the model trained with IWAE, however, is lower than its counterpart on the model trained with the ELBO. For the model trained with FIVO, both IWAE and ELBO bounds seem to diverge, but the FIVO bound outperforms the FIVO bounds on both of the other models. As in the figure, we generally found that the same model evaluated with FIVO, IWAE, and ELBO produced values descending in that order.

We suspect that $q$ distributions trained under the FIVO bound are more entropic than those trained under ELBO or IWAE because of the resampling operation. During training under FIVO, $q$ is able to propose state transitions that could poorly explain the observations because the bad states will be resampled away without harming the final bound value. Then, when a FIVO-trained $q$ is evaluated with ELBO or IWAE it proposes poor states that are not resampled away, leading to a poor final bound value. Conversely, $q$s trained with ELBO and IWAE are not able to fully leverage the resampling operation when evaluated with the FIVO bound.

Because of this behavior, we chose to optimistically evaluate models trained with IWAE and ELBO by reporting the maximum across all the bounds. For models trained with FIVO, we reported only the FIVO bound. We felt this evaluation scheme provided the strongest comparison to existing bounds.

Evaluating TIMIT Log-Likelihoods

We reported log-likelihood scores for TIMIT relative to an ELBO baseline instead of raw log-likelihoods. Previous papers (e.g., ) report the log-likelihood of data that have been mean centered and variance normalized, but it would be more proper to report the results on the un-standardized data. Specifically, if the training set has mean $\mu$ and variance $\sigma^{2}$ and the model outputs $\hat{\mu}$ and $\hat{\sigma}^{2}$, then the un-standardized test data would be evaluated under a $\mathcal{N}(\hat{\mu}\sigma+\mu,\hat{\sigma}^{2}\sigma^{2})$ distribution.

Log-likelihoods produced by these approaches differ by a constant offset that depends on $\sigma$. Because the offset is a function of only training set statistics, it does not affect relative comparison between methods. Because of this we chose to report log-likelihoods relative to a baseline instead of absolute numbers. Absolute numbers calculated on standardized data are reported in Tables 3, 4, and 5 to allow for comparisons with other papers.