Variational Autoencoders for Collaborative Filtering

Introduction

Recommender systems are an integral component of the web. In a typical recommendation system, we observe how a set of users interacts with a set of items. Using this data, we seek to show users a set of previously unseen items they will like. As the web grows in size, good recommendation systems will play an important part in helping users interact more effectively with larger amounts of content.

Collaborative filtering is among the most widely applied approaches in recommender systems. Collaborative filtering predicts what items a user will prefer by discovering and exploiting the similarity patterns across users and items. Latent factor models (Salakhutdinov and Mnih, 2008; Hu et al., 2008; Gopalan et al., 2015) still largely dominate the collaborative filtering research literature due to their simplicity and effectiveness. However, these models are inherently linear, which limits their modeling capacity. Previous work (Liang et al., 2016) has demonstrated that adding carefully crafted non-linear features into the linear latent factor models can significantly boost recommendation performance. Recently, a growing body of work involves applying neural networks to the collaborative filtering setting with promising results (Zheng et al., 2016; Sedhain et al., 2015; Wu et al., 2016; He et al., 2017).

Here, we extend variational autoencoders (vaes) (Kingma and Welling, 2013; Rezende et al., 2014) to collaborative filtering for implicit feedback. Vaes generalize linear latent-factor models and enable us to explore non-linear probabilistic latent-variable models, powered by neural networks, on large-scale recommendation datasets. We propose a neural generative model with multinomial conditional likelihood. Despite being widely used in language modeling and economics (Blei et al., 2003; McFadden et al., 1973), multinomial likelihoods appear less studied in the collaborative filtering literature, particularly within the context of latent-factor models. Recommender systems are often evaluated using ranking-based measures, such as mean average precision and normalized discounted cumulative gain (Järvelin and Kekäläinen, 2002). Top-$N$ ranking loss is difficult to optimize directly and previous work on direct ranking loss minimization resorts to relaxations and approximations (Weimer et al., 2008; Weston et al., 2011). Here, we show that the multinomial likelihoods are well-suited for modeling implicit feedback data, and are a closer proxy to the ranking loss relative to more popular likelihood functions such as Gaussian and logistic.

Though recommendation is often considered a big-data problem (due to the huge numbers of users and items typically present in a recommender system), we argue that, in contrast, it represents a uniquely challenging “small-data” problem: most users only interact with a tiny proportion of the items and our goal is to collectively make informed inference about each user’s preference. To make use of the sparse signals from users and avoid overfitting, we build a probabilistic latent-variable model that shares statistical strength among users and items. Empirically, we show that employing a principled Bayesian approach is more robust regardless of the scarcity of the data.

Although vaes have been extensively studied for image modeling and generation, there is surprisingly little work applying vaes to recommender systems. We find that two adjustments are essential to getting state-of-the-art results with vaes on this task:

First, we use a multinomial likelihood for the data distribution. We show that this simple choice realizes models that outperform the more commonly used Gaussian and logistic likelihoods.

Second, we reinterpret and adjust the standard vae objective, which we argue is over-regularized. We draw connections between the learning algorithm resulting from our proposed regularization and the information-bottleneck principle and maximum-entropy discrimination.

The result is a recipe that makes vaes practical solutions to this important problem. Empirically, our methods significantly outperform state-of-the-art baselines on several real-world datasets, including two recently proposed neural-network approaches.

Method

The log-likelihood for user $u$ (conditioned on the latent representation) is:

This multinomial likelihood is commonly used in language models, e.g., latent Dirichlet allocation (Blei et al., 2003), and economics, e.g., multinomial logit choice model (McFadden et al., 1973). It is also used in the cross-entropy lossThe cross-entropy loss for multi-class classification is a multinomial likelihood under a single draw from the distribution. for multi-class classification. For example, it has been used in recurrent neural networks for session-based sequential recommendation (Hidasi et al., 2015; Tan et al., 2016; Smirnova and Vasile, 2017; Hidasi and Karatzoglou, 2017; Chatzis et al., 2017) and in feedward neural networks applied to Youtube recommendation (Covington et al., 2016). The multinomial likelihood is less well studied in the context of latent-factor models such as matrix factorization and autoencoders. A notable exception is the collaborative competitive filtering (CCF) model (Yang et al., 2011) and its successors, which take advantage of more fine-grained information about what options were presented to which users. (If such information is available, it can also be incorporated into our vae-based approach.)

We believe the multinomial distribution is well suited to modeling click data. The likelihood of the click matrix (Eq. 2) rewards the model for putting probability mass on the non-zero entries in $\boldsymbol{\mathbf{x}}_{u}$. But the model has a limited budget of probability mass, since $\pi(\boldsymbol{\mathbf{z}}_{u})$ must sum to 1; the items must compete for this limited budget (Yang et al., 2011). The model should therefore assign more probability mass to items that are more likely to be clicked. To the extent that it can, it will perform well under the top-$N$ ranking loss that recommender systems are commonly evaluated on.

By way of comparison, we present two popular choices of likelihood functions used in latent-factor collaborative filtering: Gaussian and logistic likelihoods. Define $f_{\theta}(\boldsymbol{\mathbf{z}}_{u})\equiv[f_{u1},\dots,f_{uI}]^{\top}$ as the output of the generative function $f_{\theta}(\cdot)$. The Gaussian log-likelihood for user $u$ is

We adopt the convention in Hu et al. (2008) and introduce a “confidence” weight $c_{x_{ui}}\equiv c_{ui}$ where $c_{1}>c_{0}$ to balance the unobserved 0’s which far outnumber the observed 1’s in most click data. This is also equivalent to training the model with unweighted Gaussian likelihood and negative sampling. The logistic log-likelihoodLogistic likelihood is also cross-entropy loss for binary classification. for user $u$ is

where $\sigma(x)=1/(1+\exp(-x))$ is the logistic function. We compare multinomial likelihood with Gaussian and logistic in Section 4.

2. Variational inference

To learn the generative model in Eq. 1, we are interested in estimating $\theta$ (the parameters of $f_{\theta}(\cdot)$). To do so, for each data point we need to approximate the intractable posterior distribution $p(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})$. We resort to variational inference (Jordan et al., 1999). Variational inference approximates the true intractable posterior with a simpler variational distribution $q(\boldsymbol{\mathbf{z}}_{u})$. We set $q(\boldsymbol{\mathbf{z}}_{u})$ to be a fully factorized (diagonal) Gaussian distribution:

With variational inference the number of parameters to optimize $\{\boldsymbol{\mathbf{\mu}}_{u},\boldsymbol{\mathbf{\sigma}}_{u}^{2}\}$ grows with the number of users and items in the dataset. This can become a bottleneck for commercial recommender systems with millions of users and items. The variational autoencoder (vae) (Kingma and Welling, 2013; Rezende et al., 2014) replaces individual variational parameters with a data-dependent function (commonly called an inference model):

parametrized by $\phi$ with both $\mu_{\phi}(\boldsymbol{\mathbf{x}}_{u})$ and $\sigma_{\phi}(\boldsymbol{\mathbf{x}}_{u})$ being $K$-vectors and sets the variational distribution as follows:

That is, using the observed data $\boldsymbol{\mathbf{x}}_{u}$ as input, the inference model outputs the corresponding variational parameters of variational distribution $q_{\phi}(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})$, which, when optimized, approximates the intractable posterior $p(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})$.In the implementation, the inference model will output the log of the variance of the variational distribution. We continue to use $\sigma_{\phi}(\boldsymbol{\mathbf{x}}_{u})$ for notational brevity. Putting $q_{\phi}(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})$ and the generative model $p_{\theta}(\boldsymbol{\mathbf{x}}_{u}\,|\,\boldsymbol{\mathbf{z}}_{u})$ together in Figure 2c, we end up with a neural architecture that resembles an autoencoder — hence the name variational autoencoder.

Vaes make use of amortized inference (Gershman and Goodman, 2014): they flexibly reuse inferences to answer related new problems. This is well aligned with the ethos of collaborative filtering: analyze user preferences by exploiting the similarity patterns inferred from past experiences. In Section 2.4, we discuss how this enables us to perform prediction efficiently.

Learning vaes: As is standard when learning latent-variable models with variational inference (Blei et al., 2017), we can lower-bound the log marginal likelihood of the data. This forms the objective we seek to maximize for user $u$ (the objective function of the dataset is obtained by averaging the objective function over all the users):

This is commonly known as the evidence lower bound (elbo). Note that the elbo is a function of both $\theta$ and $\phi$. We can obtain an unbiased estimate of elbo by sampling $\boldsymbol{\mathbf{z}}_{u}\sim q_{\phi}$ and perform stochastic gradient ascent to optimize it. However, the challenge is that we cannot trivially take gradients with respect to $\phi$ through this sampling process. The reparametrization trick (Kingma and Welling, 2013; Rezende et al., 2014) sidesteps this issue: we sample $\boldsymbol{\mathbf{\epsilon}}\sim\mathcal{N}(0,\boldsymbol{\mathbf{I}}_{K})$ and reparametrize $\boldsymbol{\mathbf{z}}_{u}=\mu_{\phi}(\boldsymbol{\mathbf{x}}_{u})+\boldsymbol{\mathbf{\epsilon}}\odot\sigma_{\phi}(\boldsymbol{\mathbf{x}}_{u})$. By doing so, the stochasticity in the sampling process is isolated and the gradient with respect to $\phi$ can be back-propagated through the sampled $\boldsymbol{\mathbf{z}}_{u}$. The vae training procedure is summarized in Algorithm 1.

2.2. Alternative interpretation of elbo.

We can view elbo defined in Eq. 5 from a different perspective: the first term can be interpreted as (negative) reconstruction error, while the second KL term can be viewed as regularization. It is this perspective we work with because it allows us to make a trade-off that forms the crux of our method. From this perspective, it is natural to extend the elbo by introducing a parameter $\beta$ to control the strength of regularization:

While the original vae (trained with elbo in Eq. 5) is a powerful generative model; we might ask whether we need all the statistical properties of a generative model for tackling problems in recommender systems. In particular, if we are willing to sacrifice the ability to perform ancestral sampling, can we improve our performance? The regularization view of the elbo (Eq. 6) introduces a trade-off between how well we can fit the data and how close the approximate posterior stays to the prior during learning.

We propose using $\beta\neq 1$. This means we are no longer optimizing a lower bound on the log marginal likelihood. If $\beta<1$, then we are also weakening the influence of the prior constraint $\frac{1}{U}\sum_{u}q(\boldsymbol{\mathbf{z}}\,|\,\boldsymbol{\mathbf{x}}_{u})\approx p(\boldsymbol{\mathbf{z}})=\mathcal{N}(\boldsymbol{\mathbf{z}};0,\boldsymbol{\mathbf{I}}_{K})$ (Hoffman and Johnson, 2016); this means that the model is less able to generate novel user histories by ancestral sampling.

But ultimately our goal is to make good recommendations, not to maximize likelihood or generate imagined user histories. Treating $\beta$ as a free regularization parameter therefore costs us nothing, and, as we will see, yields significant improvements in performance.

Selecting $\beta$: We propose a simple heuristic for setting $\beta$: we start training with $\beta=0$, and gradually increase $\beta$ to 1. We linearly anneal the KL term slowly over a large number of gradient updates to $\theta,\phi$ and record the best $\beta$ when its performance reaches the peak. We found this method to work well and it does not require the need for training multiple models with different values of $\beta$, which can be time-consuming. Our procedure is inspired by KL annealing (Bowman et al., 2015), a common heuristic used for training vaes when there is concern that the model is being underutilized.

Figure 1 illustrates the basic idea (we observe the same trend consistently across datasets). Here we plot the validation ranking metric without KL annealing (blue solid) and with KL annealing all the way to $\beta=1$ (green dashed, $\beta$ reaches 1 at around 80 epochs). As we can see, the performance is poor without any KL annealing. With annealing, the validation performance first increases as the training proceeds and then drops as $\beta$ gets close to 1 to a value that is only slightly better than doing no annealing at all.

2.3. Computational Burden

Previous collaborative filtering models with neural networks (He et al., 2017; Wu et al., 2016) are trained with stochastic gradient descent where in each step a single (user, item) entry from the click matrix is randomly sampled to perform a gradient update. In Algorithm 1 we subsample users and take their entire click history (complete rows of the click matrix) to update model parameters. This eliminates the necessity of negative sampling (and consequently the hyperparameter tuning for picking the number of negative examples), commonly used in the (user, item) entry subsampling scheme.

A computational challenge that comes with our approach, however, is that when the number of items is huge, computing the multinomial probability $\pi(\boldsymbol{\mathbf{z}}_{u})$ could be computationally expensive, since it requires computing the predictions for all the items for normalization. This is a common challenge for language modeling where the size of the vocabulary is in the order of millions or more (Mikolov et al., 2013). In our experiments on some medium-to-large datasets with less than 50K items (Section 4.1), this has not yet come up as a computational bottleneck. If this becomes a bottleneck when working with larger item sets, one can easily apply the simple and effective method proposed by Botev et al. (2017) to approximate the normalization factor for $\pi(\boldsymbol{\mathbf{z}}_{u})$.

3. A taxonomy of autoencoders

In Section 2.2, we introduced maximum marginal likelihood estimation of vaes using approximate Bayesian inference under a non-linear generative model (Eq. 1). We now describe our work from the perspective of learning autoencoders. Maximum-likelihood estimation in a regular autoencoder takes the following form:

There are two key distinctions of note: (1) The autoencoder (and denoising autoencoder) effectively optimizes the first term in the vae objective (Eq. 5 and Eq. 6) using a delta variational distribution $q_{\phi}(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})=\delta(\boldsymbol{\mathbf{z}}_{u}-g_{\phi}(\boldsymbol{\mathbf{x}}_{u}))$ — it does not regularize $q_{\phi}(\boldsymbol{\mathbf{z}}_{u}\,|\,\boldsymbol{\mathbf{x}}_{u})$ towards any prior distribution as the vae does. (2) the $\delta(\boldsymbol{\mathbf{z}}_{u}-g_{\phi}(\boldsymbol{\mathbf{x}}_{u}))$ is a $\delta$ distribution with mass only at the output of $g_{\phi}(\boldsymbol{\mathbf{x}}_{u})$. Contrast this to the vae, where the learning is done using a variational distribution, i.e., $g_{\phi}(\boldsymbol{\mathbf{x}}_{u})$ outputs the parameters (mean and variance) of a Gaussian distribution. This means that vae has the ability to capture per-data-point variances in the latent state $\boldsymbol{\mathbf{z}}_{u}$.

To provide a unified view of different variants of autoencoders and clarify where our work stands, we depict variants of autoencoders commonly found in the literature in Figure 2. For each one, we specify the model (dotted arrows denote a sampling operation) and describe the training objective used in parameter estimation.

4. Prediction

It is easy to see the advantage of using autoencoders. We can effectively make predictions for users by evaluating two functions – the inference model (encoder) $g_{\phi}(\cdot)$ and the generative model (decoder) $f_{\theta}(\cdot)$. For most of the latent factor collaborative filtering model, e.g., matrix factorization (Hu et al., 2008; Gopalan et al., 2015), when given the click history of a user that is not present in the training data, normally we need to perform some form of optimization to obtain the latent factor for this user. This makes the use of autoencoders particularly attractive in industrial applications, where it is important that predictions be made cheaply and with low latency.

Related work

Vaes on sparse data. Variational autoencoders (vaes) (Kingma and Welling, 2013; Rezende et al., 2014) have seen much application to images since their inception. Doersch (2016) presents a review on different applications of vae to image data. Miao et al. (2016) study vae s on text data. More recent results from Krishnan et al. (2017) find that vaes (trained with Eq. 5) suffer from underfitting when modeling large, sparse, high-dimensional data. We notice similar issues when fitting vae without annealing (Figure 1) or annealing to $\beta=1$. By giving up the ability to perform ancestral sampling in the model, and setting $\beta\leq 1$, the resulting model is no longer a proper generative model though for collaborative filtering tasks we always make predictions conditional on users’ click history.

Information-theoretic connection with vae. The regularization view of the elbo in Eq. 6 resembles maximum-entropy discrimination (Jaakkola et al., 2000). Maximum-entropy discrimination attempts to combine discriminative estimation with Bayesian inference and generative modeling. In our case, in Eq. 6, $\beta$ acts as a knob to balance discriminative and generative aspects of the model.

The procedure in Eq. 6 has information-theoretic connections described in Alemi et al. (2017). The authors propose the deep variational information bottleneck, which is a variational approximation to the information bottleneck principle (Tishby et al., 2000). They show that as a special case they can recover the learning objective used by vaes. They report more robust supervised classification performance with $\beta<1$. This is consistent with our findings as well. Higgins et al. (2017) proposed $\beta$-vae, which leads to the same objective as Eq. 6. They motivate $\beta$-vae for the goal of learning disentangled representations from images (basic visual concepts, such as shape, scale, and color). Their work, however, sets $\beta\gg 1$, effectively imposing a stronger independent prior assumption on the latent code $\boldsymbol{\mathbf{z}}$. While their motivations are quite different from ours, it is interesting to note orthogonal lines of research emerging from exploring the full spectrum of values for $\beta$.

Neural networks for collaborative filtering. Early work on neural-network-based collaborative filtering models focus on explicit feedback data and evaluates on the task of rating predictions (Salakhutdinov et al., 2007; Georgiev and Nakov, 2013; Sedhain et al., 2015; Zheng et al., 2016). The importance of implicit feedback has been gradually recognized, and consequently most recent research, such as this work, has focused on it. The two papers that are most closely related to our approaches are collaborative denoising autoencoder (Wu et al., 2016) and neural collaborative filtering (He et al., 2017).

Collaborative denoising autoencoder (cdae) (Wu et al., 2016) augments the standard denoising autoencoder, described in Section 2.3, by adding a per-user latent factor to the input. The number of parameters of the cdae model grows linearly with both the number of users as well as the number of items, making it more prone to overfitting. In contrast, the number of parameters in the vae grows linearly with the number of items. The cdae also requires additional optimization to obtain the latent factor for unseen users to make predicion. In the paper, the authors investigate the Gaussian and logistic likelihood loss functions — as we show, the multinomial likelihood is significantly more robust for use in recommender systems. Neural collaborative filtering (ncf) (He et al., 2017) explore a model with non-linear interactions between the user and item latent factors rather than the commonly used dot product. The authors demonstrate improvements of ncf over standard baselines on two small datasets. Similar to cdae, the number of parameters of ncf also grows linearly with both the number of users as well as items. We find that this becomes problematic for much larger datasets. We compare with both cdae and ncf in Section 4.

Asymmetric matrix factorization (Paterek, 2007) may also be interpreted as an autoencoder, as elaborated in Steck (2015). We can recover this work by setting both $f_{\theta}(\cdot)$ and $g_{\phi}(\cdot)$ to be linear.

Besides being applied in session-based sequential recommendation (see Section 2.1), various approaches (van den Oord et al., 2013; Liang et al., 2015; Almahairi et al., 2015; Wang et al., 2015) have applied neural networks to incorporate side information into collaborative filtering models to better handle the cold-start problem. These approaches are complementary to ours.

Empirical Study

For the denoising and variational autoencoder, the multinomial likelihood compares favorably over the more common Gaussian and logistic likelihoods.

The source code to reproduce the experimental results is available on GitHubhttps://github.com/dawenl/vae_cf.

We study three medium- to large-scale user-item consumption datasets from various domains:

MovieLens-20M (ML-20M): These are user-movie ratings collected from a movie recommendation service. We binarize the explicit data by keeping ratings of four or higher and interpret them as implicit feedback. We only keep users who have watched at least five movies.

Netflix Prize (Netflix): This is the user-movie ratings data from the Netflix Prizehttp://www.netflixprize.com/. Similar to ML-20M, we binarize explicit data by keeping ratings of four or higher. We only keep users who have watched at least five movies.

Million Song Dataset (MSD): This data contains the user-song play counts released as part of the Million Song Dataset (Bertin-Mahieux et al., 2011). We binarize play counts and interpret them as implicit preference data. We only keep users with at least 20 songs in their listening history and songs that are listened to by at least 200 users.

Table 1 summarizes the dimensions of all the datasets after preprocessing.

2. Metrics

The expression in the denominator is the minimum of $R$ and the number of items clicked on by user u. This normalizes Recall@$R$ to have a maximum of 1, which corresponds to ranking all relevant items in the top $R$ positions.

Truncated discounted cumulative gain (DCG@$R$) is

NDCG@$R$ is the DCG@$R$ linearly normalized to after dividing by the best possible DCG@$R$, where all the held-out items are ranked at the top.

3. Experimental setup

We study the performance of various models under strong generalization (Marlin, 2004): We split all users into training/validation/test sets. We train models using the entire click history of the training users. To evaluate, we take part of the click history from held-out (validation and test) users to learn the necessary user-level representations for the model and then compute metrics by looking at how well the model ranks the rest of the unseen click history from the held-out users.

This is relatively more difficult than weak generalization where the user’s click history can appear during both training and evaluation. We consider it more realistic and robust as well. In the last row of Table 1, we list the number of held-out users (we use the same number of users for validation and test). For each held-out user, we randomly choose 80% of the click history as the “fold-in” set to learn the necessary user-level representation and report metrics on the remaining 20% of the click history.

4. Baselines

We compare results with the following standard state-of-the-art collaborative filtering models, both linear and non-linear:

Weighted matrix factorization (wmf) (Hu et al., 2008): a linear low-rank factorization model. We train wmf with alternating least squares; this generally leads to better performance than with SGD. We set the weights on all the 0’s to 1 and tune the weights on all the 1’s in the click matrix among $\{2,5,10,30,50,100\}$, as well as the latent representation dimension $K\in\{100,200\}$ by evaluating NDCG@100 on validation users.

Neural collaborative filtering (ncf) (He et al., 2017): explores non-linear interactions (via a neural network) between the user and item latent factors. Similar to cdae, the number of parameters for ncf grows linearly with the number of users and items. We use the publicly available source code provided by the authors, yet cannot obtain competitive performance on the datasets used in this paper — the validation metrics drop within the first few epochs over a wide range of regularization parameters. The authors kindly provided the two datasets (ML-1M and Pinterest) used in the original paper, as well as the training/test split, therefore we separately compare with ncf on these two relatively smaller datasets in the empirical study. In particular, we compare with the hybrid NeuCF model which gives the best performance in He et al. (2017), both with and without pre-training.

We also experiment with Bayesian personalized ranking (bpr) (Rendle et al., 2009). However, the performance is not on par with the other baselines above. This is consistent with some other studies with similar baselines (Sedhain et al., 2016). Therefore, we do not include bpr in the following results and analysis.

5. Experimental results and analysis

In this section, we quantitatively compare our proposed methods with various baselines. In addition, we aim to answer the following two questions:

How does multinomial likelihood compare with other commonly used likelihood models for collaborative filtering?

Table 4 summarizes the results of different likelihoods on ML-20M (the results on the other two datasets are similar.) We tune the hyperparameters for each likelihood separately.Surprisingly, partial regularization seems less effective for Gaussian and logistic. The multinomial likelihood performs better than the other likelihoods. The gap between logistic and multinomial likelihood is closer — this is understandable since multinomial likelihood can be approximated by individual binary logistic likelihood, a strategy commonly adopted in language modeling (Mikolov et al., 2013; Xu et al., 2011).

Conclusion

In this paper, we develop a variant of vae for collaborative filtering on implicit feedback data. This enables us to go beyond linear factor models with limited modeling capacity.

We introduce a generative model with a multinomial likelihood function parameterized by neural network. We show that multinomial likelihood is particularly well suited to modeling user-item implicit feedback data.

References