White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Introduction

Ateniese et al. (2015) state that “it is unsafe to release trained classifiers since valuable information about the training set can be extracted from them”. The problem that we address in this paper, i.e., to determine whether a sample has been used to train a given model, is related to the privacy implications of machine learning systems. They were first discussed in the context of support vector machines (Rubinstein et al., 2009; Biggio et al., 2014). The problem of “unintended memorization” (Carlini et al., 2018) appears in most applications of machine learning, such as natural language processing systems (Carlini et al., 2018) or image classification (Yeom et al., 2018).

More specifically, we consider the problem of membership inference, i.e., we aim at determining if a specific image was used to train a model, given only the (image, label) pair and the model parameters. This question is important to protect both the privacy and intellectual property associated with data. For neural networks, the privacy issue was recently considered by Yeom et al. (2018) for the MNIST and CIFAR datasets. The authors evidence the close relationship between overfitting and privacy of training images. This is reminiscent of prior membership inference attacks, which employ the output of the classifier associated with a particular sample to determine whether it was used during training or not (Shokri et al., 2017).

At this stage, it is worth defining the different levels of information to which the “attacker”, i.e., the membership inference system, has access to. We assume that the attacker knows the data distribution and the specifications of the model (training procedure, architecture of the network, etc), even though they are not necessarily required for all methods. We refer to the white-box setting as the case where the attacker knows all the network parameters. On a side note, the setup commonly adopted in differential privacy (Dwork et al., 2006) corresponds to the white-box setting, where the attacker additionally knows all the training samples except the one to be tested.

The black-box setting is when these parameters are unknown. For classification models, the attacker has only access to the output for a given input, in one of the following forms:

(i) the classifier decision; (ii) the loss of the correct label; (iii) the full response for all classes.

Prior works on membership inference commonly assume (i) or (iii). Our paper focuses on the black-box case (ii), in which we know the loss incurred by the correct label. The state of the art in this setting are the shadow models proposed by Shokri et al. (2017).

In our work, we use a probabilistic framework to derive a formal analysis of the optimal attack. This framework encompasses both Bayesian learning, and noisy training, where the noise is injected (Welling & Teh, 2011) or comes from the stochasticity of SGD. Under mild assumptions on the distribution of the parameters, we derive the optimal membership inference strategy. This strategy only depends on the classifier through evaluation of the loss, thereby showing that black-box attacks will perform as well as white-box attacks in this optimal asymptotic setting. This result may explain why, to the best of our knowledge, the literature does not report white-box attacks outperforming the state-of-the-art black-box-(ii) attacks.

The aforementioned optimal strategy is not tractable, therefore we introduce approximations to derive an explicit method for membership inference. As a byproduct of this derivation, we show that state-of-the-art approaches (Shokri et al., 2017; Yeom et al., 2018) are coarser approximations of the optimal strategy. One of the approximation drastically simplifies the membership inference procedure by simply relying on the loss and a calibration term. We employ this strategy to the more complex case of neural networks, and show that it outperforms all approaches we are aware of.

In summary, our main contributions are as follows:

We show, under a few assumptions on training, that the optimal inference only depends on the loss function, and not on the parameters of the classifier. In other terms, white-box attacks don’t provide any additional information and result in the same optimal strategy.

We employ different approximations to derive three explicit membership attack strategies. We show that state-of-the-art methods constitute other approximations. Simple simulations show the superiority of our approach on a simple regression problem.

We apply a simplified, tractable, strategy to infer the membership of images to the train set in the case of the public image classification benchmarks CIFAR and Imagenet. It outperforms the state of the art for membership inference, namely the shadow models.

The paper is organized as follows. Section 2 reviews related work. Section 3 introduces our probabilistic formulation and derives our main theoretical result. This section also discusses the connection between membership inference and differential privacy. Section 4 considers approximations for practical use-cases, which allow us to derive inference strategies in closed-form, some of which are connected with existing methods from the literature. Section 5 summarizes the practical algorithms derived from our analysis. Finally, Section 6 considers the more difficult case of membership inference for real-life neural networks and datasets.

Related work

Our work is related to the topics of overfitting and memorization capabilities of classifiers. Determining what neural networks actually memorize from their training set is not trivial. A few recent works (Zhang et al., 2017; Yeom et al., 2018) evaluate how a network can fit random labels. Zhang et al. (2017) replace true labels by random labels and show that popular neural nets can perfectly fit them in simple cases, such as small datasets (CIFAR10) or Imagenet without data augmentation. Krueger et al. (2017) extend their analysis and argue in particular that the effective capacity of neural nets depends on the dataset considered. In a privacy context, Yeom et al. (2018) exploit this memorizing property to watermark networks. As a side note, random labeling and data augmentation have been used for the purpose of training a network without any annotated data (Dosovitskiy et al., 2014; Bojanowski & Joulin, 2017).

In the context of differential privacy (Dwork et al., 2006), recent works (Wang et al., 2016; Bassily et al., 2016) suggest that guaranteeing privacy requires learning systems to generalize well, i.e., to not overfit. Wang et al. (2015) show that Bayesian posterior sampling offers differential privacy guarantees. Abadi et al. (2016) introduce noisy SGD to learn deep models with differential privacy.

Membership Inference.

A few recent works (Hayes et al., 2017; Shokri et al., 2017; Long et al., 2018) have addressed membership inference. Yeom et al. (2018) propose a series of membership attacks and derive their performance. Long et al. (2018) observe that some training images are more vulnerable than others and propose a strategy to identify them. Hayes et al. (2017) analyze privacy issues arising in generative models. Dwork et al. (2015) and Sankararaman et al. (2009) provide optimal strategies for membership inference in genomics data.

Shadow models.

Shadow models were introduced by Shokri et al. (2017) in the context of black-box attacks. In this setup, an attacker has black-box-(iii) access (full response for all classes) to a model trained on a private dataset, and to a public dataset that follows the same distribution as the private dataset. The attacker wishes to perform membership inference using black-box outputs of the private model. For this, the attacker simulates models by training shadow models on known splits from the public set. On this simulated models, the attacker can analyze the output patterns corresponding to samples from the training set and from a held-out set. Shokri et al. (2017) propose to train an attack model that learns to predict, given an output pattern, whether it corresponds to a training or held-out sample. If the attack model simply predicts “training” when the output activations fire on the correct class, this strategy is equivalent to Yeom et al. (2018)’s adversary. Salem et al. (2019) further show that shadow models work under weaker assumptions than those of Shokri et al. (2017).

Membership inference model

In this section, we derive the Bayes optimal performance for membership inference (Theorem 1). We then make the connection with differential privacy and propose looser guarantees that prevent membership inference.

We assume that the machine learning algorithm has some randomness, and we model it with a posterior distribution over parameters $\theta|z_{1},\dots,z_{n}$. The randomness in $\theta$ either comes from the training procedure (e.g., Bayesian posterior sampling), or arises naturally, as is the case with Stochastic Gradient methods.

In general, we assume that the posterior distribution follows:

2 Membership inference

Given $\theta$ produced by such a machine learning algorithm, membership inference asks the following question: What information does $\theta$ contain about its training set $z_{1},\dots,z_{n}$?

Taking the case of $z_{1}$ without loss of generality, membership inference determines, given parameters $\theta$ and sample $z_{1}$, whether $m_{1}=1$ or $m_{1}=0$.

Inferring the membership of sample $z_{1}$ to the training set amounts to computing:

Notation. We denote by $\sigma$ the sigmoid function $\sigma(u)=(1+e^{-u})^{-1}$. We collect the knowledge about the other samples and their memberships into the set $\mathcal{T}=\{z_{2},...,z_{n},m_{2},...,m_{n}\}$.

3 Optimal membership inference

In Theorem 1, we derive the explicit formula for $\mathcal{M}(\theta,z_{1})$.

Given a parameter $\theta$ and a sample $z_{1}$, the optimal membership inference is given by:

with $t_{\lambda}=\log\left(\frac{\lambda}{1-\lambda}\right)$.

By the law of total expectation, we have:

which gives the expression for $\mathcal{M}(\theta,z_{1})$. ∎

Theorem 2 uses the assumption in Equation (2) to further explicit $\mathcal{M}(\theta,z_{1})$; we give its formal expression below, prove it, and analyze the expression qualitatively. Let us first define the posterior over the parameters given samples $z_{2},\dots,z_{n}$ and memberships $m_{2},\dots,m_{n}$:

Given a parameter $\theta$ and a sample $z_{1}$, the optimal membership inference is given by:

Singling out $m_{1}$ in Equation (2) yields the following expressions for $\alpha$ and $\beta$:

Then, Equation (8) yields the expected result. ∎

4 Differential privacy and guarantees

In this subsection we make the link with differential privacy.

Differential privacy (Dwork et al., 2006) is a framework that allows to learn model parameters $\theta$ while maintaining the confidentiality of data. It ensures that even if a malicious attacker knows parameters $\theta$ and samples $z_{i}$, $i\geq 2$, for which $m_{i}=1$, the privacy of $z_{1}$ is not compromised.

A machine learning algorithm is $\epsilon$-differentially private if, for any choice of $z_{1}$ and $\mathcal{T}$,

Note that this definition is slightly different from the one of Dwork et al. (2006) in that we consider the removal of $z_{1}$ rather than its substitution with $z^{\prime}$. Additionally we consider probability densities instead of probabilities of sets, without loss of generality.

If the training is $\epsilon$-differentially private, then:

Combining Equation (20) and the fact that $\sigma(u)\leq\sigma(v)+\max(u-v,0)/4$ (Appendix A.3), we have:

Combining this expression with Theorem 1 yields the result. ∎

Note that this bound gives a tangible sense of $\epsilon$. In general, decreasing $\epsilon$ increases privacy, but there is no consensus over “good” values of $\epsilon$; this bound indicates for instance that $\epsilon=0.01$ would be sufficient for membership privacy.

$\epsilon$-differential privacy gives strong membership inference guarantees, at the expense of a constrained training procedure resulting generally in a loss of accuracy (Abadi et al., 2016). However, if we assume that the attacker knows the $z_{i},i\geq 2$ for which $m_{i}=1$, $\epsilon$-differential privacy is required to protect the privacy of $z_{1}$. Depending on the information we have on $z_{i},i\geq 2$, there is a continuum between differential privacy (all $z_{i}$’s are known) and membership inference (only prior knowledge on $z_{i}$). In the case of membership inference, it suffices to have the following guarantee:

The training is $(\epsilon,\delta)$-membership private for some $\epsilon>0,\delta>0$ if with probability $1-\delta$ over the choice of $\mathcal{T}$:

If the training is $(\epsilon,\delta)$-membership private, then:

Jensen’s inequality states that for any distribution $p$ and any function $f$:

hence the score $s$ from Equation (15) verifies:

Thus, distinguishing the cases $\delta$ and $1-\delta$ in the expectation in Equation (13),

Membership privacy provides a post-hoc guarantee on $\theta,z_{1}$. Guarantees in the form of Equation (23) can be obtained by PAC (Probably Approximately Correct) bounds.

Approximations for membership inference

Estimating the probability of Equation (15) mainly requires to compute the term $\tau_{p}$. Since its expression is intractable, we use approximations to derive concrete membership attacks (MA). We now detail these approximations, referred to as MAST (MA Sample Threshold), MALT (MA Loss Threshold) and MATT (MA Taylor Threshold).

We first make the mean-field assumption that $p_{\mathcal{T}}(t)$ does not depend on $\mathcal{T}$ (we note it $p$), and define

2 MALT: Constant τ𝜏\tau

In Shokri et al. (2017), we argue that the attack model essentially performs such an estimation, albeit in a non-explicit way. In particular, we believe that the gap between Shokri et al. (2017)’s method and ours is due to instabilities in the estimation of $\tau$ and the numerical computation of the $\log$, as the model is given only $\phi_{\theta}(x)$. As a side note, the expectation term in $\mathcal{T}={z_{2},\dots,z_{n},m_{2},\dots,m_{n}}$ is very similar in spirit to the shadow models, and they can be viewed as a Monte-Carlo estimation of this quantity.

We illustrate the difference between a MALT (global $\tau$) and MAST (per-sample $\tau(\cdot)$) on a simple toy example. Let’s assume we estimate the mean $\mu$ of Gaussian data with unit variance.

We sample $n$ values $z_{1},\dots,z_{n}$ from $\mathcal{D}=\mathcal{N}(\mu,I)$. The estimate of the mean is $\theta=\frac{1}{n^{\prime}}\sum_{i=1}^{n}m_{i}z_{i}$ where $n^{\prime}=|\{i~{}|~{}m_{i}=1\}|$. We have (see Appendix A.2 for derivations):

The expression of $\tau(z_{i})$ shows that the “difficulty” of sample $z_{i}$ is its distance to $\mu$, i.e., how untypical this sample is.

Figure 1 shows the results with a global $\tau$ or a per-sample $\tau$: the per-sample $\tau$ better separates the two distributions, leading to an increased membership inference accuracy.

MATT: Estimation with Taylor expansion

We denote by $\theta_{0}^{*}$ (resp. $\theta_{1}^{*}$) the mode of the Gaussian corresponding to $\{z_{2},...,z_{n}\}$ (resp. {$z_{1},...,z_{n}$}), and $H_{0}$ (resp. $H_{1}$) the corresponding Hessian matrix. We assume that $H$ is not impacted by removing $z_{1}$ from the training set, and thus $H:=H_{0}\approx H_{1}$ (cf. appendix A.4 for a more precise justification). The log-ratio is therefore

The difference $\theta_{1}^{*}-\theta_{0}^{*}$ can be estimated using a Taylor expansion of the loss gradient around $\theta_{0}^{*}$ (see e.g. Koh & Liang (2017)):

Combining this with Equation (37) leads to

We study the asymptotic behavior of this expression when $n\rightarrow\infty$. On the left-hand side, the parameters $\theta$ and $\theta_{0}^{*}$ are estimates of the optimal $\theta^{*}$, and under mild conditions, the error of the estimated parameters is of order $1/\sqrt{n}$. Therefore the difference $\theta-\theta_{0}^{*}$ is of order $1/\sqrt{n}$. On the right-hand side, the matrix $H$ is the summation of $n$ sample-wise Hessian matrices. Therefore, asymptotically, the right-hand side shrinks at a rate $1/n$, which is negligible compared to the other, which shrinks at $1/\sqrt{n}$. In addition to the asymptotic reasoning, we verified this approximation experimentally. Thus, we approximate Equation (39) to give the following score:

Membership inference algorithms

In this section, we detail how the approximations of $s(\theta,z_{1},p)$ are employed to perform membership inference.

We assume that a machine learning model has been trained, yielding parameters $\theta$. We assume also that similar models can be re-trained with different training sets. Given a sample $z_{1}$, we want to decide whether $z_{1}$ belongs to the training set.

We consider as a baseline the “0-1” heuristic, which predicts that $z_{1}$ comes from the training set if the class is predicted correctly, and from the test set if the class is predicted incorrectly. We note $p_{\text{train}}$ (resp. $p_{\text{test}}$) the classification accuracy on the training (resp. held-out) set. The accuracy of the heuristic is (see Appendix A.1 for derivations):

For example when $\lambda=1/2$, since $p_{\text{train}}\geq p_{\text{test}}$ this heuristic is better than random guessing (accuracy $1/2$) and the improvement is proportional to the overfitting gap $p_{\text{train}}-p_{\text{test}}$.

2 Making hard decisions from scores

Variants of our method provide different estimates of $s(\theta,z_{1},p)$. Theorem 2 shows that this score has to be passed through a sigmoid function, but since it is an increasing function, the threshold can be chosen directly on these scores. Estimation of this threshold has to be conducted on simulated sets, for which membership information is known. We observed that there is almost no difference between chosing the threshold on the set to be tested and cross-validating it. This is expected, as a one-dimensional parameter the threshold is not prone to overfitting.

3 Membership algorithms

MAST: Estimating $\tau(z_{1})$. To estimate $\tau(z_{1})$ in Equation (29), we train several models with different subsamples of the training set. This yields a set of per-sample losses for $z_{1}$ that are averaged into an estimate of $\tau(z_{1})$.

MATT: the Taylor approximation. We run the training on a separate set to obtain $\theta_{0}^{*}$. Then we take a gradient step over the loss to estimate the approximation in Equation (40). Note that this strategy is not compatible with neural networks because the assumption that parameters lie around a unique global minimum does not hold. In addition, parameters from two different networks $\theta$ and $\theta_{0}^{*}$ cannot be compared directly as neural networks that express the same function can have very different parameters (e.g. because channels can be permuted arbitrarily).

Experiments

In this section we evaluate the membership inference methods on machine-learning tasks of increasing complexity.

We evaluate three metrics: the accuracy of the attack, and the mean average precision when detecting either from train (mAPtrain) or test (mAPtest) images. For the mean average precision, the scores need not to be thresholded, the metric is invariant to mapping by any increasing function.

2 Logistic regression

CIFAR-10 is a dataset of $32\times 32$ pixel images grouped in 10 classes. In this subsection, we consider two of the classes (truck and boat) and vary the number of training images from $n=400$ to $6,000$.

We train a logistic regression to separate the two classes. The logistic regression takes as input features extracted from a pretrained Resnet18 on CIFAR-100 (a disjoint dataset). The regularization parameter $C$ of the logistic regression is cross-validated on held-out data.

We assume that $\lambda=1/2$ ($n/2$ training images and $n/2$ test images). We also reserve $n/2$ images to estimate $\theta_{0}^{*}$ for the MATT method. In both experiments, we report the peak accuracy obtained for the best threshold (cf. Section 5.2).

Table 1 shows the results of our experiments, in terms of accuracy and mean average precision. In accuracy, the Taylor expansion method MATT outperforms the MALT method, for any number of training instances $n$, which itself obtains much better results than the naive 0-1 attack.

Interestingly, it shows a difference between MALT and MATT: both perform similarly in terms of mAPtest, but MATT slightly outperforms MALT in mAPtrain. The main reason for this difference is that the MALT attack is asymmetric: it is relatively easy to predict that elements come from the test set, as they have a high loss, but elements with a low loss can come either from the train set or the test set.

3 Small convolutional network

In this section we train a small convolutional network 2 convolutional and 2 linear layers with Tanh non-linearity. with the same architecture as Shokri et al. (2017) on the CIFAR-10 dataset, using a training set of $15,000$ images. Our model is trained for $50$ epochs with a learning rate of $0.001$. We assume a balanced prior on membership ($\lambda=1/2$).

We run the MALT and MAST attacks on the classifiers. As stated before, the MATT attack cannot be carried out on convolutional networks. For MAST, the threshold is estimated from 30 shadow models: we train these 30 shadow models on $15,000$ images chosen among the train+held-out set ($30,000$ images). Thus, for each image, we have on average 15 models trained on it and 15 models not trained on it: we estimate the threshold for this image by taking the value $\tau(z)$ that separates the best the two distributions: this corresponds to a non-parametric estimation of $\tau(z)$.

Table 2 shows that our estimations outperform the related works. Note that this setup gives a slight advantage to MAST as the threshold is estimated directly for each sample under investigation, whereas MALT first estimates a threshold, and then applies it to never-seen data. Yet, in contrast with the experiment on Gaussian data, MAST performs only slightly better than MALT. Our interpretation for this is that the images in the training set have a high variability, so it is difficult to obtain a good estimate of $\tau(z_{1})$. Furthermore, our analysis of the estimated thresholds $\tau(z_{1})$ show that they are very concentrated around a central value $\tau$, so their impact when added to the scores is limited.

Therefore, in the following experiment we focus on the MALT attack.

4 Evaluation on Imagenet

We evaluate a real-world dataset and tackle classification with large neural networks on the Imagenet dataset (Deng et al., 2009; Russakovsky et al., 2015), which contains 1.2 million images partitioned into 1000 categories. We divide Imagenet equally into two splits, use one for training and hold out the rest of the data.

We experiment with the popular VGG-16 (Simonyan & Zisserman, 2014) and Resnet-101 (He et al., 2016) architectures. The model is learned in $90$ epochs, with an initial learning rate of $0.01$, divided by $10$ every $30$ epochs. Parameter optimization is conducted with SGD with a momentum of $0.9$, a weight decay of $10^{-4}$, and a batch size of $256$.

We conduct the membership inference test by running the 0-1 attack and MALT. An important factor for the success of the attacks is the amount of data augmentation. To assess the effect of data augmentation, we train different networks with varying data augmentation: None, Flip+Crop$\pm$5, Flip+Crop (by increasing intensity).

Table 3 shows that data augmentation reduces the gap between the training and the held-out accuracy. This decreases the accuracy of the Bayes attack and the MALT attack. As we can see, without data augmentation, it is possible to guess with high accuracy if a given image was used to train a model (about $90$% with our approach, against $77$% for existing approaches). Stronger data augmentation reduces the accuracy of the attacks, that still remain above $64\%$.

Conclusion

This paper has addressed the problem of membership inference by adopting a probabilistic point of view. This led us to derive the optimal inference strategy. This strategy, while not explicit and therefore not applicable in practice, does not depend on the parameters of the classifier if we have access to the loss. Therefore, a main conclusion of this paper is to show that, asymptotically, white-box inference does not provide more information than an optimized black-box setting.

We then proposed two approximations that lead to three concrete strategies. They outperform competitive strategies for a simple logistic problem, by a large margin for our most sophisticated approach (MATT). Our simplest strategy (MALT) is applied to the more complex problem of membership inference from a deep convolutional network on Imagenet, and significantly outperforms the baseline.

References

Appendix A Derivations

We note $g_{1}$ the binary random variable that indicates whether $z_{1}$ was classified correctly, and thus considered part of the training set by the 0-1 attack. The attack is accurate if $g_{1}=1$ on training images and $g_{1}=0$ on other images. This happens with probability

A.2 Gaussian data

We assume without loss of generality that $\mu=0$. $\theta$ is the mean of $n$ Gaussian variables, centered on $\mu$ with covariance $I$. Thus, $\theta$ follows a Gaussian distribution, of variance $\frac{1}{n}I$.

Denoting $\omega:=\frac{z}{n+1}$, we have

A.3 Bound on variations of a sigmoid

Since $\sigma$ is increasing, the relation is obvious for $v>u$.

Thus, $\sigma$ is Lipschitz-continuous with constant $1/4$, which entails Equation (47).

A.4 Hessian approximations

We give here a rough justification of the approximation conducted in the MATT paragraph of Section 5.

This approximation holds up to the following quantity:

We reason qualitatively in orders of magnitude. $\theta_{0}^{*}-\theta_{1}^{*}$ has order of magnitude $1/n$, and $H_{1}-H_{0}$ has order of magnitude $1$, so $\delta_{2}$ has order of magnitude $1/n^{2}$. As for $\delta_{1}$, we observe that $H_{0}^{-1}(H_{1}-H_{0})$ has order of magnitude $1/n$ and therefore

Hence, $\delta_{1}$ has order of magnitude $1/n$ as well. Since the main term in Equation (37) is in the order of $1/\sqrt{n}$, $\delta_{1}$ and $\delta_{2}$ can be safely neglected.