Reconstructing Training Data with Informed Adversaries

I Introduction

Machine learning (ML) models have the capacity to memorize their training data , and such memorization is sometimes unavoidable while training highly accurate models . When the training data is sensitive, sharing models that exhibit memorization can lead to privacy breaches. To design mitigations enabling privacy-preserving deployment of ML models we must understand how these breaches arise and how much information they leak about individual data points.

Membership leakage is considered the gold standard for privacy in ML, both from the point of view of empirical privacy evaluation (e.g., via membership inference attacks (MIA) ) as well as mitigation (e.g., differential privacy (DP) ). Membership information represents a minimal level of leakage: it allows an adversary to infer a single bit determining if a given data record was present in the training dataset. Models trained on health data represent a prototypical application where membership can be considered sensitive: the presence of an individual’s record in a dataset might itself be indicative of whether they were tested or treated for a medical condition.

Reconstruction of training data from ML models sits at the other extreme of the individual privacy leakage spectrum: a successful attack enables an adversary to reconstruct all the information about an individual record that a model might have seen during training. The possibility of extracting training data from models can pose a serious privacy risk even in applications where membership information is not directly sensitive. For example, reconstruction of individual images from a model trained on pictures that were privately shared in a social network can be undesirable even if that individual’s membership in the social network is public information.

Existing evidence of the feasibility of reconstruction attacks is sparse and focuses on specialized use cases. For example, recent work on generative language models highlights their capacity to memorize and regurgitate some of their training data , while works on gradient inversion show that adversaries with access to model gradients (e.g. in federated learning (FL) ) can use this information to reconstruct training examples . Similarly, attribute inference attacks reconstruct a restricted subset of attributes of a training data point given the rest of its attributes , while property inference attacks infer global information about the training distribution rather than individual points .

Our work proposes a general approach to study the feasibility of reconstruction attacks against ML models without assumptions on the type of model or access to intermediate gradients, and initiates a study of mitigation strategies capable of preventing this kind of attacks. The starting point is the instantiation of an informed adversary that, knowing all the records in a training data set except one, attempts to reconstruct the unknown record after obtaining white-box access to a released model. This choice of adversary is inspired by the (implicit) threat model in DP .

Working with such a powerful, albeit unrealistic, adversary enables us to demonstrate the feasibility of reconstruction, both in theory against convex models as well as experimentally against standard neural network architectures for image classification. Furthermore, the use of an informed adversary makes our work relevant for provable mitigations: effective defenses against optimal informed adversaries will also protect against attacks run by less powerful and more realistic adversaries.

We start by introducing and motivating the informed adversary threat model (Section II). Our first contribution is a theoretical analysis of reconstruction attacks against simple ML models like linear, logistic, and ridge regression (Section III). We show that for a broad class of generalized convex linear models, access to the maximum likelihood solution enables an informed adversary to recover the target point exactly.

In the convex setting, the attack relies on solving a simple system of equations. Extending reconstruction attacks to neural networks requires a different approach due to the inherent non-convexity of the learning problem. In Section IV, we propose a generic approach to reconstruction attacks based on reconstructor networks (RecoNN): networks that are trained by the adversary to output a reconstruction of the target point when given as input the parameters of a released model.

Our second contribution is to show that it is feasible to attack standard neural network classifiers using reconstructor networks; we present effective RecoNN architectures and training procedures, and show they can extract high-fidelity training images from classifiers trained on MNISTA minimal implementation of our reconstruction attack on MNIST is available at https://github.com/deepmind/informed_adversary_mnist_reconstruction. and CIFAR-10. Figure 1 provides an illustration of reconstructions produced by a RecoNN-based attack against a convolutional neural network (CNN) classifier trained on CIFAR-10. These experiments provide compelling evidence that image classification models can store in their weights enough information to reconstruct individual training data points.

Section VI describes our third contribution: an in-depth analysis around what factors affect the success of our RecoNN-based attack. These include hyper-parameter settings in the model training pipeline, degree of access to model parameters, and quality and quantity of side knowledge available to the adversary. We also explore how different levels of knowledge about the internal randomness of stochastic gradient descent (SGD) affect reconstruction; we observe that knowing the model’s initialization significantly improves the quality of reconstructions, while knowing the randomness used for mini-batch sampling is not necessary for good reconstruction.

As part of our experiments, we also investigate the use of DP-SGD as a mitigation to protect against reconstruction attacks. We find that large values of $\epsilon$ suffice to defend against our best RecoNN-based attacks – in fact, values that are much larger than what is necessary to protect against membership inference attacks by informed adversaries . Section VII supports this observation by introducing a definition of reconstruction robustness, analyzing its relation to the (Rényi) DP parameters of the training algorithm, and showing that, under mild conditions on the adversary’s side knowledge, $\epsilon=o(d)$ suffices to prevent reconstruction of $d$-dimensional data records.

II Reconstruction with Informed Adversaries

We start by instantiating and justifying the informed adversary threat model for reconstruction attacks against ML models, and by comparing it to related attacks in the literature. Notation for the most important concepts introduced in this section is summarized in Table I. At its core, our threat model assumes a powerful adversary with white-box access to a model released by a model developer. The developer owns a dataset $D\in\mathcal{Z}^{n}$ of $n$ training records from some domain $\mathcal{Z}$, and a (possibly randomized) training algorithm $A:\mathcal{Z}^{n}\to\Theta$. They train (the parameters of) a model $\theta=A(D)$, and then release it as part of a system or service. For example, records in $D$ may be feature-label pairs in standard supervised learning settings, and $A$ may implement an optimization algorithm (e.g. SGD or Adam) for a loss function associated with $D$ and $\Theta$.

A reconstruction adversary with access to the released model aims to infer enough information about its training data to reconstruct one of the examples in $D$. In this paper, we consider a powerful adversary who already has full knowledge about all but one of the training points. Formally, they have access to the following information to carry out the attack.

Let $\theta$ be a model trained on dataset $D$ of size $n$ using algorithm $A$. Let $z\in D$ be an arbitrary training data point and $D_{\mathchar 45\relax}=D\setminus\{z\}$ denote the remaining $n-1$ points; we refer to $z$ as the target point. An informed reconstruction adversary has access to:

The fixed dataset $D_{\mathchar 45\relax}$;

The released model’s parameters $\theta$;

(Optional) Side knowledge $\mathsf{aux}$ about the target point.

We first discuss each piece of knowledge we give to our attacker, and then analyze in depth how our adversary relates to other threat models arising in other privacy attacks.

Arguably, the assumption that gives our attacker the greatest advantage is knowing all the training data except for the target point. There are two main reasons to consider such a stringent threat model. First, since our ultimate goal for studying ML vulnerabilities is to design effective mitigations, by evaluating the resilience of ML models in this strong threat model we ensure their resilience against weaker (and more realistic) attackers. Second, our setup captures the implicit threat model used in the DP definition; indeed, DP bounds the ability of a mechanism at preventing the disclosure of membership information about one data record from an adversary who knows all the other records in the database.

White-box model access

White-box access to the model is motivated by several real-world scenarios. First, the practice of publishing models online (e.g. to facilitate their use or favor public scrutiny) is increasingly widespread. Second, proprietary models shipped as part of hardware or software components can be vulnerable to reverse-engineering; it would be naive to assume that sufficiently motivated adversaries will never obtain white-box access to such models. Finally, FL settings may give real-world attackers access to similar information to the one we capture in our threat model.

Training algorithm

Privacy (and security) through obscurity is generally regarded as a bad practice. Thus, we assume the adversary has access to the model developer’s training algorithm $A$, including any associated hyper-parameters (e.g. learning rate, regularization, batch size, number of iterations, etc). Access to $A$ can be in the form of a concrete (e.g. open source) implementation. Nevertheless, black-box access (e.g. through a SaaS API) suffices for the general reconstruction attack presented in Section IV. In cases where $A$ is randomized, we will evaluate attacks with and without knowledge of the different sources of randomness used when training the released model. In stochastic optimization algorithms these typically include model initialization and mini-batch sampling. Knowledge of $A$’s internal randomness could come from the model developer using a hard-coded random seed in a public implementation. Alternatively, knowledge about the model’s initialization will also be available whenever the released model is obtained by fine-tuning a publicly available model (e.g. in transfer learning scenarios), or in FL settings where the adversary has successfully compromised an intermediate model by taking part in the training protocol.

Side knowledge about target point

Privacy attacks do not happen in a vacuum, so adversaries will often have some prior information about the target point before observing the released model. For starters, knowledge of $D_{\mathchar 45\relax}$ and $A$ provides the adversary with syntactic and semantic context for a learning task in which the model developer deemed it useful to include the target point. In our investigations, we often consider adversaries with additional side knowledge abstractly represented by $\mathsf{aux}$. From a practical perspective, the attack presented in Section IV takes $\mathsf{aux}$ to be a dataset $\bar{D}$ of points disjoint from $D_{\mathchar 45\relax}$. For example, these could come from a public academic dataset or from scraping relevant websites. Our experiments in Section VI-B show that these additional points do not necessarily need to come from the same distribution as the training data. In our theoretical investigation (Section VII), we model the adversary’s side knowledge as a probabilistic prior $\pi$ from which the target is assumed to be sampled.

II-B Reconstruction Attack Protocol and Error Metric

II-C Relation to Attribute Inference

Reconstruction can be seen as a generalization of attribute inference attacks (AIA) , also sometimes referred to as model inversion attacks. In AIA, an attacker that knows part of a data record $z$ aims to reconstruct the entire record by exploiting (white-box or black-box) access to a model $\theta$ whose training dataset contained $z$. It is also common for the attack goal of a model inversion attack to try and reveal training data information in aggregate, possibly isolated to a specific target label. Although no individual training records are reconstructed through this attack, privacy can be leaked if aggregated training information with respect to a target label is sensitive (e.g. facial recognition where each label is associated with an identity). The standard threat model in AIA does not include an informed adversary, but we can get a more direct comparison with our model by considering an informed AIA adversary. Such an adversary is identical to Definition 1 but also receives as input partial information about the target point $z$, which we denote by $\eta(z)$. This can be incorporated in Definition 1 via the side knowledge $\mathsf{aux}$, showing that informed AIA corresponds to reconstruction in our model with a particular type of side knowledge. We conclude that any investigation into mitigating general reconstruction attacks in our threat model will also be useful in protecting against informed AIA, and, by extension, standard AIA.

II-D Relation to Membership Inference

In membership inference attacks (MIA) , an attacker with access to a released model $\theta$ and a challenge example $z\in\mathcal{Z}$ guesses if $z$ was part of the model’s training data. Like in AIA, standard MIA does not assume an informed adversary. Introducing an informed MIA adversary yields a model matching the adversary in the threat model behind DP . This adversary is identical Definition 1, with the exception that it also receives two candidates $z_{0},z_{1}\in\mathcal{Z}$ for the additional data point that was used for training the model, and the developer decides which one to use uniformly at random. The corresponding interaction protocol between model developer and adversary is summarized in Algorithm 2, where the adversary uses a MIA algorithm $M$ and the result provides a bit representing whether it guessed correctly.

The contrapositive implication of the above is that if this powerful notion of MIA is not possible, then accurate reconstruction is also not possible. Furthermore, the existence of a standard MIA attacker implies the existence of an informed one. This argument indicates that protecting against informed MIA will protect against both standard MIA and accurate reconstruction, thus motivating the use of DP – a mitigation against informed MIA – as a strong privacy protection. The experiments in Section VI and the theoretical investigation developed in Section VII will, however, illustrate that values of the DP parameter $\epsilon$ that are too large to protect against informed MIA can still protect against accurate reconstruction.

II-E Further Related Work

Attacks for reconstructing training data have been studied in the context of generative language models (LM). Carlini et al. proposed a targeted black-box reconstruction attack where the adversary knows part of a training example (i.e. a text prompt) and infers the rest (e.g. a credit card number). Their attack assumes partial knowledge of the target record (as with AIA) and a threat model where the adversary has significant computational power but no additional knowledge of the training data. An untargeted version of this attack was later performed against GPT-2 by repeatedly sampling from the model and comparing the samples with the training data . Both works crucially exploit the generative aspect of LMs to carry out reconstruction; our attacks are more general and require no such assumptions, making them suitable to attack standard image classification models.

Many works have investigated what an attacker can infer from inspecting the intermediate gradients in FL settings or multiple model snapshots during training . These attacks focus on inferring training points, their labels, or related properties. The task our reconstruction adversary has to solve is harder: whilst a gradient leakage adversary has access to information involving only a mini-batch of training points, our attacks needs to invert the entire training procedure.

Finally, property inference attacks (PIA) are a generalization of AIA where the adversary infers properties about the training set . These attacks are effective at recovering overall statistics (e.g. the percentage of training records coming from a minority group, the average value of a feature across the data) but in general do not compromise the privacy of individuals.

III Reconstruction in Convex Settings

In this section, we focus on attacking convex supervised learning models. We discuss a general reconstruction attack strategy against a broad family of convex models when the empirical risk minimization (ERM) problem has a unique minimum and is solved to optimality. Specifically, we show there exists a closed form solution to perform reconstruction attacks against Generalized Linear Models (GLMs) without any additional side knowledge about the target point. This attack applies to popular models such as linear regression, ridge regression, and logistic regression.

This simplified scenario enables a direct strategy to perform a reconstruction attack. Recall the adversary has white-box access to the released model $\theta$ and knowledge of the fixed dataset $D_{\mathchar 45\relax}$. This allows them to write the following system of equations which will be satisfied by the target point $z$:

III-B Closed-Form Reconstruction Against GLMs

Consider fitting a GLM derived from a canonical exponential family with canonical link function $g$ (see, e.g. ). The GLM parameters are trained via (regularized) ERM by minimizing the maximum likelihood objective $C(\hat{\theta})=-\sum_{(x,y)\in D}\left(b(\langle x,\hat{\theta}\rangle)-\langle x,\hat{\theta}\rangle y\right)+\lambda\|\hat{\theta}\|^{2}$, where $b$ is a function satisfying $b^{\prime}=g^{-1}$, and $\lambda\geq 0$ is a regularization parameter. For example, $g^{-1}$ is the identify function for linear regression and the sigmoid function for logistic regression. This optimization admits a unique minimum when either $\lambda>0$, $b$ is strictly concave (as in the examples above) or the data is in general position . In any of these cases (1) connects the unknown $z=(x,y)$ with $\theta$ and $D_{\mathchar 45\relax}$. Assuming the model is trained with an intercept parameterIn appendix, we show an attack against linear regression without intercept parameter (Theorem 8), which although assumes the adversary knows $y$. (i.e. the first coordinate of each feature vector is equal to $1$) this results in a system of $d$ equations with $d$ unknowns. The following solution for this system gives an effective reconstruction attack.

We defer all proofs to the appendix. Two important takeaways from this result are: 1) an informed adversary needs no additional side knowledge about $z$ to effectively attack a GLM trained with intercept; and, 2) whether the model overfits the data or generalizes well plays no role in the attack’s success.

IV A General Reconstruction Attack

We describe a reconstruction attack against general ML models. Intuitively, our attack stems from the observation that the influence of the target point $z$ on the released model $\theta$ is similar to the influence an alternative point $\bar{z}$ would have on the model $\bar{\theta}=A(D_{\mathchar 45\relax}\cup\{\bar{z}\})$. By repeatedly training models on different points, our attack collects enough information about the mapping from training points to model parameters to invert it at the model of interest $\theta$. We give a high-level introduction to our attack strategy using reconstructor networks (RecoNN).

Let us use the shorthand notation $A_{D_{\mathchar 45\relax}}:\mathcal{Z}\to\Theta$ with $A_{D_{\mathchar 45\relax}}(z)=A(D_{\mathchar 45\relax}\cup\{z\})$ to emphasize that, from the point of view of an informed adversary, when $D_{\mathchar 45\relax}$ is fixed $A$ effectively becomes a mapping from target points to model parameters. An ideal reconstruction attack would invert the training procedure and output $\hat{z}=A_{D_{\mathchar 45\relax}}^{-1}(\theta)$; whenever $A$ is easy to invert, this will produce a perfect reconstruction as in the setting analyzed in Section III. In general, however, the training process is not (easily) invertible, due to the non-convexity of the optimization problem solved by $A$, or to the presence of randomness in the training process. In such settings, our general reconstruction attack relies on approximately solving this inverse problem by producing a function $\phi:\Theta\rightarrow\mathcal{Z}$ that associates model weights to a guess for the target point in a similar way to the (ideal) inverse mapping $A_{D_{\mathchar 45\relax}}^{-1}$. Note that the adversary in this threat model is extremely powerful; for example, they could enumerate (a fine discretization of) $\mathcal{Z}$ and pick the candidate $\hat{z}$ that produces the model $\hat{\theta}=A_{D_{\mathchar 45\relax}}(\hat{z})$ closest to $\theta$. However, for high-dimensional data this enumerative approach is infeasible, so we focus on attacks that can be executed in practice.

In this paper, we instantiate the search for $\phi$ as a learning problem, effectively using “neural networks to attack neural networks”. To solve this learning problem, we first design a RecoNN architecture for neural networks whose inputs lie in the parameter space $\Theta$ of the released model and outputs lie in the domain $\mathcal{Z}$ of the training data; typically we can encode both using numerical vectors. The adversary then uses its knowledge of $D_{\mathchar 45\relax}$ and $A$, together with side knowledge in the form of shadow target points $\bar{D}$ disjoint from $D_{\mathchar 45\relax}$, to generate a collection of shadow models. These shadow model and target pairs comprise the training data for the RecoNN, which is then applied to the released model to obtain a candidate reconstruction $\hat{z}$ for the (previously unseen) target point $z$.

IV-B Training Reconstructor Networks

Consider an informed adversary in our threat model (Definition 1). As side knowledge about $z$, we assume the attacker has $k$ additional shadow targets $\bar{D}=\{\bar{z}_{1},\ldots,\bar{z}_{k}\}$ from $\mathcal{Z}$. Ideally, if we think that the attack’s success will depend on the RecoNN’s ability to exhibit statistical generalization, these points would be sampled from the same distribution as the target point $z$. Nonetheless, we will see in our experimental evaluation that this requirement is not strictly necessary to achieve good reconstructions (Section VI-B). The general reconstruction attack proceeds as follows (see also Figure 2):

For $i=1,\ldots,k$, train model $\bar{\theta}_{i}=A_{D_{\mathchar 45\relax}}(\bar{z}_{i})$ on the fixed dataset plus the $i$th shadow target from the adversary’s side knowledge pool $\bar{D}$. Together, we refer to the collection of shadow model-target pairs $S=\{(\bar{\theta}_{i},\bar{z}_{i})\}_{i=1}^{k}$ as the attack training data.

Train a RecoNN $\phi$ using $S$ as examples of successful reconstrutions. Abusing our notation, we use $R$ to denote the training algorithm used by the adversary: $\phi=R(S)$.

Obtain a reconstruction candidate by applying the RecoNN to the target model: $\hat{z}=\phi(\theta)$.

The idea of using “neural networks to attack neural networks” has been used in the literature to implement a number of attacks, including (black-box and white-box) membership inference , model inversion , and property inference . Our use of RecoNNs is related to , where an invariant representation of a released neural network parameters is fed into another neural network to perform a PIA, although the output of our attack is often a high-dimensional object (e.g. an image) instead of single scalar. In preliminary experiments we did not see an improvement from using this invariant representation as a pre-processing step; standard normalization was sufficient for a successful attack. Similarly, the use of shadow models trained by the adversary to imitate the behavior of the released model is a common approach in MIA and AIA, although most works do not consider an informed adversary with knowledge of $D_{\mathchar 45\relax}$. Despite the attack being an instantiation of the shadow model technique, it is not a foregone conclusion that this approach will work for reconstruction attacks. Reconstruction is a more difficult task than membership inference, and it entails a considerable amount of engineering, data curation, and ML training insight to carry out, as we will discuss.

V Experimental Setup

We discuss the default experimental settings, and how we will evaluate reconstruction attacks.

We evaluate our reconstruction attacks on the MNIST and CIFAR-10 datasets using fully connected (i.e. multi-layer perceptron) and convolutional neural networks (CNN) as the released (and shadow) models. Our experiments investigate the influence that training hyperparameters for $A$ have on the effectiveness of reconstruction. Default model architectures and hyperparameters for both released and reconstructor models are summarized in Table VI. Most of these choices are standard and were selected based on preliminary experiments. In the following we highlight the most important details.

We split each dataset into three disjoint parts: fixed dataset ($D_{\mathchar 45\relax})$, shadow dataset ($\bar{D}$), and test targets dataset; the latter contains $1K$ points, both for MNIST and CIFAR-10. We train one released model per test target and report average performance of our attack across test targets.

Released model training

The training algorithm for released and shadow models is standard gradient descent with momentum. By default, we use full batches (i.e. no mini-batch sampling) to keep the algorithm deterministic. Additionally, by default we assume the adversary knows the model initialization step, so both released and shadow models are trained from the same starting point. We explore the effect of mini-batching and random initialization separately in Section VI-B.

The architecture is an MLP for MNIST and a CNN for CIFAR-10. On average, the released models achieve over 94% accuracy on MNIST and 40% on CIFAR-10 without significant overfitting (generalization gap is less 1% on MNIST and 5% on CIFAR-10). The reason for the subpar performance on CIFAR-10 is partiallyTraining without random mini-batches, no regularization and a small CNN architecture also contribute to this effect. because the models are trained with only 10% of the data used in standard evaluations – this constraint comes from the need to reserve a large disjoint set of shadow points to train RecoNN. We experiment with a larger CIFAR-10 fixed set size ($50K$) in Section VI-B; in this setting the released models achieve $\sim 50\%$ test accuracy.

We expect reconstructing CIFAR-10 targets will be a more challenging task than MNIST. CIFAR-10 images have a richer, more complex structure, and so capturing and reconstructing the intricacies of such an image may be difficult. Additionally, the underlying released model is larger; hence: 1) a larger reconstructor network is required, which comes with higher computational costs for the adversary; 2) the shadow dataset may need to be larger, to facilitate learning on high dimensional data (i.e. on the shadow models’ weights).

Reconstructor network training

When training the reconstructor, shadow model parameters across layers are flattened and concatenated together. We also re-scale each coordinate in this representation to zero mean and unit variance; we found this pre-processing step to be important, as some of the parameters can be extremely small. For MNIST, we use a mean absolute error (MAE) + mean squared error (MSE) loss between shadow targets and reconstructor outputs as the training objective. For CIFAR-10 we modify the reconstructor training objective by adding an LPIPS loss and a GAN-like Discriminator loss to improve visual quality of reconstructed images. We use a patch-based Discriminator with the architecture given in Table IX, and train it using mean squared error loss and a learning rate of $10^{-5}$. The patch-based discriminator aims to distinguish shadow targets from reconstructor generated candidates. At a high-level, we can view the reconstructor network as a generative model with a latent space defined over a distribution of shadow models; this enables us to apply ideas from Generative Adversarial Networks (GANs) training. Our discriminator training set-up is as in – we alternate between one gradient descent step on the discriminator, and one step on the reconstructor network. From visual inspection, we found using a discriminator improves sharpness of CIFAR-10 reconstructed images, even if it does not strictly improve the MSE metric.

V-B Criteria for Attack Success

We report the MSE between a target and its reconstruction. In the context of images, while discovery of private information does not necessarily perfectly coincide with a decreasing MSE between the original and reconstructed training point, in general the two are correlated (Section VI-A).

LPIPS

We report the LPIPS metric as it has been shown to be closer to the human’s visual systems determination of image similarity in comparison to the MSE distance. LPIPS is measured by comparing deep feature representations from visual models trained with similarity judgements made by human annotators.

KL

After running the attack, a real-world adversary may need to post-process the reconstructed image; e.g. if they wanted to extract a license plate from the reconstructed image, they may need to run a downstream image classifier. We therefore include a similarity metric between the outputs of a highly accurate classifier on the target and reconstructed image based on the Kullback–Leibler (KL) divergence between predicted class probabilities. For MNIST, we use a LeNet classifier achieving 99.4% test accuracy, and for CIFAR-10 use a Wide ResNet achieving 94.7% test accuracy.

Nearest Neighbor Oracle

To contextualize MSE reconstruction metrics we consider an oracle that exploits all the data available to the adversary in the default setting and guesses the point $\hat{z}\in D_{\mathchar 45\relax}\cup\bar{D}$ that has the smallest MSE distance to $z$. The MSE distance between $z$ and its nearest neighbor $\hat{z}$ serves as a conservative threshold for successful reconstruction: although faithful reconstructions with larger MSE are certainly possible, falling below the threshold means the reconstruction is closer to the target than to any other point previously available to the adversary, so the attack must have extracted unique information about the target point from the released model. Figure 3 provides average histograms (over 1K test targets) of MSEs between a target point and all points in $D_{\mathchar 45\relax}\cup\bar{D}$. The green line corresponds to the average MSE to the nearest neighbor across all test targets (0.0232 on MNIST and 0.0291 on CIFAR-10); if reconstructions have a smaller MSE than this distance we will judge the target to have been successfully reconstructed. For reference, we also highlight the 1st, 10th and 50th percentile MSEs, which will be helpful to contextualize experiments throughout Section VI.

VI Empirical Studies in Reconstruction

We now conduct extensive experiments investigating how the released model architecture and its training hyperparameters impact reconstruction quality. We first demonstrate the feasibility of the reconstruction attack against models trained on our default experimental setup. Then we discuss an in-depth study on which factors, such as training set size or released model’s hyperparameters, affect the success of reconstruction. Finally, we investigate DP as a mitigation against reconstruction attacks. Our findings are summarized in Table II.

We first carry out the general reconstruction attack under the default experimental settings (cf. Section V).

Figure 4 shows examples of targets and respective reconstructions; we use the nearest neighbor (NN) oracle as a baseline. We observe a good overall reconstruction quality on both datasets. Running the attack against 1K test targets, we observe an average reconstruction MSE of $0.0089$ (MNIST) and $0.0049$ (CIFAR-10). These numbers, compared to the NN oracle baselines, demonstrate our attack is effective. To account for the variance across experimental runs (e.g. different random selections of fixed sets across experiments), we repeated this experimental procedure ten times with differing fixed sets, initial released model parameters, and evaluation sets. We saw minimal variance in results; importantly, reconstructions were consistently better than the NN oracle.

To help the reader calibrate MSE values to reconstruction quality, Figure 4 shows poor reconstructions with MSE close to the oracle NN’s MSE (third column) and to its 1st percentile (fourth column); these reconstructions were obtained in preliminary experiments with weaker RecoNN instances. More examples are in Figure 24.

With the same experimental setup as above, we also evaluate results across our other metrics (Section V-B) on MNIST. We observe that MSE and LPIPS are strongly correlated (Figure 5(a)). Figure 5(b) also shows that a small MSE implies a small KL but the converse is not true; in other words, it is possible for two images that are not identical to have similar predictions. Since these metrics exhibit significant correlations, we only report a subset of them in subsequent experiments, focusing mostly on comparing the MSE metric with the NN oracle. We observe similar trends on CIFAR-10, although MSE vs LPIPS correlation is weaker; this partially motivated including the LPIPS loss when training RecoNNs on CIFAR-10 (c.f. Section -I).

VI-B What Factors Affect Reconstruction

We study which factors may improve or impact reconstruction success; these are summarized in Table II.

Recall that the general reconstruction attack assumes the attacker has access to $k$ shadow data points $\bar{D}$ from the same distribution as the target point. From this knowledge, the attacker generates a collection of shadow model-target pairs (the attack training data), which is used to train the RecoNN. Note that the size of the attack training data depends both on the knowledge of the attacker (simply, the attacker may not have access to many examples), and on their computational power: they need to train one shadow model per data point to create the attack training data.

We explore the fidelity of reconstruction on MNIST as the amount of attack training data $k$ ranges from $100$ to $59K$. Figure 5(c) shows the average MSE between reconstructions over the $1K$ released model targets as $k$ varies. Clearly, the attack becomes better as more training data is available. However, high fidelity reconstructions occur already with $1K$ shadow models; in our plots we include reconstructed examples at different values of $k$ illustrating this. Reconstructions that are (on average) better than the NN oracle only require $8K$ shadow models. Because the correlation between MSE and KL is not symmetric, we also plot the average KL against attack training set size and observe a similar monotonic decrease (Figure 5(d)). We observe similar trends on CIFAR-10 when increasing the attack training set size; $5K$ shadow models is enough to generate reconstructions below the 1st percentile oracle MSE (~0.05) and $10K$ shadow models will generate reconstructions below the NN oracle MSE (~0.03). See Section -B for full results on CIFAR-10.

Out-of-distribution (OOD) data on CIFAR-10

The previous experiment indicates that reconstructions are poor when an adversary has relatively little side-information ($<1K$ points) to create shadow models. We now investigate if these additional points must come from the same distribution as the fixed set and target sample. If the attack succeeds even when $\bar{D}$ comes from a different distribution, they can potentially create a larger pool of shadow targets for the attack. In addition, when reasonable OOD data is scarce or not available, the attacker could instead use $D_{\mathchar 45\relax}$ to train a generative model and use it to generate shadow targets from a similar distribution.

To relax the assumption that shadow targets come from the same distribution as the released model’s training data we use CIFAR-100, a standard OOD benchmark for CIFAR-10 , to construct the adversary’s side knowledge.

Figure 6 shows the difference between creating an attack training set with a fixed dataset $D_{\mathchar 45\relax}$ sampled from in-distribution data (CIFAR-10) and the additional shadow target points sampled from either in-distribution data (CIFAR-10) or out-of-distribution data (CIFAR-100). We measure attack success on the 1$K$ released models with in-distribution targets (CIFAR-10). We observe a negligible difference between the two, and so conclude the success of the attack is not predicated on access to the correct prior distribution. We further exploit OOD data in the next part, when we evaluate how the size of the fixed set affects reconstruction.

Influence of training hyper-parameters

Table II summarizes what factors in training affect reconstruction. The appendix expands upon these and gives empirical insights.

Fixed set size. We measure the role of the fixed set size by reducing from $10K$ to $1K$ (MNIST) and increasing from $5K$ to $50K$ (CIFAR-10). We observe almost no difference in MSE in both cases; e.g. CIFAR-10 target points can be reconstructed even if there are $50K$ other points in the training set.

Model size and architecture. We assess whether the size and architecture of the released model affect reconstruction. For MNIST, we increase the size of the hidden layer from $10$ to $100$; this increases the number of trainable parameters tenfold. For CIFAR-10, we double the size from $50K$ to $100K$ by increasing the width of the first linear layer. The rest of the architecture is kept to the defaults (Table VII). We observe almost no difference in reconstruction success when attacking these larger released models. Nevertheless, this attack has a bigger computational cost: the size of the RecoNN for CIFAR-10 increases from $226M$ to over $400M$ parameters.

Layers. Instead of allowing the RecoNN to process all parameters from a released model, we restrict to only the second layer for MNIST and convolutional layers for CIFAR-10. This significantly reduces the input size to the reconstructor network, by $98\%$ on MNIST and $84\%$ on CIFAR-10. We observe that this does not substantially affect the reconstruction fidelity, demonstrating that memorization of training points is not localized to a specific layer or small group of neurons.

Epochs. The number of epochs has a small impact on reconstruction. For both MNIST and CIFAR-10 there is a slight increase in MSE if we more than double the number of training epochs, although targets are still successfully reconstructed. We investigate this relationship in more detail in Section -J and Section -L.

Activation. One may wonder why we used ELU activations in the released model instead of the more common ReLUs. We noticed that released models with ReLU activations tend to be harder to attack in comparison to other activation functions, resulting in poor quality reconstructions on CIFAR-10 (i.e. MSE larger than the NN oracle). It is well known that ReLUs induce sparse gradients; we observed that $>60\%$ of weights are not updated during training when the loss is computed with respect to the target. We suspect this is why RecoNN is less effective against ReLU activated models: there is less mutual information between the model parameters and the target in comparison to models trained with other activations. We discuss this in further detail in Section -M.

Learning rate. Decreasing the learning rate of the released model did not affect the attack in the deterministic training setting. If randomness is introduced via mini-batch sampling, we will see that the learning rate impacts reconstruction. In Section -H, we better investigate the role of learning rate; we find that a larger rate can harm the success of the attack in settings where the released is trained with mini-batches.

Randomness from data sub-sampling

We explore how randomness stemming from data sub-sampling affects the attack on MNIST, by removing the assumption that the released model is trained with full batch gradient descent. We consider settings where the adversary knows the random seed used to shuffle the data (this corresponds to SGD but with no randomness), and settings where the adversary does not know the random seed. Results in Figure 7 indicate that when the adversary knows the data shuffling seed, reconstruction attacks are successful even for small batch sizes. Without knowing the seed, attack success depends on the training hyper-parameters, such as the choice of the learning rate. It appears that attacking models with randomness from sub-sampling is more difficult than determinstically trained released models, and that larger learning rates also increase the hardness of the reconstruction task. Loss landscapes of neural networks are extremely non-convex and contain many local optima ; if more randomness is introduced, this will increase the opportunity for different shadow models to reach different optima. This increases the difficulty of reconstruction as these shadow models will not be representative of the optima attained by the released model, and training with a larger learning rate will exacerbate this issue. In Figure 8, we show plot TSNE embeddings of parameters for all $1K$ released models for each of the two learning rates given in Figure 7 and the two randomness settings (known and unknown seed) for a batch size of $1024$. We represent each released model with a color depending on the label of the respective target. For a small learning rate, labels are grouped together in both known and unknown seed settings, implying the local optima these models realize are similar; this makes it easier for the RecoNN to learn and subsequently generalize to the released model. Conversely, in the large learning rate setting there is a stark difference between known and unknown seed settings: if the seed is known, groupings of labels still happen, and a successful attack is possible; however, if the seed is unknown, the local optima reached by each released model has less structure that the reconstructor network can learn on.

In Section -H we show comprehensive results with more learning rates and evaluated on more metrics.

Randomness from model initialization

We explore how initialization randomness can affect the attack on MNIST. Firstly, we remove the assumption that the adversary knows the initial parameters of the released model; in practice, this means training each released and shadow model with a new random seed controlling the model’s initial parameters. By default, each linear and convolutional layer is initialized with Lecun Normalization, which is the default in the Haiku library . In our experiments, we evaluated other common initialization procedures (e.g., Glorot, He), which did not change any of our findings; we omit these results. We refer the reader to Figure 4 for visual inspection of reconstructions at the two error rates reported in Table II, and conclude that the attack is unable to successfully reconstruct without knowledge of initialization, as they are far larger than the NN oracle described in Section V-B.

One may conjecture that the current attack pipeline is not suitable for this setting: we only train a single shadow model per shadow target, which may fail to capture the variance in shadow model parameters over different initializations for the same shadow target. For this reason, we further created an attack training set of $5M$ shadow model-target pairs, consisting of $10K$ shadow targets, where each target has $500$ shadow models all differing in initial parameters. Even so, this approach did not improve the MSE reported in Table II. In Section -A, we discuss evidence suggesting that reconstruction may not be possible without knowing the initial released model parameters. A similar observation was made by Jagielski et al. , who run attacks to find lower bounds of the privacy budget $\epsilon$ in DP-SGD. They observed that the bounds become tighter with less randomness from model initialization.

VI-C Black-box Access to Released Model

The attack assumes white-box access to the released model parameters, and so a natural question arises: can we construct an attack that achieves a similar MSE distance without white-box access? Our attack is constructed by learning the relation between the released model parameters and the unknown target point; the white-box attack uses these parameters directly by flattening released model parameters, concatenating each layer, normalizing, and passing this to the reconstructor network. However, we could instead use other representations that contain information about the released model. We design a black-box attack by limiting the adversary’s access to only the logits predicted by the released model. For each shadow model, using a set of 200 (500) images from $\bar{D}$ for MNIST (CIFAR-10), the adversary collects the logit outputs of each image, concatenates them together, and uses this as the feature representation of the model, instead of the flattened weights. This reduces the dimensionality of the feature vector from 8K to 2K for MNIST and 55K to 5K for CIFAR-10. The average MSE using this logit representation approach is $0.011$ for MNIST and $0.0198$ for CIFAR-10, which is only marginally worse than the MSE of white-box attacks with default settings, and still much better that the NN oracle. We conclude that black-box reconstruction attacks are feasible and have comparable performance to white-box ones.

VI-D Released Model Trained with Differential Privacy

VII Towards Formal Guarantees Against Reconstruction Attacks

Mitigations that (provably) protect released models against reconstruction attacks can (and should) be implemented within the training algorithm used by the model developer. Protections that defend against effective reconstruction by informed adversaries will also protect against attacks by weaker, more realistic adversaries. In this section, we propose a definition of reconstruction robustness against informed adversaries, and compare it to the privacy guarantees afforded by DP. As will soon become apparent, the strength of mitigations against reconstruction is necessarily going to be relative to the strength of the prior information available to the adversary.

Suppose $M$ is an $(\eta,\gamma)$-ReRo mechanism. The definition prevents any reconstruction attack with knowledgeKnowledge of $\pi$ and $D$ in the attack is implicit through the fact that (2) has to hold for any reconstruction attack. of the prior $\pi$, the dataset $D_{\mathchar 45\relax}$ and the output $\theta=M(D_{\mathchar 45\relax}\cup\{Z\})$ to attain a reconstruction error lower than $\eta$ on an unknown target $Z\sim\pi$ with probability larger than $\gamma$. A good ReRo mechanism is one with large $\eta$ and very small $\gamma$, i.e. one where even “decent” reconstructions are impossible with high probability. In practice a tension between these two parameters is expected, at least for mechanisms providing some form of utility when computing a function depending on all the inputs.

Similarly, the definition protects against adversaries with full knowledge of the prior $\pi$. Since the optimal attack run by an adversary with a wrong prior is necessarily weaker than the optimal attack with a correct prior, assuming the adversary knows $\pi$ is preferable when designing mitigations.

VII-B From DP to ReRo

We now show that differentially private mechanisms provide reconstruction robustness. Let us recall the definitions of approximate and Rényi DP.

Let $M:\mathcal{Z}^{n}\to\Theta$ be a randomized mechanism, $\epsilon>0$, $\delta\in$ and $\alpha>1$. We say that:

$M$ is $(\epsilon,\delta)$-DP if for any datasets $D,D^{\prime}\in\mathcal{Z}^{n}$ differing in a single record and any event $E\subseteq\Theta$ we have

When $\delta=0$ we simply say the mechanism is $\epsilon$-DP.

$M$ is $(\alpha,\epsilon)$-RDP if for any datasets $D,D^{\prime}\in\mathcal{Z}^{n}$ differing in a single record we have

Taking $\alpha\to\infty$ and recalling that $(\infty,\epsilon)$-RDP is equivalent to $\epsilon$-DP we obtain the following corollary.

Another way to interpret Theorem 2 is through the lens of zero-concentrated DP (zCDP) . A mechanism is $\rho$-zCDP if it satisfies $(\alpha,\alpha\rho)$-RDP for every $\alpha>1$. This definition provides a natural and convenient way to express the privacy afforded by the ubiquitous Gaussian mechanism . Applying Theorem 2 to a $\rho$-zCDP mechanism and optimizing $\alpha$ to minimize the upper bound yields the following.

VII-C From ReRo to DP

Next we investigate the reverse implication: does a strong enough level of reconstruction robustness imply a standard definition of privacy protection like DP? We show that this is indeed the case if one insists on protecting against exact reconstruction simultaneously for a family of priors concentrated on pairs of data points. From this lens, the result says that as soon as a mechanism exhibits strong enough reconstruction robustness to prevent membership inference it must necessarily satisfy DP.

VII-D ReRo Against High-Dimensional, High-Uncertainty Priors

A standard “rule of thumb” says that DP only provides a meaningful protection when $\epsilon$ is a small constant. On the other hand, our experiment on models trained with DP-SGD (Section VI-D, Section -E) shows that much larger values of $\epsilon$ are successful at mitigating our RecoNN-based attack. This could be interpreted as a limitation of our attack in the presence of weak levels of DP protection. An alternative explanation is that DP with large values of $\epsilon$ can protect against reconstruction attacks if the reconstruction target is high-dimensional and the adversary’s prior knowledge contains a large degree of uncertainty. We formalize this intuition by instantiating the bounds from Section VII-B on two natural priors where $\kappa$ is easy to bound: uniform and Gaussian priors. A similar analysis in the context of local DP was presented in (see Section VII-F for a detailed comparison).

This result shows that, in high-dimensional settings where an informed adversary’s knowledge about the target datapoint is only in the form a syntactic constraint like $\|z\|_{2}\leq 1$, privacy parameters sub-linear in the dimension suffice to make the reconstruction success probability negligible.

Gaussian priors

The idea that large values of $\epsilon$ can protect against reconstruction when the adversary’s prior contains significant uncertainty (i.e. it is diffused) was previously noticed in in the context of local DP (LDP) with priors close to uniform. Inspired by FL applications where adversaries get access to LDP gradients, the authors propose a notion of protection against reconstruction breaches that is more stringent than Definition 2: it asks that the adversary cannot effectively reconstruct a particular feature of interest about the target point no matter what the output of the mechanism is – in contrast, ReRo uses an average-case requirement over the outputs of the mechanism. Technically, [41, Lemma 2.2] shows that the bound in Corollary 3 also holds for this worst-case notion of protection against reconstruction.Although the bound in is stated in terms of $\epsilon$-LDP, it is easy to see that the same holds for central $\epsilon$-DP in the presence of an informed adversary. Such worst-case guarantees, however, are not attainable under relaxations of $\epsilon$-DP like RDP because the latter does not enforce an almost sure bound on the privacy loss: instead, it just guarantees that the privacy loss will be small with high probability. Thus, Theorem 2 and Proposition 6 are natural generalizations of the results from to RDP, which is the default notion of privacy provided by DP-SGD and other popular private ML algorithms .

VII-E Is Reconstruction Robustness Useful in Practice?

VII-F Further Related Work

The use of informed adversaries in formal privacy analyses can be tracked back to the sub-linear queries (SuLQ) framework . SuLQ was later subsumed by DP , where mentions to a concrete adversary were expressly avoided in the definition that is widely used nowadays . Nonetheless, [6, Appendix A] provides a “semantically flavored” definition equivalent to DP which involves the likelihood ratio between the prior and posterior beliefs of an informed adversary about any property of the target data point. The adversarial model put forward in Section II uses the same notion of informed adversary.

In other frameworks where the adversary is not (necessarily) informed (e.g. Pufferfish privacy and inferential privacy ), side knowledge about the whole dataset is encoded in a probabilistic prior capturing information about the individual entries in the dataset as well as their statistical dependencies. These frameworks extend the semantic approach to DP by replacing the prior-vs-posterior condition with an odds ratio condition – such modification is motivated by the observation that prior-vs-posterior bounds cannot hold in general for uninformed adversaries unless the prior distribution over the dataset assumes the records are mutually independent. Alternatively, provides posterior-vs-posterior semantics for DP in the presence of an uninformed adversary with an arbitrary prior. In the definition of reconstruction robustness, our use of an informed adversary with a prior over the target data point circumvents the complications arising from dependencies between points in the training data: the prior captures the adversary’s residual uncertainty about the target point after observing the fixed dataset. On the opposite direction, several authors have proposed approaches where the adversary’s uncertainty with respect to the input data of a mechanism is leveraged to increase the privacy provided to individuals . Implicitly, these works assume a less powerful adversary than the one considered in this paper.

The connection between DP and protection against membership inference is perhaps best understood via its hypothesis testing interpretation . A comprehensive discussion of the adversary implicit in the definition of DP from the hypothesis testing standpoint can be found in . Interestingly, shows that, unlike standard DP, RDP does not admit a hypothesis testing interpretation. A semantic (Bayesian) interpretation of RDP in terms of moment bounds on the odds ratio is presented in . Theorem 2 provides an alternative characterization of the privacy protection afforded by RDP in terms of resilience to reconstruction attacks.

DP and protection against reconstruction

How standard DP offers concrete protection against reconstruction attacks has been studied in other contexts. Indeed, one of the original motivations for the definition of DP was to defeat database reconstruction attacks in the context of interactive query mechanisms . In such attacks, the adversary receives (noisy) answers to a sequence of specially crafted queries against a database and, if the noise is small enough, uses the answers to (partially) reconstruct every record in the database. The success of these attacks is contingent on the adversary’s ability to control these queries; in contrast, in ML applications like the ones we consider the computation performed by the mechanism is completely under the model developer’s control.

VIII Conclusions

Our work provides compelling evidence that standard ML models can memorize enough information about their training data to enable high-fidelity reconstructions in a very stringent threat model. By instantiating an informed adversary that learns to map model parameters to training images, we successfully attacked standard MNIST and CIFAR-10 classifiers with up to $100K$ parameters, and showed the attack is significantly robust to changes in the training hyper-parameters. Two aspects of our attack we would like to improve in future work are its data and computational efficiency, and its scalability to larger, more performant released models. This would not lead to real-world adversaries mounting practical attacks due to the nature of our threat model, but it would enable model developers to assess potential privacy leakage in models before deployment. Extending our attacks to reconstruct $N>1$ targets simultaneously would also be interesting, but we expect this to be substantially harder. For example, in this setting our attacks against convex models lead to a problem with more unknowns than equations. On the defenses side, we empirically showed that DP training with large values of $\epsilon$ can effectively mitigate our reconstruction attacks. Our theoretical discussion, stemming from a new definition of reconstruction robustness and a study of its connection to (R)DP, shows this is a general phenomenon: informed reconstruction attacks can be prevented with large values of $\epsilon$ under some assumptions on the adversary. Validating such assumptions in particular applications would open the door to practical models which are accurate and resilient against reconstruction attacks.

Acknowledgment

The authors would like to thank: Leonard Berrada, Adrià Gascón and Shakir Mohamed for feedback on an earlier version of this manuscript; Brendan McMahan for suggesting the idea that random initialization in SGD might make privacy attacks harder which inspired some of our experiments; and Olivia Wiles for discussions on how to improve reconstructor network training on CIFAR-10. This work was done while G.C. was at the Alan Turing Institute.

References

-A Randomness from Released Model Initialization

In Table III, we show results both with and without randomization from initialization for different released model learning rates on the MNIST dataset. The average MSE with and without random initialization was 0.0089 and 0.0695, respectively. The choice of learning rate negatively impacts the attack in settings where initial parameters are known, but not when initial parameters are unknown, since the attack fails for any choice.

The attack fails when the adversary does not have knowledge of the initial parameters of the released model and so must instantiate each shadow model used to train the attack with a new seed that controls the selection of initial parameters. We provide evidence that it may not be possible to perform a reconstruction attack in this setting by appealing to a simpler task of inferring membership, and demonstrating this problem is also difficult without knowledge of the initial parameters. We instantiate an informed MIA as described in Section II on the MNIST dataset. Specifically, given a target point $z$ we train $1K$ released models with and without this point included (but with the same fixed set) in two settings: when each model is initialized with a new seed (differing initial parameters), and when each model is initialized with the same seed (identical initial parameters). In Figure 11, we plot the distribution of losses on this target point in these two settings. Clearly, when there is no initial parameter randomization the distributions are perfectly separable and so membership is easy to infer, while in the random setting, the distributions nearly perfectly overlap implying membership may be more difficult, if not impossible. Note that if released model training was fully deterministic, the distribution of losses on the target point in the setting with no random initialization would collapse to a point distribution. However, all our models are trained with JAX on GPUs that compile with non-deterministic reductions, introducing a small source of randomness .

-B Transfer Learning from a Reconstructor Network Trained on a Different Fixed Set

Given a reconstructor network, $\phi$, trained to attack released models of the form $\theta=A_{D_{\mathchar 45\relax}}(z)$, can the adversary amortize the cost training a new $\phi^{\prime}$ that aims to attack a released model $\theta^{\prime}=A_{D_{\mathchar 45\relax}^{\prime}}(z)$, where $D_{\mathchar 45\relax}^{\prime}\cap D_{\mathchar 45\relax}=\emptyset$? On both MNIST and CIFAR-10, in Figure 12 we show that fine-tuning the reconstructor $\phi$ on only a small number of shadow models can reach comparative performance to a reconstructor trained from scratch on substantially more data.

-C Adversary Knowledge of Starting Point: Initialization vs Near Convergence

By default we assume the adversary knows the initial released model parameters, motivated by scenarios where the random seed used to generate initial parameters is made public or is leaked. Another motivating example is that of federated learning, where an adversary participates in the learning protocol. However, in such a setting, it is not guaranteed the adversary will observe a model at it’s initial state. If the adversary is only included in the protocol after a sufficient number of time steps, the state at which they first observe released model parameters may be close to convergence. Here, we measure how reconstructions are affected by this subtle assumption. We pre-train a released model on $10K$ MNIST images (this model already achieves $>92\%$ MNIST test set accuracy), and then following the experimental set-up reported in Section V on the remaining MNIST data, and compare to a released model in the standard setting where no pre-training occurs. Figure 13 shows the MSE for each 1K released model target in both settings. Clearly there is a difference in reconstruction fidelity that depends on the step at which the adversary first observes the released model parameters. A model that has nearly converged may be less dependent / not memorize it’s newly seen training data, making reconstructions more challenging.

-D Visualization of Easy and Hard CIFAR-10 Reconstructions

In Figure 14 we show the six reconstructed CIFAR-10 examples with smallest MSE and six examples with largest MSE out the $1K$ targets used for evaluation. The easiest targets to reconstruct correspond to structurally simple images with a constant background, while the most difficult often have complex background and color schemes.

-E Reconstructing Against a Released Model Trained with DP on CIFAR-10

-F Size of Released Model

Here we expand on our investigation around the interplay between size of the released model and reconstruction success by varying the width of the released model hidden layer between 1, 10, and 100 for the MNIST dataset. Figure 16 and Table IV show that the width of the hidden layer does not significantly affect reconstruction, as all have an average MSE on the $1K$ targets far below the NN oracle of 0.0232. We also investigate the case in which the attacker only trains and evaluates the attack using the second layer of the shadow model (and released model) for the MNIST dataset. Because the architecture of the 1-hidden layer released model has 7850 parameters in the first layer and 110 in the second, by only using the second layer as inputs to the attack, we reduce the dimensionality of attack inputs by 98%. Using only the second layer of released model for the input to the reconstructor network marginally increases average MSE, but substantially reduces the dimensionality of inputs, thereby improving efficiency during training the attack.

-G Variance over Different Initializations

We repeat our default reconstruction attack on MNIST ten times, where at each repetition we will sample a new seed controlling initial parameters over that experiment run. We the measure how consistent our experimental results are for different choices of seeds. From Figure 17 we can see that while there is variance in results, almost all reconstructions lie close or below the NN oracle distance in Figure 3(a). One may wonder if certain seeds are more amenable to attacks than others? That is, are there configurations of initial released model parameters that result in better reconstructions across all target points in comparison to other initializations? We show this is not the case in Figure 17(b) by marking each seed in a different color and showing the MSE for ten test targets over the ten different seeds; this highlights that the ordering of seeds with respect to MSE changes for different images.

-H Effect of Batch Size, Learning Rate, and Fixed Set Size on MNIST Reconstructions

We summarize other aspects of the training procedure of the released model that impact the quality of reconstructions on the MNIST dataset. Firstly, larger learning rates when training the released model can negatively influence the quality of reconstructions in SGD (i.e. using mini-batches), but in full-batch gradient descent we observed similar a MSE across different learning rates (cf. Figure 23). Secondly, in Figure 18, we measure the influence of the size of the fixed set, by varying it between 100, 1$K$, and 10$K$ (which is the default in our MNIST experiments). Interestingly, we only observe a small decrease in MSE as the fixed set size decreases.

-I Correlation between Metrics on CIFAR-10

We measure the correlation between MSE, LPIPS, and KL on the 1K test targets for CIFAR-10 in Figure 19, and observe similar relationships between the metrics as in Figure 5(a) and Figure 5(b).

-J Expanded Investigation into Factors that Affect CIFAR-10 Reconstructions

In Table X, we investigate reconstructions under different released model optimizers, fixed set sizes and number of training epochs. We find that the dominant factor in reconstruction fidelity is the number of training epochs of the released model. A smaller number of training epochs and smaller fixed set size improves the quality of reconstructions. Note also that the ability to reconstruct does not seem to be correlated with overfitting or (standard) membership inference success, and that the attack succeeds for different choices of optimizer.

Due to the size of the released model, and other restrictions such as full-batch training with no regularizers, the test accuracy of the released model for which we can successfully perform attacks is approximately 35-50%. Improving the efficiency of the attack such that it can scale to larger released models is a challenge to address in future work.

-K Correlation between Gradient Norm and Ease of Reconstruction

Following on from Section -C, we investigate the relationship between reconstruction and the gradient norm of loss with respected to released model parameters computed on the target point through training. Recent work on training data memorization and individual privacy accounting in differential privacy have used the gradient norm of a model with respect to the loss induced by a training point as a measure of memorization or privacy leakage. In Figure 20, we evaluate the MSE between target and reconstructions for each 1K target point on MNIST, and also plot the sum of gradient norms over training. The two quantities are weakly correlated with one another, however one may expect that the two would be inversely correlated if examples that are strongly memorized are easier to reconstruct – targets with a larger gradient norm throughout training are outliers that the released model must necessarily memorize to perform well on (). We conjecture that the effect we are observing stems from “outliers” that are harder to reconstruct not because the released model memorizes them less, but because they’re also outliers for the RecoNN, and therefore target where the reconstructor also performs poorly.

-L Fine-Grained Analysis of CIFAR-10 Reconstructions over Released Model Training Epochs

As we observed in Section -J, reconstructing CIFAR-10 images is sensitive to the number of training epochs of the released model. We perform a fine-grained analysis to inspect at what epoch the attack becomes unsuccessful. This can be seen in Figure 21, where we plot average MSE over 1K released model targets as a function of the number of training epochs. MSE slowly increases with number of epochs up until approximately 240-250 epochs, at which point we observe that “reconstructability” undergoes a phase transition. Initially, we conjectured this was due to non-determinism from GPU training increasing the variance of shadow model parameters for a larger number of training epochs. However, when we implemented shadow model training in a deterministic set-up (using TPUs) we observed no difference in experimental outcomes. We leave a more in-depth investigation into the relationship between reconstruction success and number of training epochs for future work.

-M ReLU Activations in Released Model

As we saw in Section VI-B, released models with ReLU activations tend to be harder to attack in comparison to other activation functions with non-zero gradients almost everywhere, and result in poor quality reconstructions (an MSE larger than the NN oracle distance). We conjecture that this is caused by a large fraction of parameters receiving zero gradients at each step of training, thereby diminishing the mutual information shared between model parameters and the unknown target training point. In Figure 22, for each layer of the released model, we show the fraction of parameters that received zero gradient when computing the loss of the unknown training point. Over 80% of the parameters in the convolutional layers have zero gradients. Additionally, in Table V we compare reconstructions against released models that employ different activation functions, and find that ReLU remains the outlier. Note that we also reconstruct against a released model that uses a modified version of ReLU that has zero gradient for $x<-0.5$, and find that allowing a small negative signal is enough to reach parity with reconstruction MSE on smooth activations or activations that contain a non-zero signal almost everywhere.