Individual Privacy Accounting via a Renyi Filter

Introduction

Understanding how privacy of an individual degrades as the number of analyses using their data grows is of paramount importance in privacy-preserving data analysis. This allows individuals to participate in multiple disjoint statistical analyses, all the while knowing that their privacy cannot be compromised by aggregating the resulting reports. Furthermore, this feature is crucial for privacy-preserving algorithm design—instead of having to reason about the privacy properties of a complex algorithm, it allows reasoning about the privacy of the subroutines that make up the final algorithm.

For differential privacy , this accounting of privacy losses is typically done using composition theorems. Importantly, given that statistical analyses often rely on the outputs of previous analyses, and that algorithmic subroutines feed into one another, the composition theorems need to be adaptive, namely, allow the choice of which algorithm to run next to depend on the outputs of all previous computations. For example, in gradient descent, the computation of the gradient depends on the value of the current iterate, which itself is the output of the previous steps of the algorithm.

Given the central role that adaptive composition theorems play in differentially private data analysis, they have been investigated in numerous works (e.g. ). While they differ in some aspects, they also share one limitation. Namely, all of these theorems reason about the worst-case privacy loss for each constituent algorithm in the composition. Here, “worst-case” refers to the worst choice of individual in the dataset and worst choice of value for their data. This pessimistic accounting implies that every algorithm is summarized via a single privacy parameter, shared among all participants in the analysis.

In most scenarios, however, different individuals have different effects on each of the algorithms, as measured by differential privacy. More precisely, the output of an analysis may have little to no dependence on the presence of some individuals. For example, if we wish to report the average income in a neighborhood, removing an individual whose income is close to the average has virtually no impact on the final report after noise addition. Similarly, when training a machine learning model via gradient descent, the norm of the gradient given by a data point is often much smaller than the maximum norm (typically determined by a clipping operation). As a result, in many cases no single individual is likely to have the worst-case effect on all the steps of the analysis. This means that accounting based on existing composition theorems may be unnecessarily conservative.

In this work, we present a tighter analysis of privacy loss composition by computing the associated divergences at an individual level. In particular, to achieve a pre-specified privacy budget, we keep track of a personalized estimate of the privacy loss divergence for each individual in the analyzed dataset, and ensure that the respective estimate is maintained under the budget for all individuals throughout the composition. We do so by applying each analysis only to the points that are estimated to have sufficient leftover privacy budget.

The rest of the paper is organized as follows. In the remainder of this section, we give an overview of our main results and discuss related work. In the next section, we introduce the preliminaries necessary to state our results. In Section 3, we prove our main adaptive composition theorem for Rényi differential privacy. We build off this result in Section 4, where we develop a Rényi privacy filter—an object for budgeting privacy loss—and apply it to individual privacy accounting. In Section 5 we present an application of our theory to differentially private optimization, as well as some experimental results.

It is feasible to measure the worst-case effect of a specific data point on a given analysis in terms of any of the divergences used to define differential privacy. One can simply replace the supremum over all datasets in the standard definition of (removal) differential privacy with the supremum over datasets that include that specific data point (see Definition 2.5). Indeed, such a definition is given by Ebadi et al. and a related definition is given by Wang . However, a meaningful application of adaptive composition with such a definition immediately runs into the following technical challenge. Standard adaptive composition theorems require that the privacy parameter of each step be fixed in advance. For individual privacy parameters, this approach requires using the worst-case value of the individual privacy loss over all the possible analyses at a given step. Individual privacy parameters tend to be much more sensitive to the analysis being performed than worst-case privacy losses, and thus using the worst-case value over all analyses is likely to negate the benefits of using individual privacy losses in the first place.

Thus the main technical challenge in analyzing composition of individual privacy losses is that they are themselves random variables that depend on the outputs of all previous computations. More specifically, if we denote by $a_{1},\dots,a_{t-1}$ the output of the first $t-1$ adaptively composed algorithms ${\cal A}_{1},\dots,{\cal A}_{t-1}$, then the individual privacy loss of any point incurred by applying algorithm ${\cal A}_{t}$ is a function of $a_{1},\dots,a_{t-1}$. Therefore, to tackle the problem of composing individual privacy losses we need to understand composition with adaptively-chosen privacy parameters in general. We refer to this kind of composition as fully adaptive.

The setting of fully adaptive privacy composition is rather subtle and even defining privacy in terms of the adaptively-chosen privacy parameters requires some care. This setting was first studied by Rogers et al. , who introduced the notion of a privacy filter. Informally, a privacy filter is a stopping time rule that halts a computation based on the adaptive sequence of privacy parameters and ensures that a pre-specified privacy budget is not exceeded. Rogers et al. define a filter for approximate differential privacy that asymptotically behaves like the advanced composition theorem , but is substantially more involved and loses a constant factor. Moreover, several of the tighter analyses of Gaussian noise addition require composition to be done in Rényi differential privacy . Converting them to $(\varepsilon,\delta)$-differential privacy would incur an additional $\sqrt{\log(1/\delta)}$ factor in the final bound.

Our main result can be seen as a privacy filter for Rényi differential privacy (RDP) which justifies stopping the analyses based on the sum of privacy parameters so far even under fully adaptive composition.

Fix any $B\geqslant 0,\alpha\geqslant 1$. Suppose that ${\cal A}_{t}$ is $(\alpha,\rho_{t})$-Rényi differentially private, where $\rho_{t}$ is an arbitrary function of $a_{1},\dots,a_{t-1}$. If $\sum_{t=1}^{k}\rho_{t}\leqslant B$ holds almost surely, then the adaptive composition of ${\cal A}_{1},\dots,{\cal A}_{k}$ is $(\alpha,B)$-Rényi differentially private.

Note that, when all privacy parameters are fixed, Theorem 1.1 recovers the usual composition result for RDP . Our RDP filter immediately implies a simple filter for approximate differential privacy that is as tight as any version of the advanced composition theorem obtained via concentrated differential privacy (see Theorem 4.9). These Rényi-divergence-based composition analyses are known to improve upon the classical rate of Dwork et al. and, in particular, improve on the rate in .

We instantiate our general result for fully adaptive composition in the setting of individual privacy accounting. This allows us to define an individual privacy filter, which, given a fixed privacy budget, adaptively drops points from the analysis once their personalized privacy loss estimate exceeds the budget. Therefore, instead of keeping track of a single running privacy loss estimate for all individuals, we track a less conservative, personalized estimate for each individual in the dataset. Individual privacy filtering allows for better, adaptive utilization of data points for a given budget. It can also naturally be applied to privacy accounting in the local differential privacy model, whereby each user stops responding once their local implementation of the filter indicates that their personal privacy budget is exhausted.

Individual privacy parameters are particularly easy to compute for linear queries, as well as their high-dimensional generalizations. We show that our technique gives an algorithm for answering a sequence of adaptively-chosen linear queries that are sparse across time, meaning that, for any user, the number of queries that are non-zero on that user’s data is small. Such queries arise, for example, when a platform counts the number of users that participate in certain activities (the type of activity being adaptive to the data collected in the previous days) and users generally participate in a small number of activities. Formally, a special case of our result implies the following theorem.

There exists an algorithm ${\cal A}$ that, given a dataset $S=(X_{1},\ldots,X_{n})\in{\cal X}^{n}$, sparsity parameter $s$ and privacy level $\kappa$, for any adaptively-chosen sequence of queries $q_{1},\ldots,q_{k}$ of arbitrary length $k$, where $q_{i}\colon{\cal X}\to\{0,1\}$, provides a sequence of answers $a_{1},\ldots,a_{k}$ such that: $(1)$ ${\cal A}$ is $(\alpha,\alpha\kappa)$-RDP for all $\alpha\geqslant 1$; $(2)$ for all $t$ and any $\delta\in(0,1)$, the probability that $|a_{t}-\sum_{X_{i}\in S_{t}}q_{t}(X_{i})|>\sqrt{s\log(1/\delta)/\kappa}$ is at most $\delta$, where $S_{t}=(X_{i}\in S:\sum_{j=1}^{t}q_{j}(X_{i})\leqslant s)$.

We note that the provided answers are guaranteed to be accurate only as long as the queries are truly sparse, meaning $\sum_{j=1}^{t}q_{j}(X_{i})\leqslant s$ for (almost) all $i\in[n]$. This follows because the queries are accurate on the set $S_{t}$, hence $S_{t}$ needs to be similar to $S$ for the queries to be accurate on $S$. The privacy guarantee, on the other hand, holds for any sequence of queries of any length $k$. We describe a more general version of this result in Section 4.2. A natural application of our general theorem is the setting of high-dimensional linear queries generated by gradient descent. We apply our theory to the analysis of private gradient descent , and show—both theoretically and empirically—that individual accounting can be easy to implement and can only make the resulting privacy-utility tradeoff tighter. Independently, without any individual accounting, in our empirical evaluations we also observe that private batch gradient descent, when tuned appropriately, outperforms private stochastic gradient descent in terms of the privacy-utility tradeoff. While we make this observation only on MNIST, we believe this phenomenon holds more generally and is worth further investigation.

2 Related work

The main motivation behind our work is obtaining tighter privacy accounting methods through, broadly speaking, “personalized” accounting of privacy losses. Existing literature in differential privacy discusses several related notions , although typically with an incomparable objective. Ghosh and Roth discuss individual privacy in the context of selling privacy at auction and their definition does not depend on the value of the data point but only on its index in the dataset. Cummings and Durfee rely on a similar privacy definition, investigate an associated definition of individual sensitivity, and demonstrate a general way to preprocess an arbitrary function of a dataset into a function that has the desired bounds on individual sensitivities.

Ebadi et al. introduce personalized differential privacy in the context of private database queries and describe a system which drops points when their personalized privacy loss exceeds a budget. In their system personalized privacy losses result from record selection operations applied to the database. While this type of accounting is similar to ours in spirit, their work only considers basic and non-adaptive composition. The work of Wang considers the privacy loss of a specific data point relative to a fixed dataset and provides techniques for evaluating this “per-instance” privacy loss for several statistical problems. Wang also briefly discusses adaptive composition of per-instance differential privacy as a straightforward generalization of the usual advanced composition theorem , but the per-instance privacy parameters are assumed to be fixed. As discussed above, having fixed per-instance privacy parameters, while allowing adaptive composition, is likely to negate the benefits of personalized privacy estimates. The work of Ligett et al. tightens individuals’ personalized privacy loss by taking into account subsets of analyses in which an individual does not participate. Our work naturally captures this setting while allowing full adaptivity. Moreover, they consider the usual worst-case privacy loss, rather than an individual one, and the analyses in which a user participates are determined in a data-independent way.

Our work can be seen as related to data-dependent approaches to analyses of privacy-preserving algorithms such as smooth sensitivity , the propose-test-release framework , and ex-post privacy guarantees . Our results are complementary in that we aim to capture the dependence of the output on the value of each individual’s data point as opposed to the “easiness” of the entire dataset. Our approach also relies on composition to exploit the gains from individual privacy loss accounting.

Finally, adaptive composition of differentially private algorithms is a key tool for establishing statistical validity of an adaptively-chosen sequence of statistical analyses . In this context, Feldman and Steinke show that individual KL-divergence losses (or RDP losses for $\alpha=1$) compose adaptively and can be used to derive tighter generalization results. However, their results still require that the average of individual KL-divergences be upper bounded by a fixed worst-case value and the analysis appears to be limited to the $\alpha=1$ case.

Preliminaries

We start by reviewing some preliminaries on differential privacy.

A randomized algorithm ${\cal A}$ is $(\varepsilon,\delta)$-differentially private (DP) if for all datasets $S=(X_{1},\dots,X_{n})$,

for all $i\in[n]$ and all measurable sets $E$.

Our analysis will rely on Rényi differential privacy (RDP), a relaxation of DP based on Rényi divergences which often leads to tighter privacy bounds than analyzing DP directly. Formally, the Rényi divergence of order $\alpha\in(1,\infty)$ between two measures $\mu$ and $\nu$ such that $\mu\ll\nu$ is defined as:

The Rényi divergence of order $\alpha=1$ is defined by continuity, and recovers the Kullback-Leibler (KL) divergence. Relying on a common abuse of notation, we will use ${\cal A}(\cdot)$ to refer to the output distribution of a randomized algorithm. Thus, $D_{\alpha}({\cal A}(S)\|{\cal A}(S^{-i}))$ denotes the divergence between the output distribution of ${\cal A}$ on input $S$ and the output distribution of ${\cal A}$ on input $S^{-i}$. Similarly, we will use $a\sim{\cal A}(S)$ to denote $a$ being sampled randomly from the output distribution of ${\cal A}$ on $S$. We also use the following shorthand notation for the maximum of the two directions of Rényi divergence:

A randomized algorithm ${\cal A}$ is $(\alpha,\rho)$-Rényi differentially private (RDP) if for all datasets $S=(X_{1},\dots,X_{n})$,

A related notion that we will make use of is zero-concentrated differential privacy (zCDP).

A randomized algorithm ${\cal A}$ satisfies $\kappa$-zero-concentrated differential privacy (zCDP) if it satisfies $(\alpha,\alpha\kappa)$-RDP for all $\alpha\geqslant 1$.

Rényi differential privacy implies differential privacy; therefore, although our guarantees will be stated in terms of RDP, the conversion to DP is immediate.

If algorithm ${\cal A}$ is $(\alpha,\rho)$-RDP, then it is also $\left(\rho+\frac{\log(1/\delta)}{\alpha-1},\delta\right)$-DP, for any $\delta\in(0,1)$.

One of the successes of differential privacy (and RDP as well) lies in its adaptive composition property. In Algorithm 1 we define adaptive composition, which is at the center of our analysis.

If ${\cal A}_{t}(a_{1},\dots,a_{t-1},\cdot)$ is $(\alpha,\rho_{t})$-RDP for all values of $a_{1},\dots,a_{t-1}$, then the standard adaptive composition theorem for RDP says that ${\cal A}^{(k)}$ is $(\alpha,\sum_{t=1}^{k}\rho_{t})$-RDP . Note that, by definition, the parameters $\rho_{1},\dots,\rho_{k}$ are independent of the specific reports $a_{1},\dots,a_{k}$ obtained in the adaptive computation. In other words, they are fixed in advance.

Our individual accounting relies on measuring the maximum possible effect of an individual data point on a dataset statistic in terms of Rényi divergence. This measure is equivalent to an RDP version of personalized differential privacy . For convenience we will refer to it as individual Rényi differential privacy, or individual RDP for short. We note, however, that, by itself, a bound on this divergence does not imply any formal privacy guarantee for an individual, since the individual RDP parameter depends on the sensitive value of the data point.

Therefore, to satisfy the standard definition of RDP, an algorithm needs to satisfy individual RDP for all data points $X$.

Our main focus will be on individual privacy losses as introduced in Definition 2.5, however some of our results also hold under a weaker notion of individual privacy loss, which measures the effect of a data point on the output of a statistical analysis, relative to a fixed dataset. This notion is an RDP version of per-instance differential privacy .

Fix a dataset $S=(X_{1},\dots,X_{n})$. We say that a randomized algorithm ${\cal A}$ satisfies $(\alpha,\rho)$-individual Rényi differential privacy for $(S,X_{i})$ if it holds that

We will note which results hold under Definition 2.6, in addition to being valid under Definition 2.5.

Before we turn to analyzing composition, we give a simple example of individual RDP computation. For simplicity, we focus on Gaussian noise addition. Similar computations can be carried out for other randomization mechanisms.

individual RDP for $X_{i}$. Note that in this case individual RDP (Definition 2.5) and per-instance RDP (Definition 2.6) have the same value.

The analysis above extends to arbitrary Lipschitz functions.

Fully adaptive composition for Rényi differential privacy

Our main technical contribution is a new adaptive composition theorem for Rényi differential privacy, which bounds the overall privacy loss in terms of the individual privacy losses of all data points. As argued earlier, the main challenge in understanding how individual privacy parameters compose is the fact that these parameters are random, rather than fixed. In what follows, we first state a general version of our main theorem, which bounds the privacy loss in adaptive composition in terms of a bound on the sequence of possibly random privacy parameters. Then, we instantiate this result in the context of individual privacy.

Similarly, for fixed $a^{(t)}$ we also define

We let $\rho_{t}$ denote the RDP parameter of order $\alpha$ of ${\cal A}_{t}$, conditional on the past reports. For the sake of generality and simplicity of exposition, we introduce an abstract space ${\cal S}$ over pairs of datasets and let

In the context of individual privacy (Definition 2.5), we will instantiate ${\cal S}$ to be the space of all dataset pairs where either dataset is obtained by deleting $X_{i}$ from the other. For the per-instance notion (Definition 2.6), we will set ${\cal S}=\left\{(S,S^{-i})\right\}$. In the context of usual RDP, ${\cal S}$ will be the space of all pairs of datasets that differ in the presence of one element.

The classical composition theorem for Rényi differential privacy—while allowing ${\cal A}_{t}$ to depend on the previous reports—constrains ${\cal A}_{t}$ to be $(\alpha,\rho_{t})$-RDP for some fixed $\rho_{t}$. Here, we make no such constraints on $\rho_{t}$; hence, $\rho_{t}$ will in general be a random variable, due to the randomness in $a_{1},\dots,a_{t-1}$.

Theorem 3.1 states that, as long as $\sum_{t=1}^{k}\rho_{t}$ is maintained under a fixed budget, the output of adaptive composition preserves privacy.

Fix any $B\geqslant 0,\alpha\geqslant 1$, and a set of pairs of datasets ${\cal S}$. For any sequence of algorithms ${\cal A}_{1},\ldots,{\cal A}_{k}$, if $\sum_{t=1}^{k}\rho_{t}\leqslant B$ holds almost surely, where the sequence $\rho_{1},\ldots,\rho_{k}$ is defined in eq. (1), then the adaptive composition ${\cal A}^{(k)}$ given in Algorithm 1 satisfies

where the third equality uses the fact that $(\rho_{j})_{j=1}^{t}\in\Gamma_{t-1}$, and the inequality applies the definition of $\rho_{t}$. Therefore, by applying iterated expectations, we can conclude

Since $\sum_{j=1}^{k}\rho_{j}\leqslant B$ by assumption, this inequality implies

The same argument can be used to bound the other direction of the divergence. Since the choice of $(S,S^{\prime})$ was arbitrary, we can conclude $\sup_{(S,S^{\prime})\in{\cal S}}D^{\leftrightarrow}_{\alpha}\left({\cal A}^{(k)}(S),{\cal A}^{(k)}(S^{\prime})\right)\leqslant B$, as desired. ∎

A related argument is presented by Cesar and Rogers (see Lemma 3.1), who analyze privacy composition when a pre-specified set of concentrated differential privacy (CDP) parameters is adaptively ordered.

We remark that Theorem 3.1 is satisfied for any set ${\cal S}$, including the singleton ${\cal S}=\{(S,S^{\prime})\}$. Denote by $\rho_{t}(S,S^{\prime})$ the privacy parameter as defined in equation (1) when ${\cal S}=\{(S,S^{\prime})\}$. Then, Theorem 3.1 implies that the adaptive composition ${\cal A}^{(k)}$ satisfies $(\alpha,B)$-RDP if for all datasets $(S,S^{\prime})$ differing in the presence of one individual, $\sum_{t=1}^{k}\rho_{t}(S,S^{\prime})\leqslant B$. In other words, in principle it is possible to place the maximum over $(S,S^{\prime})\in{\cal S}$ from the definition (1) in front of the sum over privacy parameters. However, while this is formally a less conservative privacy accounting method, it remains unclear if it can be efficiently implemented in practice.

We now instantiate Theorem 3.1 in the context of individual privacy.

To simplify notation, for a fixed point $X$, we let ${\cal S}(X,n)$ denote the set of all dataset pairs $(S,S^{\prime})$ such that $|S|\leqslant n$ and $S^{\prime}$ is obtained by deleting element $X$ from $S$. More precisely, $(S,S^{\prime})\in{\cal S}(X,n)$ if $S=(X_{1},\dots,X_{m})$, where $m\leqslant n$ and $X_{i}=X$ for some $i$, and $S^{\prime}=S^{-i}$.

We use $\rho_{t}^{(i)}$ to denote the individual privacy parameter of the $t$-th adaptively composed algorithm ${\cal A}_{t}$ with respect to $X_{i}$, conditional on the past reports. Formally, for fixed $\alpha\geqslant 1$ and for any data point $X_{i}\in S$ we let:

Since $\rho_{t}^{(i)}$ is an instance of the general definition (1) obtained by specifying ${\cal S}$, a direct corollary of Theorem 3.1 is as follows.

Fix any $B\geqslant 0$. If for any input dataset $S=(X_{1},\dots,X_{n})$, $\sum_{t=1}^{k}\rho_{t}^{(i)}\leqslant B$ holds almost surely for all individuals $i\in[n]$, then the adaptive composition ${\cal A}^{(k)}$ given in Algorithm 1 is $(\alpha,B)$-Rényi differentially private.

By Theorem 3.1, $\sum_{t=1}^{k}\rho_{t}^{(i)}\leqslant B$ implies that

Since this holds for all $S\in{\cal X}^{n}$ and $i\in[n]$, we conclude that ${\cal A}^{(k)}$ is $(\alpha,B)$-Rényi differentially private. ∎

Notice that $\sum_{t=1}^{k}\rho_{t}^{(i)}\leqslant B$ is a data-specific requirement, while classical composition results consider all hypothetical datasets. In Section 4.1 we will show how Corollary 3.3 can be operationalized.

It is worth mentioning that Corollary 3.3 also holds under the per-instance notion of individual privacy (Definition 2.6). This result is obtained by simply taking ${\cal S}=\{(S,S^{-i})\}$ in the proof, where $S$ is the analyzed dataset. However, our main application of Corollary 3.3—individual privacy filtering, stated in the following section—requires individual privacy loss accounting according to Definition 2.5.

Rényi privacy filter

We now show that Theorem 3.1 immediately implies a simple RDP analogue of a privacy filter. Specifically, we show that by simply adding up privacy parameters, as in the usual composition where all privacy parameters are fixed up front, we obtain a filter for RDP. Our individual privacy accounting method can naturally be seen as a privacy filter applied to each data point individually. We formalize this in Section 4.1. Then, in Section 4.3, we show that existing conversions of RDP guarantees to DP guarantees imply a new filter for $(\varepsilon,\delta)$-DP that both simplifies and improves on the results of Rogers et al. .

We now define an RDP filter formally. This general definition is used primarily to explain the relationship of our results to the notions and results in . Our individual privacy filtering application can be derived from Theorem 3.1 directly.

As in equation (1), we let $\rho_{t}$ denote the possibly random RDP parameter of order $\alpha$ of ${\cal A}_{t}$, conditional on the past reports. We again assume an implicit space ${\cal S}$ over pairs of datasets, which we instantiate in different ways depending on the privacy accounting method.

As a remark, although in Algorithm 2 we write “compute $\rho_{k+1}$”, sometimes the exact computation is infeasible and we actually only require the computed quantity to be an upper bound on the exact divergence. We suppress this distinction for readability. The same remark applies to other algorithm displays in the paper.

Let $S_{\infty}$ denote the set of all positive, real-valued finite sequences.

We remark that the analyst might choose an algorithm at time $t$ that exceeds the privacy budget, which will trigger the filter ${\cal F}_{\alpha,B}$ to halt. However, the analyst can then decide to change the computation at time $t$ retroactively and query the filter again, which then might allow continuation. This way, one can ensure a sequence of $N$ computations with formal privacy guarantees, for any target number of rounds $N$. In the following subsection, we present an application of RDP filters to individual privacy loss accounting which relies on this reasoning.

Then, ${\cal F}_{\alpha,B}$ is a valid Rényi privacy filter.

The only difference between Theorem 3.1 and this theorem is that a privacy filter halts at a random round, meaning the length of the output is random rather than fixed. Therefore, in this proof we formalize the fact that Theorem 3.1 is valid even under adaptive stopping.

Let $T$ be the last round before Algorithm 2 halts; that is,

Note that $T$ is a stopping time with respect to $\Gamma_{t}$, that is $\{T=t\}\in\Gamma_{t}$, due to the fact that $\rho_{t+1}\in\Gamma_{t}$. Since $T$ is almost surely bounded by construction, we can apply the optional stopping theorem for supermartingales to get

By definition of the RDP filter, we know that $\sum_{j=1}^{T}\rho_{j}\leqslant B$ almost surely; otherwise the filter would have halted earlier. Thus, we can conclude

After rearranging and normalizing, this implies

The same argument can be used to bound the other direction of the divergence. Therefore, ${\cal F}_{\alpha,B}$ is a valid RDP filter. ∎

We also remark that Rogers et al. define a privacy filter somewhat more generally, by treating the analyst as an adversary who is allowed to pick “bad” neighboring datasets at every step (see Algorithm 2 in ). Theorem 4.3 holds under this setting as well, however we opted for a simpler presentation.

Just like Corollary 3.3 applies Theorem 3.1 in the individual privacy setting, we can apply Rényi privacy filters to individual RDP parameters, in which case the filter indicates whether the privacy loss of a specific individual is potentially violated.

Now we design an individual privacy filter, which monitors individual privacy loss estimates across all individuals and all computations, and ensures that the privacy of all individuals is preserved. The filter guarantees privacy by adaptively dropping data points once their cumulative individual privacy loss estimate is about to cross a pre-specified budget. More specifically, at every step of adaptive composition $t$, it determines an active set of points $S_{t}\subseteq S$ based on cumulative estimated individual losses, and applies ${\cal A}_{t}$ only to $S_{t}$.

Here, ${\cal F}_{\alpha,B}$ is the Rényi privacy filter from Theorem 4.3. Given its validity, one can observe that Algorithm 3 preserves Rényi differential privacy.

Adaptive composition with individual privacy filtering (Algorithm 3) satisfies $(\alpha,B)$-Rényi differential privacy.

We argue that the privacy loss of point $X_{i}$ in round $t$, conditional on the past reports, is upper bounded by $\rho_{t}^{(i)}$ (after $\rho_{t}^{(i)}$ has been updated):

To do so, we reason about the active set of points at time $t$ when the input to adaptive composition is $S$, and when the input is $S^{-i}$. Denote by $S_{t}$ the active set given input $S$, and by $S_{t}^{(i)}$ the active set given input $S^{-i}$. Observe that, conditional on $a_{1},\dots,a_{t-1}$, we have $(S_{t},S_{t}^{(i)})\in{\cal S}(X_{i},n)$; that is, $S_{t}$ and $S_{t}^{(i)}$ differ only in the presence on $X_{i}$. This follows because the sequence $(\rho_{j}^{(i)})_{j=1}^{t}$ is measurable with respect to $a_{1},\dots,a_{t-1}$, and whether point $X_{i}$ is active at time $t$ is in turn determined based only on $(\rho_{j}^{(i)})_{j=1}^{t}$. In particular, whether any given point is active does not depend on the rest of the input dataset ($S$ or $S^{-i}$), given $a_{1},\dots,a_{t-1}$. Therefore, if $X_{i}\not\in S_{t}$, then $X_{i}$ loses no privacy in round $t$, because ${\cal A}^{\text{filt}}_{t}(a_{1},\dots,a_{t-1},S)\stackrel{{\scriptstyle d}}{{=}}{\cal A}^{\text{filt}}_{t}(a_{1},\dots,a_{t-1},S^{-i})$, conditional on $a_{1},\dots,a_{t-1}$. On the other hand, if $X_{i}\in S_{t}$, then its privacy loss can be bounded as

With this, we have showed that $\rho_{t}^{(i)}$ is a valid estimate of the privacy loss of $X_{i}$, for all $i\in[n]$.

and since this holds for all $S$ and all $i\in[n]$, we conclude that Algorithm 3 is $(\alpha,B)$-RDP. ∎

We can now justify the use of a supremum over all datasets that include $X$ in Definition 2.5 and, in particular, why the per-instance notion in Definition 2.6 does not suffice. By the current design of Algorithm 3, $X_{i}\not\in S_{t}$ implies no privacy loss for point $X_{i}$. This is true because, conditional on $a_{1},\dots,a_{t-1}$, $X_{i}$ being inactive ensures that $S_{t}$ would be the same regardless of whether the input to Algorithm 3 is $S$ or $S^{-i}$. Consequently, the output at time $t$ would be insensitive to the value of $X_{i}$. Under the more fine-grained definition of individual privacy, even if $X_{i}\not\in S_{t}$, its privacy could still leak at round $t$. The reason is that, under two different inputs $S$ and $S^{-i}$, the running privacy loss estimates for all points are different, and hence the active set $S_{t}$ in the two hypothetical scenarios could be different as well. This fact, in turn, implies two different distributions over reports $a_{t}$. In short, dropping $X_{i}$ from the analysis does not prevent its further privacy leakage if accounting is done according to Definition 2.6.

Algorithm 3 can naturally be applied to privacy accounting in the local differential privacy model. Here, each user would have a local implementation of the individual privacy filter and would stop responding when the filter halts. This is possible because the decision to halt for any given data point does not depend on the other data points other than through the sequence of reports $a_{1},\dots,a_{t}$.

In Section 5, we apply the individual privacy filter to differentially private optimization via gradient descent, and demonstrate how this object ensures utilization of data points as long as their realized gradients have low norm.

2 Answering linear queries

Corollary 4.7 follows from Theorem 4.5, by setting each ${\cal A}_{t}$ to be the Gaussian mechanism with $\sigma^{2}=B_{\text{norm}}/(2\kappa)$. Note that, due to $\rho_{t}^{(i)}\leqslant\rho_{t}$, all points are active in $S_{t}$ for at least the first $k_{0}$ computations, as prescribed by the usual worst-case analysis, and during those $k_{0}$ steps the answers are guaranteed to be accurate. Therefore, individual privacy provides a more fine-grained way of quantifying privacy loss by taking into account the value of the point whose loss we aim to measure. While $k$ is technically allowed to be arbitrarily large, after a certain number of reports we expect few points to remain active; we discuss stopping criteria in the following section.

3 (ε,δ)𝜀𝛿(\varepsilon,\delta)-differential privacy filter via Rényi filter

By connections between Rényi differential privacy and approximate differential privacy , we can translate our Rényi privacy filter into a filter for approximate differential privacy.

We define a valid DP filter analogously to a valid RDP filter, the difference being that it takes as input DP, rather than RDP parameters, and that it is parameterized by a global DP budget $\varepsilon_{g}\geqslant 0,\delta_{g}\in(0,1)$. We denote by $\varepsilon_{t}$ the possibly adaptive differential privacy parameter of ${\cal A}_{t}$:

Here, $D_{\infty}$ denotes the max-divergence, obtained as the limit of Rényi divergence of order $\alpha$ by taking $\alpha\rightarrow\infty$.

We focus on advanced composition of pure differentially private algorithms ${\cal A}_{t}$. As shown in , a DP filter for approximately differentially private algorithms ${\cal A}_{t}$ can be obtained by an immediate extension of a filter for pure DP algorithms.

As before, $S_{\infty}$ denotes the set of all positive, real-valued finite sequences.

We note that Rogers et al. define a differential privacy filter somewhat differently—for example, in their definition a filter admits a sequence of privacy parameters of fixed length—however Definition 4.8 is essentially equivalent to theirs.

By invoking standard conversions between DP and Rényi-divergence-based privacy notions, our analysis implies a simple stopping condition for a DP filter, in terms of any zero-concentrated differential privacy (zCDP) level which ensures $(\varepsilon_{g},\delta_{g})$-DP. For clarity, we give one particularly simple such translation from zCDP to DP, however one could in principle invoke more sophisticated analyses such as those of Bun and Steinke . In general, we can reproduce any rate for advanced composition of DP that utilizes Rényi-divergence-based privacy definitions, in the setting of fully adaptive composition.

Let $B^{\star}$ be the largest $B>0$ such that $B$-zero-concentrated differential privacy (zCDP) implies $(\varepsilon_{g},\delta_{g})$-differential privacy. Let

Then, ${\cal G}_{\varepsilon_{g},\delta_{g}}$ is a valid DP filter. For example,

By conversions between DP and zCDP , we know that $\varepsilon_{t}$-DP implies $\frac{1}{2}\varepsilon_{t}^{2}$-zCDP, that is $(\alpha,\frac{1}{2}\varepsilon_{t}^{2}\alpha)$-RDP, for all $\alpha\geqslant 1$. Thus, a Rényi filter with parameters $(\alpha,\alpha B^{\star})$ would stop once $\frac{1}{2}\sum_{t=1}^{k}\varepsilon_{t}^{2}>B^{\star}$. Since this condition is independent of $\alpha$, the output of adaptive composition with this stopping condition satisfies $(\alpha,B^{\star})$-RDP for all $\alpha\geqslant 1$. This guarantee is equivalent to $B^{\star}$-zCDP, and by assumption this implies $(\varepsilon_{g},\delta_{g})$-DP as well.

By Fact 2.4, $B^{\star}$-zCDP implies $\left(\min_{\alpha}\alpha B^{\star}+\frac{\log(1/\delta_{g})}{\alpha-1},\delta_{g}\right)$-DP. Optimizing over $\alpha$ and solving for $B^{\star}$ such that $\min_{\alpha}\alpha B^{\star}+\frac{\log(1/\delta_{g})}{\alpha-1}=\varepsilon_{g}$ yields $B^{\star}=\left(-\sqrt{\log(1/\delta_{g})}+\sqrt{\log(1/\delta_{g})+\varepsilon_{g}}\right)^{2}$. ∎

If the privacy parameters are fixed up front and $\varepsilon_{t}\equiv\varepsilon$, simplifying the stopping criterion of the above DP filter implies that adaptive composition of $k$ $\varepsilon$-differentially private algorithm satisfies

for all $\delta>0$. This tightens the rate of Rogers et al. , whose filter halts when

where $C(\varepsilon_{g},\delta_{g})=\frac{\varepsilon_{g}^{2}}{28.04\log(1/\delta_{g})}$. The factor $C(\varepsilon_{g},\delta_{g})$ essentially determines the gap between our analysis and the analysis of Rogers et al., and our filter is noticeably tighter for non-negligible values of $C(\varepsilon_{g},\delta_{g})$. In addition, our filter has an arguably simpler stopping criterion.

Further improvements on the rate are possible via a more intricate conversion between zCDP and DP, as presented in .

4 Tracking privacy loss via multiple filters

We denote by $\rho_{t}$ the RDP parameter of order $\alpha$ of ${\cal A}_{t}$ conditional on the past reports, as in equation (1).

The algorithm ${\cal A}^{(T_{m})}$ can be written as an adaptive composition of $m$ algorithms, each of which outputs $(a_{T_{j-1}+1},\dots,a_{T_{j}})$, $j\in\{1,\dots,m\}$. Therefore, by the standard adaptive composition theorem for RDP, it suffices to argue that each of these $m$ algorithms is RDP, conditional on the outputs of the previous algorithms. Since ${\cal F}_{\alpha,\Delta}$ is a valid Rényi privacy filter by Theorem 4.3, each of these $m$ algorithms is indeed $(\alpha,\Delta)$-RDP, which completes the proof. ∎

Notice that Proposition 4.10 immediately implies that Algorithm 5 is also valid for any $T$ such that $T_{m-1}\leqslant T\leqslant T_{m}$, since $(a_{1},\dots,a_{T})$ is a post-processing of $(a_{1},\dots,a_{T_{m}})$.

In the context of individual privacy, Proposition 4.10 allows designing a personalized privacy tracker for all analyzed data points. Here, we track $O_{t}^{(i)}$ for all points $X_{i}\in S$. The update is analogous to that of Algorithm 5, the difference being that a separate privacy filter is applied to the individual privacy parameters for all points separately. Naturally, each data point has its own random times of filter exceedances, formally defined as

Here, $\rho_{t}^{(i)}$ are individual privacy parameters, measured according to equation (2).

It is worth pointing out that the individual values $O_{t}^{(i)}$ are sensitive, as they depend on the value of the data point $X_{i}$. Importantly, they can be disclosed to the respective user without violating the other users’ privacy; $O_{t}^{(i)}$ depends on $X_{i}$, but it does not depend on the other data points (other than through $a_{1},\dots,a_{t-1}$, which are reported in a privacy-preserving manner).

Below we state an immediate corollary of Proposition 4.10.

Moreover, the same guarantee holds if accounting is done according to the per-instance notion of individual privacy (Definition 2.6). In that case, however, the values $O_{t}^{(i)}$ depend on the whole dataset $S$, and not just $X_{i}$. Consequently, reporting these values to users, without violating the other users’ privacy, would require greater care.

Private gradient descent with individual privacy accounting

We discuss an application of the individual privacy filter from Section 4 to differentially private optimization.

A popular approach to differentially private model training via gradient descent is to clip the norm of individual gradients at every time step and add Gaussian noise to the clipped gradients . Existing analyses compute the overall privacy spent up to a given round by using a uniform upper bound on the gradient norms, determined by the clipping value. Using the individual privacy filter from Section 4, we develop a less conservative version of private gradient descent, one which takes into account the realized norms of the gradients, rather than just their upper bound.

In Algorithm 6, all gradients get clipped to have norm at most $C$ at every time step. In Algorithm 7, the gradient for point $X_{i}$ gets clipped to have norm at most $\min\left(C,\sqrt{B_{\text{norm}}-\sum_{j=1}^{t-1}\|\bar{g}_{j}(X_{i})\|_{2}^{2}}\right)$. This means that, at least for the first $\lfloor B_{\text{norm}}/C^{2}\rfloor$ rounds, all gradients get clipped to have norm at most $C$. After round $\lfloor B_{\text{norm}}/C^{2}\rfloor$, points adaptively get filtered out once the accumulated squared norm of their (clipped) gradients reaches $B_{\text{norm}}$. Therefore, we observe that for $B_{\text{norm}}=kC^{2}$ and $k_{\max}=k$, Algorithm 7 recovers Algorithm 6.

The standard privacy guarantees of Algorithm 6 are given as follows.

Private gradient descent (Algorithm 6) satisfies $\frac{k}{2\sigma^{2}}$-zCDP, or, equivalently, $\left(\alpha,\frac{\alpha k}{2\sigma^{2}}\right)$-RDP for all $\alpha\geqslant 1$.

We prove the privacy guarantees of Algorithm 7 as a corollary of our individual privacy filter.

Private gradient descent with filtering (Algorithm 7) satisfies $\frac{B_{\text{norm}}}{2\sigma^{2}C^{2}}$-zCDP, or, equivalently, $\left(\alpha,\frac{\alpha B_{\text{norm}}}{2\sigma^{2}C^{2}}\right)$-RDP for all $\alpha\geqslant 1$.

When $B_{\text{norm}}=kC^{2}$, the privacy guarantees of Algorithm 7 are the same as those of Algorithm 6. However, they do not depend on the total number of steps $k_{\max}$—in particular, $k_{\max}$ need not be equal to $k$, which raises the question of how to set $k_{\max}$. (Certainly $k_{\max}$ should be at least $\lfloor B_{\text{norm}}/C^{2}\rfloor$, otherwise the privacy budget is not used up for any data point.) If $k_{\max}$ is relatively small, we might stop the optimization process too early, and thus forgo a potentially higher accuracy. If $k_{\max}$ is too large, then a lot of points might get filtered out and we might add high amounts of noise relative to the number of active points.

One solution is to periodically estimate the number of active points in a privacy-preserving fashion. In particular, after round $\lfloor B_{\text{norm}}/C^{2}\rfloor$, the analyst can estimate the size of the active set $\{i\ :\ \sum_{j=1}^{t}\|\bar{g}_{j}(X_{i})\|_{2}^{2}\leqslant B_{\text{norm}}\}$ (which is just a linear query) and use it to stop. To reduce the privacy cost of such estimates one can use the continual monitoring technique since each point is filtered out only once. Alternatively, if one only wants to ensure that the size of the active set exceeds some fixed threshold, one can use the sparse vector technique and thus incur an even smaller privacy loss due to adaptive stopping. Another solution is to periodically check the training accuracy, which is again a linear query, and stop once it plateaus.

As proof of concept, we compare the performance of private gradient descent (Algorithm 6) and its generalization with filtering (Algorithm 7) by training a convolutional neural network on MNIST . We use the default architecture from the MNIST example of the PyTorch Opacus library .

We remark that, in practice, it is more common to use private SGD, rather than batch GD. While in principle it is possible to compute individual privacy parameters for SGD, random subsampling of points requires computing gradients for all points at every step to observe gains from individual accounting. As a result, SGD is no less computationally expensive than batch GD in the context of individual accounting. Nevertheless, GD requires fewer steps and—importantly— we observe that it achieves a significantly better privacy-utility tradeoff. For example, using the same architecture, the Opacus library reports accuracy $(94.63\%\pm 0.34\%)$ for $\varepsilon=1.16$ and $\delta=10^{-5}$, which is almost the same accuracy we obtain with $\varepsilon=0.5$ and $\delta=10^{-5}$ (see table below). Recent large-scale experiments on differentially private training of language models similarly point to larger batches leading to higher utility, and we believe this phenomenon likely holds in many other settings and is worth further exploration.

All reported average accuracies and deviations are estimated over 10 trials. We fix the target differential privacy parameters $(\varepsilon,\delta)$, and evaluate the test accuracy. We set $\delta=10^{-5}$, and vary the value of $\varepsilon$. In the first set of evaluations, for every $\varepsilon$ we tune all algorithm hyperparameters to achieve high test accuracy with private gradient descent. For private gradient descent with filtering, to make the comparison as clear as possible, we adopt the same hyperparameters. After $B_{\text{norm}}/C^{2}$ steps, we query the training accuracy a fixed number of times in intervals of 5 steps and stop adaptively, when the training accuracy is observed to be the highest. We provide the specifics of the stopping rule and other experimental details in the Appendix.

Overall we observe modest accuracy improvements with individual filtering in the tuned regime. The benefits are more noticeable for small $\varepsilon$, while for large $\varepsilon$, the accuracy plateaus after $B_{\text{norm}}/C^{2}$ steps and hence we do not add extra steps. In Figure 1 we plot the number of active points, i.e. those that have not yet exhausted their privacy budget, for $\varepsilon\in\{0.3,0.5\}$. Due to extensive hyperparameter tuning in this specific application, private GD is implicitly tuned to clip the gradients in such a way that hard-to-classify points exhaust their privacy budget. Such tuning, however, is not possible when queries are chosen by human analysts (which is also hard to run experiments on) and in federated settings where data is held by the clients and finding the optimal setting of hyperparameters is typically infeasible. Therefore we examine the benefits of individual filtering in examples of such suboptimally tuned settings. For example, we evaluate the performance when the clipping value $C$ is chosen to be larger than in the optimal setting (while keeping the noise level the same). Specifically, for $\varepsilon\in\{0.3,0.5\}$, we set $C$ to be $1.5$ the optimally tuned value, and for $\varepsilon=1.0$ we set $C$ to be double the tuned value. We reduce the number of optimization steps accordingly to achieve the same privacy guarantee. In the suboptimal regime, the benefits of individual filtering become much more significant as a large fraction of points remain in the active pool after $B_{\text{norm}}/C^{2}$ steps.

Similar results are obtained when the noise rate is not set optimally. Here, we decrease $\sigma$ by a factor of $1.5$ for $\varepsilon\in\{0.3,0.5\}$ and by a factor of $2$ for $\varepsilon=1.0$. We adjust the number of optimization steps accordingly, and keep all other hyperparameters as in the tuned regime. As before, we observe benefits to performing additional steps, especially for small values of $\varepsilon$.

Altogether, we view this application as a useful proof of concept: it demonstrates that individual accounting is practical, easy to implement, and can only make the results better.

Acknowledgements

We thank Katrina Ligett, Kunal Talwar, and Neil Vexler for insightful discussions on individual notions of privacy and feedback on this work. We are grateful to Ryan McKenna for providing code and suggestions for the experiments.

References

Appendix A Experimental details

We train a convolutional neural network using the implementation of private gradient descent from the Opacus PyTorch library . We use the same architecture as in the MNIST example of the library. Since we run batch gradient descent and not SGD as in the library example, we tune all hyperparameters from scratch.

For $\varepsilon=0.3$, we set $\sigma=170$, $C=10$, $\eta_{t}\equiv\eta=0.2$, and $k=112$ for private GD without filtering. To achieve the same privacy guarantees using private GD with individual filtering, we set $B_{\text{norm}}=kC^{2}=11200$.

For $\varepsilon=0.5$, we set $\sigma=130$, $C=15$, $\eta_{t}\equiv\eta=0.15$, and $k=180$ for private GD without filtering. For GD with individual filtering, we set $B_{\text{norm}}=kC^{2}=40500$.

For $\varepsilon=1.0$, we set $\sigma=100$, $C=10$, $\eta_{t}\equiv\eta=0.2$, and $k=420$ for private GD without filtering. This parameter configuration achieves accuracy of $(96.25\pm 0.23)\%$. When private GD achieves such high accuracies, we observe little benefit to individual filtering. This is due to the fact that the proportion of points filtered out right after round $\lfloor B_{\text{norm}}/C^{2}\rfloor$ is comparable to the proportion of points yet misclassified, suggesting that few misclassified points remain in the active pool. Therefore, we set $B_{\text{norm}}=kC^{2}=42000$, and $k_{\max}=k$.

To apply the individual filter, after $kC^{2}$ steps we continue running gradient descent while adaptively dropping points when their privacy budget is exhausted. In particular, starting with step $\lfloor kC^{2}\rfloor$, we query the training accuracy 8 times in intervals of 5 steps (hence, the total number of additional steps is 35). As the final model we take the iterate when the queried training accuracy is highest.

Note that, technically, these additional queries should be reported in a privacy-preserving manner to formally ensure DP. However, these are simple linear queries that can be reported with high accuracy at little additional privacy cost. For $\varepsilon\in\{0.3,0.5\}$, it suffices to report the training accuracy with $1\%$ resolution. Eight such reports require a smaller privacy cost than, say, one or two additional optimization steps. For $\varepsilon=1.0$ it suffices to report the accuracy with $0.1\%$ resolution since the accuracy improvements after $kC^{2}$ steps are generally smaller for large $\varepsilon$. The additional reports for $\varepsilon=1.0$ have the cost of a few dozen extra steps. In either case, the privacy cost of the additional reports is less than $1\%$ of the intended privacy parameter.

In the suboptimal regime, we increase $C$ by a factor of $1.5$ for $\varepsilon\in\{0.3,0.5\}$ and accordingly decrease $\sigma$ by the same factor. The number of steps is similarly decreased by a factor of $1.5^{2}$. We do a similar adjustment for $\varepsilon=1.0$, where we increase $C$ by a factor of $2$. Similarly, when we decrease $\sigma$ by a factor of $1.5$ for $\varepsilon\in\{0.3,0.5\}$ and by a factor of $2$ for $\varepsilon=1.0$, we adjust the number of optimization steps accordingly, and keep all other hyperparameters as in the tuned regime.