Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity

Introduction

A frequent task in data analysis is the monitoring of the statistical properties of evolving data in a manner that requires repeated computation on the entire evolving dataset. Software applications commonly apply online monitoring, e.g., to establish trends in the software configurations or usage patterns. However, such monitoring may impact the privacy of software users, as it may directly or indirectly expose some of their sensitive attributes (e.g., their location, ethnicity, gender, etc.), either completely or partially. To address this, recent work has proposed a number of mechanisms that provide users with strong privacy-protection guarantees in terms of of differential privacy and, specifically, mechanisms that provide local differential privacy (LDP) have been deployed by Google, Apple, and Microsoft .

The popularity and practical adoption of LDP monitoring mechanisms stems largely from their simple trust model: for any single LDP report that a user contributes about one of their sensitive attributes, the user will benefit from strong differential privacy guarantees even if the user’s report becomes public and all other parties collude against them.

However, this apparent simplicity belies the realities of most monitoring applications. Software monitoring, in particular, near always involves repeated collection of reports over time, either on a regular basis or triggered by specific software activity; additionally, not just one, but multiple, software attributes may be monitored, and these attributes may all be correlated, as well as sensitive, and may also change in a correlated fashion. Hence, a user’s actual LDP privacy guarantees may be dramatically lower than they might appear, since LDP guarantees can be exponentially reduced by multiple correlated reports (see Tang et al. for a case study). Furthermore, lower accuracy is achieved by mechanisms that defend against such privacy erosion (e.g., the memoized backstop in Google’s RAPPOR ). Thus, to square this circle, and make good privacy/utility tradeoffs, practical deployments of privacy-preserving monitoring rely on additional assumptions—in particular, the assumption that each user’s reports are anonymous at each timestep and unlinkable over time.

In this work, we formalize how the addition of anonymity guarantees can improve differential-privacy protection. Our direct motivation is the Encode, Shuffle, Analyze (ESA) architecture and Prochlo implementation of Bittau et al. , which relies on an explicit intermediary that processes LDP reports from users to ensure their anonymity. The ESA architecture is designed to ensure a sufficient number of reports are collected at each timestep so that any one report can “hide in the crowd” and to ensure that those reports are randomly shuffled to eliminate any signal in their order. Furthermore, ESA will also ensure that reports are disassociated and stripped of any identifying metadata (such as originating IP addresses) to prevent the linking of any two reports to a single user, whether over time or within the collection at one timestep. Intuitively, the above steps taken to preserve anonymity will greatly increase the uncertainty in the analysis of users’ reports; however, when introducing ESA, Bittau et al. did not show how that uncertainty could be utilized to provide a tighter upper bound on the worst-case privacy loss.

Improving on this, this paper derives results that account for the benefits of anonymity to provide stronger differential privacy bounds. First, inspired by differential privacy under continual observation, we describe an algorithm for high-accuracy online monitoring of users’ data in the LDP model whose total privacy cost is polylogarithmic in the number of changes to each user’s value. This algorithm shows how LDP guarantees can be established in online monitoring, even when users report repeatedly, over multiple timesteps, and whether they report on the same value, highly-correlated values, or independently-drawn values.

Second, and more fundamentally, we show how—when each report is properly anonymized—any collection of LDP reports (like those at each timestep of our algorithm above) with sufficient privacy ($\varepsilon<1$) is actually subject to much stronger privacy guarantees in the central model of differential privacy. This improved worst-case privacy guarantee is a direct result of the uncertainty induced by anonymity, which can prevent reports from any single user from being singled out or linked together, whether in the set of reports at each timestep, or over time.

Differential privacy is a quantifiable measure of the stability of the output of a randomized mechanism in the face of changes to its input data—specifically, when the input from any single user is changed. (See Section 2 for a formal definition.)

In the local differential privacy model, formally introduced by Kasiviswanathan et al. , the randomized mechanism’s output is the transcript of the entire interaction between a specific user and a data collector (e.g., a monitoring system). Even if a user arbitrarily changes their privately held data, local differential privacy guarantees will ensure the stability of the distribution of all possible transcripts. Randomized response, a disclosure control technique from the 1960s , is a particularly simple technique for designing single-round LDP mechanisms. Due to their attractive trust model, LDP mechanisms have recently received significant industrial adoption for the privacy-preserving collection of heavy hitters , as well as increased academic attention .

Anonymous data collection.

As a pragmatic means for reducing privacy risks, reports are typically anonymized and often aggregated in deployments of monitoring by careful operators (e.g., RAPPOR )—even though anonymity is no privacy panacea .

To guarantee anonymity of reports, multiple mechanisms have been developed, Many, like Tor , are based on the ideas of cryptographic onion routing or mixnets, often trading off latency to offer much stronger guarantees . Some, like those of Prochlo , are based on oblivious shuffling, with trusted hardware and attestation used to increase assurance. Others make use of the techniques of secure multi-party computation, and can simultaneously aggregate reports and ensure their anonymity . Which of these mechanisms is best used in practice is dictated by what trust model and assumptions apply to any specific deployment.

Central differential privacy.

The traditional, central model of differential privacy applies to a centrally-held dataset for which privacy is enforced by a trusted curator that mediates upon queries posed on the dataset by an untrusted analyst—with curators achieving differential privacy by adding uncertainty (e.g., random noise) to the answers for analysts’ queries. For differential privacy, answers to queries need only be stable with respect to changes in the data of a single user (or a few users); these may constitute only a small fraction of the whole, central dataset, which can greatly facilitate the establishment of differential privacy guarantees. Therefore, the central model can offer much better privacy/utility tradeoffs than the LDP setting. (In certain cases, the noise introduced by the curator may even be less than the uncertainty due to population sampling.)

Longitudinal privacy.

Online monitoring with privacy was formalized by Dwork et al. as the problem of differential privacy under continual observation . That work proposed a privacy-preserving mechanisms in the central model of differential privacy, later extended and applied by Chan et al. and Jain et al. .

Continual observations constitute a powerful attack vector. For example, Calandrino et al. describe an attack on a collaborative-based recommender system via passive measurements that effectively utilizes differencing between a sequence of updates.

In the local model, Google’s RAPPOR proposed a novel memoization approach as a backstop against privacy erosion over time: A noisy answer is memorized by the user and repeated in response to the same queries about a data value. To avoid creating a trackable identifier, RAPPOR additionally randomizes those responses, which may only improve privacy. (See Ding et al. for alternative approach to memoization.) Although memoization prevents a single data value from ever being fully exposed, over time the privacy guarantees will weaken if answers are given about correlated data or sequences of data values that change in a non-independent fashion.

More recently, Tang et al. performed a detailed analysis of one real-world randomized response mechanisms and examined its longitudinal privacy implications.

2 Our contributions

Motivated by the gap in accuracy between central and local differential privacy under continual observations, we describe a general technique for obtaining strong central differential privacy guarantees from (relatively) weak privacy guarantees in the local model. Specifically, our main technical contribution demonstrates that random shuffling of data points ensures that the reports from any LDP protocol will also satisfy central differential privacy at a per-report privacy-cost bound that is a factor $\sqrt{n}$ lower than the LDP privacy bound established in the local model. Here, $n$ is the total number of reports, which can reach into the billions in practical deployments; therefore, the privacy amplification can be truly significant.

An immediate corollary of our amplification result is that composing client-side local differential privacy with server-side shuffling allows one to claim strong central differential privacy guarantees without any explicit server-side noise addition.

For this corollary to hold, the LDP reports must be amenable to anonymization via shuffling: the reports cannot have any discriminating characteristics and must, in particular, all utilize the same local randomizer (since the distribution of random values may be revealing). However, even if this assumption holds only partially—e.g., due to faults, software mistakes, or adversarial control of some reports—the guarantees degrade gracefully. Each set of $n^{\prime}$ users for which the corollary is applicable (e.g., that utilize the same local randomizer) will still be guaranteed a factor $\sqrt{n^{\prime}}$ reduction in their worst-case privacy cost in the central model. (See Section 4.1 for a more detailed discussion.)

It is instructive to compare our technique with privacy amplification by subsampling . As in the case of subsampling, we rely on the secrecy of the samples that are used in nested computations. However, unlike subsampling, shuffling, by itself, does not offer any differential privacy guarantees. Yet its combination with a locally differentially private mechanism has an effect that is essentially as strong as that achieved via known applications of subsampling . An important advantage of our reduction over subsampling is that it can include all the data reports (exactly once) and hence need not modify the underlying statistics of the dataset.

In concurrent and independent work, Cheu et al. formally define an augmented local model of differential privacy that includes an anonymizing shuffler and examine the sample complexity of solving several problems in this model. In this model they demonstrate privacy amplification for a fixed one-bit randomized response mechanism. The analysis in this simple case reduces to a direct estimation of $(\varepsilon,\delta)$-divergence between two binomial distributions. This case was also the starting point of our work. Importantly, in the binary case the bound is $O(\min\{1,\varepsilon_{0}\}e^{\varepsilon_{0}/2}\sqrt{\log(1/\delta)/n}$), where $\varepsilon_{0}$ is the privacy of an individual’s randomized response. This bound is asymptotically the same as our general bound of $O((e^{\varepsilon_{0}}-1)e^{2\varepsilon_{0}}\sqrt{\log(1/\delta)/n})$ when $\varepsilon_{0}\leq 1$ but is stronger than the general bound when $\varepsilon_{0}>1$. Finding stronger bounds for general local randomizers in the $\varepsilon_{0}>1$ regime is a natural direction for future work.

color=blue!40]VF: It seems worth including this analysis in our paper. It has some advantages and we did derive it independently.

We also remark that another recent privacy amplification technique, via contractive iteration relies on additional properties of the algorithm and is not directly comparable to results in this work.

Lower bounds in the local model.

Our amplification result can be viewed, conversely, as giving a lower bound for the local model. Specifically, our reduction means that lower bounds in the central model translate—with a $\Omega(\sqrt{n})$ penalty factor in the privacy parameter—to a large class of local protocols. In particular, this suggests that our results in the local model are near-optimal unless the corresponding results in the central model can be improved.

LDP monitoring with longitudinal privacy guarantees.

We introduce an online monitoring protocol that guarantees longitudinal privacy to users that report over multiple timesteps, irrespective of whether their reports are about independent or correlated values. By utilizing our protocol, users need not worry about revealing too much over time, and if they are anonymous, their reports may additionally “hide in the crowd” and benefit by amplification-by-shuffling, at least at each timestep.

As a motivating task, we can consider the collection of global statistics from users’ mobile devices, e.g., about users’ adoption of software apps or the frequency of users’ international long-distance travel. This task is a natural fit for a continual-observations protocol with LDP guarantees—since both software use and travel can be highly privacy sensitive—and can be reduced to collecting a boolean value from each user (e.g., whether they are in a country far from home). However, our protocol can be extended to the collection of multi-valued data or even data strings by building on existing techniques .

Concretely, we consider the collection of user statistics across $d$ time periods (e.g., for $d$ days) with each user changing their underlying boolean value at most $k$ times for some $k\leq d$. This is the only assumption we place on the data collection task. For software adoption and international travel, the limit on the number of changes is quite natural. New software is adopted, and then (perhaps) discarded, with only moderate frequency; similarly, speed and distance limit the change rate for even the most ardent travelers. Formally, we show the following: Under the assumption stated above, one can estimate all the $d$ frequency statistics from $n$ users with error at most $O((\log d)^{2}k\sqrt{n}/\varepsilon)$ in the local differential privacy model.

Motivated by similar issues, in a recent work Joseph et al. consider the problem of tracking a distribution that changes only a small number of times. Specifically, in their setting each user at each time step $t$ receives a random and independent sample from a distribution $P_{t}$. It is assumed that $P_{t}$ changes at most $k$ times and they provide utility guarantees that scale logarithmically with the number of time steps. The key difference between this setting and ours is the assumption of independence of each user’s inputs across time steps. Under this assumption even a fixed unbiased coin flip would result in each user receiving values that change in most of the steps. Therefore the actual problems addressed in this work are unrelated to ours and we rely on different algorithmic techniques.

3 Organization of the paper

Section 2 introduces our notation and recalls the definition of differential privacy. Section 3 provides an algorithm for collecting statistics and proves its accuracy and privacy in the local model under continual observations. Section 4 contains the derivation of our amplification-by-shuffling result. Section 5 concludes with a discussion.

Technical Preliminaries and Background

Differential privacy: The notion of differential privacy was introduced by Dwork et al. . We are using the (standard) relaxation of the definition that allows for an additive term $\delta$.

A randomized algorithm $\mathcal{A}\colon\mathcal{D}^{n}\to\mathcal{S}$ satisfies $(\varepsilon,\delta)$-differential privacy (DP) if for all $S\subset\mathcal{S}$ and for all adjacent $D,D^{\prime}\in\mathcal{D}$ it holds that

The notion of adjacent inputs is application-dependent, and it is typically taken to mean that $D$ and $D^{\prime}$ differ in one of the $n$ elements (that corresponds to the contributions of a single individual). We will also say that an algorithm satisfies differential privacy at index $1$ if the guarantees hold only for datasets that differ in the element at index $i$. We assume the $\varepsilon$ parameter to be a small constant, and $\delta$ is set to be much smaller than $1/|D|$. We repeatedly use the (advanced) composition property of differential privacy.

If $\mathcal{A}_{1},\ldots,\mathcal{A}_{k}$ are randomized algorithms satisfying $(\varepsilon,\delta)$-DP, then their composition, defined as $\left(\mathcal{A}_{1}(D),\ldots,\mathcal{A}_{k}(D)\right)$ for $D\in\mathcal{D}$ satisfies $(\varepsilon^{\prime},k\delta+\delta^{\prime})$ differential privacy where $\varepsilon^{\prime}=\varepsilon\sqrt{2k\log(1/\delta^{\prime})}+k\varepsilon(\exp(\varepsilon)-1)$. Moreover, $\mathcal{A}_{i}$ can be chosen adaptively depending on the outputs of $\mathcal{A}_{1},\ldots,\mathcal{A}_{i-1}$.

A straightforward corollary implies that for $\varepsilon,\varepsilon^{\prime}<1$, a $k$-fold composition of $(\varepsilon,\delta)$-DP algorithms leads to an $O(\sqrt{k\log(1/\delta)})$ overhead, i.e., $\varepsilon^{\prime}=2\varepsilon\sqrt{2k\log(1/\delta)}$.

It will also be convenient to work with the notion of distance between distributions on which $(\varepsilon,\delta)$-DP is based more directly. We define it below and describe some of the properties we will use. Given two distributions $\mu$ and $\mu^{\prime}$, we will say that they are $(\varepsilon,\delta)$-DP close, denoted by $\mu\approxeq_{(\varepsilon,\delta)}\mu^{\prime}$, if for all measurable $A$, we have

For random variables $X,X^{\prime}$, we write $X\approxeq_{(\varepsilon,\delta)}X^{\prime}$ to mean that their corresponding distributions $\mu,\mu^{\prime}$ are $(\varepsilon,\delta)$-DP close. We use $X\approxeq X^{\prime}$ to mean that the random variables are identically distributed.

For distributions $\mu_{1},\mu_{2}$ and $a\in$, we write $a\mu_{1}+(1-a)\mu_{2}$ to denote the mixture distribution that samples from $\mu_{1}$ with probability $a$ and from $\mu_{2}$ with probability $(1-a)$. The following properties are well-known properties of $(\varepsilon,\delta)$-DP.

The notion of $(\varepsilon,\delta)$-DP satisfies the following properties:

Let $\mu\approxeq_{(\varepsilon,\delta)}\mu^{\prime}$. Then for $\varepsilon^{\prime}\geq\varepsilon$ and $\delta^{\prime}\geq\delta$, $\mu\approxeq_{(\varepsilon^{\prime},\delta^{\prime})}\mu^{\prime}$.

Let $\mu_{1}\approxeq_{(\varepsilon_{1},\delta_{1})}\mu_{2}$ and $\mu_{2}\approxeq_{(\varepsilon_{2},\delta_{2})}\mu_{3}$. Then $\mu_{1}\approxeq_{(\varepsilon_{1}+\varepsilon_{2},\delta_{1}+\delta_{2})}\mu_{3}$.

Let $\mu_{1}\approxeq_{(\varepsilon,\delta)}\mu^{\prime}_{1}$ and $\mu_{2}\approxeq_{(\varepsilon,\delta)}\mu^{\prime}_{2}$, then for any $a\in$, it holds that $(1-a)\mu_{1}+a\mu_{2}\approxeq_{(\varepsilon,\delta)}(1-a)\mu^{\prime}_{1}+a\mu^{\prime}_{2}$.

The following lemma is a reformulation of the standard privacy amplification-by-sampling result (with the tighter analysis from ).

Let $q<\frac{1}{2}$ and let $\mu_{0},\mu_{1}$ be distributions such that $\mu_{1}\approxeq_{(\varepsilon,\delta)}\mu_{0}$. For $\mu=(1-q)\mu_{0}+q\mu_{1}$, it holds that $\mu\approxeq_{(\varepsilon^{\prime},q\delta)}\mu_{0}$, where $\varepsilon^{\prime}=\log(q(e^{\varepsilon}-1)+1)\leq q(e^{\varepsilon}-1)$.

Locally Private Protocol for Longitudinal Data

Recall the motivation for collecting statistics from user devices with the intent of tracking global trends. We remain in the local differential privacy model, but our protocol addresses the task of collecting reports from users to derive global statistics that are expected to change across time. We consider the simplified task of collecting a boolean value from each user, e.g., their device being at an international location, far from the user’s home. However, our protocol can be straightforwardly extended to collecting richer data, such as strings, by building on existing techniques .

In what follows, we consider a natural model of collecting user statistics across time periods. We make two minimal assumptions: we are given the time horizon $d$, or the number of time periods (or days) ahead of time, and each user changes their underlying data at most $k\leq d$ times. The first assumption is mild: a loose upper bound on $d$ suffices, and the error depends only polylogarithmically on the upper bound. The second assumption can be enforced at the client side to ensure privacy, while suffering some loss in accuracy.

Our approach is inspired by the work on privacy under continual observations by Dwork et al. , who give a (central) DP mechanism to maintain a counter that is incrementally updated in response to certain user-driven events. Its (central) differential privacy is defined in respect to a single increment, the so-called event-level privacy. The naïve solution of applying additive noise to all partial sums introduces error $\Theta(\sqrt{d})$, proportional to the square root of the time horizon. The key algorithmic contribution of Dwork et al. is an elegant aggregation scheme that reduces the problem of releasing partial sums to the problem of maintaining a binary tree of counters. By carefully correlating noise across updates, they reduce the error to $O(\text{polylog}\,{d})$. (In related work Chan et al. describe a post-processing procedure that guarantees output consistency; Xiao et al. present a conceptually similar algorithm framed as a basis transformation.)

We adapt these techniques to the local setting by pushing the tree-of-counters to the client (Algorithm 1 below). The server aggregates clients’ reports and computes the totals (Algorithm 2).

We more formally define the problem of collecting global statistics based on reports from users’ devices. Given a time horizon $d$, we consider a population of $n$ users reporting a boolean value about their state at each time period $t\in[d]$. (Without loss of generality, we assume that $d$ is a power of 2.) Let $\vec{\mathsf{st}}_{i}=\left[\mathsf{st}_{i},\ldots,\mathsf{st}_{i}[d]\right]$ denote the states of the $i$’th user across the $d$ time periods with at most $k$ changes. The task of collecting statistics requires the server to compute the sum $\sum_{i\in[n]}\mathsf{st}_{i}[t]$ for every time periods $t$.

An online client-side algorithm for reporting statistics runs on each client device and produces an output for each time period. Correspondingly, the online server-side algorithm receives reports from $n$ clients and outputs estimates for the marginal $f_{t}$ at each time period $t$.

Outline.

To demonstrate the key techniques in the design of our algorithm, consider a version of the data collection task with every client’s data known ahead of time. Given $\vec{x}_{i}$ for user $i$, the client-side algorithm produces (up to) $d$ reports and the server computes estimates of the marginal sum $f_{t}$ for all $t\in[d]$. Our algorithm is based on the tree-based aggregation scheme used previously for releasing continual statistics over longitudinal data in the central model. Each client maintains a binary tree over the $d$ time steps to track the (up to) $k$ changes of their state. The binary tree ensures that each change affects only $\log_{2}d$ nodes of the tree. We extend the construction of Dwork et al. in a natural manner to the local model by having each client maintain and report values in this binary tree with sub-sampling as follows.

In the beginning, the client samples uniformly from $[k]$ the $\kappa^{*}$’th change they would like to report on. Changes other than the $\kappa^{*}$’th one are ignored. The client builds a tree with leaves corresponding to an index vector capturing the $\kappa^{*}$’th change ( everywhere except $\pm 1$ at the change). The rest of the nodes are populated with the sums of their respective subtrees. The client then chooses a random level of the tree to report on. Then, the client runs randomized response on each node of the selected level (with noise determined by $\varepsilon$) and reports the level of the tree along with the randomized response value for each node. In actual implementations and our presentation of the protocol (Algorithm 1) the tree is never explicitly constructed. The state maintained by the client consists of just four integer values ($\kappa^{*}$, the level of the tree, and two counters).

Notation.

The sequence of $d$ outputs of $\mathcal{A}_{\sf client}$ (Algorithm 1) satisfies $\varepsilon$-local differential privacy.

To show the local differential privacy of the outputs, we consider two aspects of the client’s reports: (1) the (randomized) values at the nodes output in Step 20, and (2) the timing of the report. The latter entirely depends on the choice of $h^{*}$ which is independent of the client’s underlying data record and hence does not affect the privacy analysis. Furthermore, $\kappa^{*}$ is also independently sampled and for the rest of the proof, we fix $\kappa^{*}$ and $h^{*}$ and focus only on the randomized values output in Step 20.

By construction, the client chooses only the $\kappa^{*}$’th change to report on. This would imply that two inputs would affect at most two nodes at level $h^{*}$ with each of the values changing by at most one.

This bound on the sensitivity of the report enables us to use standard arguments for randomized response mechanisms to Steps 17 and 18 to show that the noisy values $T[h^{*},1],\ldots,T[h^{*},H^{*}]$ satisfy $\varepsilon$-local differential privacy.

where $c_{\varepsilon}=\frac{e^{\varepsilon/2}+1}{e^{\varepsilon/2}-1}$.

Consider the set $C$ constructed in Steps 7–10 of Algorithm 2. We observe that $C$ satisfies the following properties:

The set $C$ is uniquely defined, i.e., it is independent of the order in which pairs of intervals are selected in the while loop.

The size of the set $|C|$ is at most $\log_{2}d$.

The leaf nodes of the subtrees $[h,i]$ in $C$ partition $[1,t]$.

The marginal counts $f_{t}$ totaling events in the interval $[1,t]$ can be computed by adding the corresponding $\log_{2}d$ nodes in the tree whose subtrees partition the interval $[1,t]$.

It follows that largest error in any $f_{t}$ is at most $\log_{2}d$ times the largest error in any of the subtree counts. We proceed to bound that error.

Scaling by $c^{\prime}_{\varepsilon}k\log_{2}d$, and multiplying by $\log_{2}d$ to account for the size of $C$, we conclude that except with probability $\beta$,

Privacy Amplification via Shuffling

Local differential privacy holds against a strictly more powerful adversary than central differential privacy, namely one that can examine all communications with a particular user. What can we say about central DP guarantees of a protocol that satisfies $\varepsilon$-local DP? Since local DP implies central DP, this procedure satisfies $\varepsilon$-central DP. Moreover, without making additional assumptions, we cannot make any stronger statement than that. Indeed, the central mechanism may release the entire transcripts of its communications with all its users annotated with their identities, leaving their local differential privacy as the only check on information leakage via the mechanism’s outputs.

To make the discussion more formal we define the following general form of locally differentially private algorithms that allows picking local randomizers sequentially on the basis of answers from the previous randomizers (Algorithm 3).

Our main amplification result is for Algorithm 4 that applies the local randomizers after randomly permuting the elements of the dataset. For algorithms that use a single fixed local randomizer it does not matter whether the data elements are shuffled before or after the application of the randomizer. We will discuss the corollaries of this result for algorithms that shuffle the responses later in this section.

For a domain $\mathcal{D}$, let $\mathcal{A}_{\mathsf{ldp}}^{(i)}\colon\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}\times\mathcal{D}\to\mathcal{S}^{(i)}$ for $i\in[n]$ (where $\mathcal{S}^{(i)}$ is the range space of $\mathcal{A}_{\mathsf{ldp}}^{(i)}$) be a sequence of algorithms such that $\mathcal{A}_{\mathsf{ldp}}^{(i)}$ is $\varepsilon_{0}$-differentially private for all values of auxiliary inputs in $\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}$. Let ${\mathcal{A}_{\mathsf{sl}}}\colon\mathcal{D}^{n}\to\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(n)}$ be the algorithm that given a dataset $x_{1:n}\in\mathcal{D}^{n}$, samples a uniform random permutation $\pi$ over $[n]$, then sequentially computes $z_{i}=\mathcal{A}_{\mathsf{ldp}}(z_{1:i-1},x_{\pi(i)})$ for $i=1,2,\ldots,n$ and outputs $z_{1:n}$ (see Algorithm 4). For any integer $n>1$, $\varepsilon_{0},\delta>0$, ${\mathcal{A}_{\mathsf{sl}}}$ satisfies $(\varepsilon,\delta)$-differential privacy in the central model, where $\varepsilon\leq\varepsilon_{1}\sqrt{2n\log(1/\delta)}+n\varepsilon_{1}(e^{\varepsilon_{1}}-1)$ for $\varepsilon_{1}=2e^{2\varepsilon_{0}}(e^{\varepsilon_{0}}-1)/n$. In particular, for $\varepsilon_{0}\leq\ln(n/4)/3$, $\varepsilon\leq e^{2\varepsilon_{0}}(e^{\varepsilon_{0}}-1)\sqrt{\frac{8\log(1/\delta)}{n}}+6e^{4\varepsilon_{0}}(e^{\varepsilon_{0}}-1)^{2}/n$ and for any $n\geq 1000$, $0<\varepsilon_{0}<1/2$ and $0<\delta<1/100$, $\varepsilon\leq 12\varepsilon_{0}\sqrt{\frac{\log(1/\delta)}{n}}$.

We remark that while we state our amplification result for local randomizers operating on a single data element, the result extends immediately to arbitrary $\varepsilon_{0}$-DP algorithms that operate on disjoint batches of data elements. Also note that for $\varepsilon_{0}>\ln(n/2)/3$, the resulting bound on $\varepsilon$ is larger than $\varepsilon_{0}$ and thus our bound does not provide any amplification.

A natural approach to proving this result is to use privacy amplification via subsampling (Lemma 4). At step $i$ in the algorithm, conditioned on the values of $\pi(1),\ldots,\pi(i-1)$, the random variable $\pi(i)$ is uniform over the remaining $(n-i+1)$ indices. Thus the $i$’th step will be $O((e^{\varepsilon_{0}}-1)/(n-i+1))$-DP. The obvious issue with this argument is that it gives very little amplification for values of $i$ that are close to $n$, falling short of our main goal. It also unclear how to formalize the intuition behind this argument.

Instead our approach crucially relies on a reduction to analysis of the algorithm ${\mathcal{A}_{\mathsf{swap}}}$ that swaps the first element in the dataset with a uniformly sampled element in the dataset before applying the local randomizers (Algorithm 5). We show that ${\mathcal{A}_{\mathsf{swap}}}$ has the desired privacy parameters for the first element (that is, satisfies the guarantees of differential privacy only for pairs of datasets that differ in the first element). We then show that for every index $i^{\ast}$, ${\mathcal{A}_{\mathsf{sl}}}$ can be decomposed into a random permutation that maps element $i^{\ast}$ to be the first element followed by ${\mathcal{A}_{\mathsf{swap}}}$. This implies that the algorithm ${\mathcal{A}_{\mathsf{sl}}}$ will satisfy differential privacy at $i^{\ast}$.

To argue about the privacy properties of the ${\mathcal{A}_{\mathsf{swap}}}$ we decompose it into a sequence of algorithms, each producing one output symbol (given the dataset and all the previous outputs). It is not hard to see that the output distribution of the $i$’th algorithm is a mixture $\mu=(1-p)\mu_{0}+p\mu_{1}$, where $\mu_{0}$ does not depend on $x_{1}$ (the first element of the dataset) and $\mu_{1}$ is the output distribution of the $i$’th local randomizer applied to $x_{1}$. We then demonstrate that the probability $p$ is upper bounded by $e^{2\varepsilon_{0}}/n$. Hence, using amplification by subsampling, we obtain that $\mu$ is close to $\mu_{0}$. The distribution $\mu_{0}$ does not depend on $x_{1}$ and therefore, by the same argument, $\mu_{0}$ (and hence $\mu$) is close to the output distribution of the $i$’th algorithm on any dataset $D^{\prime}$ that differs from $D$ only in the first element. Applying advanced composition to the sequence of algorithms gives the claim.

We now provide the full details of the proof starting with the description and analysis of ${\mathcal{A}_{\mathsf{swap}}}$.

For a domain $\mathcal{D}$, let $\mathcal{A}_{\mathsf{ldp}}^{(i)}\colon\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}\times\mathcal{D}\to\mathcal{S}^{(i)}$ for $i\in[n]$ (where $\mathcal{S}^{(i)}$ is the range space of $\mathcal{A}_{\mathsf{ldp}}^{(i)}$) be a sequence of algorithms such that $\mathcal{A}_{\mathsf{ldp}}^{(i)}$ is $\varepsilon_{0}$-differentially private for all values of auxiliary inputs in $\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}$. Let ${\mathcal{A}_{\mathsf{swap}}}\colon\mathcal{D}^{n}\to\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(n)}$ be the algorithm that given a dataset $D=x_{1:n}\in\mathcal{D}^{n}$, samples a uniform index $I\in[n]$, swaps element $1$ with element $I$ and then applies the local randomizers to the resulting dataset sequentially (see Algorithm 5). For any integer $n>1$, $\varepsilon_{0},\delta>0$, ${\mathcal{A}_{\mathsf{swap}}}$ satisfies $(\varepsilon,\delta)$-differential privacy at index $1$ in the central model, where $\varepsilon\leq\varepsilon_{1}\sqrt{2n\log(1/\delta)}+n\varepsilon_{1}(e^{\varepsilon_{1}}-1)$ for $\varepsilon_{1}=2e^{2\varepsilon_{0}}(e^{\varepsilon_{0}}-1)/n$.

The algorithm ${\mathcal{A}_{\mathsf{swap}}}$ defines a joint distribution between $I$ and the corresponding output sequence of ${\mathcal{A}_{\mathsf{swap}}}$, which we denote by $Z_{1},Z_{2},\ldots,Z_{n}$. We first observe that $Z_{1:n}$ can be seen as the output of a sequence of $n$ algorithms with conditionally independent randomness: $\mathcal{B}^{(i)}\colon\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}\times\mathcal{D}^{n}\to\mathcal{S}^{(i)}$ for $i\in[n]$. On input $s_{1:i-1}$ and $D$, $\mathcal{B}^{(i)}$ produces a random sample from the distribution of $Z_{i}$ conditioned on $Z_{1:i-1}=s_{1:i-1}$. The outputs of $\mathcal{B}^{(1)},\ldots,\mathcal{B}^{(i-1)}$ are given as the input to $\mathcal{B}^{(i)}$. By definition, this ensures that random bits used by $\mathcal{B}^{(i)}$ are independent of those used by $\mathcal{B}^{(1)},\ldots,\mathcal{B}^{(i-1)}$ conditioned on the previous outputs. Therefore, in order to upper bound the privacy parameters of ${\mathcal{A}_{\mathsf{swap}}}$, we can analyze the privacy parameters of $\mathcal{B}^{(1)},\ldots,\mathcal{B}^{(n)}$ and apply the advanced composition theorem for differential privacy (Theorem 2).

Next we observe that, by the definition of ${\mathcal{A}_{\mathsf{swap}}}$, conditioned on $I$, $Z_{i}$ is the output of $\mathcal{A}_{\mathsf{ldp}}^{(i)}(s_{1:i-1};x)$ with its internal randomness independent of $Z_{1:i-1}$ (where $x$ is the $i$-th element in $\sigma_{I}(D)$). In other words, for $i\geq 2$, $\mathcal{B}^{(i)}$ can equivalently be implemented as follows. First, sample an index $T$ from the distribution of $I$ conditioned on $Z_{1:i-1}=s_{1:i-1}$. Then, if $T=i$ output $\mathcal{A}_{\mathsf{ldp}}^{(i)}(s_{1:i-1};x_{1})$. Otherwise, output $\mathcal{A}_{\mathsf{ldp}}^{(i)}(s_{1:i-1},x_{i})$. To implement $\mathcal{B}^{(1)}$ we sample $T$ uniformly from $[n]$ and then output $\mathcal{A}_{\mathsf{ldp}}^{(1)}(x_{T})$.

We now prove that for every $i\in[n]$, $\mathcal{B}^{(i)}$ is $\left(2e^{\varepsilon_{0}}(e^{\varepsilon_{0}}-1)/n,0\right)$-differentially private at index $1$. Let $D=x_{1:n}$ and $D^{\prime}=(x^{\prime}_{1},x_{2:n})$ be two datasets that differ in the first element. Let $s_{1:i-1}$ denote the input to $\mathcal{B}^{(i)}$. We denote by $\mu$ the probability distribution of $\mathcal{B}^{(i)}(s_{1:i-1},D)$, denote by $\mu_{0}$ (or $\mu_{1}$) the probability distribution of $\mathcal{B}^{(i)}(s_{1:i-1},D)$ conditioned on $T\neq i$ ($T=i$, respectively) and by $p$ the probability that $T=i$ (where $T$ is sampled from the distribution of $I$ conditioned on $Z_{1:i-1}=s_{1:i-1}$ as described above). We also denote by $\mu^{\prime},\mu^{\prime}_{0},\mu^{\prime}_{1}$ and $p^{\prime}$ the corresponding quantities when $\mathcal{B}^{(i)}$ is run on $D^{\prime}$. By the definition, $\mu=(1-p)\mu_{0}+p\mu_{1}$ and $\mu^{\prime}=(1-p^{\prime})\mu^{\prime}_{0}+p^{\prime}\mu^{\prime}_{1}$.

For $i=1$, $T$ is uniform over $[n]$ and hence $p=p^{\prime}=1/n$. Further, $\mu_{0}$ is equal to $\mu^{\prime}_{0}$ (since both are equal to the output distribution of $\mathcal{A}_{\mathsf{ldp}}^{(1)}(x_{T})$ conditioned on $T\neq 1$). By $\varepsilon_{0}$-local differential privacy of $\mathcal{A}_{\mathsf{ldp}}^{(1)}$ and quasi-convexity of $(\varepsilon,\delta)$-DP we obtain that $\mu_{0}\approxeq_{(\varepsilon_{0},0)}\mu_{1}$. Therefore, privacy amplification by subsampling (Lemma 4) implies that $\mu_{0}\approxeq_{((e^{\varepsilon}_{0}-1)/n,0)}\mu$. Similarly, we obtain that $\mu^{\prime}_{0}\approxeq_{((e^{\varepsilon}_{0}-1)/n,0)}\mu^{\prime}$ and therefore, by the triangle inequality, $\mu\approxeq_{(2(e^{\varepsilon}_{0}-1)/n,0)}\mu^{\prime}$. In other words, $\mathcal{B}^{(1)}$ is $(2(e^{\varepsilon_{0}}-1)/n,0)$-differentially private at index $1$.

For $i\geq 2$, we again observe that $\mu_{0}=\mu^{\prime}_{0}$ since in both cases the output is generated by $\mathcal{A}_{\mathsf{ldp}}^{(i)}(s_{1:i-1};x_{i})$. Similarly, $\varepsilon_{0}$-local differential privacy of $\mathcal{A}_{\mathsf{ldp}}^{(i)}$ implies that $\mu_{0}\approxeq_{(\varepsilon_{0},0)}\mu_{1}$ and $\mu^{\prime}_{0}\approxeq_{(\varepsilon_{0},0)}\mu^{\prime}_{1}$.

We now claim that $p\leq e^{2\varepsilon_{0}}/n$. To see this, we first observe that the condition $Z_{1:i-1}=s_{1:i-1}$ is an event defined over the output space of ${\mathcal{A}_{\mathsf{local}}}$. Conditioning on $T=i$ reduces ${\mathcal{A}_{\mathsf{swap}}}$ to running ${\mathcal{A}_{\mathsf{local}}}$ on $\sigma_{i}(D)$. Note that for $j\neq i$, $\sigma_{i}(D)$ differs from $\sigma_{j}(D)$ in at most two positions. Therefore, by $\varepsilon_{0}$-differential privacy of ${\mathcal{A}_{\mathsf{local}}}$ and group privacy (e.g. ), we obtain that

By quasi-convexity of $(\varepsilon,\delta)$-DP we obtain that

Privacy amplification by sampling implies that $\mu_{0}\approxeq_{(e^{2\varepsilon_{0}}(e^{\varepsilon}_{0}-1)/n,0)}\mu$. Applying the same argument to $D^{\prime}$ and using the triangle inequality we get that $\mu\approxeq_{(2e^{2\varepsilon_{0}}(e^{\varepsilon}_{0}-1)/n,0)}\mu^{\prime}$.

Applying the advanced composition theorem for differential privacy with $\varepsilon_{1}=2e^{2\varepsilon_{0}}(e^{\varepsilon_{0}}-1)/n$ and $n$ steps we get that ${\mathcal{A}_{\mathsf{swap}}}$ satisfies $(\varepsilon,\delta)$-DP at index 1 for

Finally, we describe a reduction of the analysis of ${\mathcal{A}_{\mathsf{sl}}}$ to the analysis of ${\mathcal{A}_{\mathsf{swap}}}$.

Let $D$ and $D^{\prime}$ be two datasets of length $n$ that differ at some index $i^{\ast}\in[n]$. The algorithm ${\mathcal{A}_{\mathsf{sl}}}$ can be seen as follows. We first pick a random one-to-one mapping $\pi^{\ast}$ from $\{2,\ldots,n\}\to[n]\setminus\{i^{\ast}\}$ and let

Namely, we move $x_{i^{\ast}}$ to the first place and apply a random permutation to the remaining elements. In the second step we apply ${\mathcal{A}_{\mathsf{swap}}}$ to $\pi^{\ast}(D)$. It is easy to see that for a randomly and uniformly chosen $\pi^{\ast}$ and uniformly chosen $I\in[n]$ the distribution of $\sigma_{I}(\pi^{\ast}(D))$ is exactly a random and uniform permutation of elements in $D$.

For a fixed mapping $\pi^{\ast}$, the datasets $\pi^{\ast}(D)$ and $\pi^{\ast}(D^{\prime})$ differ only in the element with index $1$. Therefore ${\mathcal{A}_{\mathsf{swap}}}(\pi^{\ast}(D))\approxeq_{(\varepsilon,\delta)}{\mathcal{A}_{\mathsf{swap}}}(\pi^{\ast}(D^{\prime}))$ for $\varepsilon$ and $\delta$ given in Theorem 8. Using the quasi-convexity of $(\varepsilon,\delta)$-DP over a random choice of $\pi^{\ast}$ we obtain that ${\mathcal{A}_{\mathsf{sl}}}(D)\approxeq_{(\varepsilon,\delta)}{\mathcal{A}_{\mathsf{sl}}}(D^{\prime})$.

To obtain the bounds in the special cases we note that for $\varepsilon_{0}\leq\ln(n/4)/3$ we have that $\varepsilon_{1}\leq 2e^{3\varepsilon_{0}}/n\leq 1/2$ and thus $(e^{\varepsilon_{1}}-1)\leq 3\varepsilon_{1}/2$. Therefore

Further, for $\varepsilon_{0}\leq 1/2$ we get that $\varepsilon_{1}\leq 8\varepsilon_{0}/n$ and the first term

Using the fact that $n\geq 1000$ we have that $\varepsilon_{1}\leq 1/250$. This implies that $e^{\varepsilon_{1}}-1\leq\frac{65}{64}\varepsilon_{1}$ and using $n\geq 1000$ and $\delta\leq 1/100$, we get the following bound on the second term

Combining these terms gives the claimed bound.

Our bound can also be easily stated in terms of Renyi differential privacy (RDP) . Using the fact that $\varepsilon$-DP implies $(\alpha,\alpha\varepsilon^{2}/2)$-RDP and the composition properties of RDP , we obtain that for all $\alpha\geq 1$, the output of ${\mathcal{A}_{\mathsf{sl}}}$ satisfies $(\alpha,2\alpha e^{4\varepsilon_{0}}(e^{\varepsilon_{0}}-1)^{2}/n)$-RDP. Such results can be used to obtain tighter $(\varepsilon,\delta)$ guarantees for algorithms that produce multiple shuffled outputs.

The proof of Theorem 7 relies crucially on shuffling the data elements before applying the local randomizers. However implementing such an algorithm in a distributed system requires trusting a remote shuffler with sensitive user data, thus negating the key advantage of the LDP model. Conversely, even if shuffling is performed on a set of already-randomized LDP responses, no additional privacy guarantees will be achieved if some attribute of each report (e.g., the choice of a randomizer) can reveal the identity of the reporting user.

Fortunately, it is possible design software systems where reports are randomized before shuffling and in which the reports coming from large groups of users are indistinguishable, e.g., because they apply the same local randomizer. In such constructions, the privacy of each user’s report still have its privacy amplified, by a factor proportional to the square root of the cardinality of indistinguishable reports. This follows immediately from the fact that shuffling the responses from the same local randomizers is equivalent to first shuffling the data points and then applying the local randomizers.

We make this claim formal in the following corollary.

For a domain $\mathcal{D}$, let $\mathcal{A}_{\mathsf{ldp}}^{(i)}\colon\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}\times\mathcal{D}\to\mathcal{S}^{(i)}$ for $i\in[n]$ (where $\mathcal{S}^{(i)}$ is the range space of $\mathcal{A}_{\mathsf{ldp}}^{(i)}$) be a sequence of algorithms such that $\mathcal{A}_{\mathsf{ldp}}^{(i)}$ is $\varepsilon_{0}$-differentially private for all values of auxiliary inputs in $\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(i-1)}$. Let ${\mathcal{A}_{\mathsf{post}}}\colon\mathcal{D}^{n}\to\mathcal{S}^{(1)}\times\cdots\times\mathcal{S}^{(n)}$ be the algorithm that given a dataset $D\in\mathcal{D}^{n}$, computes $z_{1:n}={\mathcal{A}_{\mathsf{local}}}(D)$, samples a random and uniform permutation and outputs $z_{\pi(1)},\ldots,z_{\pi(n)}$. Let $S\subseteq[n]$ be any set of indices such that for all $i,j\in S$, $\mathcal{A}_{\mathsf{ldp}}^{(i)}\equiv\mathcal{A}_{\mathsf{ldp}}^{(j)}$. Then for $|S|\geq 1000$, $0<\varepsilon_{0}<1/2$ and $0<\delta<1/100$ and every $i\in S$, ${\mathcal{A}_{\mathsf{post}}}$ satisfies $(\varepsilon,\delta)$-differential privacy at index $i$ in the central model, where $\varepsilon=12\varepsilon_{0}\sqrt{\frac{\log(1/\delta)}{|S|}}$.

We note that for the conclusion of this corollary to hold it is not necessary to randomly permute all the $n$ randomized responses. It suffices to shuffle the elements of $S$. We also clarify that for $i<j$, by $\mathcal{A}_{\mathsf{ldp}}^{(i)}\equiv\mathcal{A}_{\mathsf{ldp}}^{(j)}$ we mean that for all sequences $z_{1:j-1}$ and $x\in\mathcal{D}$, the output distributions of $\mathcal{A}_{\mathsf{ldp}}^{(i)}(z_{1:i-1},x)$ and $\mathcal{A}_{\mathsf{ldp}}^{(j)}(z_{1:j-1},x)$ are identical (and, in particular, the output distribution $\mathcal{A}_{\mathsf{ldp}}^{(j)}$ does not depend on $z_{i:j-1}$).

2 Lower Bound for Local DP Protocols

The results of this section give us a natural and powerful way to prove lower bounds for protocols in the local differential privacy model. We can apply Theorem 7 in the reverse direction to roughly state that for any given problem, lower bounds on the error of $\Omega(\alpha/\varepsilon)$ (for some term $\alpha$ that might depend on the parameters of the system) of an $\varepsilon$-centrally differentially private protocol translate to a $\Omega(\alpha\sqrt{n}/\varepsilon)$ lower bound on the error of any $\varepsilon$-locally differentially private protocol of the kind that our techniques apply to.

As an exercise, a lower bound of $\Omega(\sqrt{k}\,\text{polylog}\,(d)/\varepsilon)$ for the problem of collecting frequency statistics from users across time in the central DP framework with privacy guarantee $\varepsilon$ directly implies that the result in Theorem 6 is tight. We note here that the results of Dwork et al. do show a lower bound of $\Omega(\log(d)/\varepsilon)$ for the setting when $k=1$ in the central DP framework. This strongly suggests that our bounds might be tight, but we cannot immediately use this lower bound as it is stated only for the pure $\varepsilon$-differential privacy regime. It is an open problem to extend these results to the approximate differential privacy regime.

Discussion and Future Work

Our amplification-by-shuffling result is encouraging, as it demonstrates that the formal guarantees of differential privacy can encompass intuitive privacy-enhancing techniques, such as the addition of anonymity, which are typically part of existing, best-practice privacy processes. By accounting for the uncertainty induced by anonymity, in the central differential privacy model the worst-case, per-user bound on privacy cost can be dramatically lowered.

Our result implies that industrial adoption of LDP-based mechanisms may have offered much stronger privacy guarantees than previously accounted for, since anonymization of telemetry reports is standard privacy practice in industry. This is gratifying, since the direct motivation for our work was to better understand the guarantees offered by one industrial privacy-protection mechanism: the Encode, Shuffle, Analyze (ESA) architecture and Prochlo implementation of Bittau et al. .

However, there still remain gaps between our analysis and proposed practical, real-world mechanisms, such as those of the ESA architecture. In particular, our formalization assumes the user population to be static, which undoubtedly it is not. On a related note, our analysis assumes that (most) all users send reports at each timestep and ignores the privacy implications of timing or traffic channels, although both must be considered, since reports may be triggered by privacy-sensitive events on users’ devices, and it is infeasible to send all possible reports at each timestep. The ESA architecture addresses traffic channels using large-scale batching and randomized thresholding of reports, with elision, but any benefits from that mitigation are not included in our analysis.

Finally, even though it is a key aspect of the ESA architecture, our analysis does not consider how users may fragment their sensitive information and at any timestep send multiple LDP reports, one for each fragment, knowing that each will be anonymous and unlinkable. The splitting of user data into carefully constructed fragments to increase users’ privacy has been explored for specific applications, e.g., by Fanti et al. which fragmented users’ string values into overlapping $n$-grams, to bound sensitivity while enabling an aggregator to reconstruct popular user strings. Clearly, such fragmentation should be able to offer significantly improved privacy/utility tradeoffs, at least in the central model. However, in both the local and central models of differential privacy, the privacy implications of users’ sending LDP reports about disjoint, overlapping, or equivalent fragments of their sensitive information remain to be formally understood, in general.

References