Calibrating Noise to Variance in Adaptive Data Analysis

Introduction

The central challenge in statistical data analysis is to infer the properties of some unknown population given only a small number of samples from that population. While a plethora of techniques for guaranteeing statistical validity are available, few techniques can account for the effects of adaptivity. Namely, if a single dataset is used multiple times, then the choice of which subsequent analyses to perform may depend on the outcomes of previous analyses. This adaptive dependence increases the risk of overfitting — that is, inferring a conclusion that does not generalize to the underlying population.

To formalize this problem, \AtNextCite\AtEachCitekey\@nocounterrmaxnames [DFHPRR14] and subsequent works [[]etc.]HardtU14,SteinkeU15,BassilyNSSSU16,FeldmanS17 study the following question: How many data samples are necessary to accurately answer a sequence of queries about the data distribution when the queries are chosen adaptively – that is, each query can depend on answers to previous queries? Each query corresponds to a procedure that the analyst wishes to execute on the data. The goal is to design an algorithm that provides answers to these adaptive queries that are close to answers that would have been obtained had each corresponding analysis been run on independent samples freshly drawn from the data distribution.

A common and relatively simple class of queries are statistical queries [Kea98]. A statistical query is specified by a function $\psi:\mathcal{X}\rightarrow$ and corresponds to analyst wishing to compute the true mean ${\mathop{\mathbf{E}}_{X\sim{\mathcal{P}}}\left[\psi(X)\right]}$ of $\psi$ on the data distribution ${\mathcal{P}}$. (This is usually done by using the empirical mean $\frac{1}{n}\sum_{i=1}^{n}\psi(S_{i})$ on a dataset $S$ consisting of $n$ i.i.d. draws from the distribution ${\mathcal{P}}$.) For example, such queries can be used to to estimate the true loss (or error) of a predictor, the gradient of the loss function, or the moments of the data distribution. Standard concentration results imply that, given $n$ independent samples from ${\mathcal{P}}$, $k$ fixed (i.e. not adaptively-chosen) statistical queries can be answered with an additive error of at most $O\left(\sqrt{\log(k)/n}\right)$ with high probability by simply using the empirical mean of each query. At the same time it is not hard to show that, for a variety of simple adaptive sequences of queries, using the empirical mean to estimate the expectation leads to an error of $\Omega(\sqrt{k/n})$ [DFHPRR14]. Equivalently, in the adaptive setting, the number of samples required to ensure fixed error scales linearly (rather than logarithmically in the non-adaptive setting) with the number of queries and, in particular, in the worst case, using empirical estimates gives the same guarantees as using fresh samples for every query (by splitting the dataset into $k$ parts).

This quadratic relationship between $n$ and $k$ was also shown to be optimal in the worst case [HU14, SU15].

The approach of [DFHPRR14] relies on properties of differential privacy [DMNS06, DKMMN06] and known differentially private algorithms. Differential privacy is a stability property of an algorithm, namely it requires that replacing any element in the input dataset results in a small change in the output distribution of the algorithm. Specifically, a randomized algorithm $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $(\varepsilon,\delta)$-differentially private if, for all datasets $s,s^{\prime}\in\mathcal{X}^{n}$ that differ on a single element and all events $E\subseteq\mathcal{Y}$,

This stability notion implies that a function output by a differentially private algorithm on a given dataset generalizes to the underlying distribution [DFHPRR14, BNSSSU16]. Specifically, if a differentially private algorithm is run on a dataset drawn i.i.d from any distribution and the algorithm outputs a function, then the empirical mean of that function on the input dataset is close to the expectation of that function on sample from the same distribution.

The second crucial property of differential privacy is that it composes adaptively: running several differentially private algorithms on the same dataset still is differentially private (with somewhat worse parameters) even if each algorithm depends on the output of all the previous algorithms. This property makes it possible to answer adaptively-chosen queries with differential privacy and a number of algorithms have been developed for answering different types of queries. The generalization property of differential privacy then implies that such algorithms can be used to provide answers to adaptively-chosen queries while ensuring generalization [DFHPRR14]. Specifically, the algorithm for answering statistical queries mentioned above is based on the most basic differentially private algorithm: perturbation by adding Laplace or Gaussian noise [DMNS06].

Differential privacy requires that the output distribution of an algorithm does not change much when any element of a dataset is replaced with an arbitrary other element in the domain $\mathcal{X}$. As a result, the amount of noise that needs to be added to ensure differential privacy scales linearly with the range of the function $\psi$ whose expectation needs to be estimated. If the range of $\psi$ is comparable to the standard deviation of $\psi(x)$ on $x$ drawn from ${\mathcal{P}}$ (such as when $\psi$ has range $\{0,1\}$ and mean $1/2$) then the error resulting from addition of noise is comparable to the standard deviation of $\psi$. However, for queries whose standard deviation is much lower than the range, the error introduced by noise is much worse than the sampling error. Variance is much smaller than the range for a variety of common settings, for example, difference between candidate predictors for the same problem or individual input features when the input is usually sparse.

Achieving error guarantees in the adaptive setting that scale with the standard deviation instead of range is a natural problem. Recently, [FS17] gave a different algorithm that achieves such a guarantee. Specifically, their algorithm ensures that with probability at least $1-\beta$,

In this work, we ask: does the natural algorithm that perturbs the empirical answers with noise scaled to the standard deviation suffice to answer adaptive queries with accuracy scaling to sampling error? To answer this seemingly simple question, we address a more fundamental problem: does there exist a notion of stability that has the advantages of differential privacy (namely, allows adaptive composition and implies generalization) but avoids the poor dependence on the worst-case sensitivity of the query. This algorithm was analyzed by [BF16] via a notion of typical stability they introduced. Their analysis shows that the algorithm will ensure the correct scaling of the error with standard deviation but it does not improve on the naive mechanisms in terms of scaling with $k$. Several works have considered relaxations of differential privacy in this context. For example, \AtNextCite\AtEachCitekey\@nocounterrmaxnames [BNSSSU16] considered a notion of stability based on using KL divergence or total variation distance in place of differential privacy (which can be defined in terms of approximate max divergence). \AtNextCite\AtEachCitekey\@nocounterrmaxnames [WLF16] considered the expected KL divergence between the output of the algorithm when run on a random i.i.d dataset versus the same dataset with one element replaced by a fresh sample; unfortunately, their stability definition does not compose adaptively. Notions based on the mutual information between the dataset and the output of the algorithm and their relationship to differential privacy have also been studied [DFHPRR15, RZ16, RRST16, RRTWX16, XR17]. However, to the best of our knowledge, these approaches do not give a way to analyze the calibrated noise addition that ensures correct dependence on $k$.

We introduce new stability-based and information-theoretic tools for analysis of the generalization of algorithms in the adaptive setting. The stability notion we introduce is easier to satisfy than differential privacy, yet has the properties crucial for application in adaptive data analysis. These tools allow us to demonstrate that calibrating the variance of the perturbation to the empirical variance of the query suffices to ensure generalization, as long as the noise rate does not become too small. To ensure this lower bound on the noise rate we simply add a second order term to the variance of the perturbation. Specifically, our algorithm is described in Figure 1. The only difference between our algorithm and previous work [DFHPRR14, BNSSSU16] is that in prior work the variance of the Gaussian perturbation is fixed.

We prove that this algorithm has the following accuracy guarantee.

More precisely, applying Markov’s inequality to the conclusion of Theorem 1.1, shows that, with probability at least $90\%$,

This guarantee is directly comparable to the earlier bound (2) of [FS17] – though it is weaker in two ways: First, Theorem 1.1 is a bound on the expectation and does not readily yield high probability bounds (other than via Markov’s inequality). Second, the second term in the maximum (which we think of as a low-order term) still depends linearly on the sensitivity and is potentially larger. The advantage of this algorithm is that it is substantially simpler than the earlier work.

The key to our analysis is the following stability notion.

An algorithm $M:\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Y}$ is $\varepsilon$-ALKL stable if, for all $s\in\mathcal{X}^{n}$,

Our notion differs from differential privacy in three significant ways.These relaxations mean that ALKL stability is not a good privacy definition, in contrast to differential privacy. In particular, because of the averaging, ALKL stability cannot distinguish between an algorithm that offers good privacy to all individuals and one that offers great privacy for $n-1$ individuals but terrible privacy for the last individual. Compromising a single data point is, however, not an issue for generalization. First, we use stability to leaving one out (LOO) rather than replacing one element. Second, we average the stability parameter across the $n$ dataset elements. Third, we use KL divergence instead of (approximate) max divergence. This is necessary to obtain stronger bounds for our calibrated noise addition as our algorithm does not satisfy differential privacy with parameters that would be suitable to ensure generalization. We note that average LOO stability is a well-studied way to define algorithmic stability for the loss function (e.g. , [BE02, PRMN04]). The use of KL divergence appears to be necessary to ensure adaptive composition of our averaged notion. Specifically, the following composition result is easy to prove.

Suppose $M:\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Y}$ is $\varepsilon$-ALKL stable and $M^{\prime}:\mathcal{Y}\times\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Z}$ is such that $M^{\prime}(y,\cdot):\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Z}$ is $\varepsilon^{\prime}$-ALKL stable for all $y\in\mathcal{Y}$. Then the composition $s\mapsto M^{\prime}(M(s),s)$ is $(\varepsilon+\varepsilon^{\prime})$-ALKL stable.

Using composition, we can show that our algorithm (Figure 1, with the parameters set as in Theorem 1.1) is $\frac{kt}{n^{2}}$-ALKL stable. In particular, we show that each one of the $k$ answers is computed in a way that is $\frac{t}{n^{2}}$-ALKL stable. This follows from the properties of the KL divergence between Gaussian distributions and the way we calibrate the noise. (Alternatively, we could use Laplace noise to obtain similar results.)

We note that $\sqrt{2\varepsilon}$-differential privacy [DMNS06], notions based on Renyi differential privacy [BS16, Mir17], and $\varepsilon$-KL-stability [BNSSSU16] all imply $\varepsilon$-ALKL stabilityIt may be necessary to extend an algorithm satisfying one of these definitions to inputs of size $n-1$ to satisfy ALKL stability. This can be done by simply padding such an input with one arbitrary item. Thus we can also compose any ALKL stable algorithm with any of the many algorithms satisfying one of the aforementioned definitions.

Crucially, average KL-divergence is strong enough to provide a generalization guarantee that scales with the standard deviation of the queries, as we require. Our proof is based on the high-level approach introduced by [DFHPRR15] who first convert a stability guarantee to an upper bound on information between the input dataset and the output of the algorithm and then derive generalization bounds from the bound on information. Here, we demonstrate that ALKL stability implies a bound on the mutual information between the input and output of the algorithm when run on independent samples and then derive generalization guarantees from the bound on mutual information We thank Adam Smith for suggesting that we try this approach to proving generalization for ALKL stable algorithms.

Let $M:\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Q}$ be $\varepsilon$-ALKL stable. Let $S\in\mathcal{X}^{n}$ consist of $n$ independent samples from some distribution ${\mathcal{P}}$. Then

To prove Proposition 1.4, we introduce an intermediate notion of stability:

A randomized algorithm $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-MI stable if, for any random variable $S$ distributed over $\mathcal{X}^{n}$ (including non-product distributions),

This notion is based on the notion of stability studied in [RRTWX16] that considers only product distributions over the datasets and, as a result, does not compose adaptively.

We prove Proposition 1.4 by combining the following two facts.

$\varepsilon$-ALKL stability implies $\varepsilon$-MI stability. (Lemma 3.6) To show this, we express $I(M(S);S_{i}|S_{-i})$ as the expectation over $S$ of the KL divergence of the distribution (over the randomness of $M$) of $M(S)$ from an appropriately weighted convex combination of distributions $M(S^{\prime})$. (Specifically, $S^{\prime}$ is $S$ with $S_{i}$ “resampled.”) The “mean-as-minimizer” property of KL divergence (Lemma 2.9) means we can simply replace this convex combination with $M(S_{-i})$ to complete the proof.

$\varepsilon$-MI stability implies the mutual information bound (4). (Lemma 3.7) To prove this, we invoke the chain rule for mutual information along with the fact that $S_{i}$ is independent from $S_{-i}$ (which helps resolve the conditioning).

Further, we point out that mutual information stability composes adaptively in the same way as ALKL stability and hence could be useful for understanding adaptive data analysis for more general queries (e.g. unlike ALKL stability it does not require $M(S_{-i})$ to be defined).

As first shown in the context of PAC-Bayes bounds [McA13] and more recently in [RZ16], a bound on mutual information implies generalization results. Using a similar technique, we show that, if the mutual information $I(S;\psi_{j})$ is small (with $S$ consisting of $n$ i.i.d. draws from ${\mathcal{P}}$), we have ${\mathop{\mathbf{E}}\left[\frac{1}{n}\sum_{i=1}^{n}\psi_{j}(S_{i})\right]}\approx{\mathop{\mathbf{E}}_{X\sim{\mathcal{P}}}\left[\psi_{j}(X)\right]}$. Moreover, the quality of the approximation scales with the standard deviation. (Specifically, the approximation bound depends on the moment generating function ${\mathop{\mathbf{E}}\left[e^{\lambda\mu_{j}}\right]}$ of $\mu_{j}=\frac{1}{n}\sum_{i=1}^{n}\psi_{j}(S_{i})$, which we bound using both the variance and the range of $\mu_{j}$.) We can similarly relate the empirical variance $\sigma_{j}^{2}$ to the true variance. Thus a bound on mutual information suffices to bound generalization error and, thus, prove Theorem 1.1.

Another known implication of bounded mutual information is that any event that would happen with sufficiently low probability on fresh data will still happen with low probability [RZ16, RRST16]. In particular, if $E$ is some “bad” event – such as overfitting the data or making a false discovery – and we know that we are exponentially unlikely to overfit fresh data $S^{\prime}$, then the probability of $M$ overfitting its input data $S$ is also small, provided the mutual information is small. (See Section 3.3 for additional details.)

One downside of using mutual information is that does not allow us to prove high probability bounds, as can be done with differential privacy and the notion of approximate max-information [DFHPRR15]. We note, however, that our analysis still upper bounds the expectation of the largest error among all the queries that were asked. In other words, a union bound over queries is built into the guarantees of the algorithm. Using known techniques, the confidence can be amplified at the expense of a somewhat more complicated algorithm. In addition, our algorithm yields stronger stability guarantees than just ALKL stability. For example, the minimum noise level of $1/T$ ensures differential privacy (albeit with relatively large parametersSpecifically, with the parameter setting from Theorem 1.1, our algorithm satisfies $\left(O\left(\sqrt{\log(1/\delta)}\right),\delta\right)$-differential privacy for all $\delta>k^{-\Omega(k)}$.). The parameters can be improved using the averaging over the indices that we use in ALKL stability but that leads to a notion that does not appear to compose adaptively. Using a different analysis technique it might be possible to exploit the stronger stability properties of our algorithm to prove high probability generalization bounds. We leave this as an open problem. On the other hand, stability with KL divergence is easier to analyze and allows a potentially wider range of algorithms to be used.

2 Related work

Our use of mutual information to derive generalization bounds is closely related to PAC-Bayes bounds first introduced by [McA99] and extended in a number of subsequent works (see [McA13] for an overview). In this line of work, the expected generalization error of a predictive model (such as classifier) randomly chosen from some data-dependent distribution ${\mathcal{Q}}(S)$ is upper-bounded by the KL divergence between ${\mathcal{Q}}$ and an arbitrary data-independent prior distribution ${\mathcal{P}}_{0}$. One natural choice of ${\mathcal{Q}}(S)$ is the output distribution of a randomized learning algorithm ${\mathcal{A}}$ on $S$. By choosing the prior ${\mathcal{P}}_{0}$ to be the distribution of the output of ${\mathcal{A}}$ on a dataset drawn from ${\mathcal{P}}^{n}$ one obtains that the expected generalization error is upper-bounded by the expected KL divergence between ${\mathcal{Q}}(S)$ an ${\mathcal{P}}_{0}$ [McA13]. While this has not been pointed out in [McA13], this is exactly the mutual information between $S$ and ${\mathcal{A}}(S)$.

Recently, interest in using information-based generalization bounds was revived by applications in adaptive data analysis [DFHPRR15]. Specifically, [DFHPRR15] demonstrate that approximate max-information between the input dataset and the output of the algorithm (a notion based on the infinity divergence between the joint distribution and the product of marginals) implies generalization bounds with high probability. They also showed that $(\epsilon,0)$-differential privacy implies an upper bound on approximate max-information (and this later extended to $(\epsilon,\delta)$-differential privacy by \AtNextCite\AtEachCitekey\@nocounterrmaxnames [RRST16]). [RZ16] show that mutual information can also be used to derive bounds on expected generalization error and discuss several applications of these bounds. [XR17] show how to derive “low-probability” bounds on the generalization error in this context. (We note that [RZ16, XR17] use the same technique as that used in PAC-Bayes bounds and appear to have overlooked the direct connection between their results and the PAC-Bayes line of work.)

Recent work [BMNSY18] studies learning algorithms in the PAC model whose output has low mutual information with the input dataset. They also discuss generalization bounds based on mutual information and (independently) derive results similar to those we give in Section 3.3.

Notation, Definitions, & Key Properties

Before continuing, we first establish some relevant properties of the KL divergence. See the textbook by [CT12] for an introduction to the properties of KL divergence (a.k.a. relative entropy).

First we state the definition of KL divergence for completeness.

Let ${\mathcal{P}}$ and ${\mathcal{Q}}$ be probability distributions on a space $\Omega$. Suppose ${\mathcal{P}}$ is absolutely continuous with respect to ${\mathcal{Q}}$. Then the KL divergence from ${\mathcal{Q}}$ to ${\mathcal{P}}$ is

where ${\mathcal{P}}(x)$ and ${\mathcal{Q}}(x)$ denote the probability mass or density functions of ${\mathcal{P}}$ and ${\mathcal{Q}}$ respectively evaluated at the point $X$. (More generally, ${\mathcal{P}}(x)/{\mathcal{Q}}(x)$ denotes the Radon-Nikodym derivative of ${\mathcal{P}}$ with respect to ${\mathcal{Q}}$ evaluated at $x$.)

Let ${\mathcal{P}}$ and ${\mathcal{Q}}$ be two distributions over some domain $\mathcal{X}\times{\mathcal{Y}}$. Then

Here ${\mathcal{P}}(x)$ (or ${\mathcal{Q}}(x)$) denotes the marginal distributions of ${\mathcal{P}}$ (or ${\mathcal{Q}}$) over $\mathcal{X}$ and ${\mathcal{P}}(y|x=x^{\prime})$ denotes the marginal distribution of ${\mathcal{P}}$ on ${\mathcal{Y}}$ conditioned on $x=x^{\prime}$.

We begin by looking at the KL divergence between two Gaussian distributions, as this is what our mechanism uses. Recall that the Gaussian (or normal) distribution with mean $\mu$ and variance $\sigma^{2}$ — denoted $\mathcal{N}(\mu,\sigma^{2})$ — has a probability density at $x$ given by $\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{(x-\mu)^{2}}{2\sigma^{2}}\right)$.

This follows from Lemma 2.3 and the inequalities $x-1-\ln x\leq(1/x-1)^{2}\cdot x$ and $x-1-\ln x\leq(1/x-1)^{2}\cdot x\cdot(2+x)/6$ for all $x>0$.

An analogous result holds for the Laplace distribution. Although we do not work this out, it implies that our results can be extended to work for the Laplace distribution (with slightly different constants and a higher power of $\ln k$, since the Laplace distribution has heavier tails). Recall that the Laplace distribution with mean $\mu$ and variance $2\sigma^{2}$ — denoted ${\mu+}\mathsf{Lap}\left(\sigma\right)$ — has a probability density at $x$ given by $\frac{1}{2|\sigma|}\exp\left(\left|\frac{x-\mu}{\sigma}\right|\right)$.

Next we have a technical lemma relating expectations to KL divergence.

Let ${\mathcal{P}}$ and ${\mathcal{Q}}$ be probability distributions on $\Omega$. Then

Setting $f(x)=tx$ and rearranging gives the bound we will use:

Let $X$ and $Y$ be real-valued random variables and $t>0$. Then

Next we note that KL divergence is a convex function.

Let ${\mathcal{P}}_{0}$, ${\mathcal{P}}_{1}$, ${\mathcal{Q}}_{0}$, ${\mathcal{Q}}_{1}$ be probability distributions on the same space $\Omega$. For $t\in(0,1)$, let ${\mathcal{P}}_{t}=(1-t){\mathcal{P}}_{0}+t{\mathcal{P}}_{1}$ and ${\mathcal{Q}}_{t}=(1-t){\mathcal{Q}}_{0}+t{\mathcal{Q}}_{1}$ be the convex combinations interpolating between these distributions. Then, for all $t\in$,

This lemma immediately extends to convex combinations of more than two distributions.

Next we have a geometric statement about KL divergence:

Let $\{{\mathcal{P}}_{y}\}$ be a family of distributions indexed by $y\in{\mathcal{Y}}$ and let ${\mathcal{Q}}$ be a distribution on ${\mathcal{Y}}$. Let ${\mathcal{P}}_{\mathcal{Q}}={\mathop{\mathbf{E}}_{Y\sim{\mathcal{Q}}}\left[{\mathcal{P}}_{Y}\right]}$ denote the convex combination of the distributions $\{{\mathcal{P}}_{y}\}$ weighted by ${\mathcal{Q}}$. Then

Lemma 2.9 shows that the “center” of a collection of probability distributions — as measured my minimizing average KL divergence to one distribution — is none other than the mean of those distributions.

2 Mutual Information

A key quantity that we use is mutual information:

For two random variables $X$ and $Y$ jointly distributed according to a distribution ${\mathcal{P}}$ over $\mathcal{X}\times{\mathcal{Y}}$, the mutual information between $X$ and $Y$ is

where ${\mathcal{P}}(x)\times{\mathcal{P}}(y)$ denotes the product of the marginal distributions of ${\mathcal{P}}$.

Note that mutual information is symmetric – $I(X;Y)=I(Y;X)$.

For three random variables $X$, $Y$, and $Z$. The mutual information between $X$ and $Y$ conditioned on $Z$ is given by

where ${\mathcal{P}}_{Z}$ is the marginal distribution of $Z$.

For random variables $X$, $Y$, and $Z$, we have

Average KL Stability & Generalization

In this section, we cover our theoretical tools, which center around our definition of average leave-one-out KL stability, which we restate here. For a randomized algorithm $M$ and input $s$ we use $M(s)$ denote the random variable obtained by running on $M$ on $s$ and the distribution of this random variable (according to the context).

An algorithm $M:\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Y}$ is $\varepsilon$-ALKL stable if, for all $s\in\mathcal{X}^{n}$,

where $s_{-i}\in\mathcal{X}^{n-1}$ denotes $s$ with the $i^{\text{th}}$ element removed.

More generally, an algorithm $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-ALKL stable if for every $i\in[n]$ there exists an algorithm $M_{i}:\left(\mathcal{X}^{n-1}\right)\to\mathcal{Y}$ such that under the same conditions

If an algorithm $M$ is $\varepsilon$-KL stable then it is $\varepsilon$-ALKL stable.

The key property of our definition is composition. This lemma allows us to account for the accumulation of information through multiple adaptive queries. The following lemma only considers the composition of two algorithms. Induction allows this to be extended to $k$ algorithms.

Suppose $M:\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Y}$ is $\varepsilon$-ALKL stable and $M^{\prime}:\mathcal{Y}\times\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Z}$ is such that $M^{\prime}(y,\cdot):\left(\mathcal{X}^{n}\cup\mathcal{X}^{n-1}\right)\to\mathcal{Z}$ is $\varepsilon^{\prime}$-ALKL stable for all $y\in\mathcal{Y}$. Then the composition $s\mapsto M^{\prime}(M(s),s)$ is $(\varepsilon+\varepsilon^{\prime})$-ALKL stable.

Let $M_{i}$ and $M_{i}^{\prime}$ be the algorithms whose existence is assumed by Definition 3.1. Fix $s\in\mathcal{X}^{n}$. By the chain rule for KL divergence (Lemma 2.2),

Another key property of our definition of average leave-one-out KL stability is postprocessing. That is, if $M$ is $\varepsilon$-ALKL stable, then applying an arbitrary function to the output of $M$ continues to be $\varepsilon$-ALKL stable. This can be seen by taking $\varepsilon^{\prime}=0$ in the above composition lemma or by using the data processing inequality for KL divergence [VEH14, Theorem 1].

In order to show that our notion of average leave-one-out KL stability implies generalization, we first show that it implies a bound on mutual information:

Let $M:\mathcal{X}^{n}\to\mathcal{Y}$ be $\varepsilon$-ALKL stable. Let $S\in\mathcal{X}^{n}$ be a product distribution. Then $I(M(S);S)\leq\varepsilon\cdot n$.

To prove Proposition 3.4, we introduce an intermediate notion of stability that is based on that of \AtNextCite\AtEachCitekey\@nocounterrmaxnames [RRTWX16]. Specifically, mutual information stability is defined as follows.

A randomized algorithm $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-MI stable if, for any random variable $S$ distributed over $\mathcal{X}^{n}$ (including non-product distributions),

We show that mutual information stability has the following properties.

Average leave-one-out KL stability implies mutual information stability.

Mutual information stability implies a mutual information bound.

Mutual information stability composes adaptively.

Combining properties (1) and (2) yields Proposition 3.4. The adaptive composition property of mutual information stability implies that it might be useful for analysis of adaptive procedures which are not ALKL stable (although we do not use this property since ALKL stability itself composes adaptively).

If $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-ALKL stable, then it also is $\varepsilon$-MI stable.

Let $S$ be a random variable distributed according to some distribution ${\mathcal{P}}$ on $\mathcal{X}^{n}$. Let ${\mathcal{P}}_{i}$ and ${\mathcal{P}}_{-i}$ denote the marginal distribution of $S_{i}$ and $S_{-i}$, respectively. For $z\in\mathcal{X}^{n-1}$ we use ${\mathcal{P}}(s_{i}|z)$ to denote the distribution of $S_{i}$ conditioned on $S_{-i}=z$. Now, by the definition of (conditional) mutual information,

Here $z\circ_{i}x$ refers to the vector $s\in\mathcal{X}^{n}$ such that $s_{-i}=z$ and $s_{i}=x$. Here the inner expectation is over $x$ drawn from the distribution of $S_{i}$ conditioned on $S_{-i}=z$ — of the KL divergence from the distribution of $M(z\circ_{i}x)$ to the distribution of $M(S)$ conditioned on $S_{-i}=z$. The latter distribution is exactly the convex combination of the distribution of $M(z\circ_{i}x)$ weighted by the distribution of $x\sim{\mathcal{P}}(s_{i}|z)$.

Now the key observation: the convex combination — $M(S)|S_{-i}=z$ — is the distribution that minimizes the inner expectation. Hence, we can replace it by $M_{i}(z)$ and only increase the expression. Formally, by Lemma 2.9, for all $z\in\mathcal{X}^{n-1}$,

Note that $M_{i}(z)$ refers either to execution of $M$ itself or (if $M$ is not defined over inputs of length $n-1$) to the algorithms whose existence is promised by the second half of Definition 3.1. The result now follows, as we have established that

Suppose $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-MI stable. Let ${\mathcal{P}}$ be a distribution over $\mathcal{X}$ and $S$ be distributed according to ${\mathcal{P}}^{n}$. Then $I(M(S);S)\leq\varepsilon n$.

Denote $S_{<i}=(S_{1},\cdots,S_{i-1})$, $S_{>i}=(S_{i+1},\cdots,S_{n})$ and $S_{\leq i}=(S_{1},\cdots,S_{i})$. By the chain rule for mutual information (Lemma 2.12 and induction),

By the definition of (conditional) mutual information,

Here $z\circ x\in\mathcal{X}^{i}$ denotes the concatenation of $z\in\mathcal{X}^{i-1}$ with $x\in\mathcal{X}$. Now, by the convexity of KL divergence (Lemma 2.8), we can move the randomness of $S_{>i}$ from the divergence and into the expectation. Namely,

Here we use the fact that $M(S)|S_{\leq i}=z\circ x$ and $M(S)|S_{<i}=z$ are convex combinations of the distribution of $M(s)$ weighted by $S_{>i}$ (note that independence is crucial here). Plugging this into eq. (5) and using the definition of (conditional) mutual information we get

Combining these (in)equalities yields the result:

Suppose $M:\mathcal{X}^{n}\to\mathcal{Y}$ is $\varepsilon$-MI stable and $M^{\prime}:\mathcal{Y}\times\mathcal{X}^{n}\to\mathcal{Z}$ is such that $M^{\prime}(y,\cdot):\mathcal{X}^{n}\to\mathcal{Z}$ is $\varepsilon^{\prime}$-MI stable for all $y\in\mathcal{Y}$. Then the composition $s\mapsto M^{\prime}(M(s),s)$ is $(\varepsilon+\varepsilon^{\prime})$-MI stable.

Let $S$ be a random variable on $\mathcal{X}^{n}$ and let ${\mathcal{P}}_{M(S)}$ denote the probability distribution of $M(S)$. By the chain rule,

The key is that the stability property holds for all distributions, which means it holds for the distribution of $S$ conditioned on $M(S)$. Note that if we defined mutual information stability only to quantify over product distributions, then this proof would not carry through, as $S$ conditioned on $M(S)$ is not necessarily a product distribution anymore. ∎

A natural question to ask is whether instead of using stability notions we can directly use mutual information for our analysis. Specifically, one could prove a bound on the mutual information of adding calibrated noise and then use composition properties of mutual information to bound the error of the entire algorithm for answering adaptive queries. For this approach to work one needs to prove a bound on the mutual information of adding calibrated noise for arbitrary input distributions (or at least for all distributions that might result from conditioning on the previous answers to queries). However, it is not hard to see that for non-product distributions over $S$ mutual information can be much larger than the bounds we will get via stability. (For example if the distribution on $S$ is such that the answer to the query is with probability $1/2$ and $1$ with probability $1/2$ then adding noise with variance $1$ will reveal some positive constant amount of information. At the same time this algorithm is $1/n^{2}$-KL stable so in our approach will contribute only $1/n$ to the final bound on mutual information.) As a result, this simpler approach is unlikely to lead to useful generalization bounds.

2 Generalization in expectation

In this section we translate an upper bound on $I(S;M(S))$ into an upper bound on the expectation of the generalization error. As in earlier work [RZ16], our main technical tool is Corollary 2.7. However we deal with more general random variables (not just subgaussian) and also prove bounds that are scaled to standard deviation of the random variable as opposed to the subgaussian constant. In Section. 3.3 we describe an alternative approach to generalization which is based on bounding the probability of any “bad” event.

The following proposition bounds the expected generalization error.

Let $M$ be a randomized algorithm with input from $\mathcal{X}^{n}$ and output in $\mathcal{Q}$, where $\mathcal{Q}$ is the set of functions $\psi:\mathcal{X}\to$. Let ${\mathcal{P}}$ be a distribution on $\mathcal{X}$ and $S\sim{\mathcal{P}}^{n}$. Let $\tau>0$. Suppose $I(S;M(S))\leq\varepsilon n$. Then

Thus it only remains to bound ${\mathop{\mathbf{E}}\left[e^{\lambda Y}\right]}$. We have

Let $Y$ be a random variable supported on $[-1/\tau,1/\tau]$. Suppose ${\mathop{\mathbf{E}}\left[Y\right]}=0$ and ${\mathop{\mathbf{E}}\left[Y^{2}\right]}\leq 1$. Then for $\lambda\in[0,\tau]$, ${\mathop{\mathbf{E}}\left[e^{\lambda Y}\right]}\leq e^{\lambda^{2}}$.

This lemma is similar to the proof of Bernstein’s inequality [Ber24].

Since $|Y|\leq 1/\tau$, we have $|Y|^{k}\leq(1/\tau)^{k-2}Y^{2}$ for all $k\geq 2$. Thus, for all $\lambda\geq 0$, we have

If $\lambda/\tau\leq 1$, then $e^{\lambda/\tau}-1-\lambda/\tau\leq(\lambda/\tau)^{2}$, which yields the result. ∎

Proposition 3.10 gives a bound in terms of variance. Using the PAC-Bayes framework, we can also attain an additive-multiplicative bound:

Let $M$ be a randomized algorithm that takes an input from $\mathcal{X}^{n}$ and outputs a function $\psi:\mathcal{X}\to$. Let $\mathcal{P}$ be a distribution on $\mathcal{X}$ and let $S\sim\mathcal{P}^{n}$ consist of $n$ i.i.d. samples therefrom. Fix $\lambda>1/2$. Then

To analyse our algorithm, we also need to bound the empirical error in terms of the standard deviation. Note that the empirical error – the noise we add – scales with the empirical standard deviation. Thus we must bound the empirical variance in terms of the true variance:

Let $M$ be a randomized algorithm with input from $\mathcal{X}^{n}$ and output in $\mathcal{Q}$, where $\mathcal{Q}$ is the set of functions $\psi:\mathcal{X}\to$. Let ${\mathcal{P}}$ be a distribution on $\mathcal{X}$ and $S\sim{\mathcal{P}}^{n}$. Let $\tau>0$. Suppose $I(S;M(S))\leq\varepsilon n$. Then

To prove Proposition 3.13 we make use of the following two standard facts.

Let $X$ be a random variable supported on $$. Suppose${\mathop{\mathbf{E}}\left[X\right]}=\sigma$. Then for$s\in$,

To bound ${\mathop{\mathbf{E}}\left[e^{\lambda Y}\right]}$ we note that $Y$ is determined by $S$ and $\psi$. Since these are independent, we may consider an arbitrary fixed $\psi$. We let $Z$ denote $Y$ conditioned on $\psi$ being equal to a fixed $\phi\in{\mathcal{Q}}$. We can write $Z$ as a sum of $n$ independent terms $Z_{i}=\frac{(\phi(S_{i})-{\mathcal{P}}[\phi])^{2}}{\max\left\{{\mathop{\mathbf{Var}}\left[\phi({\mathcal{P}})\right]},\tau^{2}\right\}}$, and hence ${\mathop{\mathbf{E}}\left[e^{\lambda Z}\right]}=\prod_{i}^{n}{\mathop{\mathbf{E}}\left[e^{\lambda Z_{i}}\right]}$. For each $i$, we have ${\mathop{\mathbf{E}}\left[Z_{i}\right]}=\frac{{\mathop{\mathbf{Var}}\left[\phi({\mathcal{P}})\right]}}{\max\left\{{\mathop{\mathbf{Var}}\left[\phi({\mathcal{P}})\right]},\tau^{2}\right\}}\leq 1$ and $0\leq Z_{i}\leq 1/\tau^{2}$. Thus, by (8) (with $X=Z_{i}\tau^{2}$, $\sigma\leq\tau^{2}$, and $s=\lambda/\tau^{2}$), ${\mathop{\mathbf{E}}\left[e^{\lambda Z_{i}}\right]}\leq e^{2\lambda}$ for $\lambda\in[0,\tau^{2}]$.

This implies that ${\mathop{\mathbf{E}}\left[e^{\lambda Z}\right]}\leq e^{2\lambda n}$ for $\lambda\in[0,\tau^{2}]$ for every $\phi$ and hence ${\mathop{\mathbf{E}}\left[e^{\lambda Y}\right]}\leq e^{2\lambda n}$ under the same condition. Substituting this into eq. (9) yields

3 Preservation of low-probability events

Propositions 3.10 and 3.13 bound the expected generalization error given a bound on mutual information. An alternative approach to analysis of generalization is to use a bound on mutual information to upper bound the increase in probability of any “bad” event that results from the dependence between the dataset and algorithm’s output. Specifically, we prove the following simple lemma:

Let $S$ consist of $n$ independent samples from some distribution ${\mathcal{P}}$. Let $S^{\prime}$ be an independent copy of $S$. Let $M:\mathcal{X}^{n}\to\mathcal{Y}$ and let $E$ be an event on $\mathcal{X}^{n}\times\mathcal{Y}$ satisfying

Intuitively, Lemma 3.14 says that if an event happens with very low probability on fresh data, then it happens with somewhat low probability on non-fresh data, as long as the mutual information between the event and the data is low. Note however, that the probability grows from $\delta$ to $\frac{I(S;M(S))+\ln 2}{\ln(1/\delta)}$. In particular, the inverse of the probability decreases exponentially. For example, Lemma 3.14 can be used to correct a $p$-value obtained under the assumption that the data is independent from the choice of the test (since $p$-value is the probability that a test statistic satisfies a chosen condition) [RZ16, RRST16].

The same approach to generalization is used in [DFHPRR14, DFHPRR15, RRST16] for differential privacy and max-information and in [RZ16, RRST16] for mutual information. The bound implicit in [RZ16] is $\mathop{\mathbf{Pr}}\left[(S,M(S))\in E\right]\leq\delta+\sqrt{\frac{I(S;M(S))}{\ln(1/(2\delta))}}$ which is asymptotically worse than our bound. The bound in [RRST16] is derived by first using mutual information to bound approximate max-information [DFHPRR15]. Their approach yields the following bound

which is comparable to the bound in Lemma 3.14.

As a more concrete application, we demonstrate how Lemma 3.14 can be used to derive a bound on the probability of generalization error being large (or, equivalently, to construct a valid confidence interval for the true expectation of a real-valued function).

Recall the setting of Proposition 3.10. Here $S$ consists of $n$ independent samples from ${\mathcal{P}}$ and $M$ outputs a function $\psi:\mathcal{X}\to$ and has $I(S;M(S))\leq\varepsilon n$. By Bernstein’s inequality, for $n$ fresh samples $S^{\prime}$ (independent from $M(S)$), we have

for all $t>0$. Thus, by Lemma 3.14, for all $t>0$,

In our application of Proposition 3.10, we have $\varepsilon=\tau^{2}\geq 1/n$; setting $t=3\tau/\beta$ and simplifying yields

Note that this bound appears to correspond to an application of Markov’s inequality to the conclusion Proposition 3.10. However, since the random variable in question may take both positive and negative values, Markov’s inequality cannot be applied. Namely, Lemma 3.14 corresponds to a strengthening of Proposition 3.10 that bounds the expectation of the absolute value of the random variable. This approach can also be easily used to get a bound (based on Markov’s inequality) on the tail of the largest error we state in Theorem 1.1. This follows from the fact that a high probability bound on this tail is easy to prove when the dataset is independent from the algorithm’s answers.

To prove Lemma 3.14, we observe that for any random variable $S$ and any randomized algorithm $M$,

Let $B(p)$ denote the Bernoulli random variable with bias $p$. Then for any $p,q\in(0,1]$,

where $\mathsf{H}(p)=p\ln(1/p)+(1-p)\ln(1/(1-p))$ is the binary entropy function. Rearranging yields the result. ∎

Analysis of our Algorithm

Now we assemble the tools developed in the previous section to analyse our algorithm (Figure 1). To do this we must introduce some formalisms for dealing with adaptive algorithms.

Our algorithm $M$ answers adaptively-chosen queries. We call the entity $A$ choosing these queries the analyst (or adversary to connote worst-case behaviour). The interaction between $A$ and $M$ defines a function mapping inputs (the sample) to a transcript of queries and answers. Figure 2 defines how this function is computed.

With this formalism in hand, we can extend our definition of average leave-one-out KL stability from non-interactive algorithms (Definition 1.2) to interactive algorithms:

An interactive algorithm $M$ is $\varepsilon$-ALKL stable if $A{\rightarrow\atop\leftarrow}M$ (as defined in Figure 2) is $\varepsilon$-ALKL stable for all interactive algorithms $A$.

We now show that our algorithm is (interactive) average leave-one-out KL stable:

Our algorithm (Figure 1) is $\frac{kt}{n^{2}}$-ALKL stable for any $n\geq 20$ and $T\leq\min\{t^{2},tn/10\}$.

To establish that our algorithm is average leave-one-out KL stable, we first only consider one query $\psi=\psi_{j}$. We show that, for each query $\psi$, the answer given by our algorithm is $\frac{t}{n^{2}}$-ALKL stable. Using composition (Lemma 1.3), we can extend this to $k$ queries $\psi_{1},\ldots,\psi_{k}$. That is, we prove that our algorithm is $\frac{kt}{n^{2}}$-ALKL stable.

First we recall how our algorithm answers a query $\psi:\mathcal{X}\to$: The algorithm is given as input a sample $s\in\mathcal{X}^{n}$ and, for each statistical query $\psi=\psi_{j}$ the algorithm $M$ outputs a sample from $\mathcal{N}(\mu,\max\left\{\sigma^{2}/t,1/T\right\})$ where

Here $t,T>0$ are parameters controlling the accuracy-stability tradeoff.

We also consider the following quantities in the analysis so that $M(s_{-i})$ outputs a sample from $\mathcal{N}(\mu_{-i},\max\left\{\sigma_{-i}^{2}/t,1/T\right\})$.

Before giving the full proof we give a simplified sketch.

We make three simplifications for our sketch of the analysis:

Ignore constant factors. (Take $n$ and $\frac{t}{T}n$ to be sufficiently large.)

Consider $\psi:\mathcal{X}\to\{0,1\}$ instead of $\psi:\mathcal{X}\to$.

Assume $\frac{\sigma^{2}}{t}\geq\frac{1}{T}$ and $\frac{\sigma_{-i}^{2}}{t}\geq\frac{1}{T}$ for all $i$.

The last assumption is not really an assumption, since we perforce ensure this by using $\max\left\{\frac{\sigma^{2}}{t},\frac{1}{T}\right\}$ in place of $\frac{\sigma^{2}}{t}$ and likewise for $\frac{\sigma_{-i}^{2}}{t}$. This assumption is only to simplify notation here in this sketch.

Begin by considering a fixed index $i\in[n]$. By standard properties of the KL divergence between Gaussians (Corollary 2.4) and assuming $\frac{\sigma^{2}}{\sigma_{-i}^{2}}\leq 2$, we have

Since we assumed (for simplicity) that $\psi$ takes only values and $1$, the distribution of $\psi(s_{i})$ (for a random $i$) is characterized by its mean $\mu=s[\psi]$. In particular, we can express the variance as $\sigma^{2}=\mu(1-\mu)$ and, likewise, $\sigma_{-i}^{2}=\mu_{-i}(1-\mu_{-i})$. Thus, we have $|\sigma_{-i}^{2}-\sigma^{2}|\leq|\mu-\mu_{-i}|$ and

where the final inequality follows from the assumptions $\frac{\sigma^{2}}{t}\geq\frac{1}{T}$ and $T\leq t^{2}$. Also

and, hence, (now we make critical use of the averaging ALKL stability affords us)

Combining the above equations (10,11,12) yields

as desired to show $O(t/n^{2})$-ALKL stability. ∎

The key facts that drive this simplified proof (and which hold without the simplifying assumptions) are the KL divergence between Gaussians (10),

Indeed, we can redefine $\mu$ and $\sigma^{2}$ (e.g., to answer different types of queries, rather than statistical queries) and still retain ALKL stability, as long as the above two inequalities hold.

Now we prove the general result with sharp constants. Theorem 4.2 is implied by the following result.

Let $s\in\mathcal{X}^{n}$, $n\geq 2$, $t,T>0$, and $\psi:\mathcal{X}\to$. For $i\in[n]$, define

where $1+\zeta=\left(1+\frac{1}{n-1}\right)^{2}\left(1+\frac{T}{tn}\left(1+\frac{1}{n-1}\right)^{2}\right)$.

In particular, $\zeta=O\left(\frac{1}{n}+\frac{T}{tn}\right)$ and, if $n\geq 20$ and $\frac{T}{t}\leq\frac{n}{10}$, then $\eqref{eqn:AKL-bound}\leq\frac{1}{n^{2}}\max\{t,T/t\}$.

Combining (20), (15), (18), and (19), we have

2 Accuracy guarantees

Note that, by the postprocessing property of average leave-one-out KL stability, applying any function $f$ to the transcript of an average leave-one-out KL stable algorithm still is average leave-one-out KL stable. More precisely, for a $\varepsilon$-ALKL stable interactive algorithm $M$ and any interactive algorithm $A$, the composed algorithm mapping input $s$ to output $f(A{\rightarrow\atop\leftarrow}M(s))$ is $\varepsilon$-ALKL stable. We now invoke the generalization properties of average leave-one-out KL stability:

This follows from our generalization results (Propositions 3.10 and 3.13), postprocessing, and the connection between average leave-one-out KL stability and mutual information (Proposition 3.4). ∎

A natural choice of function $f$ is to simply pick out one of the queries — that is, $f(\psi_{1},\ldots,\psi_{k},v_{1},\ldots,v_{k})=\psi_{j}$ for some fixed $j$. However, $f$ can also pick out the “worst” query. For example, the monitor technique of [BNSSSU16] takes

The monitor technique allows us to reason about a single query and derive bounds that apply to all queries simultaneously as this query is the worst query. Since we have a more refined notion of error, we must use a slightly different argument.

Note that we flip the sign to ensure that the error is always positive.

Now we give a bound on the expected scaled error of our algorithm. We use the following simple technical lemma bounding the maximum of standard Gaussians.

Let $\xi_{1},\ldots,\xi_{k}$ be independent samples from $\mathcal{N}(0,1)$. Then

Let $X=\max_{i\in[k]}\xi_{i}^{2}$ and $t>0$. Since $\cosh(\sqrt{z})=\frac{1}{2}(e^{\sqrt{z}}+e^{-\sqrt{z}})$ is a convex function of $z\geq 0$, Jensen’s inequality gives

as $\cosh(v)\geq e^{v}/2$ and, hence, $\cosh^{-1}(u)\leq\ln(2u)$. Setting $t=\sqrt{2\ln(2k)}$ completes the proof. ∎

Fix $n,k\geq 20$. Let $M$ be our algorithm from Figure 1 with $T=n^{2}/k$ and $t=n\sqrt{2\ln(2k)/k}$ that answers $k$ statistical queries given $n$ samples.

Let ${\mathcal{P}}$ be a distribution on $\mathcal{X}$ and let $A$ be an interactive algorithm that asks $k$ statistical queries. Then

where $\tau=\sqrt{\frac{\sqrt{2k\ln(2k)}}{n}}$.

Set $\varepsilon=\tau^{2}=\frac{kt}{n^{2}}=\frac{t}{T}=\frac{\sqrt{2k\ln(2k)}}{n}$. Since $n\geq 20$ and $T\leq\min\{t^{2},nt/10\}$, $A{\rightarrow\atop\leftarrow}M$ is $\varepsilon$-ALKL stable by Theorem 4.2. Let $S\sim{\mathcal{P}}^{n}$ and $(\psi,v)\sim A{\rightarrow\atop\leftarrow}M(S)$. Define

Let $\sigma_{*}=\sigma_{j_{*}}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(\psi_{j_{*}}(S_{i})-S[\psi_{j_{*}}])^{2}}$ be the empirical standard deviation corresponding to the query $\psi_{*}$ and the sample $S$. By Lemma 4.4,

Let $\xi_{1},\ldots,\xi_{k}$ be the independent standard Gaussians sampled by $M$ (Figure 1). Let $\xi_{*}=\xi_{j_{*}}$ if $v_{j_{*}}\geq{\mathcal{P}}[\psi_{j_{*}}]$ and $\xi_{*}=-\xi_{j_{*}}$ if $v_{j_{*}}<{\mathcal{P}}[\psi_{j_{*}}]$. By the definition of our algorithm,

Combining with (25) completes the proof. ∎

Acknowledgements

We thank Adam Smith for his suggestion to analyze the generalization of ALKL stable algorithms via mutual information. This insight greatly simplified our initial analysis and allowed us to derive additional corollaries presented in Section 3.3. We also thank Nati Srebro for pointing out the connection between our results and the PAC-Bayes generalization bounds.

References