DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Introduction

The deep learning architecture of transformer (Vaswani et al., 2017) is now gaining popularity in computer vision and has been widely utilized in natural language processing (NLP). Transformer-based language models (LMs), such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018, 2019), have remarkably achieved state-of-the-art performance in almost every NLP task. They are first pre-trained on massive (public) self-labeled corpora and then fine-tuned for various tasks using much smaller, potentially private corpora. It avoids training from scratch and the possible shortage of task-specific corpora while earning versatility.

Training data contributing to the improved utility of fine-tuned LMs can be sensitive. LMs can (unintentionally) memorize them (Carlini et al., 2019) and become vulnerable to membership inference attacks (MIAs) (Shokri et al., 2017) that identify whether an example is in the training set. Worse still, verbatim training text (e.g., SSNs) can be extracted via only black-box access to GPT-2 (Carlini et al., 2021). It is also possible to recover personal health information (e.g., patient-condition pairs) from BERT trained over a clinical corpus (Lehman et al., 2021) based on the extraction attack (Carlini et al., 2021).

Differential privacy (DP) (Dwork et al., 2006) has emerged as the de facto privacy standard for protecting individual privacy. To thwart MIAs on individuals’ training data, DP stochastic gradient descent (DP-SGD) (Abadi et al., 2016) can be used. It clips the gradients of each example in a batch and adds random Gaussian noise to the aggregated gradient. It is more general than earlier attempts (Chaudhuri and Monteleoni, 2008; Chaudhuri et al., 2011) that focus on convex problems and has been implemented in modern ML frameworks, such as PyTorch and TensorFlow. One can apply it to fine-tune LM-based NLP pipelines while ensuring example-level privacy, assuming each individual contributes an example, typically a sequence-label pair.

Unfortunately, DP-SGD often uses a trusted party to curate users’ sensitive training data. Although it can be done distributively (Bonawitz et al., 2017; McMahan et al., 2018) via secure aggregation (Chase and Chow, 2009) with extra costs and trust assumptions, it offers central DP (CDP) at its core.Distributed DP-SGD adds local noise too small to achieve LDP. But it is protected by secret sharing. When all shares are aggregated, they cancel out each other, assuming an honest majority. It thus faces a “synchronization” issue begging for identification and recovery mechanisms with computation and communication overheads (Bonawitz et al., 2017). Instantiating per-example gradients as large as entire pipelines (e.g., ${>}110$M parameters for BERT-Base) is obliviously costly. Moreover, maintaining the utility of pipelines trained by the noisy aggregated one is tricky due to the dimensional “curse.” A recent study (Yu et al., 2021, Table $4$) shows that the average accuracy in fine-tuning LMs for four NLP tasks at moderate privacy is $65.7\%$ (vs. $91.8\%$ without DP). Finally, the inference-time embeddings are not perturbed by the noise added during training, leaving inference queries vulnerable to various recovery attacks (Song and Raghunathan, 2020; Pan et al., 2020), ranging from sensitive attributes (e.g., authorship) to raw text.

We propose DP-Forward, a radically different approach that perturbs forward-pass signals: Users can locally inject noise into the embeddings of (labeled) sequences before sharing them for training, in contrast to perturbing gradients in back-propagation (possibly by an untrusted party). It is meant for provable local DP (LDP) guarantees, thus protecting against stronger adversaries than DP-SGD.

Our approach also naturally fits the federated learning (FL) setting that does not gather users’ data but with substantial differences – FL typically shares noiseless local model updates. Note that any subsequent computation (e.g., gradient computation) on noisy embeddings incurs no extra privacy loss due to the free post-processing of LDP. One might force DP-SGD to offer LDP by adding “enough” noise to the orders-of-magnitude larger per-example gradient from a user, but it may yield unusable models at a similar privacy level.

DP-Forward also extends its applicability to inference via adding noise to users’ test-sequence embeddings, ensuring LDP as in training. As a “side” benefit, it can effectively mitigate emerging embedding-based privacy risks (Pan et al., 2020; Song and Raghunathan, 2020) beyond MIAs.

It is evident that the design goals of DP-Forward naturally align in tandem with our overarching objectives: LDP (vs. CDP), more direct protection of raw data (vs. gradients) against new threats (Pan et al., 2020; Song and Raghunathan, 2020), and can be as efficient as regular non-private training (allowing batch processing of noisy embeddings). The foundation supporting these desiderata, unfortunately, was unavailable. A dedicated mechanism to perturb the forward-pass signals is indispensable.

Specifically, we need to derive noises for embeddings of training/inference text sequences obtained through the forward pass of LM-based pipelines as a real- and matrix-valued function. One might adopt the classical Gaussian mechanism (GM) (Dwork and Roth, 2014) to add i.i.d. noise drawn from a univariate Gaussian distribution. Yet, GM calibrates its noise variance based solely on a sufficient condition for DP, and its variance formula is not applicable to a low privacy regime (Balle and Wang, 2018). Another candidate is the matrix-variate Gaussian (MVG) mechanism (Chanyaswad et al., 2018), tailored for matrix-valued data: It exploits possibly non-i.i.d. noise from a matrix Gaussian distribution to perturb more important rows/columns less. Although it may show better utility over GM (Chanyaswad et al., 2018), it is still sub-optimal due to the sufficient condition.

To optimize MVG, we propose an analytic matrix Gaussian mechanism (aMGM) by integrating a necessary and sufficient condition from the analytic GM (aGM) (Balle and Wang, 2018) for non-i.i.d. noise calibration. Our challenge lies in manipulating the two covariance matrices instead of a single variance. We deduce a constraint only on the two smallest singular values (Section 4.2), indicating that i.i.d. noise (as in aGM) may already be optimal for general applications like DP-Forward. With extra assumptions, dedicated allocation of other singular values by optimizing/maximizing utility functions specific to applications could help.

A transformer-based pipeline contains an input embedding layer, encoders, and task layers. All these layers prominently manipulate embeddings of text inputs in training and subsequent inference. We investigate adding aMGM noise to embeddings output by any hidden (sub-)layer before task layers (Figure 1). To ensure sequence-level LDP, we need to estimate the $L_{2}$-sensitivity (Dwork and Roth, 2014) of “pre-noise” functions for any two sequences. It is non-trivial since the functions can include different (sub-)layers that may not even be Lipschitz (Kim et al., 2021). Our strategy is to normalize the function outputs to have a fixed Frobenius (or $L_{2}$) norm, similar to gradient clipping (Abadi et al., 2016). It works especially well for deeper sub-layers, achieving comparable task accuracy to the non-private baseline (Section 5). For the first few (sub-)layers, we also make two specializations in relaxing LDP to the token level, elaborated in Appendix A.2, to improve accuracy.

2. Our Contributions

Motivated by prevailing privacy concerns in LM fine-tuning and inference and inherent shortcomings of DP-SGD, we initiate a formal study of an intuitive but rarely studied approach and explore its integration with a transformer-based NLP pipeline. Specifically:

1) We propose DP-Forward fine-tuning, which perturbs the forward-pass embeddings of every user’s (labeled) sequence. It offers more direct protection than DP-SGD perturbing aggregated gradients. Its provable guarantee (Theorem 3) is a new sequence-level LDP notion (SeqLDP, Definition 2), with the more stringent $(\epsilon,\delta)$-LDP guarantee to hold w.r.t. only sequences. Moreover, DP-Forward can naturally extend to inference, ensuring the standard LDP (Theorem 5) for test sequences without labels, whereas DP-SGD cannot.

2) To instantiate an optimal output perturbation mechanism for DP-Forward, we propose aMGM, owning independent interests for any matrix-valued function. By exploiting a necessary and sufficient DP condition from aGM (Balle and Wang, 2018), it can draw possibly non-i.i.d. noise from a matrix Gaussian distribution like MVG (Chanyaswad et al., 2018) while producing orders-of-magnitude smaller noise for high-dimensional data (Section 5.3).

3) We conduct experimentsOur code is available at https://github.com/xiangyue9607/DP-Forward. on three typical NLP tasks in Section 5, showing how crucial hyperparameters (e.g., the sequence length) impact task accuracy. To fairly compare with DP-SGD on privacy-vs.-utility: i) We perturb labels by the randomized response (Warner, 1965) such that DP-Forward fine-tuning offers the standard LDP for sequence-label pairs (Theorem 4). ii) We “translate” DP-Forward with standard LDP to (example-level) CDP (as offered by DP-SGD) via shuffling (Erlingsson et al., 2019). Our accuracy gain (for deep-layer DP-Forward instantiations) is up to $7.7$ percentage points (pp), compared to DP-SGD or its recent improvements (Yu et al., 2021, 2022) (reviewed in Section 7.3), at a similar privacy level. Efficiency-wise, DP-SGD incurs ${>}3\times$ time and GPU-memory costs even with the latest Opacus library (Yousefpour et al., 2021).

4) We evaluate three classes of privacy threats. Like DP-SGD, DP-Forward (including the two token-level designs in Appendix A.3) can effectively defend against sequence-level MIAs, but only DP-Forward can thwart the two threats on (inference-time) embeddings. Specifically, Section 6 shows that DP-SGD totally fails in two embedding inversion attacks, while DP-Forward remarkably reduces their success rates by up to $88$pp. For a neural-network-based attribute inference attack, DP-SGD reduces its success rates by only $15$pp on average, while DP-Forward achieves ${\sim}41$pp reduction, making the attack predict like assigning all labels to the majority class.

In short, DP-Forward is a better alternative to DP-SGD in training (and testing) deep-learning models, e.g., gigantic LM-based ones.

Preliminaries and Notations

Modern transformer-based LMs, including BERT (Devlin et al., 2019) and GPT (Radford et al., 2018), are first pre-trained on enormous unannotated (public) corpora to learn contextualized text representations. Later, they can be fine-tuned for various downstream NLP tasks (e.g., sentiment analysis, question answering) using much smaller, task-specific datasets.

We consider BERT (Figure 1), which comprises a stack of $L$ identical layers (i.e., bidirectional transformer encoders (Vaswani et al., 2017)). Each layer has two sub-layers: the dot-product multi-head attention (MHA) (Vaswani et al., 2017) with $h$ heads and a feed-forward network (FFN). Each sub-layer has an extra residual connection, followed by layer normalization (Ba et al., 2016).

FFN is composed of two linear mappings with a ReLU activation in between. It separately and identically operates on each $x_{i\in[1,n]}$,

where $W_{1}$, $W_{2}$, $b_{1}$, and $b_{2}$ are trainable matrix/vector-valued parameters. Its output on $X$ is $\mathsf{FFN}(X)=[\mathsf{FFN}(x_{1})^{\top}||\cdots||\mathsf{FFN}(x_{n})^{\top}]$. The residual connection for sub-layers is $X+\mathsf{MHA}(X)/\mathsf{FFN}(X)$. The layer normalization $\mathsf{LN}(x_{i})$ normalizes all $x_{i}$ entries to have zero mean and unit variance using an extra scale-then-shift step.

The pre-training of BERT is based on two self-supervised tasks: masked language model (MLM) and next sentence prediction (Devlin et al., 2019). We adopt MLM: It randomly masks out some tokens, indexed by $\mathcal{I}$, in an input sequence $X$. The objective is to predict those masked tokens using their context by minimizing the cross-entropy loss

where $\theta$ denotes all the parameters of BERT transformer encoders.

2. (Local) Differential Privacy

DP (Dwork et al., 2006) is a rigorous, quantifiable privacy notion. It has two popular models, central and local. In central DP, a trusted data curator accesses the set $\mathcal{X}$ of all individuals’ raw data and processes $\mathcal{X}$ by a randomized mechanism $\mathcal{M}$ with some random noise. Formally:

For privacy parameters $\epsilon\geq 0$ and $0\leq\delta\leq 1$, $\mathcal{M}$ fulfills $(\epsilon,\delta)$-DP if, for all neighboring datasets $\mathcal{X}$ and $\mathcal{X}^{\prime}$ (denoted by $\mathcal{X}\simeq\mathcal{X}^{\prime}$) and any subset $\mathcal{O}$ of the outputs of $\mathcal{M}$,

We call it $\epsilon$-DP or pure DP when $\delta=0$.

The neighboring notion is application-dependent (to be discussed in Section 3.1). Typically, it involves the “replace-one” relation: $\mathcal{X}^{\prime}$ can be obtained from $\mathcal{X}$ by replacing a single individual’s data point (e.g., a sequence-label pair). CDP offers plausible deniability to any individual in a dataset. In contrast, local DP (LDP) (Kasiviswanathan et al., 2008) removes the trusted curator, allowing individuals to locally perturb their data using $\mathcal{M}$ before being sent to an untrusted aggregator for analytics.

For $\epsilon\geq 0,0\leq\delta\leq 1$, $\mathcal{M}$ is $(\epsilon,\delta)$-LDP if, for any two inputs $X,X^{\prime}$ and any possible output subset $\mathcal{O}$ of $\mathcal{M}$,

Similarly, we call it $\epsilon$-LDP when $\delta=0$.

For a specific pair of inputs $\mathcal{X}\simeq\mathcal{X}^{\prime}$, the privacy loss (or the “actual $\epsilon$ value”) (Balle and Wang, 2018) incurred by observing an output $O$ is the log-ratio of two probabilities:

When $O$ varies according to $\mathcal{M}(\mathcal{X})$, we get the PLRV $\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}$. A helpful way to work with DP is to analyze tail bounds on PLRVs (Dwork and Roth, 2014), which we utilize to build our proposed mechanism in Section 4.2.

DP has two desirable properties: free post-processing and composability. The former means that further computations on the outputs of an $(\epsilon,\delta)$-DP mechanism incur no extra privacy loss. The latter allows us to build more complicated mechanisms atop simpler ones: sequentially (and adaptively) running an $(\epsilon,\delta)$-DP mechanism for $k$ times on the same input is at least $(k\epsilon,k\delta)$-DP. The two properties also hold for LDP when considering a dataset has only one row.

where $||\cdot||_{F}$ denotes the matrix Frobenius norm (Horn and Johnson, 2012).

Table 1 summarizes the acronyms throughout this work.

DP-Forward

We study BERT-based pipelines as an example due to their superior performance in classification tasks. DP-Forward can be readily applied to other (transformer-based) NLP or computer vision models that involve matrix-valued computation during the forward pass.

Suppose each user holds a sequence-label pair $(X,y)$ or only $X$ for fine-tuning or testing a pipeline at an untrusted service provider. Sharing redacted $X$ (with common PII removed) or its feature, a non-human-readable real-valued embedding matrix, is leaky (Sweeney, 2015; Song and Raghunathan, 2020; Pan et al., 2020).

For DP-Forward training, users perturb their embedding matrices locally to ensure (new notions of) LDP before being shared, and they should also perturb the corresponding labels if deemed sensitive (Section 3.4). We explore different options for splitting pipelines into pre-noise functions $f(\cdot)$ and post-noise processing $p(\cdot)$ in Section 3.2: Users can access $f(\cdot)$ to derive embedding matrices, perturbed by an output perturbation mechanism $\mathcal{M}$ (e.g., GM); the service provider runs $p(\cdot)$ on noisy (labeled) embeddings for fine-tuning (Section 3.3) or pre-training (Section 3.6). The challenge lies in analyzing $S_{2}(f)$ for different pipeline parts, which we address by normalizing $f(\cdot)$.

DP-Forward can be naturally used to protect inference sequences (Section 3.5), unlike DP-SGD. It exploits the free post-processing (i.e., inference works on noisy embeddings), incurring minimal changes to pipelines with the extra “plug-and-play” noise layer.

Embeddings $f(X)$ encode semantic information of input sequences $X$, each of which has $n$ tokens (Section 2.1). Fine-tuning (or subsequent inference of) NLP pipelines essentially processes $f(X)$. DP-Forward fine-tuning protects every $X$ by an output perturbation mechanism $\mathcal{M}$ over $f(X)$, in contrast to DP-SGD, which perturbs aggregates of gradients $f^{\prime}(X,y)$ over $X$ and label $y$. Simply put, our $(\epsilon,\delta)$-LDP holds for $X$ while DP-SGD provides CDP for $(X,y)$.

Sequence-only protection is meaningful since sequences often convey (implicit) sensitive information (e.g., authorship), whereas labels (e.g., a single bit denoting positive/negative) can be public. We defer to Section 3.4 for achieving “full” LDP over $(X,y)$. To bridge the gap between theoretical guarantees of DP-SGD and DP-Forward, we first define sequence DPOne could generalize it to “feature” (or “input”) DP, as DP-Forward also allows other types of features beyond embeddings (and its essence is input-only privacy). To keep our focus on NLP, we use “sequence” here. (PixelDP (Lécuyer et al., 2019) treats pixels as image features.) (SeqDP) in the central setting.

​​For $\epsilon\geq 0,0\leq\delta\leq 1$, $\mathcal{M}$ is $(\epsilon,\delta)$-SeqDP, if $\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$ that only differ in a sequence at some index $i$: $(X_{i},y_{i})\in\mathcal{X}$ and $(X^{\prime}_{i},y_{i})\in\mathcal{X}^{\prime},\forall X_{i},X^{\prime}_{i}$, and any possible output subset $\mathcal{O}$,

The recently proposed notion of label DP (Ghazi et al., 2021; Esmaeili et al., 2021) is originally studied in PAC learning (Chaudhuri and Hsu, 2011). It only protects labels (not the corresponding inputs/images): $(\epsilon,\delta)$-DP is only w.r.t. labels.

Our SeqDP is “more secure” than or at least “complements” label DP, which has an inherent flaw (Busa-Fekete et al., 2021): As labels typically rely on their sequences (but not vice versa), it is very likely to recover the true labels from the raw sequences, even if the labels are protected (by any label-DP mechanism). The follow-up (Wu et al., 2023) shows the impossibility of label protection under label DP even with arbitrarily small $(\epsilon,\delta)$ when models generalize. Moreover, labels can be absent (e.g., inference or self-supervised learning), for which SeqDP upgrades to the standard $(\epsilon,\delta)$-DP, whereas label DP is simply inapplicable.

1.2. Sequence Local DP (SeqLDP)

We further define SeqLDP, the local counterpart of sequence DP. Note that the above discussion of label DP in relation to SeqDP also carries over to SeqLDP.

​​For $\epsilon\geq 0,0\leq\delta\leq~{}1$, $\mathcal{M}$ satisfies $(\epsilon,\delta)$-SeqLDP, if $\forall X,X^{\prime}$ with the same $y$, and any possible output subset $\mathcal{O}$,

In theory, SeqLDP remains a strong notion (like the standard LDP). It is meant to be information-theoretic protection on sequence and bounds the indistinguishability of any $X,X^{\prime}$ (differing by up to $n$ tokens), and hence governing the “usefulness” of noisy embeddings.

1.3. Sequence-Level SeqLDP vs. Token-Level SeqLDP

In practice, as a strong notion balancing seemingly conflicting requirements (ideal theoretical guarantees and empirical utility), attaining a meaningful range of $\epsilon$ for SeqLDP is a struggle. Adding Gaussian noise to the outputs of $f(\cdot)$ for $(\epsilon,\delta)$-SeqLDP requires bounding the $L_{2}$-sensitivity $S_{2}(f),\forall X,X^{\prime}$. Our approach is to normalize the outputs (with extra benefits elaborated in Section 3.2), similar to clipping gradients in DP-SGD. It generally works better when $f(\cdot)$ has more layers (at the same meaningful range of $\epsilon$) since fewer (trainable) parameters/layers of $p(\cdot)$ are “affected” by the noisy outputs.

Unfortunately, when $f(\cdot)$ includes the first few layer(s), e.g., only the input embedding layer is available to the users (say, for saving user-side storage and computation overheads), it leads to poor utility. As a comprehensive study, we resort to row-wise normalization with the (composition of) Lipschitz constants (Kim et al., 2021) to maintain utility for those cases.One might also resort to the weaker random DP (Hall et al., 2013) – $(\epsilon,\delta)$-DP holds on all but a small $\gamma$-proportion of “unlikely” $\mathcal{X}\simeq\mathcal{X}^{\prime}$ for an extra parameter $\gamma\in(0,1)$. It is useful when the global sensitivity is hard to compute. Exploring it is left as future work. In contrast to the general normalization, it aims for weaker SeqLDP at the token level (cf. event-level vs. user-level LDP (Zhou et al., 2022)), a finer granularity in the “protection hierarchy,” protecting any neighboring sequences (vs. datasets) differing in any single token (vs. sequence). Details are deferred to Appendix A.

2. Our Approach for Sequence LDP

DP-Forward in our paper (except Appendix A) applies the general normalization approach to any $f(\cdot)$ for sequence-level (Seq)LDP.

Since $\mathcal{M}$ can work on the output of any hidden layer, estimating $S_{2}(f)$ is non-trivial. Specifically, $\mathsf{MHA}$ itself, let alone more layers included, is not Lipschitz continuous, meaning its outputs can change arbitrarily for even slight input variation (Kim et al., 2021). To address this, our approach is to normalize or clip the function outputs:

as in DP-SGD (Abadi et al., 2016), where $C$ is a tunable parameter. We then have $S_{2}(f)=2C$. Such normalization makes task utility less “sensitive” to the choice of $C$ since signal and noise increase proportionally with $C$, whereas the signal may be unchanged when $f(\cdot)$ is not clipped. It also has many other benefits, such as stabilizing training, avoiding overfitting, and accelerating convergence (Aboagye et al., 2022). Hence, we resort to normalization in our experiments. One can then calibrate Gaussian noise $Z$ and derive $f(X)+Z$ for the post-noise layers $p(\cdot)$.

Note that we remove the residual connection when adding noise to the output of the first MHA layer to avoid $p(\cdot)$ reaccessing $X$ (dashed arrow, Figure 1) to maintain free post-processing. This may lead to instability (e.g., gradient vanishing) (Yu et al., 2021), but it can be mitigated by pre-training new BERT without such a residual connection to keep consistent with later fine-tuning/inference.

3. DP-Forward Fine-tuning

Suppose we use a raw, public BERT checkpointUsing noisy BERT for fine-tuning (and subsequent inference) is deferred to Section 3.6. for fine-tuning. In the forward pass of the $i$-th ($i\geq 1$) step, it offers the latest $f^{(i-1)}(\cdot)$ to a batch of users, mimicking the regular mini-batch SGD. $f^{(0)}$ is from the raw checkpoint. Users are randomly chosen (without replacement), and their number is a fixed parameter. Users in the batch individually compute their noisy embeddings $f^{(i-1)}(X)+Z$ to ensure SeqLDP (Theorem 3). They then send them with unperturbed labels $y$ to the service provider, who runs $p^{(i-1)}(\cdot)$ over $(f^{(i-1)}(X)+Z,y)$ to compute the batch loss; any post-processing of embeddings under SeqLDP incurs no extra privacy degradation on $X$. $p^{(0)}$ here includes the rest raw BERT part and randomly initialized task layers.

During the back-propagation, the service provider can update $p^{(i-1)}(\cdot)$ to $p^{(i)}(\cdot)$ via the gradient (derived from the loss and noisy embeddings) of the post-noise layers. To avoid accessing users’ raw $X$, it needs to freeze the pre-noise layers $f^{(i-1)}(\cdot)$ as $f^{(0)}$. Parameter freezing is compatible with the more recent zero-shot or in-context learning paradigm (Min et al., 2022). It is useful when models are gigantic and full fine-tuning is expensive. However, the more layers are frozen, the worse the utility might be (even in non-private settings).

There are two general ways to update $f^{(i-1)}(\cdot)$ securely: i) We can assume an extra trusted party (as in DP-SGD), but it becomes central DP. ii) Users can first derive the gradients for the layers inside $f^{(i-1)}(\cdot)$ locally on their $X$ and then resort to secure aggregation (Bonawitz et al., 2017) for global updates at the service provider. However, it is costly. For better utility, we update $f^{(i-1)}(\cdot)$ in experiments, requiring us to consider privacy degradation across different epochs due to the composability (as detailed below). Dedicated approaches (that balance efficiency, privacy, and utility) are left as future work.

Let $f(\cdot)$ be the pre-noise function (of BERT-based pipelines) and $\mathcal{M}$ be GM with $\epsilon\geq 0,0\leq\delta\leq 1$. DP-Forward fine-tuning running $\mathcal{M}$ on normalized/clipped $f(\cdot)$ ensures $(\epsilon,\delta)$-SeqLDP.

The proof follows that of GM (Dwork and Roth, 2014). The crux is that $S_{2}(f),\forall X,X^{\prime}$ is given by the output normalization, independent of the inputs.

Privacy Accounting. An epoch refers to an entire transit of the private training corpus. Every $X$ is used once per epoch. The number of epochs $k$ is a hyperparameter, which is typically small. Repeated applications of GM over the same $X$ ask for estimating the overall privacy loss due to the composability (unless freezing $f$ for re-using $f(X)+Z$). The well-known moments accountant (Abadi et al., 2016) (or its generalization to Rényi DP (Mironov, 2017)) only provides a loose upper bound, which is even inapplicable if unbounded moments exist. Gaussian DP (Bu et al., 2019) proposes an accountant based on the central limit theorem. Yet, it leads to significant underestimation by a lower bound. Instead, we resort to a recent numerical accountant (Gopi et al., 2021), which outperforms RDP or GDP by approximating the true overall $\epsilon$ to arbitrary accuracy. It composes the privacy curve of a mechanism by truncating and discretizing PLRVs with their PDFs convoluted by FFT (Gopi et al., 2021).

4. DP-Forward with Shuffling versus DP-SGD

DP-Forward ensures SeqLDP for fine-tuning, while DP-SGD offers central DP (for sequence-label pairs). To facilitate a fair comparison (on privacy-utility tradeoffs), we make two changes. First, we also perturb the labels with a suitable mechanism for the standard LDP, i.e., extending the protection from sequence to sequence-label pairs. Second, we use shuffling (Erlingsson et al., 2019) to “translate” our (label-protected) DP-Forward with LDP to claim (example-level) CDP as DP-SGD.

Discrete Labels Perturbation. For most NLP tasks, e.g., bi-/multi-nary classification in the GLUE benchmark (Wang et al., 2019), the size $|\mathbf{y}|$ of label space is often small. A simple yet effective solution for discrete data is randomized response (RR) (Warner, 1965) proposed decades ago! Specifically, RR perturbs a true label $y$ to itself $\hat{y}=y$ with the probability

or to $\forall\hat{y}\in\mathbf{y}\setminus y$ uniformly, where $\mathbf{y}$ denotes the label space.

When $|\mathbf{y}|$ is large, we can use prior to “prune” $\mathbf{y}$ to smaller $\mathbf{y}^{\prime}$ (Ghazi et al., 2021). The prior can be publicly available (e.g., auxiliary corpora similar to the users’ data) or progressively refined from a uniform distribution via the multi-stage training (Ghazi et al., 2021). One can then estimate an optimal $|\mathbf{y}^{\prime}|$ by maximizing the probability that the output is correct, i.e., $\Pr[y=\hat{y}]$. With (prior-aided) RR (Ghazi et al., 2021), we can achieve full LDP.

Let $f(\cdot)$ be the pre-noise function (of BERT-based pipelines), $\mathcal{M}$ be GM with $\epsilon_{1}\geq 0,0\leq\delta\leq 1$, and $\mathcal{M}_{RR}$ be (prior-aided) RR with $\epsilon_{2}\geq 0$. DP-Forward fine-tuning perturbing $f(X)$ and $y$ separately by $\mathcal{M}$ and $\mathcal{M}_{RR}$ ensures $(\epsilon_{1}+\epsilon_{2},\delta)$-LDP.

The proof follows from the basic composition theorem (Dwork and Roth, 2014).

Privacy Amplification by Shuffling. If noisy embedding-label pairs are also shuffled properly, DP-Forward can claim example-level CDP (as in DP-SGD), which “amplifies” LDP guarantees by $\Theta(\sqrt{N})$ for a total number of $N$ users (without extra noise addition) (Erlingsson et al., 2019). We then show that DP-Forward qualitatively outperforms DP-SGD from the SNR perspective under a similar privacy regime.

Suppose we train for an epoch, and the normalization factor is $C$. For DP-SGD, the batch size is $b$; the subsampling probability and the number of training steps are respectively $b/N$ and $N/b$. If each step is $(\epsilon,\delta)$-DP, the overall privacy loss is $(O(\epsilon\sqrt{b/N}),\delta)$-DP using the strong composition and privacy amplification by subsampling (Abadi et al., 2016).

DP-Forward with shuffling can also be seen as composing $N$ subsamplings, each a fraction of size $1$ (Steinke, 2022). It is $(O(\epsilon\sqrt{1/N}),\delta)$-DP, which is “amplified” from $(\epsilon,\delta)$-LDP. For an easier analysis of SNR, we omit $\epsilon_{2}$ of RR since the overall $\epsilon$ is dominated by composing subsampled Gaussian. So, our Gaussian noise variance is $b\times$ smaller than DP-SGD’s in each step; the SNR of each entry in embeddings vs. the aggregation of $b$ gradients can be estimated as $O(C/\sqrt{nd})$ for DP-Forward vs. $O(C/\sqrt{d^{\prime}})$ for DP-SGD, where $d^{\prime}$ is the gradient dimension and is much larger than $nd$, the embedding-matrix size.

5. DP-Forward Inference

Given only fine-tuned pipeline parts $f(\cdot)$, users can derive the noisy embedding matrices of their test sequences for inferences at the service provider while ensuring $(\epsilon,\delta)$-LDP. Inference using noise aligned to the noisy fine-tuning is also beneficial for task accuracy.

Local inference (as in DP-SGD) without noise forces the service provider to reveal its entire pipeline, losing its intellectual property and incurring more time and storage costs for both $f(\cdot)$ and $p(\cdot)$.

Let $f(\cdot)$ be the fine-tuned pre-noise layers (of BERT-based pipelines) and $\mathcal{M}$ be GM with $\epsilon\geq 0,0\leq\delta\leq 1$. DP-Forward inference running $\mathcal{M}$ on normalized/clipped $f(\cdot)$ ensures $(\epsilon,\delta)$-LDP.

The proof is inherited from GM (Dwork and Roth, 2014). Different from DP-Forward fine-tuning, LDP holds for test sequences since the labels are absent.

6. DP-Forward Pre-training

Directly using the raw BERT might not “match” DP-Forward fine-tuning/inference, degrading task utility. Pre-training BERT with DP-Forward on publicly available text (e.g., Wikipedia), besides the private user-shared data, can make future operations “adaptive” to noise. It requires us to modify the raw MLM objective in Eq. (1):

where $\theta^{*}$ denotes the parameters of “noisy” BERT. This endows the noisy BERT with some “de-noising” ability since the objective is to predict the raw masked tokens from noisy embeddings $\mathcal{M}(f(\hat{X}))$. It does not really breach privacy due to the free post-processing; LDP is ensured for each sequence, as the pre-training is self-supervised (without labels). Such noisy pre-training can also be outsourced to dedicated GPU clusters, enabling “de-noising BERT as a service.”

De-noising as post-processing is not new, but most prior arts need prior knowledge, e.g., Bayesian prior. aGM formulates it as an unusual estimation problem since a single noisy output is observed for each input, which can then be solved by appropriate estimators, e.g., the Bayesian one (Balle and Wang, 2018). Another attempt (Lécuyer et al., 2019) trains a separate noisy auto-encoder, which learns the identity function $f(X)=X$ stacked before an image classification network, to de-noise the noisy input. It has limited applications for only noisy input embeddings and incurs extra changes when migrating it to an NLP pipeline.

Optimizing Matrix Gaussian Noise

Another candidate is the matrix-variate Gaussian (MVG) mechanism (Chanyaswad et al., 2018), tailored for matrix-valued functions. It exploits possibly non-i.i.d. noise from a matrix Gaussian distribution and outperforms GM in several usage cases (Chanyaswad et al., 2018). Yet, it is not optimal either, with the root cause still being based on a sufficient DP condition (Section 4.1). To improve it, we resort to a necessary and sufficient condition from aGM (Balle and Wang, 2018) for calibrating the matrix Gaussian noise (Section 4.2).

The PDF for an $n\times d$ random variable $Z$ following $\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$ has the form:

The definition is equivalent to the conventional form given by the matrix trace. It generalizes the univariate Gaussian used in GM; $Z$ becomes i.i.d. when $\Sigma,\Psi$ are diagonal and equal-valued. Below recites the main theorem of the MVG mechanism for $(\epsilon,\delta)$-DP.

be the vectors of (non-increasingly ordered) singular values of $\Sigma^{-1}$ and $\Psi^{-1}$, respectively. The MVG mechanism using noise from the matrix Gaussian distribution $\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$ satisfies $(\epsilon,\delta)$-DP if

where $\alpha=[H_{r}+H_{r,1/2}]\gamma^{2}+2H_{r}\gamma S_{2}(f)$, $\beta=2(nd)^{1/4}H_{r}S_{2}(f)\zeta(\delta)$, with $H_{r}$ (or $H_{r,1/2}$) being the generalized harmonic number of order $r$ (of $1/2$), $\gamma$ being $\sup_{\mathcal{X}}||f(\mathcal{X})||_{F}$, and $\zeta(\delta)=2\sqrt{-nd\ln{\delta}}-2\ln{\delta}+nd$.

Sub-optimality of MVG. Theorem 2 presents an upper bound on the product of $L_{2}$-norms of two singular-value vectors $\sigma(\Sigma^{-1})$ and $\sigma(\Psi^{-1})$, assuming $||f(\mathcal{X})||_{F}$ is bounded for any $\mathcal{X}$ by a constant $\gamma$. The upper bound monotonically decreases with $\beta$ that depends on $nd$ and approaches as $nd\rightarrow\infty$, making the sums of noise variances large. A similar situation exists in high privacy regimes $\epsilon\rightarrow 0$.

At least two slacks caused the sub-optimality. The first and foremost is due to a sufficient condition for $(\epsilon,\delta)$-DP (Dwork and Roth, 2014): $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}\geq\epsilon]\leq\delta$, which is also used in the classical GM. With the Laurent-Massart Theorem (Laurent and Massart, 2000), MVG further transforms it to $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}\leq\epsilon]=1$ for a subset of all the possible outputs. The second lies in a loose matrix-trace-based privacy analysis; a follow-up (Yang et al., 2023) derives a tighter bound from Definition 1 and a matrix-norm inequality.

2. Analytic Matrix Gaussian Mechanism

To enhance MVG while still adding possibly non-i.i.d. noise $Z\sim\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$, we put forth the analytic matrix Gaussian mechanism (aMGM) by exploiting a necessary and sufficient condition for $(\epsilon,\delta)$-DP, which is formulated using two PLRVs by the analytic GM (aGM) (Balle and Wang, 2018). It is non-trivialA recent pre-print (Yang et al., 2021) also studied using matrix Gaussian distribution. The proof of (Yang et al., 2021, Lemma $4$), pivotal for our Theorem 6, is problematic. We prove it in Appendix C. since we now need to work with two covariance matrices $\Sigma$ and $\Psi$ instead of a single variance $\sigma^{2}$ in aGM.

A mechanism $\mathcal{M}$ is $(\epsilon,\delta)$-DP iff, $\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$,

It directly implies the sufficient condition due to $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X}^{\prime},\mathcal{X}}\leq-\epsilon]\geq 0$. We next show that $\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}$ or $\mathcal{L}_{\mathcal{M},\mathcal{X}^{\prime},\mathcal{X}}$ of aMGM is also Gaussian, a similar result has been proven in aGM (Balle and Wang, 2018, Lemma 3).

The PLRVs of our aMGM follow a distribution $\mathcal{N}(\eta,2\eta)$ with $\eta=\frac{||U^{-1}\Delta V^{-\top}||^{2}_{F}}{2}$, where $\Delta=f(\mathcal{X})-f(\mathcal{X}^{\prime})$.

With Lemma 4, we can then specialize the left-hand side of Eq. (3). Particularly, we use the Gaussian cumulative density function (CDF)

to explicitly express the two probabilities (see Lemma 5) instead of approximating them by the tail bounds of a Gaussian distribution.

For any $\mathcal{X}\simeq\mathcal{X}^{\prime}$, let $\Delta^{\prime}=U^{-1}\Delta V^{-\top}$ with $\Delta=f(\mathcal{X})-f(\mathcal{X}^{\prime})$. The following holds for any $\epsilon\geq 0$:

We can further re-write the left-hand side of Eq. (3) as $g(||\Delta^{\prime}||_{F})$:

a function of $\Delta$ and $(\Sigma,\Psi)$; it is defined w.r.t. $\Delta$ and $\sigma^{2}$ for aGM (Balle and Wang, 2018). To satisfy Theorem 3, we require $g(||\Delta^{\prime}||_{F})\leq\delta,\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$. Since $g(\cdot)$ is monotonically increasing (Balle and Wang, 2018, Lemma 7), we first find the upper bound $\mathcal{B}$ of $||\Delta^{\prime}||_{F}$ as the “solution” to $g(||\Delta^{\prime}||_{F})=\delta$ and then determine $U,V$ (hence $\Sigma,\Psi$) based on $\mathcal{B}$ and $\Delta$ with $S_{2}(f)=\sup_{\mathcal{X}\simeq\mathcal{X}^{\prime}}||\Delta||_{F}$.

One could derive an analytic expression for $\mathcal{B}$ using the tail bounds of $\Phi(t)$, which is sub-optimal due to the slack in the tail bounds. Instead, we adapt a “numerical solver,” as detailed in Alg. 1, for $\mathcal{B}$ since $\Phi(t)$ can also be represented by $(1+\mathsf{erf}(t/\sqrt{2}))/2$, where $\mathsf{erf}$ is the standard error function.Its efficient implementation to extremely high accuracy is supported in most statistical and numerical software packages, e.g., Python math library.

For the first term of Eq. (4), its input $||\Delta^{\prime}||_{F}/2-\epsilon/||\Delta^{\prime}||_{F}$ changes sign at $||\Delta^{\prime}||_{F}=\sqrt{2\epsilon}$, while the other term’s input $-||\Delta^{\prime}||_{F}/2-\epsilon/||\Delta^{\prime}||_{F}$ is always negative. Therefore, we only consider $||\Delta^{\prime}||_{F}=\sqrt{2\epsilon}/\alpha$ under two cases $0<\alpha\leq 1$ and $\alpha>1$ for a variable $\alpha$.

When $\alpha=1$, $\delta_{0}=g(\sqrt{2\epsilon})$ in line $1$. If $\delta\geq\delta_{0}$ (or $0<\alpha\leq 1$), we can use $v=(1/\alpha-\alpha)^{2}/2$ to re-write $g(\cdot)$ as $g^{+}_{\epsilon}(v)$ (line 3). For $\alpha>1$, we can use $u=(\alpha-1/\alpha)^{2}/2$ to re-write $g(\cdot)$ as $g^{-}_{\epsilon}(u)$ (line 7). In either case, given the “oracle” computing $\Phi(t)$ via $\mathsf{erf}$, we derive $u^{*}$ or $v^{*}$ using Newton’s method, recover $\alpha$, and return $\mathcal{B}=\sqrt{2\epsilon}/\alpha$.

With Lemma 5, and let $\sigma_{i}(\cdot)$ be the $i^{\text{th}}$ singular value; we have

Since $\sigma_{i}(U^{-1})=1/\sigma_{n-i+1}(U)$ and $\sigma_{i}(V^{-\top})=1/\sigma_{d-i+1}(V)$ with $i\in[1,r]$, we transform the right-hand side of Eq. (5) to

where the inequality follows from Theorem 2, i.e., $\sigma_{1}(\cdot)\geq\cdots\geq\sigma_{r}(\cdot)$, and the last equality is directly from Lemma 4.

Given $\mathcal{B}\geq||U^{-1}\Delta V^{-\top}||_{F}$, it suffices to let $||\Delta||_{F}/\sigma_{n}(U)\sigma_{d}(V)\leq\mathcal{B}$ with $\Delta=f(\mathcal{X})-f(\mathcal{X}^{\prime}),\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$. Recall that $S_{2}(f)$ is the upper bound on $||\Delta||_{F},\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$, we now reach the main theorem.

Our aMGM satisfies $(\epsilon,\delta)$-DP, iff

where $\mathcal{B}=A(\epsilon,\delta)$ as in Alg. 1, $S_{2}(f)$ is the $L_{2}$-sensitivity, $\sigma_{n}(U)$ and $\sigma_{d}(V)$ are respectively the smallest singular values of $U$ and $V$.

Theorem 6 only constrains the lower bound on the product of $\sigma_{n}(U)$ and $\sigma_{d}(V)$, the two smallest singular values; it offers infinite choices for all the others with the design space for $(\Sigma,\Psi)$ even larger than that of MVG (Theorem 2). More importantly, the lower bound is independent of $nd$, which can lead to orders-of-magnitude variance reduction than MVG, confirmed by our experiments in Section 5. For $\epsilon\rightarrow 0$, we can still derive a valid $\mathcal{B}$ from $2\Phi^{-1}((1+\delta)/2)$.

To determine $\Sigma,\Psi$, another implicit constraint is to keep smaller noise for better utility. Let us first consider $\Sigma=UU^{\top}$. Since it is positive definite, we can also decompose it into $W_{\Sigma}\Lambda_{\Sigma}W^{\top}_{\Sigma}$; we then have $U=W_{\Sigma}\Lambda^{1/2}_{\Sigma}$, where $\Lambda^{1/2}_{\Sigma}=\{\sigma_{i}(U)\}^{n}_{i=1}$ specifies the row-wise noise magnitudes. Assuming that the smallest overall noise will yield the best utility, we let all the singular values be the smallest: $\sigma_{1}(U)=\cdots=\sigma_{n}(U)$. As $W_{\Sigma}$ can be any unitary matrix, we simply use the standard basis, resulting in $U=\sigma_{n}(U)\cdot I_{n}$ for an $n\times n$ identity matrix $I_{n}$ and hence the final $\Sigma$. Similarly, we can pick $\Psi=VV^{\top}$ with $V=\sigma_{d}(V)\cdot I_{d}$, where $I_{d}$ is a $d\times d$ identity matrix.

2.3. Drawing the noise Z𝑍Z

With $\Sigma$ and $\Psi$, the last step is to draw $Z$. Pragmatically, we adopt the affine transformation below.

Hence, we can first sample $nd$ i.i.d. values from $\mathcal{N}(0,1)$ to form $Z^{\prime}$, then employ the transformation $UZ^{\prime}V^{\top}$ such that

When instantiating DP-Forward using aMGM, we set $\sigma_{1}(U)=\cdots=\sigma_{n}(U)$ and $\sigma_{1}(V)=\cdots=\sigma_{d}(V)$ such that the row- and column-wise noises are the smallest, and our pilot experiments show this yields optimal task utility; aMGM actually “degenerates” to aGM with i.i.d. noise. Nevertheless, aMGM also allows non-i.i.d. noise like MVG: By tuning the corresponding singular values larger, we can add more noise to the rows/columns that negatively impact the utility. It might be helpful when $f(\cdot)$ (e.g., linear regression on a small liver dataset (Chanyaswad et al., 2018)) is simple or $p(\cdot)$ does not “mix up” noisy rows/columns. In contrast to our empirical approach (like MVG), one could theoretically formulate the allocation of singular values as optimization problems that maximize different utility functions tailored to applications. It might outperform our uniform treatment but takes more dedicated efforts, which we leave as future work.

Experiments

We use three typical datasets/tasks that are widely used in NLP/DP literature (Yu et al., 2021; Li et al., 2022b; Yu et al., 2022; Yue et al., 2021) and GLUE benchmark (Wang et al., 2019): i) Stanford sentiment treebank (SST-2), ii) Internet movie database (IMDb) (Maas et al., 2011) for binary sentiment classification of single- and multi-sentence movie reviews, and iii) Quora question pairs (QQP) for semantic equivalence test over question pairs on Quora.com. Their test sets do not have any labels; we use the original dev sets as the test sets. Table 2 summarizes their characteristics. They all carry privacy risks; e.g., stylistic features of posts may leak the author’s identity. We use task accuracy (w.r.t. the ground truth labels) as the utility metric.

Baselines. We instantiate $\mathcal{M}$ in DP-Forward by the classical GM, MVG (Chanyaswad et al., 2018), and aMGM. If not specified, all the results are based on aMGM. For MVG, we adopt its unimodal type, applicable to asymmetric functions like pre-noise layers $f(\cdot)$. Specifically, we make the row-wise noise directional and assign the same precision budget to each row, assuming that tokens share the same importance.

By default, we report the accuracy of DP-Forward inferences on tasks fine-tuned using DP-Forward (with ${\sim}2$pp gains compared to the case of “DP-Forward fine-tuning + non-private inference”). We also realize DP-SGD fine-tuning with the latest Opacus (Yousefpour et al., 2021) but do not add any noise to its inference. Another baseline is non-private (in both fine-tuning and inference).

Implementation. We run experiments on a cluster with Tesla P$100$ GPUs. We implement all the mechanisms and baselines in Python. We use a raw BERT checkpoint bert-base-uncased (Face, 2023), available in the Huggingface transformers library, for fine-tuning (Section 3.3) or further pre-train it over WikiCorpus (Section 3.6).

2. Configuring Matrix Dimensions

The sequence length $n$ is variable. While the hidden dimensionality $d$ is tied as $768$ for BERT-Base, we can resort to two linear maps for “mediating” it (see Section 3.2). Since we normalize embedding matrices of size $n\times d$ to have a fixed norm $C$, each entry’s signal magnitude relies on $(n,d)$. In contrast, the noise variance is the same given $C$ and fixed privacy parameters. The signal-to-noise ratios (SNRs) affecting accuracy can be configured based on $(n,d)$.

Figure 2 shows the evaluation accuracy of SST-2 fine-tuned using DP-Forward with $n$ tuning from $16$ to $256$. We study adding aMGM noise at five hidden layers’ outputs. The results indicate that the best accuracy is often achieved at $n=64$ or $128$, so we opt for $n=128$ (which is sufficient for most sequences) in subsequent experiments.

We fine-tuned SST-2 on noisy output embeddings under different choices of $\epsilon$ and reduced $d$. Table 3 summarizes the results. Reducing $d$ leads to larger SNRs (under fixed $C$ and $n$) but may also lose useful information, degrading accuracy. For the same $\epsilon$, most accuracy variations are within $2$pp under different choices of $d$. Balancing everything, we use the raw $d=768$ in later experiments such that no extra changes (including two linear maps) are made to pipelines.

3. Fine-tuning with Sequence LDP

Our approach also supports perturbing sub-layer outputs during fine-tuning. We study six encoders as an example, with the results shown in Figure 3. Overall, DP-Forward performs better with deeper encoders since fewer parameters are directly affected by noise during fine-tuning. Another observation is that perturbing different sub-layer outputs, even inside the same encoder, may result in huge accuracy variation; e.g., using noisy outputs of the last sub-layer in Encoder $1$ can bring ${\sim}20$pp gains over those of the first sub-layer.

We next evaluate the privacy-accuracy tradeoffs under different $\epsilon$ and compare the instantiations using the classical GM, MVG (Chanyaswad et al., 2018), and aMGM. Note that we still compute the GM variance as $\sigma^{2}=2\ln(1.25/\delta)S^{2}_{2}(f)/\epsilon^{2}$ for empirical evaluation, albeit it cannot extend to $\epsilon>1$ for a single run to ensure theoretical DP guarantees.

For the GM- and aMGM-based instantiations, Table 4 shows all three tasks’ accuracy increases with $\epsilon$. Ours has better accuracy than the GM-based one due to the smaller noise produced by aMGM in all choices of $\epsilon$. Although the noise variance gap (between GM and aMGM) widens as $\epsilon$ decreases, one cannot fine-tune effective models in a high privacy regime $\epsilon<1$. The MVG-based one behaves like random guessing for all three tasks since its noise variance is proportional to $n\cdot d$, which is even much larger than the classical GM for high-dimensional settings (see Section 4.1). For instance, under the same parameter setting (e.g., $n=128,d=768$, and $\epsilon=8$), MVG produces noise with the variance orders-of-magnitude larger than aMGM (e.g., ${>}10^{8}$ vs. ${\sim}0.6$), even assuming $\sup||f(\cdot)||_{F}=1$.

We remark that the used local $\epsilon$ value is not large. Most classical LDP works that deem such $\epsilon$ lies in a low privacy regime are for statistical analytics. In great contrast, we aim at fine-tuning large LM-based pipelines with high-dimensional signals and limited training data, which is much more complicated. Many prior works (Feyisetan et al., 2020; Qu et al., 2021; Yue et al., 2021; Feyisetan et al., 2019) use a larger $\epsilon$ to ensure even a weaker token-level LDP variant, while others (Meehan et al., 2022) categorizes $\epsilon<10$ and $10\leq\epsilon<20$ as strong and moderate privacy respectivelySuch choices can be “reduced” to smaller ones under the shuffling model (Section 5.4), cf.. U.S. census discloses demographic data at central $\epsilon=11.14$ (Abowd et al., 2022). for sequence-level LDP like ours. More importantly, they provide effective protection against various privacy threats, as detailed in Section 6.

4. DP-Forward versus DP-SGD

Fairness of comparisons on privacy-accuracy tradeoffs. As elaborated in Section 3.4, we can adopt RR (Warner, 1965) to perturb the labels and then report central $\epsilon$ values for DP-Forward, amplified by shuffling using the following parameters, ensuring that comparisons are fair under (example-level) CDP. For DP-SGD, the subsampling probability is $b/N$, with $b=32$ and the dataset size $N$; the number of fine-tuning steps is $T=k\cdot N/b$ with $k=3$. For DP-Forward, the subsampling and non-flipping probabilities are respectively $1/N$ (with $T=k\cdot N$) and $0.9$; we still process $b$ noisy embeddings as a batch. For both, we use aGM (Balle and Wang, 2018), the degenerated version of aMGM (Section 4.2), and the same accountant (Gopi et al., 2021) to report approximated overall $\epsilon$ valuesThey are dominated by composing subsampled Gaussian, e.g., composing subsampled RR only consumes $0.03$ for SST-2, which is even overestimated by AutoDP..

We study eight instances of DP-Forward, including perturbing the outputs of the input embedding layer, six different encoders, and BERT. Their accuracies on all three tasks under three privacy levels, plus those of DP-SGD and the non-private baseline, are shown in Table 5. About half or more of our instances have better accuracy than DP-SGD for each task; the largest accuracy gain is ${\sim}7.7$pp for QQP. The noisy output embeddings often lead to the best accuracy for all tasks, even comparable to the non-private baseline, due to the dimension reduction at the last encoder output (Section 2.1).

Recent DP-SGD variants (Yu et al., 2021, 2022) improve DP-SGD (Abadi et al., 2016) by perturbing partial gradient entries using additional tricks (e.g., low-rank adaption). They report the best accuracy of $92.5\%$ and $85.7\%$ on SST-2 and QQP, respectively, with $2.3$pp and $6.2$pp drops from the non-private baselines at central $\epsilon=6.7$ (Yu et al., 2022, Table $4$). DP-Forward with label privacy, incurring ${<}1.7$pp accuracy drops on the two tasks at $\epsilon\approx 3$, can still beat them, albeit their fine-tuning is based on RoBERTa-base, a robustly optimized BERT approach, which by itself outperforms BERT due to larger training set, longer training time, and better techniques (e.g., dynamic masking in MLM).

Figure 4 shows the efficiency comparisons on fine-tuning SST-2. The time and storage overheads of our approach (for all possible instances) are almost the same as the non-private baseline and ${\sim}3\times$ smaller than DP-SGD. It is because we allow batch processing as in the normal fine-tuning – no need to handle per-example gradients. Meanwhile, our normalization and noise sampling/addition are also faster since the size of embeddings is smaller than that of gradients.

5. Noisy Pre-training

Pre-training BERT using DP-Forward, aligned with the noisy fine-tuning, does help accuracy. We use SST-2 as an example and perturb the input embedding matrices. We continue pre-training BERT over English WikiCorpus, the $2006$ dump with about $600$M words, for an epoch. Table 6 shows that we can obtain $1{-}2$pp accuracy gains for most choices of $\epsilon$, compared to fine-tuning on the original BERT.

Efficiency-wise, DP-Forward pre-training also consumes much fewer resources; e.g., an existing work (Anil et al., 2022) pre-trains BERT-Large (with $340$ million parameters) using DP-SGD on Google TPUs, which requires sufficient memory for handling batch sizes of millions.

Defense against Privacy Threats

Following the recent taxonomy (Song and Raghunathan, 2020), we study MIAs and two new threats of sequence leakage from their embeddings: embedding inversion and attribute inference. We moderately adapt them to suit our context, e.g., upgrading MIAs (Song and Raghunathan, 2020) to sequence-level.

For MIAs, we follow prior arts (Shokri et al., 2017; Yeom et al., 2018; Song and Raghunathan, 2020) to consider an adversary with only black-box access to an entire (DP-SGD/DP-Forward-trained) pipeline: It can query the prediction results (e.g., each-class probability) of target sequences but cannot access the pipeline weights and architecture; the hidden embeddings are not revealed.

DP-SGD only offers CDP for training data and does not protect inference-time input.One might add the same noise to it as DP-Forward inference, which indeed mitigates the new threats. However, perturbing gradients in training, inherently “mismatches” from perturbing embeddings in inference, deteriorating task performance significantly, e.g., SST-2 accuracy will be reduced to $0.7786$ (with a ${\sim}10$pp drop) at central $\epsilon\approx 8$. What follows intends to empirically confirm a major merit of DP-Forward in protecting against stronger adversaries and threats to both training- and inference-time inputs.

2. Membership Inference Attacks

Attack Objective. MIAs predict whether a data point is in the training set (Shokri et al., 2017). They often exploit the disparity in model behavior between training data and unseen data, i.e., poor model generalization due to overfitting (Yeom et al., 2018). Inferring membership at the token/word level, e.g., a sliding window of tokens (Song and Raghunathan, 2020), is not interesting. We consider more realistic MIAs on entire sequences, which can be extended for more devastating attacks, such as extracting verbatim pre-training sequences via black-box access to GPT-2 (Carlini et al., 2021).

Prior arts (Yeom et al., 2018; Song and Mittal, 2021) suggest that threshold-based MIAs using only prediction confidence (Yeom et al., 2018) or entropy (Song and Mittal, 2021) with proper assumptions are comparable to the more sophisticated one (Shokri et al., 2017) based on shadow training. Adapting the confidence-based MIA to our context exploits that a pipeline is fine-tuned by minimizing its prediction loss: The confidence/probability of predicting a training sequence as its true label should be close to $1$. The adversary can then infer a candidate sequence $X^{*}$ as a member when the confidence for the predicted label $l$ output by pipeline $\mathcal{F}$ is larger than a pre-set threshold $\tau$:

where $\mathds{1}\{\cdot\}$ is the indicator function. We simply use a fixed $\tau$ for all possible labels in our evaluation, albeit it can be label-dependent.

The second MIA we use is based on the prediction output (i.e., a vector of probabilities) of a training sequence tends to be a one-hot vector, i.e., its entropy should be close to . Similarly, the adversary can infer $X^{*}$ as a member when its prediction entropy falls below a preset threshold $\tau$; otherwise, it is not deemed a member:

for all possible labels $\{l_{i}\}$. Note that a totally wrong prediction with probability ${\sim}1$ also leads to entropy approaching . We can address it by encoding the information of the ground-truth label of $X^{*}$ (Song and Mittal, 2021).

Numerical Results. As in (Yu et al., 2021), all the test examples and a random subset of the training examples (as many as the test ones) are evenly split into two subsets (each has half of the training/test examples), one for finding the optimal $\tau$, and the other for reporting the attack success rates. Given that the training and test examples likely share the same distribution, we randomly drop/replace tokens in the test examples to enlarge the prediction difference to make MIAs easier.

We evaluated the adapted confidence- and entropy-based MIAs on SST-2 fine-tuned by the non-private baseline, DP-Forward, and DP-SGD. For DP-Forward, we investigate five instances, perturbing input embeddings, three encoders’ outputs, and output embeddings. Table 7 presents the results, where success rates within $0.49$–$0.51$ are shown in bold. Both DP-Forward and DP-SGD can mitigate MIAs effectively. For all choices of $\epsilon$, the two MIAs’ success rates on DP-Forward are reduced to ${\sim}0.5$ (like random guessing) for deeper layers, outperforming DP-SGD by ${>}6$pp at the same privacy level.

3. Embedding Inversion Attacks

Attack Objective. These attacks aim at recovering the raw text as (unordered) tokens $\{x_{i}\}_{i\in[n]}\subseteq X$ from embeddings, highlighting the risk of directly sharing (without noise) even only text embeddings (for training/inference). They have been employed to reconstruct specific patterns, e.g., identity codes and gene segments (Pan et al., 2020).

where $z_{i}$ is the $i^{\text{th}}$ row of noise $Z$ from $\mathcal{M}$ (omitted for DP-SGD or the non-private baseline). It returns $x^{*}_{i}$ with its embedding closest to the observed one of $x_{i}$ via a nearest-neighbor search over $\mathcal{V}$.

A token’s hidden embedding from deeper layers encodes more “abstract” contextual information of the entire sequence it belongs to; the token-wise inversion may be less accurate. We thus require a more general attack (Song and Raghunathan, 2020). It first maps the observed (noisy) embedding back to a lower-layer one using a linear least square model $M$ and then selects $n$ tokens as $X^{*}$ to minimize the $L_{2}$-distance between the lower-layer representation of $X^{*}$ and the one from $M$:

where $\zeta(\cdot)$ is a lower-layer representation function than $f(\cdot)$.

Numerical Results. The gradient-based attack reports the highest recall (or precision) on inverting the lowest-layer (clear) embeddings (Song and Raghunathan, 2020, Figure 2). To show that DP-Forward can mitigate such “strongest” inversion, we implement both (nearest-neighbor and gradient-based) attacks to invert input embeddings, with the public BERT embedding lookup table as prior. We also report their success rates as recall – the ratios of correct recoveries over the raw targets.

Table 8 shows that DP-Forward can reduce their success rates to a relatively low level, most are within $0.2$. However, DP-SGD fails in defense. The results corroborate our claim: DP-Forward directly adds noise to embeddings, thus mitigating embedding inversion, whereas DP-SGD only perturbs gradients, offering no protection for the (clear) inference-time embeddings of test sequences.

4. Sensitive Attribute Inference Attacks

Attack Objective. Instead of recovering exact tokens, one can try to infer sensitive attributes about target sequences from their embeddings. The attributes are often statistically unrelated to the training/inference objective but inherent in sequences, e.g., stylometry, implying the text’s authorship for sentiment analysis (Shetty et al., 2018). We are not interested in any global property of an entire corpus (Ganju et al., 2018).

where $\mathcal{S}$ is the set of all possible sensitive attributes of interest, say, authorship. It does not care about non-sensitive attributes.

We investigate five DP-Forward instances. Table 9 shows that they “reduce” the classifier to majority-class prediction, which returns the majority class (‘action’) on all inputs. In contrast, DP-SGD only reduces success rates moderately compared to the non-private baseline. It is because the embeddings from DP-SGD-trained/noisy models still “lose” some useful information (cf., accuracy drops of DP-SGD inference on embeddings without noise). The results confirm DP-Forward is more effective in thwarting attribute inference.

Related Work

An active line of research (Pan et al., 2020; Song and Raghunathan, 2020; Béguelin et al., 2020; Carlini et al., 2021) discloses severe privacy risks in modern LMs (even used as black-box query “oracles”) concerning their (hidden/output) text embeddings. Song and Raghunathan (Song and Raghunathan, 2020) build a taxonomy of attacks that covers a broader scope than a parallel work (Pan et al., 2020). These attacks include embedding inversion (which can partially recover raw texts), membership inference (establishing the is-in relation between a target and private training data), and inferring sensitive attributes like text authorship from embeddings. A common defense for them is adversarial training, e.g., (Elazar and Goldberg, 2018).

Others (Béguelin et al., 2020; Carlini et al., 2021) study the “memorization” of training data in LMs (a.k.a. membership inference attack). In particular, Carlini et al. (Carlini et al., 2021) define $k$-eidetic memorization, where a string is extractable or memorized if it appears in at most $k$ examples. Their black-box attacks on GPT-2 (Radford et al., 2018) can extract verbatim training texts even when $k=1$ (e.g., a name that only appears once is still extractable). A smaller $k$ means a higher privacy risk. Beguelin et al. (Béguelin et al., 2020) define differential score and rank as two new metrics for analyzing the update leakage, enabling the recovery of new text used to update LMs. Incorporating DP to address memorization is a promising solution.

2. Input (Text/Feature) Perturbation for LDP

SynTF (Weggenmann and Kerschbaum, 2018) synthesizes term-frequency (feature) vectors under LDP, which have limited applications compared to sentence embeddings or text itself. Feyisetan et al. (Feyisetan et al., 2019, 2020) resort to metric-LDP (Alvim et al., 2018), a relaxed variant of LDP with a distance metric (e.g., Euclidean or Hyperbolic), which allows the indistinguishability of outputs to grow proportionally to the inputs’ distance. They first add noise to the outputs of a non-contextualized token embedding model (e.g., GLoVe (Pennington et al., 2014)), which are then projected back to “sanitized” text using the nearest neighbor search as post-processing. In contrast, Yue et al. (Yue et al., 2021) sanitize text by directly sampling token-wise replacements, avoiding adding noise to high-dimensional embeddings. All these works only achieve (variants of) token-level metric-LDP.

To offer sequence-level protection, recent studies (Lyu et al., 2020; Meehan et al., 2022) apply Laplace or exponential mechanism to perturb (the average of) sentence embeddings extracted by an LM (e.g., BERT (Devlin et al., 2019)). Both ensure pure LDP (homogeneously protecting any entire sequence), which may be too stringent and impact utility. In contrast, heterogeneous protection (Feyisetan et al., 2020; Yue et al., 2021) can strategically manage the privacy demands across inputs. Du et al. (Du et al., 2023) achieve metric-LDP (by Purkayastha and planar Laplace mechanisms) at the sequence level (unlike token-level in prior arts (Feyisetan et al., 2020; Yue et al., 2021)). To further boost the utility, they mitigate the dimensional curse via a random-projection-like approach. They also perturb sensitive sequence labels for enhanced privacy. Nevertheless, perturbing different hidden (rather than token or sentence) embeddings inside LM-based NLP pipelines remains unexplored.

3. DP-SGD (Variants) in Training LMs

An early attempt (McMahan et al., 2018) uses DP-SGD to train long short-term memory LMs in the federated learning setting. By configuring hyperparameters properly (e.g., setting the batch size to millions), one can even pre-train BERT-Large, an LM with ${\sim}340$M parameters, using DP-SGD/Adam while achieving acceptable (MLM) accuracy (Anil et al., 2022).

Using the vanilla DP-SGD in pre-training/fine-tuning large LMs leads to significant efficiency and accuracy drops due to the “curse of dimensionality.” Yu et al. (Yu et al., 2021) propose reparametrized gradient perturbation: It first reparameterizes/decomposes each high-rank weight matrix into two low-rank (gradient-carrier) ones with a residual matrix and then only perturbs the two low-rank gradients to alleviate the dimensional curse. The noisy low-rank gradients are finally projected back to update the raw high-rank weights.

Applying reparameterization to every weight in each update is still costly and may introduce instability (e.g., noises are “zoomed up” during the projection). Instead, the follow-up (Yu et al., 2022) builds atop the recent success of parameter-efficient fine-tuning (e.g., LoRA (Hu et al., 2022), Adapter (Houlsby et al., 2019), and Compacter (Mahabadi et al., 2021)): It perturbs the gradients of a much smaller number of additional “plug-in” parameters. However, Li et al. (Li et al., 2022b) empirically show that parameter-efficient fine-tuning is not necessarily better than the full one; they propose ghost clipping, a memory-saving technique (“orthogonal” to dimension reduction), to use DP-SGD in full fine-tuning without instantiating per-example gradients. Despite efficiency/accuracy gains, all these works still only protect training data by perturbing (smaller) gradients.

4. DP Mechanisms for Matrix Functions

Gaussian and Laplace mechanisms are typically for scalar-/vector-valued functions (Dwork and Roth, 2014). Vectorizing the outputs and adding i.i.d. noise could generalize them for matrix-valued functions, but the structural information of matrix functions is not exploited. The MVG mechanism (Chanyaswad et al., 2018) is thus devised, which draws directional or non-i.i.d. noise from a matrix Gaussian distribution. It injects less noise into more “informative” output directions for better utility, with only a constraint on the sum of the singular values (determining the noise magnitude) of two covariance matrices. Such a constraint is only a sufficient condition for $(\epsilon,\delta)$-DP, which is improved by the follow-up (Yang et al., 2023) with a tighter bound on the singular values.

There also exist mechanisms dedicated to restricted matrix-valued functions. The matrix mechanism (Li et al., 2015) considers a collection of linear counting queries represented by $Wx$ for query matrix $W$ and input vector $x$. It still resorts to additive Laplace/Gaussian noise but with an extra transformation solving the min-variance estimation to the noisy $Wx$. Another very recent study (Ji et al., 2021) focuses on matrix-valued queries with only binary (matrix) outputs. It then devises an exclusive-or (xor) mechanism xor-ing the outputs with noise attributed to a matrix-valued Bernoulli distribution.

Conclusion

Pre-trained LMs became pivotal in NLP. Alarmingly, fine-tuning corpora or inference-time inputs face various privacy attacks. The popular DP-SGD only provides limited protection for training data by adding noise to gradients. Raw tokens or sensitive attributes of training/inference data can be inverted or inferred from embeddings in forward-pass computation. Vanilla DP-SGD also imposes high GPU memory and computational burdens but cannot be batched.

We propose DP-Forward, which directly adds noise to embedding matrices derived from the raw training/inference data in the forward pass. Its core is the analytic matrix Gaussian mechanism, a general-purpose tool that owns independent interests. It draws optimal matrix-valued noise from a matrix Gaussian distribution in a dedicated way using a necessary and sufficient condition for DP.

Perturbing embeddings at various positions across multiple layers yields at least two benefits. DP-Forward users are only required to download pipeline parts for deriving noisy embeddings, which is more storage- and time-efficient than deriving noisy gradients. Together with our prior attempts (Yue et al., 2021; Du et al., 2023) at sanitizing input text tokens and output sentence embeddings, we provide a full suite of forward-pass signal sanitization options for users only to share their sanitized data for LM-as-a-Service APIs while protecting privacy.

Beyond the theoretical contribution of two local DP notions and the experimental comparisons with baselines (e.g., GM, MVG, and DP-SGD) across three typical NLP tasks, we investigate the hyperparameter configuration for reproducible validations of DP-Forward’s potential in terms of efficiency, accuracy, and its ability to withstand diverse against diverse attacks.

Altogether, our new perspective leads to a better approach to privacy-aware deep neural network training, challenging the traditional wisdom focusing on gradients. As a new paradigm for local DP in fine-tuning and inference, our work paves the way for a myriad of possibilities for new machine-learning privacy research (Ng and Chow, 2023), e.g., generalization to transformer-based computer vision tasks.

References

Appendix A Token-level DP-Forward

​​For $\epsilon\geq 0,0\leq\delta\leq 1$, $\mathcal{M}$ fulfills token-level $(\epsilon,\delta)$-SeqLDP, if $\forall X\simeq X^{\prime}$ that differ in any single token but with the same $y$, and any possible output subset $\mathcal{O}$,

Despite a token-level notion, our experiments (Appendix A.3) show that when $f(\cdot)$ is only the input embedding layer, our token-level SeqLDP designs can also effectively mitigate MIAs on entire sequences, with up to $20$pp accuracy gains at the same choices of $\epsilon$. It is not necessarily weaker than sequence-level CDP (as offered by DP-SGD). One might doubt its usefulness since two neighboring sequences may be too similar. Nevertheless, there are cases where a sentence, e.g., “How’s it going” may not matter in a bigger unit (paragraph/essay) of the training data either. Moreover, a token (e.g., yes/no) can play a crucial role, e.g., in named entity recognition (Li et al., 2022a). Our LDP guarantee is for any such two sequences, covering the wide spectrum between “too similar” and radically different cases.

Note that weakening privacy notions by itself is not our goalAs a related example, in image classification, PixelDP (Lécuyer et al., 2019) has been proposed for a DP notion defined upon pixels. Its motivation is robustness to adversarial examples.. Protection at the token level has been studied under metric-DP (Feyisetan et al., 2020; Qu et al., 2021), a relaxation of LDP. They require even much larger $\epsilon$, say, $175$. Our goal of studying token-level SeqLDP is to narrow the gap between theory and practice, i.e., provable privacy notions tailored to the protection targets (the first few layers vs. the whole pipeline).

A.2. Two Token-level SeqLDP Designs

For token-level SeqLDP, we need to bound a “new” $S_{2}(f),\forall X\simeq X^{\prime}$, which should be tight and smaller than the one over $\forall X,X^{\prime}$, hence producing smaller noise for better utility at meaningful token-level $\epsilon$. It is still non-trivial since $f(\cdot)$, except for being the input embedding layer, may differ in every entry for even $X\simeq X^{\prime}$. One could also normalize the entire $f(\cdot)$ for $S_{2}(f),\forall X\simeq X^{\prime}$, which “degenerates” to the token-level SeqLDP. Instead, we tailor two designs to estimate a tighter $S_{2}(f)$ than the “general” one for only the input embedding layer and the first two layers, respectively. Specifically, we employ row-wise normalization and the Lipschitz continuity (Kim et al., 2021).

In the First MHA Sub-layer. The second option could be adding $Z$ right after the first MHA sub-layer: $\mathsf{MHA}(X)+Z$, where $\mathsf{MHA}(\cdot)$ is the concatenation of $\mathsf{Att}_{i}(\cdot),i\in[h]$. Yet, it is non-trivial to estimate $S_{2}(f)$ of $\mathsf{MHA}(\cdot)$ as $\mathsf{Att}_{i}(\cdot)$, let alone $\mathsf{MHA}(\cdot)$, is not Lipschitz (Kim et al., 2021).

Given two metric spaces $(\mathcal{X},d_{\mathcal{X}})$ and $(\mathcal{Y},d_{\mathcal{Y}})$, a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ is Lipschitz continuous ($K$-Lipschitz) if there exists a constant $K\geq 0$,

The smallest $K$ is the Lipschitz constant, denoted by $\mathit{Lip}(f)$.

The non-Lipschitz continuity stems from the non-linear Softmax activation, which takes pairwise dot products as input (Kim et al., 2021). To make MHA Lipschitz, one might apply pairwise $L_{2}$-distances (hence called $L_{2}$-MHA) (Kim et al., 2021) or add a normalization step called LipschitzNorm (Dasoulas et al., 2021) in $\mathsf{softmax}(\cdot)$. Unfortunately, estimating $\mathit{Lip}(f)$ of $L_{2}$-MHA needs to solve an intractable optimization problem, and LipschitzNorm is ill-suited for the high-dimensional BERT attention.

The two instances (with row-wise normalization) for fine-tuning or inference fulfill token-level $(\epsilon,\delta)$-(Seq)LDP.

The proof is equivalent to our approach for (Seq)LDP. One just needs to compute $S_{2}(f),\forall X\simeq X^{\prime}$ properly, and we did.

For minimal changes to the pipeline, we adopt the raw WordPiece (Wu et al., 2016), which splits text into sub-words; using word-level tokenization yields word-level (Seq)LDP. Our notion can also extend to phrase-level (Seq)LDP by directly using the group privacy (Dwork and Roth, 2014) or dedicatedly computing the $L_{2}$-sensitivity smaller than $c\cdot S_{2}(f)$ for two sequences differing in (consecutive) $c$ tokens. Typically, $c$ is small since a few tokens are enough for most sensitive information. One could also add noise deeper in a pipeline using $S_{2}(f_{1}\circ f_{2})\leq S_{2}(f_{1})\cdot S_{2}(f_{2})$, where $f_{1}\circ f_{2}$ is function composition $f_{1}(f_{2}(\cdot))$. We then need to estimate $S_{2}(f)$ of each (component of) sub-layer. For example, $\mathsf{FFN}(\cdot)$ has two linear maps $W_{1}$ and $W_{2}$ with $\mathsf{ReLU}(\cdot)$ in between, where $S_{2}(f)$ of $\mathsf{ReLU}(\cdot)$ is $1$. For $W_{1,2}$, its $S_{2}(f)$ is bounded by $\sqrt{d}C\sigma_{\max}(W_{1,2})$ since $||\cdot||_{F}\leq\sqrt{d}||\cdot||_{2}$ with $d$ as the rank. We can also estimate $S_{2}(f)$ of $\mathsf{LN}(\cdot)$ from its Lipschitz constant (Kim et al., 2021). When $f(\cdot)$ is composed of more layers, we can only get a looser estimation on the final $S_{2}(f)$. Hence, our general recommendation is to add noise early when estimating a tight $S_{2}(f)$ is feasible.

A.3. More Experiment Results

We also study the privacy-accuracy tradeoff on all three tasks for our two token-level SeqLDP designs when tuning local $\epsilon$. The results are compared with the non-private baseline and fine-tuning using MVG noise. Figure 5 shows task accuracy increases with $\epsilon$. Perturbing input embeddings for token-level (vs. sequence-level) SeqLDP can achieve remarkable accuracy gain, e.g., ${\sim}0.7$ vs. $0.5$ for IMDb.

We evaluate the two MIAs on SST-2 fine-tuned by our two token-level SeqLDP instances. Table 10 shows the results, with success rates within $0.48{-}0.52$ (like random guessing) bolded. Even if the provable guarantee is at the token level, our instances can notably reduce the success rates of the confidence-based attack by ${\sim}14$pp and the entropy-based one by ${\sim}11$pp, compared to the non-private baseline.

Appendix B Relevant Matrix Algebra

The PDF defined in Eq. (1) and the matrix-trace-based one used in MVG (Chanyaswad et al., 2018) are equivalent.

For the numerator part in Eq. (1), we have

where $\operatorname{Tr}(\cdot)$ denotes the matrix trace. Denote

which is a similar matrix of $A$, and hence $\operatorname{Tr}(A)=\operatorname{Tr}(B)$. So, the two PDFs are equivalent since

We first prove that $||A||_{F}=||W_{1}A||_{F}$ by

and similarly we can prove that $||A||_{F}=||AW_{2}||_{F}$. ∎

The SVD of $A$ is $W_{1}\Lambda W_{2}^{\top}$. By Lemma 3, we have

where $W=W^{\top}_{A_{2}}W_{B_{1}}=(w_{ij})_{n\times n}$ and $W^{\prime}=W^{\top}_{B_{2}}W_{C_{1}}=(w^{\prime}_{ij})_{d\times d}$ are still two unitary matrices. We further have

where $\beta_{ij}=\sum^{r}_{k=1}\sigma_{k}(B)w_{ik}w^{\prime}_{kj}$. Hence, we need to show

Following the strategy in (Yang et al., 2021) (cf. Eq. (29), (30)), we rewrite $\sigma^{2}_{i}(A)$ and $\sigma^{2}_{j}(C)$ using non-negative values $\xi_{t}$ and $\eta_{s}$ s.t.

For $i\in[1,n],j\in[1,d]$, we denote $\gamma_{ij}=\sigma_{i}(B)$, if $i=j$; $\gamma_{ij}=0$, otherwise. Then, we transform the Eq. (7) as

Since $\xi_{t},\eta_{s}$ are non-negative, we only need to show

However, the original proof (Yang et al., 2021) has two issues: i) $t>s$ is not considered, and ii) the commutative law of matrix multiplication in Eq. (35) does not hold as $E(t)$ in Eq. (34) is not a standard diagonal matrix. To address them, we have

We then denote a sub-matrix $B^{*}=(\beta_{ij})$ for $i\in[1,t],j\in[1,s]$ of $W\Lambda_{B}W^{\prime}$. With SVD of $B^{*}$, we have

The last inequality is due to $\sigma_{k}(B^{*})\leq\sigma_{k}(B)$ for $\forall k\in[1,r]$ (Horn and Johnson, 2012). So, Inequality (8) holds. ∎

Appendix C Proofs for Our Analytic Matrix Gaussian Mechanism

This section proof Lemma 4, Lemma 5, and Theorem 6 in Section 4.2.

Recall that $\mathcal{M}(f(\mathcal{X}))=f(\mathcal{X})+Z$ with $Z\sim\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$, the probability of $\mathcal{M}(f(\mathcal{X}))=O$ is

Similarly, we can compute $\Pr[\mathcal{M}(f(\mathcal{X}^{\prime}))=O]$. By plugging them into $\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}(O)$, and let $\Delta=f(\mathcal{X})-f(\mathcal{X}^{\prime})$,

where $\mathit{vec}(\cdot)$ is the vectorization of a matrix and $\langle\cdot,\cdot\rangle$ denotes the inner product. For easy presentation, we denote $Z^{\prime}=U^{-1}ZV^{-\top}$ and $\Delta^{\prime}=U^{-1}\Delta V^{-\top}$, and then we re-write

Given Lemma 7, $Z^{\prime}\sim\mathcal{MN}_{n,d}(0,I_{n},I_{d})$ with each entry i.i.d. drawn from $\mathcal{N}(0,1)$. $\langle\mathit{vec}(\Delta^{\prime}),\mathit{vec}(Z^{\prime})\rangle$ is thus the $\Delta^{\prime}$-weighted sum of $nd$ i.i.d. Gaussian random variables, which is a Gaussian variableen.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables $\mathcal{N}(0,||\Delta^{\prime}||^{2}_{F})$ too. So $\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}\sim\mathcal{N}(\eta,2\eta)$, $\eta=\frac{1}{2}||\Delta^{\prime}||^{2}_{F}$. ∎

where we used $\mathcal{N}(\eta,2\eta)=\eta+\mathcal{N}(0,1)/\sqrt{2\eta}$ and the symmetry of the standard normal distribution $\Pr[\mathcal{N}(0,1)\geq t]=\Pr[\mathcal{N}(0,1)\leq-t]$.

A similar argument applied to $\mathcal{L}_{\mathcal{M},\mathcal{X}^{\prime},\mathcal{X}}$ yields

The proof boils down to two directions. From $(\epsilon,\delta)$-DP (Theorem 3) to Theorem 6, the proof directly follows from all the derivations in Section 4.2. For the inverse direction, it is sufficient to show that $||\Delta^{\prime}||_{F}\leq\mathcal{B}$ holds for $\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$ given Theorem 6. In particular, for $||\Delta^{\prime}||_{F}=||U^{-1}\Delta V^{-\top}||_{F}$, we have

where the first inequality is due to Lemma 5, the second one holds since $\sigma_{n}(U)$ and $\sigma_{d}(V)$ are the smallest singular values among the others, and the third one is from Theorem 6. ∎