CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU

I Introduction

Deep learning has enabled numerous applications in the form of digital voice assistants, video monitoring and surveillance systems, and even systems for disease diagnosis and treatment planning. But these new and exciting applications raise challenging questions regarding user privacy. After all, modern machine learning algorithms are largely data-driven and training image recognition, speech recognition, or disease predictor systems all rely on aggregating and analyzing sensitive user data. Even model inference raises privacy concerns as increasingly often, voice or video recordings from a mobile or IoT device are outsourced to the cloud for analysis.

To address some of the privacy challenges associated with the widespread deployment of deep learning technologies, a number of works in the last few years have introduced cryptographic frameworks based on secure multiparty computation (MPC) to enable privacy-preserving deep learning (see Section V for a more comprehensive survey). At a high level, MPC protocols allow a set of mutually-distrusting parties to compute an arbitrary function over secret inputs such that at the end of the computation, the parties only learn the output of their computation, and nothing more. In particular, all information about other parties’ inputs are completely hidden (up to what could be inferred based on the outputThere are settings where even learning the exact output is problematic and can reveal compromising information about other parties’ inputs. Techniques like differential privacy provide a defense against these types of attacks. We discuss this in greater detail in Section V.).

While there have been considerable advances in the concrete efficiency of MPC protocols, current approaches remain computationally expensive and do not scale well to the types of neural networks typically used in modern machine learning systems. Until recently, cryptographic protocols for private inference over deep neural networks have been limited to small datasets such as MNIST or CIFAR . In contrast, the current baseline for object recognition is ImageNet , a dataset that is over $1000\times$ larger than CIFAR/MNIST and contains 1000 different classes (compared to just 10 classes for MNIST and CIFAR). Similarly, state-of-the-art deep learning models for computer vision such as ResNet-152 contain over 150 layers and over 60 million parameters. In contrast, most protocols for privacy-preserving machine learning have been constrained to relatively shallow networks with just tens of layers and a few hundred thousand parameters.

Recently, two systems Falcon and CrypTFlow have made considerable headway towards scalable privacy-preserving machine learning. For the first time, they demonstrate the ability to perform privacy-preserving machine learning at the scale of ImageNet (or Tiny ImageNet in the case of Falcon) and with much larger models (e.g., AlexNet , VGG-16 , and the ResNet family of models ). In spite of these advances, there still remains considerable overhead: for example, private training of AlexNet on Tiny ImageNet is estimated to still take over a year using Falcon. CrypTFlow currently only supports private inference and not private training. Both works argue that hardware acceleration with graphics processing units (GPUs) will be essential for scaling up privacy-preserving deep learning, especially in the case of private training.

GPUs and hardware acceleration have played a critical role in the evolution of modern deep learning. Today, convolutional neural networks (CNNs) have become a staple for modern computer vision. However, in the immediate years following their introduction in the seminal work of LeCun et al. , CNNs did not see widespread adoption. This was in large part due to the high computational costs of the backpropagation training algorithm. Starting the mid-2000s, several works showed that CNN training could be greatly accelerated through the use of graphics processing units (GPUs). This culminated with the breakthrough moment when Krizhevsky et al. introduced “AlexNet” and won the ImageNet Large Scale Visual Recognition Challenge in 2012 using a large CNN trained entirely on the GPU. Since AlexNet, CNNs have become a mainstay of computer vision. Modern machine learning frameworks like PyTorch and TensorFlow all support and rely heavily on not only GPUs, but even custom-designed application-specific integrated circuits (ASICs) such as Google’s tensor processing unit .

Privacy-preserving machine learning on the GPU.

Hardware acceleration has become a core component for evaluating and training deep learning models. Given that MPC protocols necessarily incur a non-zero overhead on top of the plaintext computation, it is essential for cryptographic protocols to be able to leverage GPU acceleration in order to have any chance of scaling up to support training and inference over deep models. After all, if we are bound to CPU-based computations (as nearly all existing MPC frameworks have), then it is infeasible to even run the machine learning algorithm on plaintext data.

I-A Our Contributions

In this work, we introduce CryptGPU, a new cryptographic MPC framework built on top of PyTorch and CrypTen where all of the cryptographic operations (both linear and non-linear) are implemented on the GPU. CryptGPU operates in the standard 3-party setting where we assume that all inputs are secret-shared across three non-colluding servers who execute the MPC protocol. The inputs are secret shared using a 2-out-of-3 replicated secret sharing scheme (see Section III for the full details). Our system provides security against a single semi-honest corruption. We describe our threat model formally in Section III-A.

CryptGPU can perform private inference over modern computer vision models such as ResNet-152 on ImageNet images in just over 25s ($2.3\times$ faster than the previous state-of-the-art CrypTFlow ). For smaller networks like AlexNet, private inference over ImageNet requires just 1.5s.

Further improvements to the costs of private training are possible if we consider batch inference, which also benefits from GPU parallelism. For example, batch inference over ResNet-152 reduces the cost of private inference from 25s for a single image to 13.2s per image when amortized over a batch of $8$ images.

For private training (which has a greater potential to benefit from GPU acceleration), we demonstrate a $36\times$ speed-up for private training of AlexNet on the Tiny ImageNet database compared to Falcon. Whereas it would have taken over a year to privately train Falcon on Tiny ImageNet, our GPU-accelerated system would be able to do so in just over a week (see Section IV-B). Beyond these performance results, our work highlights the potential of leveraging GPUs to accelerate privacy-preserving deep learning in much the same way GPUs have dramatically accelerated standard deep learning. Our work also highlights the importance of developing new types of cryptographic protocols that are “GPU-friendly” and can take advantage of the parallelism provided by GPUs.

While NVIDIA’s CUDA (Compute Unified Device Architecture) platform supports general-purpose computations on the GPU, directly translating code written for the CPU onto the GPU is unlikely to translate to immediate performance gains. The architectural differences between the CPU and the GPU introduce several additional hurdles that must be overcome in order to have an efficient implementation:

Leveraging existing CUDA kernels. The first challenge is that highly optimized CUDA kernels for computing deep learning primitives (i.e., convolutions, pooling, matrix multiplication) are designed to operate on floating-point inputs, and there does not currently exist kernels for computing on integer values. In MPC, we typically compute over discrete objects (i.e., ring or field elements). To leverage optimized kernels for these basic primitives, we need a way to embed the integer-valued cryptographic operations into (64-bit) floating-point arithmetic that can in turn be operated on by these kernels. CryptGPU enables this by introducing a new abstraction called a CUDALongTensor that models tensors (i.e., multi-dimensional arrays) of integer values, but seamlessly translates the integer-valued computations into a corresponding set of floating-point computations. We describe our construction in Section II-B. The Delphi system encountered a similar challenge, but as we discuss in Remark II.3, their solution does not extend well to our setting. A critical difference is that Delphi considers private inference where the model is public while in this work, we assume that the model is also hidden (i.e., secret-shared).

“GPU-friendly” cryptography. The GPU architecture is optimized for performing a large number of simple computations on blocks of values. This means that operations like component-wise addition and multiplication of vectors/matrices are fast while operations that involve large numbers of conditional statements are slower. While there is support for integer addition and multiplication, operations like computing a modular reduction by a prime incurs considerably more overhead ; for instance, we observed a $40\times$ difference in the running time of point-wise addition vs. point-wise modular reduction. Thus, when choosing and designing cryptographic protocols for the GPU, one must carefully calibrate them for the architecture. Protocols like Yao’s garbled circuits are less well-suited for taking advantage of GPU parallelism compared to a vectorized secret-sharing-based protocol. Similarly, protocols that require extensive finite field arithmetic (and thus, require modular reductions) will incur more overhead on the GPU compared to protocols that only rely on arithmetic modulo a power of $2$. We also design protocols for common non-linear functions (e.g., exponentiation and division) that are specifically optimized for our particular setting. We describe the cryptographic protocols we use in Section III.

Systematic evaluation of GPU-based MPC.

We present a comprehensive and systematic evaluation of CryptGPU to quantify the advantages of a GPU-based MPC protocol and compare against previous protocols for privacy-preserving machine learning. We specifically measure the performance of our private training and inference protocols on a wide range of object recognition models (e.g., LeNet , AlexNet , and the ResNet family of networks ) and datasets (e.g., MNIST , CIFAR-10 , and ImageNet ). We describe our experimental methodology and measurements in Section IV.

We also collect fine-grained measurements to understand how the computational costs are split across the different layers of a network. For instance, in CPU-based systems like Falcon , the linear layers account for $86$% to $99$% of the overall computational costs of private training.While linear layers are simpler to evaluate from a cryptographic perspective (in comparison to non-linear layers), the size of the linear layers is typically much larger than that of the non-linear layers. On the same model/datasets, our GPU-based approach evaluates the same linear layers with a $25\times$ to $72\times$ speed-up; this is a major source of the performance advantage of CryptGPU compared to previous systems. Consequently, the costs of our private training protocol is more evenly split between evaluating linear layers and non-linear layers. We provide the full details in Sections IV-B and V.

In Section IV-C, we report microbenchmarks to quantify the performance advantages of using the GPU to execute all of the MPC protocols. For instance, we show that evaluating convolutions on secret-shared data (with secret-shared kernels) on the GPU is over $150\times$ faster than the corresponding protocol on the CPU. Even for non-linear operations like the ReLU (rectified linear unit) function, using a GPU-based MPC protocol still yields a $10\times$ speed-up over the same underlying CPU-based protocol.

Finally, since our MPC protocol represents real-valued inputs using a fixed-point encoding, and moreover, some of our protocols rely on approximations to non-linear functions, we also compare the accuracy of our private inference and private training algorithms to the analogous plaintext algorithms. As we show in Section IV-D, for the models and datasets we consider in this work, the behavior of our privacy-preserving algorithms closely matches their plaintext analogs.

An ML-friendly approach.

One of the guiding principles behind our system design is to make it friendly for machine learning researchers to use. We build our system on top of CrypTen , which is itself built on top of the popular machine learning framework PyTorch . Effectively, our work (much like CrypTen) provides a new cryptographic back end that supports computations on secret-shared values while retaining a similar front end as PyTorch. In fact, we note that our work on developing the CUDALongTensor module has already been integrated as part of CrypTen to support privacy-preserving GPU computations .

II System Overview

Similar to previous works on constructing efficient protocols for privacy-preserving machine learning (see also Section V), we assume that the data and model are (arbitrarily) partitioned across three parties. For example, the three parties could be three independent organizations seeking to collaboratively train a model on their joint data without revealing their inputs to each other. Our system is also applicable in the “server-aided” setting , where a group of (arbitrarily-many) clients seek to train a joint model on their data (or evaluate a secret-shared model on private inputs). In the server-aided setting, the clients first secret share their inputs to three independent cloud-service providers, who in turn run the cryptographic protocol on the secret-shared inputs. We design our protocols to provide security against a single semi-honest corruption. We provide a formal description of our threat model in Section III-A.

Our starting point in this work is the CrypTen privacy-preserving machine learning framework . CrypTen is built on top of the widely-used machine-learning framework PyTorch . We adapt the basic architecture of CrypTen, and make modifications to support three-party protocols based on replicated secret sharing. We describe the main system architecture below.

GPUs, and more recently, ASICs like Google’s tensor processing units , have played a critical role in scaling up modern deep learning. These specialized hardware platforms support massive parallelism, making them well-suited for performing standard linear algebraic operations (e.g., convolutions or average pooling) as well as point-wise evaluation of functions on large blocks of neurons (e.g., evaluating an activation function or performing batch normalization). Popular frameworks for deep learning frameworks such as PyTorch and TensorFlow natively support computations on both the CPU and GPU.

CUDA is a parallel computing platform developed by NVIDIA for general-purpose computing on GPUs . For deep learning in particular, CUDA libraries such as cuBLAS and cuDNN provide highly-optimized implementation for a wide-range of standard primitives such as convolutions, pooling, activation functions, and more. These libraries are designed for floating-point computations and do not support integer-valued analogs of these operations. Since cryptographic protocols typically operate over discrete spaces (e.g., a 64-bit ring) where the underlying algebra is implemented using integer-valued computations, one cannot directly translate an existing protocol to the GPU.

PyTorch.

PyTorch is a popular open-source machine learning framework designed for prototyping, implementing, and deploying deep neural networks. The PyTorch front end supports many standard neural network layers (e.g., convolutions, pooling, activation functions, etc.) as well as features such as automatic differentiation and gradient computation. The PyTorch back end natively supports computation on both CPUs as well as GPUs. This flexibility enables users to train complex models without needing to worry about the finer details of backpropagation. It also allows users to take advantage of GPU acceleration without needing to interface with low-level CUDA kernels. PyTorch also provides library support for distributing computations across multiple devices and/or GPUs.

Data in PyTorch is organized around tensors, which provide a general abstraction for $n$-dimensional arrays. PyTorch provides an expressive API for computing on and applying transformations to tensors. Especially importantly in our case, the PyTorch back end natively and seamlessly leverages GPU acceleration for tensor computations.

CrypTen.

CrypTen is a recent framework built on top of PyTorch for privacy-preserving machine learning. CrypTen provide a secure computing back end for PyTorch while still preserving the PyTorch front end APIs that enables rapid prototyping and experimentation with deep neural networks.

CrypTen supports general $n$-party computation and provides security against a single semi-honest corruption. At the cryptographic level, elementary arithmetic operations are handled using Beaver multiplication triples , Boolean circuit evaluation is implemented using the Goldreich-Micali-Wigderson (GMW) protocol , and low-degree polynomial approximations are used for most non-linear operations. We note that while our system builds on CrypTen, we work in a 3-party model where parties compute using replicated secret shares (as in ). We describe this in Section III.

II-B System Design and Architecture

The design of CryptGPU is centered around the following principles:

Leverage existing CUDA kernels for linear algebra. As mentioned in Section II-A, highly-optimized CUDA kernels exist for most linear algebra operations encountered in deep learning. However, these kernels only support computations on floating-point values and are not directly applicable for computing on discrete structures common in cryptographic protocols. Thus, we seek a way to keep all of the computation on the GPU itself.

Keep all computations on the GPU. While some previous works on private machine learning show how to leverage the GPU for computing linear and bilinear functions, they then move the data out of the GPU to evaluate non-linear functions. In this work, we seek to keep all of the computations on the GPU, and as we show in Section IV-C, even computing non-linear functions can benefit greatly from GPU acceleration, provided that they are implemented using “GPU-friendly” cryptographic protocols (i.e., protocols that primarily rely on point-wise or component-wise vector operations).

Integer operations using floating-point arithmetic.

Our approach for embedding 64-integer operations into 64-bit floating point operations relies on the following observations:

Bilinearity. Operations like matrix multiplication and convolutions are bilinear. This means that for any choice of inputs $\mathbf{A}_{1},\mathbf{A}_{2},\mathbf{B}_{1},\mathbf{B}_{2}$,

where $\circ$ denotes an arbitrary bilinear operation. Suppose now that we rewrite an input as an expansion in a smaller base; for example, we might write $\mathbf{A}=\mathbf{A}_{0}+2^{16}\mathbf{A}_{1}$ and $\mathbf{B}=\mathbf{B}_{0}+2^{16}\mathbf{B}_{1}$. Bilinearity ensures that $\mathbf{A}\circ\mathbf{B}$ can be expressed as a linear combination of the pairwise products $\mathbf{A}_{0}\circ\mathbf{B}_{0}$, $\mathbf{A}_{0}\circ\mathbf{B}_{1}$, $\mathbf{A}_{1}\circ\mathbf{B}_{0}$, and $\mathbf{A}_{1}\circ\mathbf{B}_{1}$. Computing $\mathbf{A}\circ\mathbf{B}$ from the pairwise products only requires element-wise additions and scalar multiplications.

CUDA kernels for element-wise operations. To complete the puzzle, we note that there are optimized CUDA kernels for performing component-wise addition and scalar multiplication on 64-bit integer values.

When performing computations using floating-point kernels, CryptGPU decomposes each input into $k=4$ blocks, where the values in each block are represented by a $w=16$-bit value. For this choice of parameters, each bilinear operation is expanded into $10$ pairwise products.

While it may be tempting to decompose 64-bit values into $k=3$ blocks, where each block consists of 22-bit values, this compromises correctness of our approach. Namely, correctness of the computation is guaranteed only if the entries in each of the intermediate pairwise products $\mathbf{A}_{i}\circ\mathbf{B}_{j}$ do not exceed the 52-bits of available floating-point precision. If the entries of $\mathbf{A}_{i}$ and $\mathbf{B}_{j}$ are 22 bits, then the entries in a single multiplication between an element in $\mathbf{A}_{i}$ and $\mathbf{B}_{j}$ will already be 44 bits. If we are evaluating a convolution (or matrix multiplication) where each output component is a sum of $2^{8}=256$ values, this exceeds the available precision and triggers an arithmetic overflow. This is problematic for larger networks. Using 16-bit blocks, we can handle bilinear operations involving up to $2^{20}$ intermediate products, which is sufficient for our applications.

While decomposing each bilinear operation on integer values into $O(k^{2})$ floating-point operations on same-sized inputs can appear costly, CryptGPU takes advantage of GPU parallelism to mitigate the computational overhead. Namely, for convolutions, CryptGPU uses group convolution (cudnnConvolutionForward) to compute the convolutions in parallel. Similarly, for matrix multiplications, CryptGPU uses a batch matrix multiplicative kernel (cublasSgemm) to multiply matrices in parallel. We observe that for small inputs (e.g., $64\times 64$ inputs), this approach only incurs a modest $2\times$ overhead (compared with evaluating a single convolution of the same size) and increases to roughly $9\times$ for larger $224\times 224$ inputs.

While the computational overhead of our embedding is partially mitigated through parallelism, this approach does increase the memory requirements of our protocol. This does not have a significant effect on privacy-preserving inference, but it does limit the batch size we can handle during privacy-preserving training (recall that during training, a single iteration of the optimization algorithm processes a batch of instances). Scaling up to support larger batch sizes during privacy-preserving training would likely necessitate distributing the computation across multiple GPUs rather than a single GPU (as is also the case for training deep models in the clear).

The Delphi system leverage GPUs for evaluating convolutions on secret-shared inputs in their private inference system. In their setting, the parameters are chosen so that the outputs of the convolution are always within the interval $[2^{-52},2^{52}]$, and as such, the existing floating-point kernels for convolution can be used without incurring any floating-point precision issues. In particular, Delphi uses a 32-bit ring and 15 bits of fixed-point precision. The system works in the setting where the model parameters are assumed to be public: namely, the convolution kernels are not secret-shared. In this way, convolutions are evaluated between a plaintext value and a secret-shared value, which ensures that the resulting outputs are bounded. In our setting, both the model and the inputs are secret-shared so we cannot directly embed the integer-valued operations into 64-bit floating-point computations. In fact, as we discuss in Section IV-C, to have sufficient precision when scaling up to deeper models and larger datasets, it is often necessary to use a larger ring (i.e., a 64-bit ring) for the arithmetic secret sharing.

The CUDALongTensor abstraction.

CryptGPU provides a new abstraction called a CUDALongTensor for embedding 64-bit integer-valued operations into 64-bit floating-point arithmetic. Similar to CrypTen’s MPCTensor, the CUDALongTensor abstractly represents a secret-shared tensor of 64-bit integers and is backed by a standard PyTorch tensor of 64-bit integers. In the back end, whenever an elementary operation needs to be evaluated on the underlying tensor, CryptGPU proceeds as follows:

If optimized CUDA kernels exist for evaluating the chosen operation on integer-valued tensors (e.g., point-wise addition or point-wise multiplication), then the corresponding CUDA kernel is directly invoked.

For bilinear operations where optimized CUDA kernels only exist for computations on floating-point inputs (e.g., convolutions, matrix multiplications), then CryptGPU applies the above technique of first decomposing the input into $k=4$ tensors of 16-bit values, computing all necessary $O(k^{2})$ pairwise products of the resulting blocks (using the floating point kernel), and re-combines the pairwise products to obtain the final output.

III Threat Model and Cryptographic Design

In this section, we provide a formal specification of our threat model and a description of the private inference and training functionalities we develop. We then describe the cryptographic sub-protocols we use to construct our privacy-preserving training and inference protocols.

We begin by introducing the notation we use in this work. For a finite set $S$, we write $x\xleftarrow{\textsc{r}}S$ to denote that $x$ is drawn uniform at random from $S$. We use boldface letters (e.g., $\mathbf{x},\mathbf{y}$) to denote vectors and use non-boldface letters (e.g., $x_{i},y_{i}$) to denote their components. We denote our three parties by $P_{1},P_{2},P_{3}$. To simplify notation, whenever we use an index $i\in\{1,2,3\}$ to denote a party (or a share), we write $i-1$ and $i+1$ to denote the “previous” party and the “next” party, respectively. For example, $P_{3+1}$ refers to $P_{1}$.

Similar to several recent 3-party protocols , we design our system in the honest-majority model. Moreover, we focus on semi-honest adversaries. Namely, we assume that each of the three computing parties follow the protocol, but may individually try to learn information about other parties’ inputs. Formally, we consider the standard simulation-based notion of security in the presence of semi-honest adversaries :

Let $f\colon(\{0,1\}^{n})^{3}\to(\{0,1\}^{m})^{3}$ be a randomized functionality and let $\pi$ be a protocol. We say that $\pi$ securely computes $f$ in the presence of a single semi-honest corruption if there exists an efficient simulator $\mathcal{S}$ such that for every corrupted party $i\in\{1,2,3\}$ and every input $\mathbf{x}\in(\{0,1\}^{n})^{3}$,

where $\mathsf{view}^{\pi}_{i}(\mathbf{x})$ is the view of party $i$ in an execution of $\pi$ on input $\mathbf{x}$, $\mathsf{output}^{\pi}(\mathbf{x})$ is the output of all parties in an execution of $\pi$ on input $\mathbf{x}$, and $f_{i}(\mathbf{x})$ denotes the $i^{\text{th}}$ output of $f(\mathbf{x})$.

In this work, we consider two main settings: private inference and private training on secret-shared inputs. We use standard 3-out-of-3 additive secret sharing as well as 2-out-of-3 replicated secret sharing. Abstractly, we model both types of secret sharing as a pair of algorithms $(\mathsf{Share},\mathsf{Reconstruct})$ with the following properties:

On input $x\in\{0,1\}^{n}$, the share algorithm $\mathsf{Share}(x)$ outputs a tuple of three shares $(x_{1},x_{2},x_{3})$.

The reconstruction algorithm $\mathsf{Reconstruct}(S)$ takes a set of shares $T$ and outputs a value $x\in\{0,1\}^{n}$ if successful and $\bot$ if not.

Correctness of a threshold secret sharing scheme with threshold $t$ says that for any subset of shares $T\subseteq\mathsf{Share}(x)$ of size at least $t$, $\mathsf{Reconstruct}(T)=x$. Perfect security says that there exists a probabilistic polynomial-time simulator $\mathcal{S}$ such that for every subset $T\subseteq\{1,2,3\}$ where $|T|<t$ and every input $x\in\{0,1\}^{n}$,

We now formally define our notion of private inference and private training on secret-shared inputs:

Private inference: Inference is the problem of evaluating a trained model $M$ on an input $x$. We denote this operation as $\mathsf{Eval}(M,x)$. In private inference, the ideal functionality $f$ maps secret shares of an input $x$ and a model $M$ to a secret share of the output $\mathsf{Eval}(M,x)$. Namely, on input $((M_{1},x_{1}),(M_{2},x_{2}),(M_{3},x_{3}))$, the ideal functionality outputs $\mathsf{Share}(\mathsf{Eval}(M,x))$ where $M\leftarrow\mathsf{Reconstruct}(\{M_{1},M_{2},M_{3}\})$ and $x\leftarrow\mathsf{Reconstruct}(\{x_{1},x_{2},x_{3}\})$. In particular, a private inference protocol ensures privacy for the model $M$, the input $x$, and the output $\mathsf{Eval}(M,x)$.

Private training: In private training, the goal is to run a training algorithm $\mathsf{Train}$ on some dataset $D$. In this case, the ideal functionality $f$ maps secret shares of the dataset $(D_{1},D_{2},D_{3})$ to a secret share of the model $\mathsf{Share}(\mathsf{Train}(D))$ where $D\leftarrow\mathsf{Reconstruct}(D_{1},D_{2},D_{3})$. In this case, each party individually learn nothing about the input dataset $D$ or the resulting learned model $\mathsf{Train}(D)$.

III-B Cryptographic Building Blocks for Private Inference

We now describe the main MPC building blocks we use for private inference on deep neural networks. First, we decompose the neural network inference algorithm into a sequence of elementary operations: linear/pooling/convolution layers and activation function evaluation (ReLU). To obtain our protocol $\pi$ for computing the ideal functionality for private inference, we sequentially compose the semi-honest secure protocols for realizing each of the elementary operations. Correctness and semi-honest security of the overall protocol then follows by correctness and security of the underlying sub-protocols together with the sequential composition theorem .

As alluded to in Sections I-A and II-B, we seek cryptographic protocols that are particularly amenable to GPU acceleration. For example, protocols that involve conditionals (such as garbled circuits ) or require extensive finite field arithmetic are more challenging to support efficiently on the GPU. For this reason, we focus primarily on secret-sharing based protocols and work over a ring with a power-of-two modulus. In the following description, we elect to use cryptographic protocols where the underlying implementations vectorize and whose evaluation can be expressed primarily in terms of point-wise or component-wise operation on blocks of data.

Secret sharing.

Fixed point representation.

Protocol initialization.

In the following description, we assume that the parties have many independent secret shares of . This will be used for “re-randomization” during the protocol execution. We implement this using the approach of Araki et al. . Specifically, let $F$ be a pseudorandom function (PRF). At the beginning of the protocol, each party $P_{i}$ samples a PRF key $k_{i}$ and sends $k_{i}$ to party $P_{i+1}$. The $j^{\text{th}}$ secret share of is the triple $(z_{1},z_{2},z_{3})$ where $z_{i}=F(k_{i},j)-F(k_{i-1},j)$.

Linear operations.

Multiplication.

Since $x,y$ are fixed-point encodings, the parties additionally need to rescale $z$ after computing the product (i.e., divide it by the scaling factor $2^{t}$). In this work, we use the truncation protocol $\Pi_{\mathsf{trunc1}}$ from $\text{ABY}^{3}$ to implement this procedure. We note that Mohassel and Rindal propose two versions of the share truncation protocol: a two-round protocol $\Pi_{\mathsf{trunc1}}$ that only relies on elementary arithmetic operations and a one-round protocol $\Pi_{\mathsf{trunc2}}$ that relies on precomputed “truncation tuples”. While generating the truncation tuples can be done in a separate offline phase, doing so requires implementing a Boolean bit extraction circuit over secret-shared values. In contrast, $\Pi_{\mathsf{trunc1}}$ relies exclusively on arithmetic operations, and naturally extends to our tensor-based computing model. For this reason, we use the two-round truncation protocol $\Pi_{\mathsf{trunc1}}$ in our implementation. This has the added advantage that we avoid a separate (and potentially expensive) preprocessing step. Both of these share-truncation protocols are not exact and may introduce $1$ bit of error in the least significant bit of the secret-shared value (i.e., with $t$ bits of fixed-point precision, the error introduced is bounded by $2^{-t}$). We provide an empirical assessment of the error (and resulting model accuracy) in Section IV-D.

Convolutions and matrix multiplication.

The above protocols for computing linear functions as well as products of secret-shared values directly vectorize to yield protocols for computing linear functions on tensors as well as bilinear operations like matrix multiplication and convolution. Linear functions on secret-shared tensors only require local computation. Bilinear operations on secret-shared tensors like matrix multiplications and convolutions are implemented by computing three separate products (as described in the multiplication protocol above). These computations over secret-shared tensors directly map to analogous computations on local shares, so we can take advantage of existing highly-optimized CUDA kernels for evaluating these operations via the technique from Section II-B.

As in several previous systems (e.g., ), when we compute products of secret-shared tensors, we only apply the truncation protocol to the result of the product and not after each individual multiplication. This has a significant impact on the performance of the protocol for two reasons: (1) we can use existing CUDA kernels optimized for matrix products and convolutions without needing to modify how the elementary multiplications are performed; and (2) the total communication in the protocol is proportional to the size of the output rather than the number of intermediate element-wise multiplications.

Most significant bit.

The majority of this computation is the evaluation of the addition circuit over binary shares on the GPU. Evaluating a Boolean addition circuit on secret-shared binary values decomposes into a sequence of bitwise and and xor operations (along with communication for the and gates), which can be computed using efficient GPU kernels. We provide microbenchmarks in Section IV-C.

ReLU activation function.

III-C Additional Building Blocks for Private Training

To support private training, we need to augment our existing toolkit with several additional protocols. Here, we consider a standard backpropagation setting with a softmax/cross-entropy loss function optimized using (minibatch) stochastic gradient descent (SGD) . As with private inference, we decompose the backpropagation algorithm into a sequence of elementary operations and build our private training protocol by sequentially composing protocols for the elementary operations.

For the ReLU layers, the gradient computation reduces to evaluating the derivative of the ReLU function. The gradients for the linear/convolution layers are themselves linear functions of the gradients from the preceding layer, and thus, can be handled using the protocols from Section III-B. In the following, we describe our protocols for evaluating the softmax and the derivative of the ReLU function on secret-shared values. Note that backpropagation does not require computing the value of the loss function (Eq. III.1), so we do not need a protocol for computing logarithms on secret-shared values.

Exponentiation.

We approximate the exponential function $e^{x}$ needed to compute softmax with its limit characterization $f_{m}$:

Using a Taylor expansion for the function $\ln(1+x)$ and assuming that $|x|<m$,

Thus, the degree-$m$ approximation $f_{m}$ provides a good approximation $e^{x}$ on an interval of size $O(\sqrt{m})$ centered at . A common alternative approximation is to use Taylor series to approximate the exponential function. The advantage of using a Taylor series approximation of degree $m$ is that it provides a good estimate in an interval of size $O(m)$ (as opposed to $O(\sqrt{m})$ using the approximation $f_{m}$). However, using a Taylor series approximation has several drawbacks:

Evaluating a degree-$m$ Taylor approximation requires $m$ multiplications over $O(\log m)$ rounds. Computing $f_{m}$, in comparison, only requires $\log m$ multiplications. For a fixed degree $m$, the cost of computing the $f_{m}$ approximation is exponentially smaller than computing the degree-$m$ Taylor series approximation.

The size of the smallest coefficient in the Taylor series of degree $m$ is $1/m!$. In a fixed-point encoding scheme with $t$ bits of precision, values less than $2^{-t-1}$ round to . This gives an upper bound on the degree of the Taylor expansion we can feasibly support. Alternatively, we could compute the terms $x^{m}/m!$ in the Taylor expansion as $\prod_{i\in[m]}\frac{x}{i}$, but this now requires $O(m)$ rounds of multiplications to compute.

In our setting, the inputs to the exponential function are drawn from the interval $(-\infty,0]$. The approximation $f_{m}(x)$ has the appealing property that as $x\to-\infty$, $f(x)\to 0$, which matches the behavior of $e^{x}$. In contrast, the Taylor approximation diverges as $x\to-\infty$. This can introduce significant errors in the computation (unless we use a Taylor approximation of sufficiently high degree). For the models and inputs we consider in Section IV-A, most inputs to the exponential function lie in the interval $$. Ensuring that the Taylor approximation does not diverge for all inputs in this interval would require a high-degree approximation.

Thus, compared to a Taylor approximation, the limit-based approximation $f_{m}$ is more efficient to evaluate (in terms of the number of multiplications) and more robust for handling large negative inputs that may arise in the computation.

Division.

Computing the softmax function requires computing a quotient $\llbracket x/y\rrbracket^{n}$ on secret-shared values $\llbracket x\rrbracket^{n}$ and $\llbracket y\rrbracket^{n}$ and where $1\leq y\leq Y$, for some bound $Y$. It suffices to compute the reciprocal $\llbracket 1/y\rrbracket^{n}$ and compute the quotient using share multiplication. Similar to previous works , we use the iterative Newton-Raphson algorithm to approximate the value of $1/y$. Very briefly, the Newton-Raphson algorithm for approximating $1/y$ starts with an initial “guess” $z_{0}$ and iteratively computes $z_{i}\leftarrow 2z_{i-1}-yz_{i-1}^{2}$. In this work, we use a fixed initialization $z_{0}=1/Y$. This provides a highly-accurate estimate for $1/y$ for all $y\in[1,Y]$ using $O(\log Y)$ iterations of Newton’s algorithm. To see this, let

where $\varepsilon_{i}=|1-z_{i}y|$. Substituting in the Newton-Raphson updates, $\varepsilon_{i}=\varepsilon_{i-1}^{2}$, so the maximum error after $i$ iterations is $(1-1/Y)^{2^{i}}\leq e^{-2^{i}/Y}$.

We note that using a more accurate initialization for Newton-Raphson will allow convergence in fewer iterations. However, methods for computing a more accurate estimate for the initialization typically rely on binary-valued operations (e.g., comparisons) and are more costly than using a fixed initialization and increasing the number of iterations. Note that a fixed initialization is possible in our setting because we are guaranteed that the values $y$ lies in a fixed interval (due to the normalization in the softmax computation).

Maximum.

Derivative of ReLU.

During backpropagation, we also need to compute the derivative of the ReLU function $\mathsf{ReLU}^{\prime}(x)$, which is if $x<0$ and $1$ if $x>0$. This again corresponds to computing the most significant bit of the fixed-point encoding of $x$, which we implement using the protocol from Section III-B.

IV System Implementation and Evaluation

We build CryptGPU on top of CrypTen, which itself builds on PyTorch. First, we introduce the CUDALongTensor data type that represents a PyTorch tensor for 64-bit integer values (see Section II-B). Our design enables us to take advantage of optimized CUDA kernels for evaluating bilinear operations such as convolutions and matrix multiplications on secret-shared tensors. This suffices for evaluating arithmetic circuits on secret-shared tensors. Using these elementary building blocks, we then implement protocols for each of the operations described in Section III (i.e., the truncation protocol for fixed-point multiplication, ReLU computation, and the softmax function). Through composing these individual protocols together, we obtain an end-to-end system for private inference and private training.

We leverage PyTorch’s torch.distributed package for point-to-point communication between parties. The default communication mode in PyTorch is a “broadcast” mode where every message sent by a party is sent to all peers. To emulate point-to-point channels (as required by our protocol), we initialize a separate communication back end between each pair of parties. In this case, a “broadcast” channel between each pair of parties functions as a point-to-point channel between the parties.

Pseudorandom generators on the GPU.

We use AES as the PRF in our protocol (used for share re-randomization in the truncation protocol). We use the torchcsprng PyTorch C++/CUDA extension (based on the Salmon et al. protocol ) which enables AES evaluation on the GPU.

IV-A Experimental Setup for System Evaluation

We now describe our experimental setup for evaluating CryptGPU as well as the specific parameters we use to instantiate our cryptographic protocols from Section III.

We evaluate CryptGPU on the following standard datasets for object recognition:

MNIST . MNIST is a dataset for handwritten digit recognition. The training set has 60,000 images and the test set has 10,000 images. Each digit is a grayscale (i.e., single-channel) $28\times 28$ image. Due to its relatively small size, it is widely used as a benchmark in many privacy-preserving ML systems .

CIFAR-10 . CIFAR-10 is a dataset with 60,000 $32\times 32$ RGB images split evenly across 10 classes.

Tiny ImageNet . Tiny ImageNet is a modified subset of the ImageNet dataset. It contains 100,000 $64\times 64$ RGB training images and 10,000 testing images split across 200 classes. Compared to CIFAR-10, Tiny ImageNet is much more challenging: each image is $4\times$ larger and there are $20\times$ more classes.

ImageNet . ImageNet is a large-scale visual recognition dataset with more than 1,000,000 training images. It is the standard benchmark for evaluating the classification performance of computer vision models. ImageNet has 1000 classes, and each example is a center-cropped $224\times 224$ RGB image. The only prior system for privacy-preserving machine learning that demonstrates performance at the scale of ImageNet is CrypTFlow .

Deep learning models.

For our experimental evaluation, we measure the cost of our private training and private inference protocols on several representative CNN architectures developed for object recognition. Each of these networks can be represented as a composition of a collection of standard layers: convolution, pooling, activation, batch normalization, softmax, and fully-connected layers.

LeNet . LeNet was proposed by LeCun et al. for handwritten digit recognition. It is a shallow network with 2 convolutional layers, 2 average pooling layers, and 2 fully connected layers. The network uses the hyperbolic tangent ($\tanh$) as its activation function.

AlexNet . AlexNet was the winner of 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC-2012) competition. It has 5 convolutional layers, 3 max pooling layers, and 2 fully connected layers for a total of 61 million parameters. AlexNet uses ReLU as its activation function.

VGG-16 . VGG-16 is the runner-up of the ILSVRC-2014 competition. It uses 16 layers consisting of convolution, ReLU, max pooling, and fully-connected layers. VGG-16 has a total of 138 million parameters.

ResNet . ResNet is the winner of ILSVRC-2015 competition. It introduces skip-connections that addresses the vanishing gradient problem when training deep neural network models. ResNet consists of convolution, max pooling, average pooling, batch normalization, and fully connected layers. Since their inception, the ResNet family of models have enjoyed wide adoption in the computer vision community. We evaluate the performance of ResNet-50, ResNet-101, and ResNet-152 on ImageNet. These networks respectively have 23, 44, and 60 million parameters and 50, 101, and 152 layers.

Architecture adjustments.

We use the standard architecture of each of these networks, except with the following modifications:

AlexNet and VGG-16 on small datasets. Since AlexNet and VGG-16 were designed for ImageNet, they are not directly compatible with smaller inputs (i.e., those from CIFAR-10 or Tiny ImageNet). Thus, when using AlexNet or VGG-16 with smaller inputs, we have to modify the network architecture. For AlexNet, we drop the final max pooling layer for CIFAR-10, and adjust the number of neurons in the fully-connected classification layers to $256$-$256$-$10$ and $1024$-$1024$-$200$ for CIFAR-10 and Tiny ImageNet, respectively. For VGG-16, we adjust the number of neurons in the fully-connected classification layers to $256$-$256$-$10$ and $512$-$512$-$200$ for CIFAR-10 and Tiny ImageNet, respectively.Previous systems like Falcon made similar adjustments when evaluating AlexNet and VGG-16 on smaller datasets. When evaluating AlexNet on ImageNet, we use the original architecture . In the case of VGG-16, we add a 2x2 average pooling layer to reduce the input dimension of the first fully connected layer from $18432$ to $4608$; this is due to memory limitations on the GPU. When we compare our system to the Falcon system on these models and datasets, we make the same adaptations. We provide the full specification of the AlexNet and VGG-16 model architectures we use in Appendix A.

Activation functions. All networks we consider except LeNet use the ReLU function as the activation function. In contrast, LeNet uses the hyperbolic tangent function $\tanh$ as the underlying activation function. Since CryptGPU does not support evaluating the $\tanh$ function and modern networks primarily use ReLU as their activation function, we replace $\tanh$ with ReLU in our experiments with LeNet.

Average pooling. Pooling is a standard way to down-sample the outputs of the convolutional layers in a CNN. Specifically, a pooling layer accumulates the output of the convolutional layers by replacing each (small) window of the feature map (from the convolutional layer) with the average of the values (i.e., average pooling) or the max of the values (i.e., max pooling). Earlier networks such as AlexNet and VGG-16 used max pooling throughout, while more recent deep networks such as the ResNets primarily use average pooling (with a single max pooling layer at the beginning). While the choice of pooling function does not make a significant difference in the computational costs of plaintext training, this is not the case in private training. The difference is due to the fact that average pooling is a linear operation while max pooling is a highly non-linear operation. To reduce the computational overhead of our system, we replace all the max pooling layers in the above networks with average pooling. This reduces the complexity at the cryptographic level and allows us to take better advantage of GPU parallelism.

We show in Section IV-B that in existing systems, the pooling layer is not the bottleneck, and the performance improvements of our protocol relative to past works is not due to our substitution of average pooling in place of max pooling. We additionally show in Section IV-D that using average pooling in place of max pooling does not significantly affect the accuracy of the models we consider.

Protocol instantiation.

We instantiate our protocols from Section III using the following parameter settings:

Exponentiation. We use the function $f_{m}$ from Eq. III.3 to approximate the exponential function. In this work, we take $m=2^{9}=512$, so evaluating $f_{m}$ requires $\log m=9$ rounds of multiplication. With $t=20$ bits of fixed-point precision, we measure the maximum error of our approximation on all inputs $x\leq 0$ to be at most $6\cdot 10^{-4}$.

Division. As described in Section III-C, we require a private division protocol to compute $\llbracket 1/y\rrbracket^{n}$ where $y\in[1,Y]$, and $Y$ is the number of classes in the classification problem. For all of the datasets we consider for private training, $Y\leq 200$. In our implementation, we use 13 iterations of Newton-Raphson (with $1/Y$ as the initialization). With $t=20$ bits of fixed-point precision, we measure the maximum absolute difference between the approximate value and the true value for inputs in the interval $[1,Y]$ to be $\approx 10^{-4}$ (and $\approx 10^{-9}$ using floating-point evaluation).

IV-B Benchmarks for Private Training and Inference

We run our experiments on three Amazon EC2 instances optimized for GPU computation (p3.2xlarge). Each instance has a single NVIDIA Tesla V100 GPU with 16 GB of GPU memory. All of the instances run Ubuntu 18.4 and have 8 Intel Xeon E5-2686 v4 (2.3 GHz) CPUs and 61 GB of RAM. We consider a local area network (LAN) environment and place all three servers in the us-east-1 (Northern Virginia) region. In this case, we measure the network bandwidth to be 1.25GB/s with an average latency of 0.2ms. For each model/dataset pair we consider in our evaluation, we measure the end-to-end protocol execution time and the total amount of communication.

We compare the performance of CryptGPU against Falcon and CrypTFlow . To our knowledge, these are the only privacy-preserving machine-learning frameworks that have demonstrated the ability to handle neural networks at the scale of AlexNet on large datasets. Since our primary focus is on the scalability of our approach and not on the performance on shallow networks (where GPUs are unlikely to shine compared to optimized CPU protocols), we focus our comparisons with Falcon and CrypTFlow.

For CrypTFlow (which supports private inference for ResNet), we use the performance numbers reported in their paper (which also operate in a LAN environment).

For Falcon (which supports private inference and private training for LeNet, AlexNet, and VGG-16), we collect benchmarks using their provided reference implementation . We run the Falcon system on three compute-optimized AWS instances (c4.8xlarge) in the Northern Virginia region.Note that we use different instances for our comparison because CryptGPU is GPU-based while Falcon is CPU-based. Each instance runs Ubuntu 18.4 and has 36 Xeon E5-2666 v3 (2.9 GHz) CPUs and 60 GB of RAM. We measure the network bandwidth between machines to be 1.16GB/s with an average latency of 0.2ms.

For the main benchmarks, we also measure the computational cost using PyTorch on plaintext data (with GPU acceleration).

Private inference.

Table I summarizes the performance of CryptGPU’s private inference protocol on the models and datasets described in Section IV-A. For shallow networks and small datasets (e.g., LeNet on MNIST or AlexNet on CIFAR), Falcon outperforms CryptGPU. However, as we scale to progressively larger datasets and deeper models (e.g., VGG-16 on Tiny ImageNet), then CryptGPU is faster ($3.7\times$ on VGG-16). The performance on small datasets is not unexpected; after all, if the computation is sufficiently simple, then the extra parallelism provided by the GPU is unlikely to benefit. Moreover, the use of more efficient cryptographic building blocks (which may not be “GPU-friendly”) can allow a CPU-based approach to enjoy superior performance.

The setting where we would expect the GPU-based approach to perform well is in the setting of large datasets and deeper models. For instance, at the scale of ImageNet, CryptGPU is able to perform private inference over the ResNet-152 network (containing over 150 layers and over 60 million parameters) in just over 25 seconds. This is about $2.2\times$ faster than CrypTFlow, which to our knowledge, is the only protocol for private inference that has demonstrated support for the ResNet family of networks on the ImageNet dataset. For the ResNet family of networks, the running time of CryptGPU scales linearly with the depth of the network.

Compared to plaintext inference on the GPU, there still remains a significant $1000\times$ gap in performance. This underscores the importance of designing more GPU-friendly cryptographic primitives to bridge this gap in performance.

Batch private inference.

We can also leverage GPU parallelism to process a batch of images. This allows us to amortize the cost of private inference. Table II shows the time and communication needed for private inference over a batch of 64 images on the CIFAR-10 dataset. Here, the amortized cost of private inference on a single image using AlexNet drops from 0.91s to 0.017s (a $53\times$ reduction). With VGG-16, batch processing reduces the per-image cost from 2.14s to 0.18s (a $12\times$ reduction).

Table III shows the time and communication needed for private inference on ImageNet using the ResNet networks with a batch of 8 images. Here, we see a $1.9\times$ reduction in the amortized per-image private inference cost for each of ResNet-50, ResNet-101, and ResNet-152. The cost reduction compared to those on the CIFAR-10 dataset (Table II) is smaller. This is likely due to the smaller batch sizes in play here (8 vs. 64). Supporting larger batch sizes is possible by either using multiple GPUs or using GPUs with more available memory. Nonetheless, irrespective of the model/input size, we observe that batch private inference allows us to amortize the cost of private inference protocol. Communication in all cases scales linearly with the batch size.

Private training.

We expect GPUs to have a larger advantage in the setting of private training (just like modern deep learning, training is much more challenging than inference and thus, more reliant on hardware acceleration). We measure the time needed for a single iteration of private backpropagation (Section III-C) on a batch size of 128 images for several dataset/model configurations and summarize our results in Table IV (together with measurements for the equivalent plaintext protocol). We only compare with Falcon because CrypTFlow does not support private training. We note that the public implementation of the Falcon system does not include support for computing the cross-entropy loss function for backpropagation. However, given the gradients for the output layer, the provided implementation supports gradient computation for intermediate layers. Thus, our measurements for the Falcon system only includes the cost of computing the gradients for intermediate layers and not for the output layer; this provides a lower bound on the running time of using Falcon for private training. Our system supports the full backpropagation training algorithm.

Our system achieves a considerable speedup over Falcon in multiple settings, especially over larger models and datasets. For instance, to train AlexNet on Tiny ImageNet, a single iteration of (private) backpropagation completes in 11.30s with CryptGPU and 6.9 minutes using Falcon. For context, privately training AlexNet on Tiny ImageNet (100,000 examples) would just take over a week ($\approx 10$ days) using CryptGPU while it would take over a year ($\approx 375$ days) using Falcon (assuming 100 epochs over the training set).

On the larger VGG-16 network, our system is constrained by the amount of available GPU memory. Our system currently supports a maximum batch size of 32 when training VGG-16 on CIFAR-10 and a maximum batch size of 8 when training on Tiny ImageNet. To establish a fair comparison when comparing our system against Falcon for privately training VGG-16, we apply the same batch size adjustment. As shown in Table IV, when training VGG-16, our system is $30\times$ faster when training on CIFAR-10 and $26\times$ when training on Tiny ImageNet. Reducing the memory overhead of our protocol and augmenting it with support for multiple GPUs (as is standard for modern deep learning) will enable better scalability. We leave this as an interesting direction for future work.

Like the setting of private inference, there still remains a large gap (roughly $2000\times$) between the costs of private training and plaintext training (on the GPU). Designing new cryptographic protocols that can take even better advantage of GPU parallelism will be important for closing this gap.

Private training breakdown.

In Table V, we provide a fine-grained breakdown of the costs of processing the different layers in a single iteration of private training. Not surprisingly, the primary advantage of our GPU-based protocol compared to the CPU-based protocol of Falcon is in the computation of the linear layers. In the settings we consider, evaluation of the linear layers is between $25\times$ and $70\times$ faster with our system. The linear layers are the primary bottleneck in Falcon, and account for $86\%$ to $99\%$ of the overall computational cost. In CryptGPU, the computational costs are more evenly split between the linear layers and the non-linear layers.

For the pooling layers, the performance difference between CryptGPU and Falcon can be partially attributed to the the fact that Falcon uses max pooling rather than average pooling. As discussed in Section IV-A, average pooling is a linear function and simpler to evaluate privately. However, our measurements show that CryptGPU maintains a (significant) performance edge even if we exclude the cost of the pooling layers from the running time of Falcon.

Finally, for the ReLU layers, the CPU-based protocol in Falcon compares very favorably with the ReLU protocol in CryptGPU, and even outperforms our protocol on the smaller models and datasets. Having a ReLU protocol that can better take advantage of GPU parallelism will likely improve the performance of our protocol. As described in Section III-B, our ReLU protocol relies on an arithmetic-to-binary share conversion, which is less GPU-friendly compared to bilinear operations. The ReLU protocol from Falcon relies on different techniques and it is interesting whether their approach can be adapted to be efficiently computed on the GPU.

Avenues for improvement.

Compared to Falcon, our private training protocol is more communication-intensive. Falcon develops a number of specialized cryptographic protocols to substantially reduce the communication in their protocols. We believe it is an interesting question to study whether the protocols developed in Falcon are “GPU-friendly” and can benefit from GPU acceleration.

CryptGPU does not currently support batch normalization during private training, so we do not report private training benchmarks on the ResNet-family of models.Note that we can still perform private inference for a model that is trained using batch normalization. Namely, the normalization parameters are secret-shared (as part of the model) and applying batch normalization just corresponds to an affine transformation. Developing a GPU-friendly protocol for batch normalization is an interesting avenue for further work and an important step towards supporting private training of the ResNet family of models. We are not aware of any system that currently supports private training over ResNet.

IV-C Microbenchmarks

To quantify the advantage of keeping all of the computation on the GPU, we compare the running time of the MPC protocols for evaluating convolutions (i.e., the linear layers) and for evaluating ReLU (i.e., the primary non-linear layer) on the CPU vs. the GPU. For convolutions, we study the effect of both the input dimension as well as the batch size. We use the same experimental setup described in Section IV-A for all of the experiments in this section.

For convolutions, we consider two types of convolutions: (1) convolutions with a large receptive field (filter size) but a relatively small number of input/output channels; and (2) convolutions with a small receptive field, but a large number of input/output channels. Convolutions of the first type are generally used in the initial layers of the CNN while filters of the second type are used in the later layers of the CNN. Note that when implementing convolutions on the CPU, we do not break up the 64-bit secret-shared tensor into 16-bit blocks (as we do in the GPU setting; see Section II-B). We provide the microbenchmarks in Fig. 1.

From Figs. 1(a) and 1(c), we see that for small inputs, the computational cost of the private convolution protocol is comparable on both the GPU and the GPU. While there is only a $10\times$ speed-up for convolutions between a small $32\times 32\times 3$ input with a stack of $64$ filters, the gap grows quickly as the input size increases; for instance, increasing the input size to that of a Tiny ImageNet instance $(64\times 64\times 3)$, the GPU-based protocol is nearly $40\times$ faster. Scaling to a $512\times 512\times 3$ image, the GPU-based protocol is $174\times$ faster than the CPU-based protocol (from 23.9s on the CPU to 0.14s on the GPU). An analogous trend holds when we consider convolutions with a large number of input/output channels: for small inputs, the running times of the CPU- and GPU-based protocols are quite comparable, but for large inputs (e.g., a $64\times 64\times 512$ input), the GPU-based protocol is $168\times$ faster (from 543s on the CPU to just 3.2s on the GPU).

We additionally note that for small instances, the protocol running time on the GPU is essentially constant—this is due to the parallelism. Only after the input becomes sufficiently large do we start seeing increases in the running time based on the size of the input. In contrast, the CPU running time always scales with the size of the input.

Similar speedups are present when we consider convolutions on batches of inputs (this is important for training and for batch inference). For a fixed input size ($32\times 32\times 3$) and kernel size ($11\times 11$), we observe a $10\times$ speed-up for running the private convolution protocol on a single input using the GPU, but a $40\times$ to $60\times$ speed-up when we consider a batch of anywhere from 32 to 512 inputs. As an example, to evaluate a convolution over a batch of 512 inputs with this set of parameters, we require 11.6s on the CPU and only 0.27s on the GPU. We refer to Fig. 1(b) for the full comparison.

Private ReLU: GPU vs. CPU.

Previous privacy-preserving ML systems like Delphi leveraged GPUs to accelerate convolutions, but still executed the non-linear steps (e.g., ReLU computations) on the CPU. Here, we argue that with a carefully-chosen set of cryptographic protocols, we can also take advantage of GPU parallelism to accelerate the non-linear computations. To illustrate this, we compare the running time of our private ReLU protocol on the CPU vs. the GPU. As described in Section III-B, private ReLU evaluation of ReLU on a large block of neurons (e.g., output by the convolutional layer) corresponds to evaluating a large number of point-wise Boolean operations on secret-shared binary tensors. Such operations naturally benefit from GPU parallelism.

We measure the time it takes to privately-evaluate ReLU on different numbers of secret-shared inputs (ranging from 50,000 to 32,000,000). The full results are shown in Fig. 2. For ReLU evaluation, we see a $16\times$ speedup when evaluating ReLU on a block of 256,000 inputs (from 2s on the CPU to 0.12s on the GPU). As we scale up to a block with 32 million inputs (250 MB of data), there is a $9\times$ speedup on the GPU, with the absolute running time dropping from 149s on the CPU to just 16.3s on the GPU.

IV-D Accuracy of Privacy-Preserving Protocols

Several of the underlying protocols in CryptGPU are not exact and can introduce a small amount of error: using fixed-point encodings to approximate floating-point arithmetic, the share-truncation protocol from $\text{ABY}^{3}$, and the approximation to the softmax function. While we have chosen our parameters (e.g., the fixed-point precision) to reduce the likelihood of errors, we validate our parameter choices with an empirical analysis. In the following, we will often measure the difference between an output $z_{\text{priv}}$ computed using CryptGPU with the output $z_{\text{plain}}$ of a plaintext version of the same computation (using 64-bit floating-point values). We define the absolute error between $z_{\text{priv}}$ and $z_{\text{plain}}$ as $|z_{\text{priv}}-z_{\text{plain}}|$ and the relative error to be $|z_{\text{priv}}-z_{\text{plain}}|/z_{\text{plain}}$.

Previous privacy-preserving protocols like Falcon and Delphi use a smaller number of bits of fixed-point precision (e.g., 13 bits and 15 bits, respectively). In turn, they are able to work with arithmetic shares over a 32-bit ring as opposed to a 64-bit ring. This reduces communication (since shares are half as large) and in our model, also saves computation (recall from Section II-B that we need to split up tensors of 64-bit integers into 4 tensors of 16-bit integers in order to use existing CUDA kernels for deep learning).

Using fewer number of bits of precision reduces the accuracy of the protocol outputs, especially when scaling to deep architectures and large inputs. To analyze the effect the number of bits of fixed-point precision $t$ has on the accuracy of the outputs of our system (i.e., the values of the output layer), we compute the average relative error between the output values output by CryptGPU to those computed using the plaintext inference protocol on a small example (AlexNet over CIFAR-10) as well as a large example (ResNet-50 on ImageNet). Our results are summarized in Fig. 3.

Fig. 3 shows that for a relatively shallow model like AlexNet on the CIFAR-10 dataset, it is sufficient to use 12 to 14 bits of fixed-point precision (e.g., the parameter setting in ). The relative error in this case between the outputs computed by the private inference protocol and the plaintext computation is around 1%. However, when we scale up to a model like ResNet-50 on ImageNet, the average relative error in the model outputs increases $5\times$ to almost 5%. We further remark that we are only measuring the relative error in a single forward pass over the network (inference). Larger errors would be expected in the case of private training when the protocol needs to run multiple forward and backward passes. In this work, we use $t=20$ bits of fixed-point precision which ensures that the average relative error for private inference over ResNet-50 on ImageNet is under 0.02%. Our analysis indicates that scaling up to deeper architectures and operating over larger datasets will require a greater number of bits of precision in the underlying fixed-point representation. For instance, to keep the average relative error under 1% for ResNet-50 on ImageNet, we require at least 15 bits of fixed-point precision. As such, to prevent overflows in the arithmetic evaluation over secret-shared data for deep networks, a 32-bit ring is no longer sufficient.

Privacy-preserving inference.

To evaluate the accuracy of our private inference protocol, we compare the average relative error between the outputs of our private inference protocol using ResNet-50, ResNet-101, and ResNet-152 on ImageNet and compare those against the values obtained from plaintext evaluation. We additionally compute the accuracy of the predictions (using the standard metrics of Top-1 and Top-5 accuracy—i.e., the model succeeds if the actual class of an example coincides with the most likely class predicted by the model or among the top 5 most likely classes predicted by the model). The results are summarized in Table VI. In particular, for our chosen set of parameters, we observe that the average relative error in the classifier output is at most 0.021%, and in all cases we tested (100 randomly-chosen images from the ImageNet test set), both the Top-1 accuracy and the Top-5 accuracy exactly match that of the plaintext model.

Privacy-preserving training.

We perform a similar set of experiments to evaluate the accuracy of our private training protocol. In Fig. 4, we plot the value of the cross-entropy loss function for a model trained using the private training protocol of CryptGPU as well as for a model trained using the plaintext training algorithm (using the same initialization and learning rate for the underlying stochastic gradient descent optimizer). Fig. 4 shows that the value of the loss function is slightly higher initially for private training, but the overall progression closely follows that of plaintext training.

In addition to comparing the evolution of the loss function, we also compare the model accuracies (as measured on the validation set) for the models trained using CryptGPU and using the plaintext training algorithm (again with same initialization and learning rate as above). Our results are summarized in Table VII. On all of the models/datasets we considered, the accuracy of the model output by CryptGPU closely matches that of the plaintext evaluation. These experiments indicate that CryptGPU efficiently and accurately supports end-to-end private training for models like AlexNet over moderately-large datasets like Tiny ImageNet.

Average pooling vs. max pooling.

As discussed in Section IV-A, we use average pooling in place of max pooling in the models we consider. To evaluate whether the choice of pooling makes a significant difference on model performance, we use PyTorch to train the AlexNet and VGG-16 networks over the CIFAR-10 dataset where we replace all of the max pooling layers with average pooling layers. The resulting model accuracy on the CIFAR-10 test set is shown in Table VIII. In particular, we observed a $3\%$ drop in accuracy (from 76% to 73%) for AlexNet and a $1\%$ increase in accuracy with VGG-16 (from 82% to 83%). This indicates that using average pooling in place of max pooling does not lead to a significant degradation of model performance. We note also that in contrast to AlexNet and VGG-16 which use max pooling exclusively, the more recent ResNets use average pooling in all but the initial layer.

V Related Work

Privacy-preserving machine learning is a special case of secure computation and can be solved via general cryptographic approaches such as secure 2-party computation (2PC) , secure multiparty computation or fully homomorphic encryption . While powerful, these general approaches incur significant overhead, and much of the work in developing concretely-efficient protocols for scalable privacy-preserving machine learning have focused on more specialized approaches (that still rely on the general building blocks for designing sub-protocols). We survey some of these techniques here.

Many recent works have developed specific protocols for the problem of private inference for deep learning models (c.f., and the references therein). These works operate in a variety of different models and architectures: some works consider a 2-party setting (e.g., ), others consider a 3-party (e.g., ) or a 4-party setting (e.g., ). Some frameworks assume that the model is held in the clear (e.g., ) while others (including this work) support secret-shared models (e.g., ). With the recent exceptions of Falcon and CrypTFlow , these existing approaches only consider privacy-preserving inference using shallow neural networks (e.g., less than 10 layers) on relatively small datasets (at the scale of MNIST or CIFAR ). Our focus in this work is designing privacy-preserving machine learning protocols that are able to support inference over modern deep learning models (which typically contain tens of millions of parameters and over a hundred layers) on large datasets (i.e., at the scale of ImageNet , one of the de facto standards for state-of-the art computer vision). As shown in Section IV-B, our system outperforms both Falcon and CrypTFlow for inference over sufficiently-large models and datasets.

Privacy-preserving training.

Compared to private inference, privacy-preserving training of deep neural networks is a considerably more challenging and computationally-intensive problem and has received comparably less attention. Of the aforementioned works, only a few support privacy-preserving training. Among these systems, the only one that scales beyond MNIST/CIFAR is Falcon , which is the first system (to our knowledge) that supports privacy-preserving training at the scale of (Tiny) ImageNet and for models as large as AlexNet and VGG-16 . Our work is the first framework to leverage GPUs to demonstrate significantly better scalability to privately train deep networks over large datasets.

Privacy-preserving machine learning using GPUs.

Most of the works on privacy-preserving machine learning are CPU-based and do not leverage GPU acceleration. We discuss some notable exceptions. Some works use GPUs to accelerate homomorphic evaluation of convolutional neural networks on MNIST. Delphi uses GPUs to compute linear layers (i.e., convolutions) to support private inference; however, they still perform non-linear operations (e.g., ReLU evaluation) on the CPU and moreover, their scheme assumes the model to be public (and only the input is hidden). Our design philosophy in this work is to keep all of the computations on the GPU through a careful choice of “GPU-friendly” cryptographic protocols. Slalom shows how to integrate a trusted computing base (e.g., Intel SGX) with GPUs to enable fast private inference of neural networks (by offloading convolutions to the GPU and performing non-linear operations within the trusted enclave). Recent works proposing scalable private training and inference protocols highlight the use of GPUs as an important way for further scalability . Our system is the first to support private training and inference entirely on the GPU.

Model stealing and inversion attacks.

We note that MPC protocols can only hide the inputs to the computation (e.g., the model or the dataset) up to what can be inferred from the output. Several recent works have shown how black-box access to a model (in the case of an private inference service) can allow an adversary to learn information about the model or even recover its training data. Differentially-private training algorithms provide one defense against certain types of these attacks. Our focus in this work is on protecting the computation itself and ensure that there is no additional leakage about the inputs other than through the output. It is an interesting question to design a private training/inference protocol that also provides robustness against specific classes of model stealing/inversion attacks.

VI Conclusion

In this paper, we introduce CryptGPU, a new MPC framework that implements all of the cryptographic operations (both linear and non-linear) on the GPU. CryptGPU is built on top of PyTorch and CrypTen to make it easy to use for machine learning developers and researchers. Our experiments show that leveraging GPUs can significantly accelerate the private training and inference for modern deep learning and make it practical to run privacy-preserving deep learning at the scale of ImageNet and with complex networks. In addition, our systematic analysis of different cryptographic protocols provides new insights for designing “GPU-friendly” cryptographic protocols for deep learning. This will be an important step towards bridging the roughly $1000\times$ gap that still remains between private machine learning and plaintext machine learning (on the GPU).

Acknowledgments

We thank Pavel Belevich, Shubho Sengupta, and Laurens van der Maaten for their feedback on system design and providing helpful pointers. D. J. Wu is supported by NSF CNS-1917414.

References

Appendix A Network Architecture

As discussed in Section IV-A, some of the models we consider (e.g., AlexNet and VGG-16) were designed for ImageNet, and are not directly compatible with smaller datasets such as CIFAR-10 and Tiny ImageNet. As such, when training or running inference with these models on the smaller datasets, we make adjustments to their “head architecture” (i.e., the fully-connected classification layers at the top of the network). In all settings, we keep the same “base architecture” (adapted from their description in the original papers ). We describe the base AlexNet architecture we use in Fig. 5 and the head architectures for the different datasets in Fig. 6. We describe the base VGG-16 architecture we use in Fig. 7 and the head architectures for the different datasets in Fig. 8.