Model Stealing Attacks Against Inductive Graph Neural Networks

Introduction

Many real-world data come in the form of graphs, such as molecular graphs and social networks . The graph-structured data contains nodes with features and edges that represent the relationship between them. To fully unleash the potential of graph data, a new family of machine learning (ML) models, namely graph neural networks (GNNs), has been proposed . Compared to classical machine learning models, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are designed to process images and texts, GNNs offer state-of-the-art performance by taking both node features as well as graph structures into consideration.

Prior work unveiled that machine learning models are vulnerable to model stealing attacks , where an adversary with query access to a target model can steal its parameters or functionality. Concretely, the adversary first crafts a number of queries as the input to the target model’s API and obtains the corresponding outputs. Then, a local surrogate model is trained on the paired data (input, output). As such, the surrogate model may not only violate the intellectual property of the target model but also serve as a stepping stone for further attacks like membership inference and adversarial examples . Notably, most of the current efforts on model stealing attacks concentrate on ML models with images and text data . On the other hand, the potential model stealing risks of GNNs have been largely understudied.

There exists some preliminary work on model stealing attacks against GNNs . They focus on transductive GNNs and assume that the attackers have access to the training process of the target model, in which the training and query graphs are used to train the target model. As such, GNN model stealing attacks in a transductive setting are unrealistic.

In this paper, we concentrate on a more realistic and popularly deployed GNN setting, i.e., inductive GNNs, which can generalize well to unseen nodes . In this setting, the adversary only queries the target model via remotely accessible API. They do not tamper with the training process of the target model. Note that in this paper, we focus on node classification tasks.

When an adversary uses a query graph (i.e., a graph induced by a set of query nodes) to query the target model, they could face different types of responses from the target model, ranging from the posterior distribution over the possible labels of query nodes (called predicted posterior probability) to 2-dimensional t-SNE projection (for graph visualization). Therefore, it is important to summarize a complete taxonomy of the threats for model stealing attacks against GNNs. Also, it is challenging to design a general attack methodology that can be applied to different attack scenarios. Moreover, in reality, the graph structural information of the query graph can be missing. It is inevitably harder for the adversary to launch attacks against GNNs.

To tackle these challenges, we make the following contributions in this paper. We first systematically define the threat model of model stealing attacks against inductive GNNs by categorizing the adversary’s knowledge into two dimensions, i.e., query graph and target model’s response (see Section 3). Concretely, we assume that the adversary has a query graph that contains a number of nodes and their features. The node features come from the same distribution of the graph used to train the target model. However, the graph structure of the query graph may be missing (i.e., the edges connecting the nodes may not be available). Regarding the target model’s response, we consider three cases, i.e., the predicted posterior probability, the node embedding vector, or the 2-dimensional t-SNE projection. In turn, we have six attack scenarios under our threat model.

We propose two types of attacks, i.e., Type I and Type II, based on the information provided by the query graph (see Section 4.1). Each type has three variants depending on the target model’s responses. We design a general attack framework that can be applied to all the scenarios. Concretely, the framework is assembled with two major components. The first component is used to learn the discrete graph structure if the structural information is not available in the query graph. Then, the second component builds a surrogate model by jointly learning from the nodes’ features and the response of the target model.

Ideally, the surrogate model should achieve both high accuracy and high fidelity whereby accuracy measures the prediction correctness and fidelity measures the prediction agreement between the target model and the surrogate model . We evaluate all our attacks on three popular inductive GNN models including GraphSAGE , Graph Attention Network (GAT) , and Graph Isomorphism Network (GIN) with six benchmark datasets, i.e., DBLP , Pubmed , Citeseer Full , Coauthor Physics , ACM , and Amazon Co-purchase Network for Photos . Extensive experiments demonstrate that our model stealing attacks consistently achieve strong performance in both types of attacks. For instance, when the target model is GIN trained on the Pubmed dataset and the response is embeddings, our Type I attack achieves a 0.877 accuracy score and a 0.906 fidelity score (see Section 5.2). In particular, when the aforementioned target model’s response is the t-SNE projection, our attack still achieves strong performance with a 0.823 accuracy score and a 0.846 fidelity score. Moreover, we empirically demonstrate that even without graph structure information, i.e., Type II attacks, the adversary could still launch effective attacks, extracting high-accuracy and high-fidelity surrogate models (see Section 5.3). This further demonstrates the severe model stealing risks of GNN models.

In summary, we make the following contributions.

Our work is the first research effort to perform model stealing attacks against inductive GNNs.

We systematically define the threat model to characterize an adversary’s background knowledge along two dimensions. Moreover, we propose six different attack scenarios based on the adversary’s different background knowledge.

Extensive evaluation on three popular inductive GNN models and six benchmark graph datasets demonstrates the efficacy of our attacks.

Background

We define a labelled, undirected, unweighted, attributed graph as $\mathbf{G}=(\mathbf{V},\mathbf{E},\mathbf{X},\mathbf{C})$, where $\mathbf{V}=\{v_{1},v_{2},...,v_{n}\}$ denotes the set of nodes, $\mathbf{E}\subseteq\{(v,u)|v,u\in\mathbf{V}\}$ denotes the set of edges, $\mathbf{x}_{i}\in\mathbf{X}$ denotes the feature of node $v_{i}$, and one-hot vectors $\mathbf{c}_{i}\in\mathbf{C}$ denotes the label of node $v_{i}$. We denote $\mathbf{A}\in\{0,1\}^{n\times n}$ as the adjacency matrix, where $\mathbf{A}_{vu}=1,\forall(v,u)\in\mathbf{E}$. As such, $\mathbf{G}$ can also be represented as $\mathbf{G}=(\mathbf{A},\mathbf{X},\mathbf{C})$. The original graph used to train a GNN model is denoted as $\mathbf{G}_{O}$ (training graph). We use $\mathcal{N}^{l}(v)$ to denote $l$-hop neighborhood of $v$, and $\mathbf{G}_{v}^{l}$ to denote the subgraph induced by $l$-hop neighborhood of $v$. Besides, we denote the target GNN model as $\mathcal{M}_{T}$ and the surrogate GNN model as $\mathcal{M}_{S}$. The notations introduced here and in the following sections are summarized in Table 1.

2 Preliminaries

Graph Neural Networks (GNNs). Representation learning of graph-structured data is challenging because both graph structure and node features carry important information. Graph Neural Networks (GNNs) provide an effective way to fuse information from network structure and node features. Most of the GNNs follow a neighborhood aggregation strategy, where the model iteratively updates the representation of a node through message passing and aggregating representations of its neighbors. After $l$ iterations of aggregation, a node’s representation, denoted as $\mathbf{h}_{v}$, captures the structural information within its $l$-hop network neighborhood. In practice, a GNN contains several graph convolutional layers. Each graph convolutional layer of a GNN model can be defined as follows:

Note that the number of graph convolutional layers is equivalent to the $l$-hop network neighborhood that a GNN model can reach in the graph. Once trained, the GNN can map each node to an embedding vector. These node embeddings can be directly used for downstream machine learning tasks that can be categorized into three levels, i.e., node-level (e.g., node classification ), link-level (e.g., link prediction ), and graph-level (e.g., graph classification ).

Inductive GNN Models. There are two settings for training the GNNs, i.e., transductive setting and inductive setting. In the transductive setting, a GNN learns from both labelled and unlabelled nodes in a single fixed graph at the training time and predicts the labels of those unlabelled nodes once the training is done, e.g., vanilla graph convolutional network (GCN) , DeepWalk , etc. However, transductive GNN models must be retrained if new nodes are introduced to the graph. A more popular one is the setting of inductive learning, where the learned GNN model can be generalized to the graphs that are previously unseen during the training procedure. The reusable GNN model avoids time-consuming retraining if a graph includes more nodes or even subgraphs. It facilitates the real-world practices of graph data analytics. We therefore focus on the inductive setting in our study. We briefly introduce three widely used inductive GNN models below.

GraphSAGE. GraphSAGE proposed by Hamilton et al. is the first inductive GNN model. Inspired by the Weisfeiler-Lehman test for graph isomorphism, GraphSAGE generalizes the original GCN into the inductive setting with different aggregation functions. Take widely used mean aggregation operator as an example, GraphSAGE can be defined as follows:

Graph Attention Network (GAT). It is straightforward to observe that GraphSAGE assigns the same weight to all neighbors (i.e., $1/{\mathcal{N}(v)}$) when aggregating $v$’s neighborhood information. However, in practice, different nodes may play different roles in the target node embedding. Inspired by the attention mechanism in deep learning , Velickovic et al. propose GAT that leverages multi-head attention to learn different attention weights and pays more attention to the important neighborhoods. Its aggregation function can be formulated as:

where $\|$ is the concatenation operation, $Z$ is the total number of projection heads in the attention mechanism, $W^{z}$ is the linear transformation weight matrix, and $\alpha_{uv}^{z}$ is the attention coefficient calculated by the $z$-th projection head.

Graph Isomorphism Network (GIN). GraphSAGE can be treated as an instance of the Weisfeiler-Lehman test. Xu et al. propose Graph Isomorphism Network (GIN) to extend GraphSAGE with arbitrary aggregation functions on multi-sets. GIN is theoretically proven to be as powerful as the Weisfeiler-Lehman test of graph isomorphism. Its aggregation function can be represented as:

where $\epsilon$ is a learnable parameter to adjust the weight of node $v$.

The inductive GNNs usually employ shared weight parameters and neighborhood sampling to speed up the computation.

Responses by Inductive GNNs. An inductive GNN model learns the parameters of aggregation functions in different layers from the training data. Once trained, the learned GNN model can infer previously unseen data. Such capability paves the way for remotely deployed GNN models in the wild (e.g., GROVER , DGL ) to make inferences on graphs for customers via publicly accessible API. Moreover, a trained GNN model is often used to perform node embedding tasks, and the resulted node embeddings can then help to perform other downstream ML tasks (e.g., fine-tuning pretrained GNNs , model partitioning ) or graph visualization. Therefore, in this paper, we consider three query responses from a target GNN model when facing a query node, namely predicted posterior probability, embedding vector, and 2-dimensional t-SNE projections from the embedding . Specifically, given a query node $v\in\mathbf{V}_{Q}$, we feed its $l$-hop subgraph (i.e., $G_{v}^{l}$) to a remote GNN model and obtain one of the three corresponding responses. All the query nodes with the edges between them can form a query graph/dataset, denoted as $\mathbf{G}_{Q}$. Note that $\mathbf{G}_{Q}$ is not necessarily connected and for each query node, the $l$-hop subgraph is extracted from $\mathbf{G}_{Q}$ only.

Threat Model

In this section, we outline the threat model to characterize the adversary’s background knowledge and the goal of the model stealing attack.

We frame our attack in a black-box setting, which is the most challenging scenario for the adversary mentioned in previous work . That is, the adversary has no knowledge of the target GNN model (e.g., model parameters, model architecture) and cannot tamper with its training process (e.g., training graph $\mathbf{G}_{O}$). Our attack setting is fundamentally different from the previous attacks . These attacks assume that the adversary can gain access to the target model’s training process. However, such a strong assumption is unrealistic in the real world as it is impractical to expect an adversary can interfere with the target model at its training time. Note that in this paper we focus on the aforementioned three node-level query responses. The target model is an inductive GNN model which accepts node $v$’s $l$-hop subgraph $\mathbf{G}_{v}^{l}$ as input and returns the corresponding response for the given node $v$, i.e., its predicted posterior probability, node embedding vector, or 2-dimensional t-SNE projection.

2 Adversary’s Goal

Following the taxonomy defined by Jagielski et al. , the adversary’s goal falls into two categories, i.e., theft and reconnaissance.

The goal of the theft adversary is to build a surrogate model $\mathcal{M}_{S}$ that matches the accuracy of the target model $\mathcal{M}_{T}$ on the target task . The theft adversary’s motivation is compromising the intellectual property and violating the confidentiality of the target model $\mathcal{M}_{T}$.

Subtly different from the theft adversary, the reconnaissance adversary aims to build a surrogate model that closely matches the behavior of the target model. That is, $\mathcal{M}_{S}$ seeks an agreement to $\mathcal{M}_{T}$ on any input. A surrogate model $\mathcal{M}_{S}$ with high fidelity to $\mathcal{M}_{T}$ enables the adversary to leverage it as a stepping stone to launch further attacks. For instance, the adversary can craft adversarial examples using this $\mathcal{M}_{S}$ instead of risking potentially detectable queries to the target model $\mathcal{M}_{T}$ .

Note that the reconnaissance adversary’s motivation is faithfully copying the behavior of $\mathcal{M}_{T}$ (e.g., $\mathcal{M}_{S}$ and $\mathcal{M}_{T}$ may make the same wrong/correct prediction of an input) though both adversaries intend to get close to the performance (i.e., accuracy) of the target model. We refer the audience to the work by Jagielski et al. for additional discussions.

3 Adversary’s Capability

We first assume an adversary can make queries to a target model $\mathcal{M}_{T}$. Given a query graph $\mathbf{G}_{Q}$, the adversary can query all its nodes with their corresponding $l$-hop subgraphs, then obtain the query responses $\mathbf{R}$. This assumption is in line with the adversarial machine learning setting, whereas the attackers exploit the remote target model $\mathcal{M}_{T}$ via publicly accessible API . The response $\mathbf{R}$ of the whole query graph $\mathbf{G}_{Q}$, depending on the target model’s specification, may be returned in the form of a node embedding matrix (denoted as $\mathbf{H}$), a predicted posterior probability matrix (denoted as $\bm{\Theta}$), or a t-SNE projection matrix of $\mathbf{H}$ (denoted as $\bm{\Upsilon}$), where each row is a 2-dimensional vector. These three responses are representative of the real-world API output. For example, t-SNE and node embeddings are widely returned in the scenarios of graph visualization , transfer learning , federated learning , fine-tuning pretrained GNNs , and model partitioning where the target model is split into local and cloud parts bridged by embeddings information . Moreover, these responses characterize three different levels of knowledge the adversary may gain access to in practice. For instance, the t-SNE projection matrix $\bm{\Upsilon}$ is usually used for data visualization. It contains the least information that an adversary can harvest. We later show that the adversary can still launch the attack through such data visualization function of a remote model.

Second, we assume that the query graph $\mathbf{G}_{Q}$ (both node features $\mathbf{X}_{Q}$ and graph structure $\mathbf{A}_{Q}$) is from the same distribution of the training graph $\mathbf{G}_{O}$ used to train the target model $\mathcal{M}_{T}$. Here we consider the term “same distribution” as $\mathbf{G}_{O}$ and $\mathbf{G}_{Q}$ are drowned randomly from the same dataset. We do not necessarily require $\mathbf{G}_{O}$ and $\mathbf{G}_{Q}$ to have the same graph characteristics. Besides, the nodes in the query graph $\mathbf{G}_{Q}$ does not need to be in the training graph $\mathbf{G}_{O}$. For example, both $\mathbf{G}_{O}$ and $\mathbf{G}_{Q}$ can be subgraphs sampled from social networks like Twitter but there is no overlap with each other. This assumption is in line with recent attacks to neural networks where an adversary uses part of a public dataset to exploit the target model and graphs are available in some domains (e.g., social networks and molecular graphs). We further relax this assumption and demonstrate that the adversary can still launch effective attacks even without the graph structural information of $\mathbf{G}_{Q}$. This weak assumption of adversary knowledge makes our attack more practical in real-world scenarios. For instance, the adversary may compromise a user database and acquire their profile information. However, the relationships among the users, i.e., the graph structure, may not be revealed. We later demonstrate that the adversary can still launch high-performance model stealing attacks in this scenario (see Section 5).

Notes. Our threat model is different from the causative and evasion adversarial attacks to GNNs . Those attacks allow the attackers to manipulate the training graphs in order to change the parameters of the target model, or modify the node features and/or graph structure to fool the inductive GNN models. Our attack is an instance of exploratory attacks. We do not tamper with the original training process. The goal is thereof not to change the parameters or fool the target GNN model. Instead, it is designed to steal a copy of a target inductive GNN model.

Our attack also differs from the existing model stealing attacks against transductive GNNs in several key aspects. First, these attacks focus on attacking transductive GNN models. Both methods assume that the query graph is part of the graph used for training the target GNN model and must be involved in the training process, hence unrealistic. Our attacks instead focus on a more realistic stealing attack scenario whereas the adversary only queries the target model via remotely accessible API. We do not tamper with the training process of the target model. In turn, our threat model is practical and fills the gap to understand if both theft and reconnaissance adversaries can steal inductive GNNs with high accuracy and high fidelity. Moreover, the existing methods are limited to the GCN model (i.e., model dependent) and rely on node predicted posterior probability scores to launch attacks. In contrast, our attack is model agnostic and can still successfully copy the target model’s behavior with marginal information. Besides, our attack setting is different from Attack-3 proposed by Wu et al. . Specifically, Attack-3 in trains the surrogate model without querying the target model, while our attacks do interact with the target model. Note that it is impractical to attain the reconnaissance goal without interacting with the target model . Also, the target model considered by Wu et al. is a transductive GNN while ours focuses on the inductive GNN (the difference between transductive and inductive GNN is described in Section 2).

Model Stealing Attack

In this section, we first outline 6 attack scenarios that can be launched by the adversary given different levels of knowledge. Then we propose our attack framework.

As outlined in Section 3, the adversary has two main pieces of information at their disposal to launch the model stealing attack: the query graph $\mathbf{G}_{Q}=(\mathbf{A}_{Q},\mathbf{X}_{Q},\mathbf{C}_{Q})$ and its corresponding query response $\mathbf{R}$ (i.e., node embedding matrix $\mathbf{H}$, predicted posterior probability matrix $\bm{\Theta}$, or t-SNE projection matrix $\bm{\Upsilon}$). Recall that the adversary may not have graph structural information of the query graph $\mathbf{G}_{Q}$ (i.e., the adjacency matrix $\mathbf{A}_{Q}$ may be missing). We thus have 6 possible attack scenarios falling into two types (see Table 2).

Type I Attack. When the adversary obtains the query graphs which are of the same distribution as the training graphs, they can launch a Type I attack against the target model. The application scenarios of Type I attack include side-effects prediction due to drug interactions , financial fraud detection , recommendation systems , etc. In these cases, the adversary can obtain the query graphs (e.g., drug-protein interaction graphs, transaction graphs, user-item purchase graphs) that are of the same distribution of the training graphs used to train the target models due to the wide availability of such graphs.

Type II Attack. When the graph structural information is missing, the adversary can resort to a Type II attack to steal the target model. For instance, social networks such as Instagram or Tinder do not reveal the social relationship of (private) user accounts. However, an adversary can crawl users’ profile information without social relations (i.e., graph structural information). Then they leverage Type II attacks to rebuild a relationship graph and steal the target models offered by these service providers. Besides, Type II attacks can also be used in stealthy insider threat scenarios. For instance, a company may enforce data segregation due to data privacy and security concerns (e.g., one department has user information and another has the relationship graph). To train a joint model, the company needs to perform vertical federated learning . The insider attackers may leverage the information they can access (e.g., user information) to steal the joint model using our Type II attack.

It is straightforward to observe in Table 2 that the attacks are increasingly harder in the row order. For instance, the Type II.3 attack is the most challenging scenario from the adversary’s perspective since they only have a set of node features to start with. As such, they first need to restore the relationships among the nodes and build the query graph $\mathbf{G}_{Q}$, then leverage the t-SNE projection matrix obtained from $\mathcal{M}_{T}$ to steal the target model. Besides, model stealing attacks against the classical ML models focus on the scenario that the remote models return the predicted posterior probability. We note that the node embedding and the t-SNE projection-based query responses are the new attack surface of the inductive GNN models and outline the technical details in Section 4.2.

2 Attack Framework

To tackle the aforementioned challenges, we propose a unified attack framework as illustrated in Figure 1 to launch both Type I and II attacks. It has two components. The first component (❶ in Figure 1) learns the missing graph structure $\mathbf{A}_{Q}$. The output of the first component is a learned query graph $\mathbf{G}_{Q}$. This component is designed to facilitate Type II attacks, hence not required for Type I attacks. It is important to note that this reconstruction process is done locally at the attacker’s side. They do not interact with the target model. The second component (❷ in Figure 1) learns a surrogate model from the response of the target model given the query graph $\mathbf{G}_{Q}$. The output of the second component is a learned surrogate model $\mathcal{M}_{S}$. We outline their details below.

Recall that the adversary does not have the adjacency matrix $\mathbf{A}_{Q}$ for the query graph $\mathbf{G}_{Q}$ at their disposal in Type II attacks (see Section 3). However, the graph structure is necessary for the adversary to conduct the attack. As such, without a graph structure to piece $\mathbf{X}_{Q}$ together, the adversary must first build the adjacency matrix $\mathbf{A}_{Q}$ from query graph $\mathbf{G}_{Q}$ and then query the target model $\mathcal{M}_{T}$. The goal of this component is thereof to learn a high-quality discrete graph structure that enables the adversary to query and gain useful knowledge from the query response returned by the target model.

One common approach is to create a $k$-nearest neighbor ($k$NN) graph from node features $\mathbf{X}_{Q}$. However, the efficacy of the resulting $k$NN graph and consequent GNN model rests on the choice of $k$ (e.g., $k$NN graphs often produce nodes with extremely high degrees) and the chosen similarity measure over the node features. To this end, we leverage the IDGL framework proposed by Chen et al. to learn a query graph $\mathbf{G}_{Q}$ by minimizing a joint loss function combining both the task-dependent prediction loss and the graph regularization loss. Note that task-dependent prediction loss is flexible, and can be tailored to use different loss functions (e.g., node classification loss or link prediction loss). The graph regularization loss controls the smoothness, connectivity, and sparsity of the resulting graph. Additional details can be found in .

To launch Type II attacks, the adversary first initiates a $k$NN graph constructed based on multi-head weighted cosine similarity, then utilizes the IDGL framework to search for a hidden graph structure that augments the initial $k$NN graph structure using the aforementioned joint loss function. Note that the $k$NN graph is only used to seed the initial graph structure. The final learned graph structure is optimized during the learning process and may not contain the edges from the original $k$NN graph. This component’s output is a learned query graph $\mathbf{G}_{Q}$.

2.2 Learn Surrogate Model (❷)

The goal of the second component is to learn a surrogate GNN model from the query response returned by the target model given a query graph $\mathbf{G}_{Q}$. To this end, we first discuss our observation of three state-of-the-art inductive GNN models, namely GraphSAGE, GIN, and GAT. We then propose a unified framework to learn surrogate models given both Type I and Type II attacks.

Observation. According to the convolution operations defined on graphs, graph neural networks can be categorized as spectral approaches and spatial approaches . For spectral approaches, the graph is represented with a Laplacian matrix according to the spectral theories with the convolution operation defined in the sequence domain via Fourier transform. Different from spectral approaches, spatial approaches define graph convolutions by collective information propagation (i.e., propagating node information along edges) and perform convolution by considering node neighborhoods. Leveraging the insights from , in spatial-based GNN methods, at $l$-th layer, node embedding $\mathbf{h}_{v}^{l}$ is iteratively updated using Equation 5 where $\Phi$ and $\Psi$ are weight functions, and $\eta$ is a normalization factor.

Similarly, GIN (Equation 4 in Section 2) can be written as Equation 8, where $\Phi=1+\epsilon$, $\Psi=1$, $\mathbf{A}$ is unnormalized adjacency matrix.

In the same way, GAT (Equation 3 in Section 2) can be written as Equation 9, where $\Psi=\bm{\Xi}$ is a learnable neighbor weight matrix and $\Phi=0$.

Given Equation 7, Equation 8, and Equation 9, we can see that there exists an intrinsic connection among GraphSAGE, GAT, and GIN (i.e., they are special cases of Equation 6). It is evidential that these models are spatial-based GNN models through adopting different designs for feature aggregation. This observation enables us to design a unified attack framework to leverage the query graph $\mathbf{G}_{Q}$ as the data and the query response $\mathbf{R}$ as the supervised information to build the surrogate model in a unified manner.

Learning Framework. Our attack framework is illustrated in Figure 1. The surrogate model $\mathcal{M}_{s}$ in ❷ consists of two modules. The first module is a customized inductive GNN model (denoted as $\mathcal{F}$) taking all nodes’ $l$-hop subgraphs from $\mathbf{G}_{Q}$ as the training data and the query response $\mathbf{R}$ as the supervised information. Since the responses from $\mathcal{M}_{T}$ are vectors in Euclidean space, they reflect the spatial connectivity among nodes either from the graph connectivity perspective (t-SNE projection or node embedding), or from the node label perspective (the predicted posterior probability). That is, the nodes that are close or connected in the query graph $\mathbf{G}_{Q}$ should be close in $\bm{\Upsilon}$ or $\mathbf{H}$, and the nodes that are of the same label should be close in $\bm{\Theta}$.

Following the above observation, the key idea of our attack is that we ignore the response types and uniformly treat all three possible responses as embedding vectors. As such, for the first module, the goal is to minimize the RMSE loss ($\mathcal{L}_{R}$) between $\hat{\mathbf{H}}_{Q}$ and $\mathbf{R}$ as shown in Equation 10.

where $n_{Q}$ denotes the number of nodes of the query graph $G_{Q}$. Here $\hat{\mathbf{H}}$ keeps the same dimension as $\mathbf{R}$. The rationale of using the RMSE loss is that $\mathcal{F}$ is optimized to maintain the similar spatial connectivity among the nodes in $\hat{\mathbf{H}}_{Q}$ as suggested by $\mathbf{R}$. Note that the output from the first component cannot be directly used for node classification tasks. In light of this, we employ an MLP as the classifier (denoted as $\mathcal{O}$). It takes the output from the first module (i.e., $\hat{\mathbf{H}}_{Q}$) as the input and $\mathbf{C}_{Q}$ as the supervision information to minimize the prediction error ($\mathcal{L}_{P}$) as shown in Equation 11.

We optimize the first module with $\mathcal{L}_{R}$, then we freeze it and optimize the second module with $\mathcal{L}_{P}$. The two modules are then chained together as our surrogate model. Followed by Orekondy et al. , we assume that $\mathbf{C}_{Q}$ is obtained by the adversary.

Our attack design enjoys two-fold flexibility. First, our attack allows the adversary to conduct the attack without knowing the target model’s architecture. This makes the threat model closer to the real-world scenario where the adversary only has query access to the target model. Second, our attack framework shows that node embedding and 2-dimensional t-SNE projection as the query response can be the new attack surface of the inductive GNN models. Notably, t-SNE projection is for data visualization purposes. It barely unveils the graph structural information to the adversary. However, we demonstrate that our attack can still successfully copy the target model’s behavior from such marginal information.

Evaluation

In this section, we perform an in-depth analysis of the proposed model stealing attack against inductive GNNs. We first introduce the experimental setup, and present the evaluation results for Type I and Type II attacks from both theft and reconnaissance adversary’s perspectives. We then explore how various query budgets may affect the attack performance. Finally, we study how different hyperparameters of the surrogate model may influence the attack performance.

Datasets. We use 6 public datasets to evaluate the performance of our attack, including DBLP , Pubmed , Citeseer Full (abbreviated as Citeseer) , Coauthor Physics (abbreviated as Coauthor) , ACM , and Amazon Co-purchase Network for Photos (abbreviated as Amazon) . These datasets are widely employed as benchmark datasets to evaluate the performance of GNNs . Among them, DBLP, Pubmed, and Citeseer are citation networks with nodes representing publications and edges indicating citations among these publications. Coauthor is a user interaction network where nodes represent the users and edges represent interactions between them. ACM and Amazon are cooperative networks where nodes represent the papers/items and there is an edge between two nodes if they have the same author or purchased together. We use these datasets to verify the efficacy of our attacks given different graph characteristics (e.g., graph size, node feature size, number of classes, etc.). Statistics of these datasets are summarized in Table 3.

Dataset Configuration. For each dataset, we split them into three parts. The first part consists of 20% randomly sampled nodes that are used to train the target model $\mathcal{M}_{T}$. The second part consists of 30% randomly sampled nodes, forming our query graph $\mathbf{G}_{Q}$. We further show that our attacks are still effective with fewer nodes to form the query graph (see Figure 6). The third part consists of the rest 50% of the nodes, functioning as the testing data for both $\mathcal{M}_{T}$ and $\mathcal{M}_{S}$. This setting matches the inductive learning on evolving graphs as laid out in .

Target Model ($\mathcal{M}_{T}$). We use GIN, GAT, and GraphSAGE as our target models’ architectures in our evaluation. For reproducibility purposes, we outline the details below.

GIN. We use a 3-layer GIN model with a fixed neighborhood sample size of 10 at each layer. For the first hidden layer, we set the hidden unit size to 256. For the second layer, we set the hidden unit size to the embedding size (i.e., 64, 128, or 256 in our experimental setting). The final layer is used for classification.

GAT. We use a 3-layer GAT model with a fixed neighborhood sample size of 10 at each layer. The first layer consist of 4 attention heads and the hidden unit size is 256. The second layer consists of 4 attention heads and the hidden unit size is the embedding size. The final layer is used for classification following the original design .

GraphSAGE. Following Hamilton et al. , we use a 2-layer GraphSAGE with neighborhood sample sizes of 25 and 10 respectively. For the first hidden layer, we set the hidden unit size to 256. For the second layer, we set the hidden unit size to the embedding size. Each layer employs a GCN aggregator and uses 0.5 dropout rate to prevent overfitting. Finally, we use a linear transformation layer for classification.

All models use cross-entropy as the loss function, ReLU as the activation function between layers, and Adam as the optimizer with an initial learning rate of 0.001. We train all models for 200 epochs and select the best models with the highest validation accuracy. All models above follow the design specifications outlined in the respective papers.

Query Response ($\mathbf{R}$). For the node embedding $\mathbf{H}$, we fix the sizes to three commonly used values, i.e., 64, 128, and 256. For the node predicted posterior probability $\bm{\Theta}$, the dimension sizes (i.e., the number of classes) are dataset dependent and outlined in Table 3. For the node projection $\bm{\Upsilon}$, we use t-SNE to project the node embedding $\mathbf{H}$ into 2-dimensional vectors, which cover most of the data visualization cases.

Surrogate Model ($\mathcal{M}_{S}$). Recall our attack design in Section 4.1, the surrogate model consists of two components to provide extensibility. The first component is a customized GNN model taking the subgraph extracted from $\mathbf{G}_{Q}$ as the input and using the RMSE as its loss function. The second component is a 2-layer MLP with the hidden unit size of 100. It takes the output from the first component as the input and uses the cross-entropy as its loss function. Both components use Adam optimizer with a learning rate of 0.001. We train the first and the second components for 200 epochs and 300 epochs, respectively. For evaluation purposes, we use customized GraphSAGE, GAT, and GIN models as the first component in our surrogate model. The details are outlined below.

GIN. We use a 2-layer GIN model with neighborhood sample sizes of 10 and 50 respectively. The hidden unit size is 256 for the first layer. For the second layer, we set the hidden unit size to the size of the query response.

GAT. We use a 2-layer GAT model with neighborhood sample sizes of 10 and 50 respectively. Both the first and the second layers consist of 4 attention heads and we follow the same hidden unit size as GIN we mentioned above.

GraphSAGE. We use a 2-layer GraphSAGE with neighborhood sample sizes of 10 and 50 respectively and follow the same hidden unit size as GIN we mentioned above.

Query Graph Reconstruction Configuration. We use the IDGL framework proposed by Chen et al. to learn the missing graph structural information, i.e., $\mathbf{A}_{Q}$, of the query graph $\mathbf{G}_{Q}$ for Type II attacks. For simplicity, we use node classification loss and leave experimenting with other loss functions as future work. The initial $k$ of $k$NN graph is set to 24. We use a 2-layer (hidden unit size is set to 256) GraphSAGE with GCN aggregator to learn the node embedding. Weighted cosine similarity matrices with 8 attention heads are employed to decide if there exists an edge between two node embeddings during the graph learning process. The graph structure and the GraphSAGE parameters are jointly and iteratively learned by minimizing a hybrid loss function combining both the node classification loss and the graph regularization loss. We set the cutoff value $\epsilon$ to 0.99 to identify the final edges in the learned adjacency matrix $\mathbf{A_{Q}}$ of the query graph $\mathbf{G}_{Q}$.

Metrics. Following the taxonomy defined by Jagielski et al. , we use two metrics, i.e., accuracy and fidelity, to evaluate the performance of our attack. We use accuracy (i.e., the number of correct predictions made divided by the total number of predictions made) as our evaluation metric of theft adversary. Accuracy has been dominantly used in evaluating node classification performance of GNNs . Recall that the goal of reconnaissance adversary is to closely match the behavior of the target model (see Section 3), we use fidelity (i.e., the number of predictions agreed by both $\mathcal{M}_{S}$ and $\mathcal{M}_{T}$) as the second evaluation metric of our attack . Both metrics are normalized between 0 and 1. Higher scores imply better performance.

Runtime Configuration. Note that we have 27 different combinations for each response in each dataset (i.e., combinations of three target models, three responses, and three surrogate models). All the experiments in this paper are repeated 5 times. For each run, we follow the same data configuration and report the mean as well as the standard deviation of the aforementioned two metrics to evaluate the attack performance.

2 Performance Evaluation: Type I Attacks

We first summarize the accuracy of the target models for the original node classification tasks in Table 4. We can observe that all GNN models achieve good performance on all datasets, which demonstrates that jointly considering node features and graph structure are effective for classification. We then show the accuracy and fidelity of Type I attacks in Table 5. Due to space limitations, we only show the attack results when the adversary uses GraphSAGE as the surrogate model. The performance results using GIN and GAT as the surrogate models follow similar patterns and can be found in Appendix A.

Accuracy. As we can see from Table 5, Type I attacks can build surrogate models close to the target models given the response is predicted posterior probability or node embeddings (i.e., Type I.1 and Type I.2 attacks respectively). Take the Pubmed dataset as an example, the target models (i.e., GIN, GAT, and GraphSAGE) respectively achieve 0.924, 0.905, and 0.909 accuracy scores (see Table 4), while the surrogate models can consistently achieve at least 0.869 accuracy score (see Table 5). This represents an approximately 0.04 accuracy score drop in all cases compared to the target models. We can also observe that Type I attacks can build surrogate models that offer usable accuracy even the response is a 2-dimensional t-SNE projection matrix (i.e., Type I.3 attack). For instance, on the Pubmed dataset, the surrogate models achieve 0.823, 0.743, and 0.844 accuracy scores respectively. This represents a 0.162 accuracy score drop in the worst case when the target model is GAT. For the rest of the five datasets, we also observe a subtle performance drop in Type I.3 attacks. Also, such performance drop, compared to Type I.1 and Type I.2 attacks, is expected since each t-SNE projection is only a 2-dimensional vector, which leads to additional information loss. Overall, our results show that all three Type I attacks can build usable surrogate models.

Given our experimental configuration (i.e., three target models and three surrogate models), there are 9 different combinations for each response type given a single dataset. Such configuration enables us to understand if the adversary can reliably steal target models in different circumstances. Using the Pubmed dataset as an example, we summarize the attack accuracy performance in Figure 2. We can see that in general, the adversary can build accurate surrogate models given different combinations of GNN architectures for each response. The subtle performance drop only occurs when GAT is the target model and the response is a t-SNE projection, i.e., Type I.3 attack. Even in this case, the adversary can steal usable surrogate models with no more than a 0.162 accuracy score drop, which indicates our attack remains effective. Our results demonstrate that the adversary does not require knowledge about the architecture of target models to conduct the attacks.

To better illustrate it, take the ACM dataset as an example, we extract the embeddings of a given set of nodes from both target and surrogate models and project them into a 2-dimensional space using t-SNE. The result is shown in Figure 3. We use the triangle (cross) to denote the embeddings extracted from the target (surrogate) model and different colors to denote different classes. We find that for both target and surrogate models, the different classes’ embeddings can be separated easily. It means that the surrogate model can also successfully map nodes from different classes into different space, which lead to high accuracy.

Fidelity. The reconnaissance adversary’s motivation is faithfully copying the behavior of the target model. We also summarize the fidelity performance of the surrogate models in Table 5. It is straightforward to see that the better accuracy performance a surrogate model can reach, the better fidelity it can achieve. For instance, on the Coauthor dataset, the surrogate models reach at least 0.816 accuracy score while these models achieve at least 0.817 fidelity score to the target models in all 9 cases. The surrogate models for other datasets also follow similar patterns. We then calculate the Pearson correlation coefficient between accuracy and fidelity of the surrogate models given three target models. The coefficient scores are 0.957, 0.988, and 0.959 respectively. The results exemplify that the fidelity of the surrogate models to the target models is highly correlated with their accuracy performance. From Figure 3, we can also observe that for each class, the embeddings extracted from target and surrogate models lie in the same region, which implies that the surrogate model has the ability to generate the node embeddings that is close to the one generated by the target model, which leads to high fidelity. Besides, we compare our attacks with Attack-3 proposed by Wu et al. using the overlapping Pubmed dataset and the results are shown in Table 6. Note that we use fewer data to train the surrogate model compared to Wu et al. . Given SAGE as $\mathcal{M}_{S}$ and GIN, GAT, and SAGE as $\mathcal{M}_{T}$, our attack achieves 0.903, 0.898, and 0.924 fidelity scores respectively while their attack only achieves 0.818 fidelity score. The results exemplify that interacting with the target model can better facilitate the adversary to attain the reconnaissance goal.

Stability. We run each combination 5 times with different graph partition seeds. It enables us to measure how widely accuracy/fidelity values are dispersed from the average value (i.e., standard deviation). A low standard deviation indicates a low volatility. As we can observe in Table 5, the standard deviation values are low in all cases. It shows that the adversary can steal from the target models with statistically stable accuracy and fidelity.

Observation. To achieve high fidelity, the adversary wants to make sure the mistakes and correct labels are the same between the surrogate and target models. When the target model achieves high accuracy, it can make the correct predictions for most of the test data while making few mistakes. If the surrogate model gets close to the accuracy performance of the target model, it would gain high fidelity to the target model due to the fact the number of correctly predicted instances by the surrogate model would considerably overlap those made by the target model. The above Pearson correlation results verify this intuitive explanation.

Takeaways. The theft adversary can reliably build accurate surrogate models close to the target models via Type I attacks. It is worth noting that, in the real world, it is hard for the reconnaissance adversary to comprehensively verify the fidelity of the surrogate models without risking a number of queries. Our results imply that the reconnaissance adversary may focus on building surrogate models that preserve high accuracy similar to the target models. In turn, these surrogate models would likely be faithful to the remote targets. Besides, our results demonstrate that the adversary does not require knowledge of the target models’ architectures.

3 Performance Evaluation: Type II Attacks

To launch Type II attacks, the adversary first builds an adjacency matrix $\mathbf{A}_{Q}$ for query graph $\mathbf{G}_{Q}$ before querying the target model $\mathcal{M}_{T}$. We follow the aforementioned graph reconstruction configuration to restore $\mathbf{A}_{Q}$ for query graph $\mathbf{G}_{Q}$ and conduct the model stealing attacks. The performance of Type II attacks is summarized in Table 7. Due to space limitations, we only show the attack results when the adversary uses GraphSAGE as the surrogate model. The performance results using GIN and GAT as the surrogate models follow similar patterns and can be found in Appendix B.

Accuracy. As we can see in Table 7, given all the datasets, the adversary can launch Type II.1/2 attacks to build surrogate models that offer accuracy on par with the target models. At the same time, we observe that the adversary can launch Type II.3 attacks and steal usable surrogate models. Take the Pubmed dataset as an example, when the response is the t-SNE projection of query nodes, the surrogate models achieve the average accuracy score of 0.836, 0.739, and 0.850 concerning different target models. This represents 0.166 accuracy drop in the worst case (compared to the target model performance). In general, Type II.3 attacks can achieve comparable accuracy performance in all 6 datasets. Also, we investigate if the adversary can build accurate surrogate models given different combinations of GNN architectures for each response in Type II attacks. As we can observe from Figure 5, when the response is predicted posterior probability or embedding, the accuracy of the surrogate model is close to the target model. Regarding the case when the response is t-SNE projection, our attack still works well. We only witness a slight performance drop when the target model is GAT. The results exemplify that, in general, Type II attacks remain effective in stealing target models with different architectures.

Fidelity. We also observe the same correlation between accuracy and fidelity in Type II attacks. That is, the better accuracy performance a surrogate model can reach, the better fidelity it can achieve. Take the Coauthor dataset as an example, the surrogate models reach at least 0.793 accuracy score while these models achieve at least 0.801 fidelity score to the target models in all cases. We also calculate the Pearson correlation coefficient between accuracy and fidelity of the surrogate models given three target models. The coefficient scores are 0.970, 0.991, and 0.967 respectively. These correlation scores are similar to what we observe from Type I attack results.

Efficacy of Learned Query Graph Structure. To investigate the effect of graph structure, we compare IDGL with two additional methods, i.e., random graph construction and $k$NN. For the two additional methods, we set the average node degree to 24, which is the same value used to initialize IDGL. Due to space limitations, we only show the accuracy of Type II attacks on the Citeseer dataset using GAT as the target model and GraphSAGE as the surrogate model. The accuracy and fidelity of other datasets follow similar patterns. As we can see from Figure 4, using IDGL to reconstruct the graph structure reaches the highest attack accuracy for all responses. For instance, the accuracy score is 0.879 when using IDGL as the reconstruction method and taking embedding as the response, while the corresponding accuracy score is only 0.411 and 0.737 respectively when using random graph construction or $k$NN as the reconstruction method. It demonstrates that an effective graph reconstruction method does benefit the final attack performance.

Stability. As we can observe in Table 7, the standard deviation values remain low in all cases. This shows that the adversary can steal the target models with statistically stable accuracy and fidelity in Type II attacks.

Observation. When comparing Table 7 to Table 5, we observe that Type II attack achieves better performance than Type I attack in certain cases. Chen et al. observed a similar phenomenon and concluded that the raw graphs are not always optimal for the downstream tasks for different reasons. For example, raw graphs may contain noisy/incomplete information due to the error-prone data collection or their structures do not reflect the ideal graph topology after feature extraction and transformation. The query graphs learned by the IDGL framework in our Type II attacks are optimized toward the downstream tasks (e.g., node classification) and may achieve better performance in some cases. We refer the audience to Chen et al. for additional details.

Takeaways. Our results show that the attack framework enables the adversary to learn a discrete graph structure and steal usable surrogate models. Coupling with the results shown in Section 5.2, we demonstrate that our model stealing attacks achieve strong performance concerning different responses.

4 Performance Evaluation: Query Budget

We then investigate the attack performance with respect to different query budgets, i.e., different sizes of query graph $\mathbf{G}_{Q}$. Due to space limitations, we only show the results for the Pubmed dataset, other datasets follow similar trends. The corresponding accuracy and fidelity of Type I and II attacks are summarized in Figure 6. We observe that in general, larger query budgets lead to better accuracy and fidelity. For instance, in Type I attack, when the response is GraphSAGE’s embedding, the accuracy score increases from 0.839 to 0.870 when the query budget increases from 3% to 27%. However, in most of the cases, we can achieve similar performance even using only 3% of randomly sampled nodes of the original dataset (10% of the $\mathbf{G}_{Q}$ used in previous experiments). The results demonstrate that the adversary can still launch effective attacks even with a low-quality query graph (less query budget and no graph structure information). Besides, when the query budget is small, the adversary can investigate the characteristics of query data (e.g., sparsity of the learned graph) and decide if edge reconstruction is necessary to launch attacks.

Discussions

Limitation. Our model stealing attack is limited to the scope that the target model returns node-level results. We did not explore the scenario when the target model accepts an arbitrary graph as input and returns graph-level results. That is, the target models represent the whole structure of graphs using various pooling methods and return a single embedding vector for downstream graph-level tasks such as graph classification . Model stealing attack under this setting would require a different approach. Besides, we do not jointly optimize both graph structure and surrogate model. Such training paradigm may cost more training epochs to converge, and the query budget would also increase, which inevitably increases the risk of being detected. We put them into our future work.

Defense. Several countermeasures to model stealing attacks have been discussed in previous literature . One straightforward countermeasure is injecting perturbations to the predicted posterior probability reported by the classifier, i.e., perturb the probability while retaining the top-1 label . To cope with different types of query responses, we consider adding random noise to the response regardless of the corresponding label, i.e. the distribution of the random noise is independent of the node class label. Concretely, we add random Gaussian noise into node embedding and t-SNE projection returned by all three target models and use GraphSAGE as the surrogate model to understand the effectiveness of such countermeasure. According to the defined threat model, the accuracy of the surrogate model measures the attack performances. A higher accuracy of the surrogate model indicates a more successful attack. We use the ACM dataset as an example and the results are shown in Figure 7. We observe that the random Gaussian noise slightly affects the accuracy performance of the surrogate model. For instance, in 7(a), when $\sigma$ (i.e., the standard deviation of the added noise) is greater than 7, we can observe the accuracy performance of the surrogate model starts to decrease for different target models. In contrast, if the query response from the target model is the node embedding vector, we can only observe much fewer fluctuations of the surrogate model’s accuracy with increasingly stronger random Gaussian noise injected to the embedding (see 7(b)). To summarize, our preliminary experiment does not establish concrete evidence that adding random noise would counter our attacks. We leave designing effective defense mechanisms for model stealing attacks against GNNs as our future work.

Related Work

In this section, we review the research work close to our proposed attacks. We refer the readers to for an in-depth overview of different GNN models, and for comprehensive surveys of existing adversarial attacks and defense strategies on GNNs.

Model Stealing Attack Against ML Models. Model extraction is in many ways similar to model distillation, but it differs in that the victim’s proprietary training set is not accessible to the adversary. In this regard, previous literature already investigated stealing various aspects of a black-box ML model such as hyperparameters , architecture , information on training data , parameters , decision boundaries , and functionality . However, most of these efforts focused on images. There exist some preliminary work on model stealing attacks against GNNs . However, they are only focusing on the transductive setting of GNNs, which cannot generalize to unseen data. Our attacks instead focus on a more popular and general setting of GNNs, i.e., inductive setting. We fill the gap and understand if both theft and reconnaissance adversaries can steal inductive GNNs with high accuracy and high fidelity.

Causative Attacks on GNNs. Many adversarial attacks to GNNs are causative attacks . These attacks assume that an adversary can manipulate the training dataset in order to change the parameters of the target model and influence their behavior. In this context, Zügner et al. was the first to introduce unnoticeable adversarial perturbations of the node’s features and the graph structure. Their goal was to reduce the accuracy of node classification via GCN. After this work, different adversarial attack strategies have been proposed. Depending on the attack objectives, they aim at reducing the accuracy of node classification (node level), link prediction (edge level), graph classification (graph level), etc. Our attack does not tamper with the training graph data and does not change the behavior of the target model or its parameters.

Exploratory Attacks on GNNs. In reality, however, it is more practical for the attacker to query the target model and leverage the model’s responses on these carefully crafted input data. Consequently, we witness the emerging of exploratory attacks on ML models. However, adversarial exploratory attacks on GNNs remain understudied. In particular, only a few studies focused on exploratory attacks on GNNs. For instance, He et al. proposed the first link stealing attack to infer if there exists an edge between a given pair of nodes in the training graph. Duddu et al. and He et al. discussed the membership inference attack that infers whether a given node in the graph was used to train the target model by leveraging different background knowledge. Note that, in membership inference attacks, the attack model in its training phase does not interact with the target model. However, in model stealing attacks, the surrogate model does have interaction with the target model since it needs the guidance of the target model to optimize its parameters.

Defense of Attacks on GNNs. To mitigate those attacks, several defense strategies have been proposed. The core idea of the existing defense strategies is reducing the sensitivity of GNNs using adversarial training , perturbation detection , graph sanitization , etc. In turn, the trained GNNs are robust to perturbation (e.g., structure perturbation , attribution perturbation ). Also, robustness certification becomes an emerging research direction. They aim at reasoning the safety posture of GNNs under adversarial perturbations. However, these defense techniques only protect GNNs from causative attacks instead of exploratory attacks.

Conclusion

In this paper, we perform the first security risk assessment against inductive GNNs through the lens of model stealing attacks. We propose a threat model to systematically categorize an adversary’s background knowledge into two dimensions, i.e., query graph and model responses. By jointly considering the two dimensions, we summarize six attack scenarios. We then propose a general attack framework that can be applied in different scenarios. Extensive experiments on three popular inductive GNN architectures and six benchmark datasets show that our model stealing attacks can handle different types of responses and achieve strong performance. Moreover, the attacks are still effective even the adversary has no knowledge about the graph structural information.

Acknowledgement

We thank the anonymous shepherd and reviewers for their feedback in improving this paper. This work is partially funded by the Helmholtz Association within the project “Trustworthy Federated Data Analytics” (TFDA) (funding number ZT-I-OO1 4). This work is also supported by the Helmholtz Association’s Initiative and Networking Fund on the HAICORE@FZJ partition.

References

Appendix

Appendix A Performance Evaluation: Type I Attacks (GIN/GAT)

The performance of Type I attacks using GIN and GAT as surrogate models are summarized in Table 8 and Table 9.

Appendix B Performance Evaluation: Type II Attacks (GIN/GAT)

The performance of Type II attacks using GIN and GAT as surrogate models are summarized in Table 10 and Table 11.

Appendix C Fidelity

The fidelity score of Type II attacks on the Citeseer dataset using different graph reconstruction methods is shown in Figure 8.

Appendix D Hyperparameter Study

GNNs are complex and their performance may be affected by the hyperparameter settings. This is especially important for our model stealing attack since we use inductive GNNs as the part of our surrogate models. At the same time, our attack is in a full black-box setting. As such, we evaluate the impact of three hyperparameters on the performance of surrogate models, i.e., hidden unit size, number of epochs, and batch size. For each hyperparameter, we only show the results on one dataset under Type I attacks given the space limitation. Other datasets follow a similar trend.

Hidden Unit Size. In general, the larger the hidden units, the greater space of representation functions a graph convolutional layer can offer. As such, we investigate how the hidden unit size may affect the attack performance. Recall that the adversary does not know the target model’s hidden unit size. They can only blindly guess the hidden unit size used by the target model. To this end, we first build a target model (i.e., GAT) with 64, 128, and 256 hidden units using Citeseer dataset. We then launch Type I attacks using a fixed surrogate model (i.e., GraphSAGE) with 64, 128, and 256 hidden units. In this way, we can observe the potential performance changes in different circumstances. The results are shown in 9(a) and 9(b), respectively. We can see that different hidden unit sizes adopted by the surrogate models have a limited impact on the accuracy performance. Our hypothesis is that the inductive GNN models employed as surrogate models are powerful enough to extract information from the responses.

Number of Epochs and Batch Size. The number of epochs and batch size are the other two hyperparameters that may affect attack performance but can be controlled by the adversary. Respectively they control the number of complete passes through the training dataset and the number of samples processed before the GNN model is updated. Batch size may affect the speed and stability of the learning process, while the number of epochs may lead to overfitting. To this end, we fix the surrogate model to GraphSAGE and the target models to GIN and GAT to understand the impact of both hyperparameters. We use 150, 200, 250, 300 for the number of epochs, and 400, 600, 800, 1000 for bath sizes. The results are summarized in Figure 10 and Figure 11, respectively. Regarding the number of epochs, the accuracy is relatively stable with respect to different numbers of epochs (see Figure 10). Meanwhile, as we can observe in Figure 11, a larger batch size may have some negative impact on the attack accuracy.

Takeaways. Inductive GNN models employed as surrogate models are powerful enough to extract information from the responses. We observe that a large batch size may have a negative impact on the attack accuracy. Other hyperparameters such as hidden unit size and the number of epochs have a limited impact on the accuracy of the surrogate models.