Membership Inference Attacks Against Text-to-image Generation Models

Introduction

With the superb power of unfastening limitless imaginative content creations, text-to-image generation has become one of the most noteworthy topics in the computer vision field and has been advanced significantly in recent years by a series of designs such as sequence-to-sequence based models (e.g., Parti ), and diffusion-based models (e.g., DALL-E 2 and Imagen ). Along with the extremely rapid development of this topic, the demand for data is highly growing. For instance, the Parti/DALL-E 2/Imagen models are trained on 6.6B/650M/860M image-text pairs, respectively. A stark reality is that such large amounts of training data collected for model builders often contain inherently privacy-sensitive information, such as facial identity, and have raised community concerns. Under the terms of the GDPR,https://gdpr-info.eu/ the LAION organization is calling on people to determine whether their private information exists in publicly released LAION datasets and providing support for the removal of their private data.https://laion.ai/gdpr/ Unfortunately, model builders are unlikely to disclose their training data due to the huge effort and resources they have put into collecting them .

Various recent studies have shown that machine learning (ML) models are vulnerable to privacy attacks against their training data, and a major attack in this area is membership inference: an adversary aims to infer whether a data sample is part of the dataset trained by the target ML model. The resulting privacy leakage caused by this attack would raise serious issues as the training data is intellectual property and contains sensitive information. In addition, data owners can also use it to audit whether their data was collected by model builders without authorization under GDPR terms, i.e., having better control over their data.

Existing membership inference attacks have been demonstrated to be a realistic threat to different type of tasks, such as classification and GAN generation . Unfortunately, the peculiarities of text-to-image generation do not allow to trivially extend the understanding of membership leakage from fully explored classification and GAN generation domains to text-to-image domain. Hence, these aforementioned realities motivate us to focus on membership leakage in text-to-image generation models.

Contribution. In this work, we take the first step towards studying membership leakage in text-to-image generation models, where an adversary aims to infer whether a given image is used to train a target text-to-image generation model. In particular, we focus on the most difficult and realistic scenario where no additional information about the target model is available to the adversary other than the output images. Based on the characteristics of the text-to-image generation models, we consider three key intuitions and design four attack methods accordingly. We conduct comprehensive experiments on two representative text-to-image generation modes, i.e., sequence-to-sequence and diffusion-based. Extensive empirical results show that all of our proposed attack methodologies achieve remarkable performance, which convincingly demonstrates that membership leakage is a severe threat to the text-to-image generation models. Furthermore, to delve into which factors and their impact on the attack performance, we conduct a comprehensive ablation study from different perspectives that can guide developers and researchers to be alert to vulnerabilities in text-to-image generation models. Our main contributions are as follows:

We pioneer in studying the privacy risks of text-to-image generation models from the perspective of membership inference.

We consider three intuitions and design four attack methodologies via differentiating attack intuitions.

We conduct an extensive evaluation on two mainstream text-to-image generation models, and the results show the effectiveness and generalizability of the proposed attacks, indicating such membership leakage poses a much more severe threat than existing work. We further perform a comprehensive ablation study from different angles to analyze the factors that may affect the attack performance, which is expected to provide instructional warnings to model inventors.

Background

Text-to-image generation targets to unlock the innovative applications covering various areas of life, including painting, design, and multimedia content creation. It generates creative images combining concepts, attributes, and styles, from expressive text descriptions. Currently, text-to-image generation models can be divided into two different designs, namely diffusion-based modeling and sequence-to-sequence modeling.

Diffusion-based. The diffusion-based models directly leverage noise as the input of the de-noising network, starting from these random points and gradually de-noising them conditioned on textual descriptions until images matching the conditional information are generated. Building on the power of diffusion models in high-fidelity image synthesis, the text-to-image generation is significantly pushed forward by the recent effort of GLIDE , LDM , DALL-E 2 and Imagen .

Sequence-to-sequence. The main idea of this design is to turn images into discrete image tokens via leveraging transformer-based image tokenizers (e.g., dVAE ), and employ the sequence-to-sequence architectures to learn the relationship between textual input and visual output from a large collection of text-image pairs. The representative works of sequence-to-sequence modeling are Parti , DALL-E and CogView .

In this work, we adopt LDM and DALL-E mini as our target models, representing diffusion-based and sequence-to-sequence models, respectively.

Membership Inference Attacks

Membership inference attacks (MIAs), aiming to infer whether a specific data sample was involved in a target model’s training phase (called member or non-member), are considered as an approach to investigating privacy leakage and detecting illegal data abuse. The basic idea is to exploit the behavioral difference of the target model on members and non-members. Depending on the characteristics of the target model, the behavioral differences can be constructed in different ways. For instance, in the classification domain, the behavioral difference exploited in most prior works is the confidence score of members over non-members. More recent works later attack in a more realistic scenario where the adversary has access only to the predicted labels of the target model. Here, the behavioral differences refer to the fact that the perturbations added to members to change the predicted labels are larger than those of non-members. In image generation domain where the models accept random latent code as input and then output images, Chen et al. and Hilprecht et al. propose a customized attack method that estimates the probability of the query sample can be generated by the generator, where the behavioral difference is that the probability of members is greater than non-members .

Unfortunately, these popular and well-explored attack methodologies in classification and image generation domains cannot be trivially extended to the text-to-image generation domain, because the text-to-image generation model accepts text as input and then outputs images, which is totally different from previous works. Hence, it is difficult for existing attack methods to evaluate whether text-to-image generation models are truly vulnerable to membership inference, which prompts the need to investigate new attack methods specifically for the text-to-image generation models.

Problem Statement

In this section, we formulate the text-to-image generation and the threat model.

As aforementioned, we focus on the membership leakage in the text-to-image generation domain. A text-to-image generation model $\mathcal{M}$ can map a text caption $t$ to the corresponding image $\boldsymbol{x}$. To construct a text-to-image generation model $\mathcal{M}$, one needs to collect a huge amount of data pairs ($\boldsymbol{t}$, $\boldsymbol{x}$) to construct the training set $\mathcal{D}$. The model is then optimized via minimizing a predefined loss function.

Threat Model

Adversary’s goal. The goal of the adversary is to infer whether the user’s image $\boldsymbol{x}$ is used to train a target text-to-image generation model $\mathcal{M}_{\textit{target}}$.

Adversary’s knowledge. Typically, the training datasets are composed of a huge number of data pairs, i.e., text caption $\boldsymbol{t}$ and the corresponding image $\boldsymbol{x}$. Here, we assume that the adversary only queries a candidate image $\boldsymbol{x}$ without its corresponding text caption to infer the membership, which is more realistic and broadly applicable. We assume the adversary only has black-box access to the target model $\mathcal{M}_{\textit{target}}$, which is the most difficult and realistic scenario. Besides, we assume that the adversary has a very small subset from the member training data of target model $\mathcal{D^{\textit{member}}_{\textit{sub\_target}}}$, as well as a small set of local non-member data $\mathcal{D}^{\textit{non\_member}}_{\textit{local}}$. The adversary then constructs an auxiliary dataset $\mathcal{D}_{\textit{auxiliary}}=\{x^{m}\cup x^{nm}:x^{m}\in\mathcal{D^{\textit{member}}_{\textit{sub\_target}}},x^{nm}\in\mathcal{D}^{\textit{non\_member}}_{\textit{local}}\}$ that can be used to train the attack model $\mathcal{A}$, i.e., a binary classifier. Note that, the assumption of auxiliary dataset also holds for previous works . The reason of the second assumption is that due to the resource limitation, i.e., insufficient GPU resource, so we are unable to leverage shadow technique , i.e., training shadow models on billions of local image-caption pairs to mimic the target model $\mathcal{M}_{\textit{target}}$.

Methodology

In this section, we start with the design intuition, and then we introduce the attack methodologies.

The key intuition of our work is a general observation about the overfitting nature of ML models. Concretely, given a query data pair (text caption $\boldsymbol{t}$ and the corresponding image $\boldsymbol{x}$), the text-to-image generation models accept text $\boldsymbol{t}$ as input and are optimized to generate image $\boldsymbol{x^{\prime}}$ that are similar or same to the original image $\boldsymbol{x}$. This derives to the following three perspective of key intuitions on membership information:

Intuition I. The quality of generated image $\boldsymbol{x^{\prime}}$ of data pair ($\boldsymbol{t}$ and $\boldsymbol{x}$) from training set should be higher than that from testing set.

Intuition II. The reconstruction error between generated image $\boldsymbol{x^{\prime}}$ and original image $\boldsymbol{x}$ from training set should be smaller than that from testing set.

Intuition III. The generated image $\boldsymbol{x^{\prime}}$ should more faithfully reflects the semantic of a textual caption $\boldsymbol{t}$ from the training set than the testing set.

Therefore, we focus on the distinguishability of membership by observing behavioral differences of these intuitions, i.e., members and non-members behave differently in the three aspects mentioned above.

Attack Methodologies

As the adversary only holds a query image $\boldsymbol{x}$ that they aim to infer, the adversary initializes the attack by leveraging a third parity image captioning tool to generate a caption $\boldsymbol{t}$ for the given query image $\boldsymbol{x}$, as illustrated in Figure 1. Then, they feed the generated caption $\boldsymbol{t}$ into the target text-to-image generation model $\mathcal{M}_{\textit{target}}$ to obtain a generated image $\boldsymbol{x^{\prime}}$. In this way, we connect the query image and generated images explicitly. In addition, we only need to query the target model once for each query image to get one corresponding generated image, largely decreasing the possibility of being detected by defense mechanisms. Building on the attack pipeline, we design 4 different types of attacks via exploiting different intuitions, i.e., Attack-I/II/III based on Intuition-I/II/III, and Attack-IV using all intuitions, as illustrated by Table 1. In the end, the adversary constructs an attack dataset and trains the attack models. The dataset is split by half as the attack training dataset and attack testing dataset. We provide the details of each attack method as follows.

Attack I. This attack is based on Intuition-I that there is a discrepancy between members and non-members in terms of the quality of generated images. For simplicity, an adversary can differentiate between members or non-members by feeding the query image directly into the attack model, rather than measuring quality and then making the distinction. Such an attack method is based on the pixel-level discrepancy, hence is named Attack I-P. Besides the pixel-level discrepancy, we further consider a semantic-level discrepancy of generated images. Concretely, the adversary applies a pre-trained vision-language models (e.g., CLIP ) to extract the embedding of the generated images and then feeds the embedding into the attack model, which is referred to as Attack I-S.

Attack II. This attack is based on the Intuition-II that there is a discrepancy between members and non-members in terms of reconstruction errors. In particular, we again consider both pixel-level and semantic-level reconstruction errors. The former, Attack II-S, directly measures the distance between the generated image and its corresponding query image, while the latter, Attack II-S, also applies a pre-trained visual language model to extract the embeddings of the generated image and its corresponding query image, respectively, and then measures the distance between them. Finally, the adversary feeds the distances into the attack model to distinguish between members and non-members.

Attack III. This attack is based on the Intuition-III that there is a discrepancy between members and non-members in terms of faithful reflection. Here, faithful reflection means that the generated images faithfully reflect the semantics of the text captions. Therefore, we only consider the semantic level attack pipeline, i.e., the adversary first applies a pre-trained visual language model to extract the embedding of the generated images and their corresponding text captions separately, and then measures the distance between them. Finally, the attacker feeds the distances into the attack model to distinguish between members and non-members.

Attack IV. This attack is based on all three intuitions. For Intuition-I and Intuition-II, we only consider semantic-level discrepancies because our experiments on Attack-I and Attack-II demonstrate that semantic-level discrepancies perform much better than pixel-level discrepancies (see Section 5.2). Thus, we take all attack features based on three semantic-level discrepancies of Intuition-I/II/III as input and feed them into the attack model to distinguish between members and non-members.

Evaluation

Target models and training datasets. We demonstrate the efficacy and generalizability of the proposed membership inference attack against two mainstream text-to-image generation models: LDM represented for diffusion-based modeling and DALL-E mini represented for sequence-to-sequence modeling. We download pre-trained LDMhttps://github.com/CompVis/latent-diffusion and DALL-E minihttps://github.com/borisdayma/dalle-mini models from the Web and evaluate our proposed attack methods directly on these real-world models, which is actually more realistic than almost all existing works only in a laboratory setting.

Each generation application benchmarks its own dataset. LDM is pretrained on the LAION-400M dataset , DALL-E mini is pretrained on the CC3M , CC12M , and a filtered subset of YFCC100M . All of them are widely used image-text pair datasets.

Attack models and auxiliary datasets. For pixel-level attacks, i.e., Attack I-P/II-P, the attack model is a CNN model. For semantic-level attacks, i.e., Attack I-S/II-S/III, the attack model is a 3-layer MLP model. The attack model of Attack-IV is composed of three sub-networks and a linear layer, as it has three types of input. Every sub-network accepts one type of input and generates a feature embedding. These three embeddings are concatenated and fed into a linear layer to make predictions. The loss function is cross-entropy and the optimizer is Adam. The learning rate is set to 0.001, and we train the attack model for 200 epochs.

To construct the auxiliary dataset, we randomly select a small set of member images from the target training set, as well as the same size non-member samples from two local datasets, i.e., MSCOCO and Visual Genome (abbreviated as VG) . For the LDM model, with the intention of better focusing on the privacy-sensitive information, we leverage LAION-Face , a human face subset of LAION-400M, as members, and create MSCOCO-Face and VG-Face as non-members. For the DALL-E mini model, as it is claimed that faces in general are not generated properly by the DALL-E mini model, we create CC3M-No-Face as members, MSCOCO-No-Face and VG-No-Face as non-members. See more details about the dataset creation in Appendix A. Then, we merge the member dataset and non-member dataset as the attack dataset and split it by half creating the attack training dataset and the attack testing dataset.

Captions and embeddings generation tools. We leverage officially pretrained BLIP , a recent state-of-the-art vision-language model to generate captions, image embeddings, and text embeddings. We later show that the choice of the caption generation tool and embedding generation tool has negligible effect on the attack performance.

Evaluation metrics. We adopt accuracy as the main evaluation metric for the attack performance, as widely used in previous works .

Results

Attack performance. We first show the attack performance of our proposed attacks when treats MSCOCO and VG as the non-member datasets on LDM and DALL-E mini models in Figure 2. We can observe that all of the proposed attacks can achieve remarkable performance, even in the worst case with an accuracy of over 0.65, which is much higher than random guesses (i.e., 0.5). These empirical results verify our key intuitions in all three perspectives.

Furthermore, we can also find that semantic-level attacks achieve much better performance than pixel-level attacks. Take the LDM model as an example, the accuracy of Attack I-S/II-S on MSCOCO-Face is 0.835/0.980, while the accuracy of Attack I-P/II-P is 0.699/0.707. Therefore, an adversary can apply semantic-level attacks to obtain higher performance if they have a semantic extraction tool, such as a well-trained BLIP. We reason that this is due to the fact that these pixel-level features make pixels independent and are unable to obtain context-dependent information, whereas semantic-level features have this capability. In addition, a query image can generate multiple images that are not identical at the pixel level but are similar at the semantic level, leading to significant variance in the performance of pixel-level attacks. Meanwhile, the results of Attack II-S are much better than those of Attack III, indicating that the relationship between the same-modality embeddings works better than that of cross-modality embeddings. These results lead to the unsurprising observation that the performance of Attack-IV and Attack II-S is very similar and that Attack IV and Attack II-S achieve the best attack performance in all cases. Take the DALL-E mini model as an example, these two attacks even successfully achieve 99.9% test accuracy on both non-member datasets.

Analysis. To figure out the reason behind the success of our attacks, we calculate the FID scores between the query images and its corresponding generated images on member and non-member datasets, respectively. As we leverage BLIP to generate embeddings used for our attacks, we also calculate the FID score based on the BLIP’s embeddings. As illustrated in Table 2, members have a lower FID score than non-members on both cases. For example, the FID score for Laion-Face is only 9.912, while that of MSCOCO-Face is 20.314. These results demonstrate that the generated images of members are perceptually closer to the original query images. We also plot the histogram of cosine similarity between the same-modality embedding in Figure 8 of Appendix A, i.e., the query image embedding and generated image embedding, showing the distributions on members and non-members have a clear difference and these same-modality embeddings of the members have a higher cosine similarity in general.

Effect of the caption/embedding generation tools. We investigate if the selection of the caption generation tool and embedding generation tool affects the attack performance. To this end, we leverage pretrained CLIP to generate embeddings for generated captions, generated images, and query images, and ClipCap , a state-of-the-art image captioning model, to generate caption for each query image. As illustrated in Figure 4, the attack performance is consistent with when we use BLIP as the caption and embedding generation tool. Concretely, Attack IV and Attack II-S still achieve the best performance, with semantic-level attacks still performing better than pixel-level attacks. Hence, we conclude that the choice of embedding and caption generation tools has negligible effect on the attack performance.

Effect of the denoising steps. At the inference time of diffusion-based models, we need to manually set the denoising step. Generally, the more denoising steps, the better the quality of the generated images, which leads to higher time and memory costs. This motivates us to explore whether more steps can bring better results, or whether a small number of steps is enough to give us good attack performance. In Figure 3, we start with 20 denoising steps as the resulting images become visually usable, and gradually increase the steps to 50, 100, and 200, where 50 is the default setting for previous experiments. We can observe that the proposed attacks work well even with 20 denoising steps except Attack II-P. Meanwhile, increasing the denoising steps does not gain much improvement of the attack performance. To explain the results, we calculate the difference between the FID scores of the members and non-members in Figure 5. We can see that the difference is already very large at 20 denoising steps on both cases. Although the difference continues increasing with the growth of denoising steps, it is sufficient to enable most of attacks successfully when the generated images are just visually available (e.g., 20 denoising steps in our cases).

Effect of the auxiliary dataset’s size. So far, we assume that the adversary can have a small subset $\mathcal{D^{\textit{member}}_{\textit{sub\_target}}}$ from the training data to construct the auxiliary dataset. In the real world, it is more likely that the adversary can only get less data. Therefore, we investigate the influence of auxiliary dataset’ size on attack performance. Specifically, we randomly sample from the above member $\mathcal{D^{\textit{member}}_{\textit{sub\_target}}}$ with different proportions $\{0.05,0.1,0,3,0.5,1\}$ and sample the same size from the local non-member $\mathcal{D}^{\textit{non\_member}}_{\textit{local}}$. As shown in Figure 6, the proposed attacks are more effective than random guessing even with 5% auxiliary dataset. Especially the Attack II-S and Attack IV, the size of auxiliary dataset only has negligible influence on them. More results on the DALL-E mini model are in Figure 9 of Appendix A, and the same conclusions can be drawn.

Operations for processing embeddings. In Attack II-S/III/IV, we calculate the element-wise distance between two embeddings. Besides leveraging traditional metrics. e.g., L1 distance and L2 distance, we also consider other three pair-wise operations. These operations are summarized in Table 3 of Appendix A. We refer to the embedding distance between the generated images and query images as same-modality embedding and to the embedding distance between the generated images and generated captions as cross-modality embedding. We explore the effect of different operations on Attack IV, as it takes both the same-modality and cross-modality embedding as input. As illustrated in Figure 7, the operation of the same-modality embedding plays a decisive role. The concatenation operation for the same-modality embedding brings the best performance of Attack IV, as it achieves over 97.7% test accuracy on both cases no matter the operation for the cross-modality embedding. We observe the concatenation operation is also the most appropriate one for the cross-modality embedding, as it can increase the attack performance when the same-modality embedding is less dominant. Thus, we use the concatenation operation for Attack II-S, Attack III, and Attack IV.

Defense

To mitigate membership leakage, the traditional way is to only output top-k predictions or the predicted labels. But this method cannot be extended to our scenarios as the output format of the text-to-image generation model is image. Chen et al. apply differential private (DP) to the target GAN model as a defense mechanism . However, due to the resource limitation, i.e., insufficient GPU resource, we are unable to train a text-to-image generation model with DP from scratch. In the previous experiment, we limit the number of data samples from the target training data. We can observe that by reducing the size of the dataset to 5% of its original size, the proposed attacks only have a slight deterioration and are still effective. Hence, we cannot limit the attack performance via restricting the number of member samples the adversary can have. Our future work will concentrate on thoroughly exploring more effective defense mechanisms against our attacks.

Conclusion

We take the first step towards studying membership leakage in text-to-image generation models, where an adversary aims to infer whether a given image is used to train a target text-to-image generation model. Based on the characteristics of the text-to-image generation models, we consider three key intuitions and design four attack methods accordingly. We conduct comprehensive experiments on two representative text-to-image generation modes. Empirical results show that all proposed attacks achieve remarkable performance, which convincingly demonstrates that membership leakage is a severe threat to the text-to-image generation models. Furthermore, to deeply investigate which factors and their effects on the attack performance, we conduct an in-depth ablation study from different perspectives that can guide developers and researchers to be alert to vulnerabilities in text-to-image generation models.

References

Appendix A Appendix

Member and non-member datasets creation for LDM. We first randomly sample 30K images from the LAION-Face dataset as the member dataset. To be in line with the member dataset, we generate the MSCOCO-Face and VG-Face datasets by following the generation process of the Laion-Face, i.e., adopting RetinaFace as a face detector to detect whether a given image contains a human face and filtering out those whose face detection scores are less than 0.9. To maintain the data balance, we randomly sample 30K image-text pairs from MSCOCO-Face as the final version of the non-member dataset. As the VG-Face only contains 26,134 image-text pairs, we resize the member dataset to the equal size when considering VG-Face as the non-member dataset. Note that we only leverage images from these image-text-pair datasets, and we can also launch these attacks by using image-only datasets.

Member and non-member datasets creation for DALL-E mini. As it is claimed that faces in general are not generated properly by the DALL-E mini model, we create a 30K subset for each dataset by leveraging the RetinaFace detector again and randomly sampling from images that no face is detected. The member dataset is referred to as CC3M-No-Face, and the non-member datasets are referred to as MSCOCO-No-Face and VG-No-Face, respectively.

The cosine similarity between same-modality embedding. We plot the histogram of the cosine similarity between the query image embeddings and generated image embeddings obtained by BLIP. As shown in Figure 8, the distributions of cosine similarity between same-modality embeddings on member dataset and non-member dataset has a clear difference on these two cases. In addition, these two embeddings of the member dataset have a higher cosine similarity in general, which again verifies the intuition that the distance between generated image $\boldsymbol{x^{\prime}}$ and query image $\boldsymbol{x}$ from the member dataset should be closer than that from non-member dataset.