FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention

Introduction

Recent advancements in text-to-image generation , particularly diffusion models , have opened new frontiers in content creation. Subject-driven text-to-image generation permits the personalization to new individuals given a few sample images , allowing the generation of images featuring specific subjects in novel scenes, styles, and actions. However, current subject-driven text-to-image generation methods suffer from two key limitations: cost of personalization and identity blending for multiple subjects. Personalization is costly because they often need model fine-tuning for each new subject for best fidelity. The computational overhead and high hardware demands introduced by model tuning, largely due to the memory consumption and computations of backpropagation, constrain the applicability of these models across various platforms. Furthermore, existing techniques struggle with multi-subject generation (Figure 1) because of the “identity blending” issue (Figure 2 left), in which the model combines the distinct characteristics of different subjects (subject A looks like subject B and vice versa).

We propose FastComposer, a tuning-free, personalized multi-subject text-to-image generation method. Our key idea is to replace the generic word tokens, such as "person", by an embedding that captures an individual’s unique identity in the text conditioning. We use a vision encoder to derive this identity embedding from a referenced image, and then augment the generic text tokens with features from this identity embedding. This enables image generation based on subject-augmented conditioning. Our design allows the generation of images featuring specified subjects with only forward passes and can be further integrated with model compression techniques to boost deployment efficiency.

To tackle the multi-subject identity blending issue, we identify unregulated cross-attention as the primary reason (Figure 4). When the text includes two "person" tokens, each token’s attention map attends to both person in the image rather than linking each token to a distinct person in the image. To address this, we propose supervising cross-attention maps of subjects with segmentation masks during training (i.e., cross-attention localization), using standard segmentation tools . This supervision explicitly guides the model to map subject features to distinct and non-overlapping regions of the image, thereby facilitating the generation of high-quality multi-subject images (Figure 2 left). We note that segmentation and cross-attention localization is only required during the training phase.

Naively applying subject-augmented conditioning leads to subject overfitting (Figure 2 right), restricting the user’s ability to edit subjects based on textual directives. To address this, we introduce delayed subject conditioning, preserving the subject’s identity while following text instructions. It employs text-only conditioning in the early denoising stage to generate the image layout, followed by subject-augmented conditioning in the remaining denoising steps to refine the subject appearance. This simple technique effectively preserves subject identity without sacrificing editability (Figure 5).

For the first time, FastComposer enables inference-only generation of multiple-subject images across diverse scenarios (Figure 1). FastComposer achieves 300$\times$-2500$\times$ speedup and 2.8$\times$-6.7$\times$ memory saving compared to fine-tuning-based methods, requiring zero extra storage for new subjects. FastComposer paves the way for low-cost, personalized, and versatile text-to-image generation.

Related Work

Subject-Driven Image Generation aims to render a particular subject unseen at the initial training stage. Given a limited number of example images of the subject, it seeks to synthesize novel renditions in diverse contexts. DreamBooth , textual-inversion , and custom-diffusion use optimization-based methods to embed subjects into diffusion models. This is achieved by either fine-tuning the model weights or inverting the subject image into a text token that encodes the subject identity . Recently, tuning-encoder reduces the total number of fine-tuning steps by first generating an inverted set latent code using a pre-trained encoder and then refines these codes through several finetuning steps to better preserve subject identities. However, all these tuning-based methods require resource-intensive backpropagation, and the hardware must be capable of fine-tuning the model, which is neither feasible on edge devices such as smartphones, nor scalable for cloud-based applications. In contrast, our new FastComposer amortizes the costly subject tuning during the training phase, enabling instantaneous personalization of multiple subjects using simple feedforward methods at test time.

A number of concurrent works have explored tuning-free methods. X&Fuse concatenates the reference image with the noisy latent for image conditioning. ELITE and InstantBooth use global and local mapping networks to project reference images into word embeddings and inject reference image patch features into cross-attention layers to enhance local details. Despite impressive results for single-object customization, their architecture design restricts their applicability to multiple subject settings, as they rely on global interactions between the generated image and reference input image. UMM-Diffusion shares a similar architecture to ours. However, it faces identity blending challenges when extended to multiple subjects . In comparison, our method supports multi-subject composition via a cross-attention localization supervision mechanism (Sec 4.2).

Custom-Diffusion enables multi-concepts composition by jointly fine-tuning the diffusion model for multiple concepts. However, it typically handles concepts with clear semantic distinctions, such as animals and their related accessories or backgrounds. The method encounters challenges when dealing with subjects within similar categories, often generating the same person twice when composing two different individuals (Figure 1). SpaText , and Collage Diffusion enable multi-object composition through a layout to image generation process. A user-provided segmentation mask determines the final layouts, which are then transformed into high-resolution images using a diffusion model. Nevertheless, these techniques either compose generic objects without customization or demand the costly textual-inversion process to encode instance-specific details . Furthermore, these techniques require a user-provided segmentation map. In contrast, FastComposer generates personalized, multi-subject images in an inference-only manner and automatically derives plausible layouts from text prompts.

Preliminaries

We use the state-of-the-art StableDiffusion (SD) model as our backbone network. The SD model consists of three components: the variational autoencoder (VAE), U-Net, and a text encoder. The VAE encoder $\mathcal{E}$ compresses the image $x$ to a smaller latent representation $z$, which is subsequently perturbed by Gaussian noise $\varepsilon$ in the forward diffusion process. The U-Net, parameterized by $\theta$, denoises the noisy latent representation by predicting the noise. This denoising process can be conditioned on text prompts through the cross-attention mechanism, while the text encoder $\psi$ maps the text prompts $\mathcal{P}$ to conditional embeddings $\psi(\mathcal{P})$. During training, the network is optimized to minimize the loss function given by the equation below:

where $z_{t}$ is the latent code at time step $t$. At inference time, a random noise $z_{T}$ is sampled from $\mathcal{N}(0,1)$ and iteratively denoised by the U-Net to the initial latent representation $z_{0}$. Finally, the VAE decoder $\mathcal{D}$ generates the final image by mapping the latent codes back to pixel space $\hat{x}=\mathcal{D}(z_{0})$.

Text-Conditioning via Cross-Attention Mechanism.

FastComposer

Figure 3 gives a concrete example of our augmentation approach.

Subject-Driven Image Generation Training.

To enable inference-only subject-driven image generation, we train the image encoder, the MLP module, and the U-Net with the denoising loss (Figure 3). We create a subject-augmented image-text paired dataset to train our model, where noun phrases from image captions are paired with subject segments appearing in the target images. We initially use a dependency parsing model to chunk all noun phrases (e.g., “a woman”) in image captions and a panoptic segmentation model to segment all subjects present in the image. We then pair these subject segments with corresponding noun phrases in the captions with a greedy matching algorithm based on text and image similarity . The process of constructing the subject-augmented image-text dataset is detailed in Sec. 5.1. In the training phase, we employ subject-augmented conditioning, as outlined in Equation 2, to denoise the perturbed target image. We also mask the subjects’ backgrounds with random noise before encoding, preventing the overfitting of the subjects’ backgrounds. Consequently, FastComposer can directly use natural subject images during inference without explicit background segmentation.

2 Localizing Cross-Attention Maps with Subject Segmentation Masks

We observe that traditional cross-attention maps tend to attend to all subjects at the same time, which leads to identity blending in multi-subject image generation (Figure 4 top). We propose to localize cross-attention maps with subject segmentation masks during training to solve this issue.

Prior research shows that the cross-attention mechanism within diffusion models governs the layout of generated images. The scores in cross-attention maps represent “the amount of information flows from a text token to a latent pixel.” We hypothesize that identity blending arises from the unrestricted cross-attention mechanism, as a single latent pixel can attend to all text tokens. If one subject’s region attends to multiple reference subjects, identity blending will occur. In Figure 4, we confirm our hypothesis by visualizing the average cross-attention map within the U-Net of the diffusion model. The unregularized model often has two reference subject tokens influencing the same generated person at the same time, causing a mix of features from both subjects. We argue that proper cross-attention maps should resemble an instance segmentation of the target image, clearly separating the features related to different subjects. To achieve this, we add a regularization term to the subject cross-attention maps during training to encourage focusing on specific instance areas. Segmentation maps and cross-attention regularization are only used during training, not at test time.

Localizing Cross-Attention with Segmentation Masks.

As discussed in Section 3, a cross-attention map $A\in^{(h\times w)\times n}$ connects latent pixels to conditional embeddings at each layer, where $A[i,j,k]$ denotes the information flow from the $k$-th conditional token to the $(i,j)$ latent pixel. Ideally, the subject token’s attention map should focus solely on the subject region rather than spreading throughout the entire image, preventing identity blending among subjects. To accomplish this, we propose localizing the cross-attention map using the reference subject’s segmentation mask. Let $\mathcal{M}=\{M_{1},M_{2},\dots M_{m}\}$ represent the reference subjects’ segmentation masks, $\mathcal{I}=\{i_{1},i_{2},\dots i_{m}\}$ be the index list indicating which subject corresponds to each word in the text prompt, and $A_{i}=A[:,:,i]\in^{(h\times w)}$ be the cross-attention map of the $i$-th subject token. We supervise the cross-attention map $A_{i_{j}}$ to be close to the segmentation mask $m_{j}$ of the $j$-th subject token, i.e., $A_{i_{j}}\approx m_{j}$. We employ a balanced L1 loss to minimize the distance between the cross-attention map and the segmentation mask:

The final training objective of FastComposer is given by:

using a localization loss ratio controlled by hyperparameter $\lambda=0.001$. Motivated by , we apply the localization loss to the downsampled cross-attention maps, i.e., the middle 5 blocks of the U-Net, which are known to contain more semantic information. As illustrated in Figure 4, our localization technique enables the model to precisely allocate attention to reference subjects at test time, which prevents identity blending between subjects.

3 Delayed Subject Conditioning in Iterative Denoising

During inference, using the augmented text representation directly often leads to images that closely resemble the subjects while ignoring the textual directives. This occurs because the image layout forms at the early phases of the denoising process, and premature augmentation from the reference image causes the resulting image to stray from the text instructions. Prior methods mitigate this issue by generating an initial latent code and refining it through iterative model finetuning. However, this process is resource-intensive and needs high-end devices for model fine-tuning. Inspired by Style Mixing , we propose a simple delayed subject conditioning, which allows for inference-only subject conditioning while striking a balance between identity preservation and editability.

Specifically, we perform image augmentation only after the layout has been created using a text-only prompt. In this framework, our time-dependent noise prediction model can be represented as:

Here, $c$ denotes the original text embedding and $c^{\prime}$ denotes text embedding augmented with the input image embedding. $\alpha$ is a hyperparameter indicating the ratio of subject conditioning. We ablate the effect of using different $\alpha$ in Figure 5. Empirically, $\alpha\in[0.6,0.8]$ yields good results that balance prompt consistency and identity preservation, though it can be easily tuned for specific instances.

Experiments

We built a subject-augmented image-text paired dataset based on the FFHQ-wild dataset to train our models. First, we use the BLIP-2 model blip2-opt-6.7b-coco to generate captions for all images. Next, we employ the Mask2Former model mask2former-swin-large-coco-panoptic to generate panoptic segmentation masks for each image. We then leverage the spaCy library to chunk all noun phrases in the image captions and expand numbered plural phrases (e.g., "two women") into singular phrases connected by “and” (e.g., "a woman and a woman"). Finally, we use a greedy matching algorithm to match noun phrases with image segments. We do this by considering the product of the image-text similarity score by the OpenCLIP model CLIP-ViT-H-14-laion2B-s32B-b79K and the label-text similarity score by the Sentence-Transformer model stsb-mpnet-base-v2. We reserve 1000 images for validation and testing purposes.

Training Details.

We start training from the StableDiffusion v1-5 model. To encode the visual inputs, we use OpenAI’s clip-vit-large-patch14 vision model, which serves as the partner model of the text encoder in SDv1-5. During training, we freeze the text encoder and only train the U-Net, the MLP module, and the last two transformer blocks of the vision encoder. We train our models for 150k steps on 8 NVIDIA A6000 GPUs, with a constant learning rate of 1e-5 and a batch size of 128. We only augment segments whose COCO label is “person” and set a maximum of 4 reference subjects during training, with each subject having a 10% chance of being dropped. We train the model solely on text conditioning with 10% of the samples to maintain the model’s capability for text-only generation. To facilitate classifier-free guidance sampling , we train the model without any conditions on 10% of the instances. During training, we apply the loss only in the subject region to half of the training samples to enhance the generation quality in the subject area.

Evaluation Metric

We evaluate image generation quality on identity preservation and prompt consistency. Identity preservation is determined by detecting faces in the reference and generated images using MTCNN , and then calculating a pairwise identity similarity using FaceNet . For multi-subject evaluation, we identify all faces within the generated images and use a greedy matching procedure between the generated faces and reference subjects. The minimum similarity value among all subjects measures overall identity preservation. We evaluate the prompt consistency using the average CLIP-L/14 image-text similarity following textual-inversion . For efficiency evaluation, we consider the total time for customization, including fine-tuning (for tuning-based methods) and inference. We also measure peak memory usage during the entire procedure.

2 Single-Subject Image Generation

Our first evaluation targets the performance of single-subject image generation. Given the lack of published baselines in our tuning-free environment, we compare with leading optimization-based approaches, including DreamBooth , Textual-Inversion , and Custom Diffusion . We use the implementations from diffusers library . We provide the detailed hyperparameters in the appendix section. We assess the capabilities of these different methods in generating personalized content for subjects derived from the Celeb-A dataset . To construct our evaluation benchmark, we develop a broad range of text prompts encapsulating a wide spectrum of scenarios, such as recontextualization, stylization, accessorization, and diverse actions. The entire test set comprises 15 subjects, with 30 unique text prompts allocated to each. An exhaustive list of text prompts is available in the appendix. We utilized five images per subject to fine-tune the optimization-based methods, given our observation that these methods overfit and simply reproduce the reference image when a single reference image is used. In contrast, our model employs a single randomly selected image for each subject. Shown in Table 1, FastComposer surpasses all baselines, delivering superior identity preservation and prompt consistency. Remarkably, it achieves 300$\times$-1200$\times$ speedup and 2.8$\times$ reduction in memory usage. Figure 6 shows the qualitative results of single-subject personalization comparisons, employing different approaches across an array of prompts. Significantly, our model matches the text consistency of text-only methods and exceeds all baseline strategies in terms of identity preservation, with only single input and forward passes used.

3 Multi-Subject Image Generation

We then consider a more complex setting: multi-object, subject-driven image generation. We examine the quality of multi-subject generation by using all possible combinations (105 pairs in total) formed from 15 subjects described in Sec. 5.2, allocating 21 prompts to each pair for assessment. Table 2 shows a quantitative analysis contrasting FastComposer with the baseline methods. Optimization-based methods frequently falter in maintaining identity preservation, often generating generic images or images that blend identities among different reference subjects. FastComposer, on the other hand, preserves the unique features of different subjects, yielding a significantly improved identity preservation score. Furthermore, our prompt consistency is on par with tuning-based approaches . Qualitative comparisons are shown in Figure 1. More visual examples for three-subject images are shown in Figure 7.

4 Ablation Study

Figure 5 shows the impact of varying the ratio of timesteps devoted to subject conditioning, a hyperparameter in our delayed subject conditioning approach. As this ratio increases, the model improves in identity preservation but loses editability. A ratio between 0.6 to 0.8 achieves a favorable balance on the tradeoff curve.

Cross-Attention Localization Loss.

Table 3 presents the ablation studies on our proposed cross-attention localization loss. The baseline is trained in the same setting but excludes the localization loss. Our method demonstrates a substantial enhancement of the identity preservation score. Figure 4 shows the qualitative comparisons. Incorporating the localization loss allows the model to focus on particular reference subjects, thereby avoiding identity blending.

Discussion and Conclusion

We propose FastComposer, a tuning-free method for personalized, multi-subject text-to-image generation. We achieve tuning-free subject-driven image generation by using a pre-trained vision encoder, making this process efficient and accessible across various platforms. FastComposer effectively tackles the identity blending issue in multi-subject generation by supervising cross-attention maps with segmentation masks during training. We also propose a novel delayed subject conditioning technique to balance the identity preservation and the flexibility of image editability.

First, the current training set is FFHQ which is small and primarily contains headshots of human faces. It also has a long-tailed distribution for the number of people, thus limiting our ability to generate images with more than three subjects. Utilizing a more diverse dataset will enable FastComposer to generate a broader range of actions and scenarios, thereby enhancing its versatility and applicability. Second, our work is primarily human-centric due to a scarcity of large-scale, multi-subject datasets featuring other subjects like animals. We believe that broadening our dataset to incorporate multi-subject imagery of other categories will significantly enrich our model’s capabilities. Finally, our model, built on the foundation of Stable Diffusion and FFHQ, also inherits their biases.

Acknowledgements

This work is supported by MIT AI Hardware Program, NVIDIA Academic Partnership Award, MIT-IBM Watson AI Lab, Amazon and MIT Science Hub, Microsoft Turing Academic Program, Singapore DSTA under DST00OECI20300823 (New Representations for Vision), NSF grant 2105819 and NSF CAREER Award 1943349.

References