SyncDreamer: Generating Multiview-consistent Images from a Single-view Image

Introduction

Humans possess a remarkable ability to perceive 3D structures from a single image. When presented with an image of an object, humans can easily imagine the other views of the object. Despite great progress brought by neural networks in computer vision or graphics fields for extracting 3D information from images, generating novel view images with multiview consistency from a single-view image of an object is still a challenging problem due to the limited 3D information available in an image.

Recently, diffusion models have demonstrated huge success in 2D image generation, which unlocks new potential for 3D generation tasks. However, directly training a generalizable 3D diffusion model usually requires a large amount of 3D data while existing 3D datasets are insufficient for capture the complexity of arbitrary 3D shapes. Therefore, recent methods resort to distilling pretrained text-to-image diffusion models for creating 3D models from texts, which shows impressive results on this text-to-3D task. Some works extend such a distillation process to train a neural radiance field (NeRF) for the image-to-3D task. In order to utilize pretrained text-to-image models, these methods have to perform textual inversion to find a suitable text description of the input image. However, the distillation process along with the textual inversion usually takes a long time to generate a single shape and requires tedious parameter tuning for satisfactory quality. Moreover, due to the abundance of specific details in an image, such as object category, appearance, and pose, it is challenging to accurately represent an image using a single word embedding, which results in a decrease in the quality of 3D shapes reconstructed by the distillation method.

Instead of distillation, some recent works apply 2D diffusion models to directly generate multiview images for the 3D reconstruction task. The key problem is how to maintain the multiview consistency when generating images of the same object. To improve the multiview consistency, these methods allow the diffusion model to condition on the input, previously generated images or renderings from a neural field . Although some impressive results are achieved for specific object categories from ShapeNet or Co3D , how to design a diffusion model to generate multiview-consistent images for arbitrary objects still remains unsolved.

In this paper, we propose a simple yet effective framework to generate multiview-consistent images for the single-view 3D reconstruction of arbitrary objects. The key idea is to extend the diffusion framework to model the joint probability distribution of multiview images. We show that modeling the joint distribution can be achieved by introducing a synchronized multiview diffusion model. Specifically, for $N$ target views to be generated, we construct $N$ shared noise predictors respectively. The reverse diffusion process simultaneously generates $N$ images by $N$ corresponding noise predictors, where information across different images is shared among noise predictors by attention layers on every denoising step. Thus, we name our framework SyncDreamer which synchronizes intermediate states of all noise predictors on every step in the reverse process.

SyncDreamer has the following characteristics that make it a competitive tool for lifting 2D single-view images to 3D. First, SyncDreamer retains strong generalization ability by initializing its weights from the pretrained Zero123 model which is finetuned from the Stable Diffusion model on the Objaverse dataset. Thus, SyncDreamer is able to reconstruct shapes from both photorealistic images and hand drawings as shown in Fig. 1. Second, SyncDreamer makes the single-view reconstruction easier than the distillation methods. Because the generated images are consistent in both geometry and appearance, we can simply run a vanilla NeRF or a vanilla NeuS without using any special losses for reconstruction. Given the generated images, one can easily reckon the final reconstruction quality while it is hard for distillation methods to know the output reconstruction quality beforehand. Third, SyncDreamer maintains creativity and diversity when inferring 3D information, which enables generating multiple reasonable objects from a given image as shown in Fig. 5. In comparison, previous distillation methods can only converge to one single shape.

We quantitatively compare SyncDreamer with baseline methods on the Google Scanned Object dataset. The results show that, in comparison with baseline methods, SyncDreamer is able to generate more consistent images and reconstruct better shapes from input single-view images. We further demonstrate that SyncDreamer supports various styles of 2D input like cartoons, sketches, ink paintings, and oil paintings for generating consistent views and reconstructing 3D shapes, which verifies the effectiveness of SyncDreamer in lifting 2D images to 3D shapes.

Related Work

Diffusion models have shown impressive results on 2D image generation. Concurrent work MVDiffusion also adopts the multiview diffusion formulation to synthesize textures or panoramas with known geometry. We propose similar formulations in SyncDreamer but with unknown geometry. MultiDiffusion and SyncDiffusion correlate multiple diffusion models for different regions of a 2D image. Many recent works try to repeat the success of diffusion models on the 3D generation task. However, the scarcity of 3D data makes it difficult to directly train diffusion models on 3D and the resulting generation quality is still much worse and less generalizable than the counterpart image generation models, though some works are trying to only use 2D images for training 3D diffusion models.

2 Using 2D diffusion models for 3D

Instead of directly learning a 3D diffusion model, many works resort to using high-quality 2D diffusion models for 3D tasks. Pioneer works DreamFusion and SJC propose to distill a 2D text-to-image generation model to generate 3D shapes from texts. Follow-up works improve such text-to-3D distillation methods in various aspects. Many works also apply such a distillation pipeline in the single-view reconstruction task. Though some impressive results are achieved, these methods usually require a long time for textual inversion and NeRF optimization and they do not guarantee to get satisfactory results.

Other works directly apply the 2D diffusion models to generate multiview images for 3D reconstruction. are conditioned on the input image by attention layers for novel-view synthesis in indoor scenes. Our method also uses attention layers but is intended for object reconstruction. resort to estimated depth maps to warp and inpaint for novel-view image generation, which strongly relies on the performance of the external single-view depth estimator. Two concurrent works generate new images in an autoregressive render-and-generate manner, which demonstrates good performances on specific object categories or scenes. In comparison, SyncDreamer is targeted to reconstruct arbitrary objects and generates all images in one reverse process. The concurrent work Viewset Diffusion shares a similar idea to generate a set of images. The differences between SyncDreamer and Viewset Diffusion are that SyncDreamer does not require predicting a radiance field like Viewset Diffusion but only uses attention to synchronize the states among views and SyncDreamer fixes the viewpoints of generated views for better training convergence.

3 Other single-view reconstruction methods

Single-view reconstruction is a challenging ill-posed problem. Before the prosperity of generative models used in 3D reconstruction, there are many works that reconstruct 3D shapes from single-view images by regression or retrieval , which have difficulty in generalizing to real data or new categories. Recent NeRF-GAN methods learn to generate NeRFs for specific categories like human or cat faces. These NeRF-GANs achieve impressive results on single-view image reconstruction but fail to generalize to arbitrary objects. Although some recent works also attempt to generalize NeRF-GAN to ImageNet , training NeRF-GANs for arbitrary objects is still challenging.

Method

Given an input view $\mathbf{y}$ of an object, our target is to generate multiview images of the object. We assume that the object is located at the origin and is normalized inside a cube of length 1. The target images are generated on $N$ fixed viewpoints looking at the object with azimuths evenly ranging from $0^{\circ}$ to $360^{\circ}$ and elevations of $30^{\circ}$, as shown in Fig. 2. To improve the multiview consistency of generated images, we formulate this generation process as a multiview diffusion model to correlate the generation of each image. In the following, we begin with a review of diffusion models .

Diffusion models aim to learn a probability model $p_{\theta}(\mathbf{x}_{0})=\int p_{\theta}(\mathbf{x}_{0:T})d\mathbf{x}_{1:T}$ where $\mathbf{x}_{0}$ is the data and $\mathbf{x}_{1:T}:=\mathbf{x}_{1},...,\mathbf{x}_{T}$ are latent variables. The joint distribution is characterized by a Markov Chain (reverse process)

where $p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{x}_{T};\mathbf{0},\mathbf{I})$ and $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\mathbf{\mu}_{\theta}(\mathbf{x}_{t},t),\sigma^{2}_{t}\mathbf{I})$. $\mathbf{\mu}_{\theta}(\mathbf{x}_{t},t)$ is a trainable component while the variance $\sigma^{2}_{t}$ is untrained time-dependent constants . The target is to learn the $\mathbf{\mu}_{\theta}$ for the generation. To learn $\mathbf{\mu}_{\theta}$, a Markov chain called forward process is constructed as

where $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$ and $\beta_{t}$ are all constants. DDPM shows that by defining

where $\alpha_{t}$ and $\bar{\alpha}_{t}$ are constants derived from $\beta_{t}$ and $\mathbf{\epsilon}_{\theta}$ is a noise predictor, we can learn $\mathbf{\epsilon}_{\theta}$ by

where $\mathbf{\epsilon}$ is a random variable sampled from $\mathcal{N}(\mathbf{0},\mathbf{I})$.

2 Multiview diffusion

Applying the vanilla DDPM model to generate novel-view images separately would lead to difficulty in maintaining multiview consistency across different views. To address this problem, we formulate the generation process as a multiview diffusion model that correlates the generation of each view. Let us denote the $N$ images that we want to generate on the predefined viewpoints as $\{\mathbf{x}^{(1)}_{0},...,\mathbf{x}^{(N)}_{0}\}$ where suffix means the time step . We want to learn the joint distribution of all these views $p_{\theta}({\mathbf{x}^{(1:N)}_{0}}|\mathbf{y}):=p_{\theta}(\mathbf{x}^{(1)}_{0},...,\mathbf{x}^{(N)}_{0}|\mathbf{y})$. In the following discussion, all the probability functions are conditioned on the input view $\mathbf{y}$ so we omit $\mathbf{y}$ for simplicity.

The forward process of the multiview diffusion model is a direct extension of the vanilla DDPM in Eq. 2, where noises are added to every view independently by

where $q(\mathbf{x}^{(n)}_{t}|\mathbf{x}^{(n)}_{t-1})=\mathcal{N}(\mathbf{x}^{(n)}_{t};\sqrt{1-\beta_{t}}\mathbf{x}^{(n)}_{t-1},\beta_{t}\mathbf{I})$. Similarly, following Eq. 1, the reverse process is constructed as

where $p_{\theta}(\mathbf{x}^{(n)}_{t-1}|\mathbf{x}^{(1:N)}_{t})=\mathcal{N}(\mathbf{x}^{(n)}_{t-1};\mathbf{\mu}^{(n)}_{\theta}(\mathbf{x}^{(1:N)}_{t},t),\sigma^{2}_{t}\mathbf{I})$. Note that the second equation in Eq. 6 holds because we assume a diagonal variance matrix. However, the mean $\mathbf{\mu}^{(n)}_{\theta}$ of $n$-th view $\mathbf{x}_{t-1}^{(n)}$ depends on the states of all the views $\mathbf{x}^{(1:N)}_{t}$. Similar to Eq. 3, we define $\mathbf{\mu}^{(n)}_{\theta}$ and the training loss by

where $\mathbf{\epsilon}^{(1:N)}$ is the standard Gaussian noise of size $N\times H\times W$ added to all $N$ views, $\mathbf{\epsilon}^{(n)}$ is the noise added to the $n$-th view, and $\mathbf{\epsilon}^{(n)}_{\theta}$ is the noise predictor on the $n$-th view.

Training procedure. In one training step, we first obtain $N$ images $\mathbf{x}^{(1:N)}_{0}$ of the same object from the dataset. Then, we sample a timestep $t$ and the noise $\epsilon^{(1:N)}$ which is added to all the images $\mathbf{x}^{(1:N)}_{0}$ to obtain $\mathbf{x}^{(1:N)}_{t}$. After that, we randomly select a view $n$ and apply the corresponding noise predictor $\mathbf{\epsilon}^{(n)}_{\theta}$ on the selected view to predict the noise. Finally, the L2 distance between the sampled noise $\mathbf{\epsilon}^{(n)}$ and the predicted noise is computed as the loss for the training.

Synchronized $N$-view noise predictor. The proposed multiview diffusion model can be regarded as $N$ synchronized noise predictors $\{\mathbf{\epsilon}^{(n)}_{\theta}|n=1,...,N\}$. On each time step $t$, each noise predictor $\mathbf{\epsilon}^{(n)}$ is in charge of predicting noise on its corresponding view $\mathbf{x}^{(n)}_{t}$ to get $\mathbf{x}^{(n)}_{t-1}$. Meanwhile, these noise predictors are synchronized because, on every denoising step, every noise predictor exchanges information with each other by correlating the states $\mathbf{x}^{(1:N)}_{t}$ of all the other views. In practical implementation, we use a shared UNet for all $N$ noise predictors and put the viewpoint difference between the input view and the $n$-th target view $\Delta\mathbf{v}^{(n)}$, and the states $\mathbf{x}^{(1:N)}_{t}$ of all views as conditions to this shared noise predictor, i.e., $\mathbf{\epsilon}^{(n)}_{\theta}(\mathbf{x}^{(1:N)}_{t},t)=\mathbf{\epsilon}_{\theta}(\mathbf{x}^{(n)}_{t};t,\Delta\mathbf{v}^{(n)},\mathbf{x}^{(1:N)}_{t})$.

3 3D-aware feature attention for denoising

In this section, we discuss how to implement the synchronized noise predictor $\mathbf{\epsilon}_{\theta}(\mathbf{x}^{(n)}_{t};t,\Delta\mathbf{v}^{(n)},\mathbf{x}^{(1:N)}_{t},\mathbf{y})$ by correlating the multiview features using a 3D-aware attention scheme. The overview is shown in Fig. 3.

Backbone UNet. Similar to previous works , our noise predictor $\mathbf{\epsilon}_{\theta}$ contains a UNet which takes a noisy image as input and then denoises the image. To ensure the generalization ability, we initialize the UNet from the pretrained weights of Zero123 given that it is based on Stable Diffusion which has seen billions of images and can generalize to images of various domains. Zero123 concatenates the input view with the noisy target view as the input to UNet. Then, to encode the viewpoint difference $\Delta\mathbf{v}^{(n)}$ in UNet, Zero123 reuses the text attention layers of Stable Diffusion to process the concatenation of $\Delta\mathbf{v}^{(n)}$ and the CLIP feature of the input image. We follow the same design as Zero123 and empirically freeze the UNet and the text attention layers when training SyncDreamer. Experiments to verify these choices are presented in Sec. 4.6.

3D-aware feature attention. The remaining problem is how to correlate the states $\mathbf{x}_{t}^{(1:N)}$ of all the target views for the denoising of the current noisy target view $\mathbf{x}_{t}^{(n)}$. To enforce consistency among multiple generated views, it is desirable for the network to perceive the corresponding features in 3D space when generating the current image. To achieve this, we first construct a 3D volume with $V^{3}$ vertices and then project the vertices onto all the target views to obtain the features. The features from each target view are concatenated to form a spatial feature volume. Next, a 3D CNN is applied to the feature volume to capture and process spatial relationships. In order to denoise $n$-th target view, we construct a view frustum that is pixel-wise aligned with this view, whose features are obtained by interpolating the features from the spatial volume. Finally, on every intermediate feature map of the current view in the UNet, we apply a new depth-wise attention layer to extract features from the pixel-wise aligned view-frustum feature volume along the depth dimension.

Discussion. There are two primary design considerations in this 3D-aware feature attention UNet. First, the spatial volume is constructed from all the target views and all the target views share the same spatial volume for denoising, which implies a global constraint that all target views are looking at the same object. Second, the added new attention layers only conduct attention along the depth dimension, which enforces a local epipolar line constraint that the feature for a specific location should be consistent with the corresponding features on the epipolar lines of other views.

Experiments

We train SyncDreamer on the Objaverse dataset which contains about 800k objects. We set the viewpoint number $N=16$. The elevation of the target views is set to 30∘ and the azimuth evenly distributes in $[0^{\circ},360^{\circ}]$. Besides these target views, we also render 16 random views as input views on each object for training, which have the same azimuths but random elevations. We always assume that the azimuth of both the input view and the first target view is 0∘. We train the SyncDreamer for 80k steps ($\sim$4 days) with 8 40G A100 GPUs using a total batch size of 192. The learning rate is annealed from 5e-4 to 1e-5. Since we need an elevation of the input view to compute the viewpoint difference $\Delta\mathbf{v}^{(n)}$, we use the rendering elevation in training while we roughly estimate an elevation angle as input in inference. To obtain surface meshes, we predict the foreground masks of the generated images using CarveKit. Then, we train the vanilla NeuS for 2k steps to reconstruct the shape, which costs about 10 mins.

2 Experiment protocol

Evaluation dataset. Following , we adopt the Google Scanned Object dataset as the evaluation dataset. To demonstrate the generalization ability to arbitrary objects, we randomly chose 30 objects ranging from daily objects to animals. For each object, we render an image with a size of 256$\times$256 as the input view. We additionally evaluate some images collected from the Internet and the Wiki of Genshin Impact.

Baselines. We adopt Zero123 , RealFusion , Magic123 , One-2-3-45 , Point-E and Shap-E as baseline methods. Given an input image of an object, Zero123 is able to generate novel-view images of the same object from different viewpoints. Zero123 can also be incorporated with the SDS loss for 3D reconstruction. We adopt the implementation of ThreeStudio for reconstruction with Zero123, which includes many optimization strategies to achieve better reconstruction quality than the original Zero123 implementation. RealFusion is based on Stable Diffusion and the SDS loss for single-view reconstruction. Magic123 combines Zero123 with RealFusion to further improve the reconstruction quality. One-2-3-45 directly regresses SDFs from the output images of Zero123 and we use the official hugging face online demo to produce the results. Point-E and Shap-E are 3D generative models trained on a large internal OpenAI 3D dataset, both of which are able to convert a single-view image into a point cloud or a shape encoded in an MLP. For Point-E, we convert the generated point clouds to SDFs for shape reconstruction using the official models.

Metrics. We mainly focus on two tasks, novel view synthesis (NVS) and single view 3D reconstruction (SVR). On the NVS task, we adopt the commonly used metrics, i.e., PSNR, SSIM and LPIPS . To further demonstrate the multiview consistency of the generated images, we also run the MVS algorithm COLMAP on the generated images and report the reconstructed point number. Because MVS algorithms rely on multiview consistency to find correspondences to reconstruct 3D points, more consistent images would lead to more reconstructed points. On the SVR task, we report the commonly used Chamfer Distances (CD) and Volume IoU between ground-truth shapes and reconstructed shapes. Since the shapes generated by Point-E and Shap-E are defined in a different canonical coordinate system, we manually align the generated shapes of these two methods to the ground-truth shapes before computing these metrics.

3 Consistent novel-view synthesis

For this task, the quantitative results are shown in Table 1 and the qualitative results are shown in Fig. 4. By applying a NeRF model to distill the Stable Diffusion model , RealFusion shows strong multiview consistency producing more reconstructed points but is unable to produce visually plausible images as shown in Fig. 4. Zero123 produces visually plausible images but the generated images are not multiview-consistent. Our method is able to generate images that not only are semantically consistent with the input image but also maintain multiview consistency in colors and geometry. Meanwhile, for the same input image, Our method can generate different plausible instances using different random seeds as shown in Fig. 5.

4 Single view reconstruction

We show the quantitative results in Table 2 and the qualitative comparison in Fig. 6. Point-E and Shap-E tend to produce incompleted meshes. Directly distilling Zero123 generates shapes that are coarsely aligned with the input image, but the reconstructed surfaces are rough and not consistent with input images in detailed parts. Magic123 produces much smoother meshes but heavily relies on the estimated depth values on the input view, which may lead to incorrect results when the depth estimator is not robust. One-2-3-45 reconstructs meshes from the multiview-inconsistent outputs of Zero123, which is able to capture the general geometry but also loses details. In comparison, our method achieves the best reconstruction quality with smooth surfaces and detailed geometry.

5 Text-to-image-to-3D

By incorporating text2image models like Stable Diffusion or Imagen , SyncDreamer enables generating 3D models from text. Examples are shown in Fig. 7. In comparison with existing text-to-3D distillation, our method gives more flexibility because users can generate multiple images with their text2image models and select the desirable one to feed to SyncDreamer for 3D reconstruction.

6 Discussions

In this section, we further conduct a set of experiments to evaluate the effectiveness of our designs.

Generalization ability. To show the generalization ability, we evaluate SyncDreamer with 2D designs or hand drawings like sketches, cartoons, and traditional Chinese ink paintings, which are usually created manually by artists and exhibit differences in lighting effects and color space from real-world images. The results are shown in Fig. 8. Despite the significant differences in lighting and shadow effects between these images and the real-world images, our algorithm is still able to perceive their reasonable 3D geometry and produce multiview-consistent images.

Performance without 3D-aware feature attention. To show how the proposed 3D-aware feature attention improves multiview consistency, we discard the 3D-aware attention module in SyncDreamer and train this model on the same training set. This actually corresponds to finetuning a Zero123 model with fixed viewpoints. As we can see in Fig. 9, such a model still cannot produce images with strong consistency, which demonstrates the necessity of the 3D-aware attention module in generating multiview-consistent images.

Initializing from Stable Diffusion instead of Zero123. An alternative strategy is to initialize our model from Stable Diffusion . However, the results shown in Fig. 9 indicate that initializing from Stable Diffusion exhibits a worse generalization ability than from Zero123 . Zero123 already enables the UNet to infer the relationship between different views, which thus reduces the difficulty in training a multiview image generator.

Training UNet. During the training of SyncDreamer, another feasible solution is to not freeze the UNet and the related layers initialized from Zero123 but further finetune them together with the volume condition module. As shown in Fig. 9, the model without freezing these layers tends to predict the input object as a thin plate, especially when the input images are 2D hand drawings. We speculate that this phenomenon is caused by overfitting, likely due to the numerous thin-plate objects within the Objaverse dataset and the fixed viewpoints employed during our training process.

Runtime. SyncDreamer uses about 2.7 minutes to sample 64 images (4 instances) with 200 DDIM sampling steps on a 40G A100 GPU. Our runtime is slightly longer than Zero123 because we need to construct the spatial feature volume on every step.

Limitations and Conclusion

Limitations and future works. Though SyncDreamer shows promising performances in generating multiview-consistent images for 3D reconstruction, there are still limitations that the current framework does not fully address. First, the current SyncDreamer only generates 16 target views for an object, while reconstructing objects from such a limited number of views is associated with compromised quality. It is possible to train a SyncDreamer to generate more dense views, which would require more GPUs and larger GPU memory to train such a model. Second, the generated images are not always plausible and we may need to generate multiple instances with different seeds and select a desirable instance for 3D reconstruction. To further increase the quality, we may need to use a larger object dataset like Objaverse-XL and manually clean the dataset to exclude some uncommon shapes like complex scene representation, textureless 3D models, and point clouds. Third, the current implementation of SyncDreamer assumes a perspective image as input but many 2D designs are drawn with orthogonal projections, which would lead to unnatural distortion of the reconstructed geometry. Applying orthogonal projection in the volume construction of SyncDreamer would alleviate this problem.

Conclusions. In this paper, we present SyncDreamer to generate multiview-consistent images from a single-view image. SyncDreamer adopts a synchronized multiview diffusion to model the joint probability distribution of multiview images, which thus improves the multiview consistency. We design a novel architecture that uses the Zero123 as the backbone and a new volume condition module to model cross-view dependency. Extensive experiments demonstrate that SyncDreamer not only efficiently generates multiview images with strong consistency, but also achieves improved reconstruction quality compared to the baseline methods. Moreover, it exhibits excellent generalization to various styles of input images.

References