V3D: Video Diffusion Models are Effective 3D Generators

Introduction

With the advancement of diffusion-based image generation, automatic 3D generation has also made remarkable progress recently. Approaches based on Score Distillation Sampling have achieved tremendous success in the field of 3D generation by extracting priors directly from text-to-image diffusion models . However, such optimization-based methods are relatively slow and suffer from mode collapse and the Janus problem .

Subsequent efforts have proposed various alternative pathways to address these issues, mainly including directly mapping images into 3D representations (such as Triplane NeRF or 3DGS ), or first generating consistent multi-view images then conducting 3D reconstruction to obtain the underlying 3D model . With the increase in 3D data size and model capacity, these methods have reduced the generation time from hours to minutes. However, the training process of these models still exhibits certain limitations; Methods that directly map to 3D representations require training on 3D datasets, making it impractical to leverage higher-quality image data and large-scale pre-trained image diffusion models. Meanwhile, approaches based on multi-view generation face difficulties in directly generating a sufficient number of images for reconstruction due to the requirement of memory-consuming cross-attention or intermediate 3D structure to ensure consistency across views.

Recently, video diffusion models have attracted significant attention due to their remarkable ability to generate intricate scenes and complex dynamics with great spatio-temporal consistency . Many videos incorporate changes in perspectives which provide natural observations of 3D objects from different views , enabling the diffusion models to perceive the physical 3D world . Current multi-view generation methods are based on image diffusion models, which can only generate a few views (less than 6). In contrast, video diffusion models can generate hundreds of frames, which is adequate for downstream 3D tasks. These advancements give us the possibility to generate consistent multi-view images using video diffusion models and then reconstruct underlying 3D assets to achieve high-quality 3D generation. However, reconstruction from multi-views imposes significant demands on consistency which is difficult for current video diffusion models. In this paper, we aim to fully unleash the potential of video diffusion models to perceive the 3D world and inject the geometrical consistency prior to make the diffusion model an effective multi-view generator. In light of this, we propose V3D, a novel approach for 3D generation with video diffusion models. Specifically, we propose to fine-tune video diffusion models on 3D datasets with additional conditions to generate novel views in both object-centric and scene-level scenarios. For object-centric 3D generation, we fine-tune the base video diffusion model on $360\degree$ orbit videos of synthetic 3D objects with the front view as a condition. We find that our fine-tuned model can generate reliable multi-views for 3D reconstruction. However, there still exists inconsistency in some details which may diminish the reconstruction quality. To address this, we adopt perceptual loss instead of pixel-wise loss as the reconstruction objective. We employ Gaussian Splatting as the 3D representation due to its reconstruction and rendering efficiency. To further accelerate the reconstruction, we propose a space-carving initialization method for Gaussians, which involves unprojecting multi-view object masks back to 3D space to roughly position the Gaussians on the object’s surface, eliminating the invalid optimizations on empty spaces. Considering the demands of real-world applications, we also propose a novel mesh extraction method that initially extracts the surface using SDF and subsequently refines the appearance on generated views with image-level loss. Furthermore, we demonstrate the effectiveness of our approach in scene-level 3D generation. To achieve precise camera path control and accommodate multi-view input, we integrate a PixelNeRF encoder to enhance the video diffusion model with robust 3D signals. Combined with auxiliary loss and fine-tuning on real-world posed videos, we extend the video diffusion model to generate novel views on an arbitrary camera path given sparse view input. Our contributions are:

We propose V3D, a framework that first employs a video diffusion model to generate consistent multi-view frames, then reconstructs the underlying 3D content. By fine-tuning on 3D datasets, we successfully boost the ability of geometrical consistency of video diffusion models. Our framework is generally applicable to both object and scene generation.

We design a reconstruction pipeline tailored for video diffusion outputs to generate high-quality 3D Gaussians or textured meshes. With our designed efficient initialization and refinement, V3D can reconstruct delicate 3D assets within 3 minutes.

We conduct experiments to validate the effectiveness of V3D. Both object-centric and scene-level experiments show that our method achieves state-of-the-art performance in reconstruction quality and multi-view consistency.

Related Work

Early efforts in 3D generation focus on per-scene optimization methods based on CLIP . DreamFusion , the pioneer work, leverages stronger diffusion prior by introducing Score Distillation Sampling that minimizes the difference between rendered images from the underlying 3D assets and the diffusion prior. Subsequent methods have further achieved substantial improvements of this paradigm in quality , optimization speed , alleviating the Janus problem , and have expanded their applications to generative tasks such as editing , texture generation , and single image-to-3D . Despite great success has been achieved, optimization-based methods still suffer from slow generation speed and low success rates. To overcome these challenges, researchers have explored some non-optimization paradigms. One stream involves first generating consistent multi-view images, and then utilizing reconstruction methods to obtain the corresponding 3D model . SyncDreamer combines explicit voxels and 3D convolution with a diffusion model to enhance the consistency of generated multi-views. Wonder3D fine-tunes image diffusion models to generate sparse views with corresponding normal maps and utilizes a NeuS to reconstruct the underlying geometry. Direct2.5 adopts a remeshing-based technique to recover geometry from generated sparse normal maps. Another line of research involves directly mapping sparse-views into 3D representations . LRM , PF-LRM , and Instant3D adopt a huge transformer to directly predict triplanes from single or sparse views. TriplaneGaussian and LGM instead map sparse views into more memory-efficient 3D Gaussian Splatting which supports much higher resolution for supervision. Our concurrent work IM-3D also explores the capability of video diffusion models in object-centric multi-view generation, we further extend this paradigm in scene-level novel view synthesis and achieve better performance.

2 Generative Models for Novel View Synthesis

While NeRF and 3D Gaussian Splatting have shown impressive performance in novel view synthesis with a sufficient number of inputs, reconstruction from sparse views requires additional priors due to the incomplete information provided. Early work mainly focuses on using regression-based or GAN-based methods for generalizable scene reconstruction. However, due to the lack of high-quality data and the limited model capacity, the generated novel views are often blurry and exhibit poor generalization ability. Subsequent work further integrates diffusion priors to achieve better scene-level novel view synthesis . SparseFusion proposes view-conditioned diffusion with epipolar feature transformer to synthesize novel views with sparse inputs. GeNVS utilizes a PixelNeRF encoder to incorporate geometry information and exploits an auto-aggressive generation strategy to enhance multi-view consistency. ZeroNVS extends Zero-1-to-3 to unbounded scenes and resolves the scale ambiguity with better camera pose conditioning. ReconFusion proposes to fine-tune image diffusion models to condition on multi-view input and utilizes a novel sample loss to reconstruct the 3D scene from sparse posed inputs. We draw inspiration from ReconFusion and GeNVS, adopting PixelNeRF-based conditioner to accommodate any number of input images and provide precise camera path control in scene-level generation.

Approach

As demonstrated in Fig. 2, the core insight of V3D lies in conceptualizing dense multi-view synthesis as video generation, thus leveraging the structure and powerful priors of the large-scale pre-trained video diffusion model to facilitate consistent multi-view generation. For object-centric image-to-3D, we fine-tune the base video diffusion model on $360\degree$ orbit videos of synthetic 3D objects rendered on fixed circular camera poses (Sec. 3.2) and design a reconstruction and mesh extraction pipeline that is tailored for generated multi-views (Sec. 3.3). For scene-level novel view synthesis, we enhance the base video diffusion model by incorporating a PixelNeRF encoder, enabling it to precisely control camera poses of generated frames and seamlessly adapt to any number of input images (Sec. 3.4). The detailed V3D methodology is presented as follows.

2 Dense View Prediction as Video Generation

Previous multi-view generation models often extend from image diffusion models, incorporating multi-view cross attention (e.g. Wonder3D and Zero123-XL ) or intermediate 3D representations (e.g. SyncDreamer with 3D voxels and Viewset Diffusion and DMV3D with triplane NeRF) to enhance the consistency of generated images across different views. The former tends to involve a limited number of views due to the use of cross-attention with quadratic complexity, whereas the latter often results in low image resolution due to memory-consuming 3D representations and volume rendering. To overcome these drawbacks, V3D adopts a fundamentally distinct approach to generate dense multi-view images given a single view. Inspired by Stable Video Diffusion (SVD) , we interpret continuous multi-view images rotating around an object as a video, thereby treating multi-view generation as a form of image-to-video generation with the front view as the condition. This approach capitalizes on the comprehensive understanding of the 3D world offered by large-scale pre-trained video diffusion models to address the scarcity of 3D data and harnesses the inherent network architecture of video diffusion models to effectively produce an adequate number of multi-view images.

Similar to SVD, we provide high-level semantic information of the front view by injecting the corresponding CLIP embedding into the diffusion U-Net via cross-attention and prompt the diffusion model low-level information by concatenating its latent along the channel dimension of the input. To better adapt to image-to-3D, we removed irrelevant conditions including motion bucket ID and FPS ID, and opted not to condition on elevation angle. Instead, we randomly rotate the object in elevation to make the generation model compatible with inputs with non-zero elevation. Our modification to the conditioning arises from experimental evidence suggesting that it is not feasible to perfectly constrain the elevation of generated multi-view images to a specified angle. The misalignment between the camera poses of the video and those used for reconstruction could lead to significant damage to the final outcomes.

2.2 Data.

We fine-tune the base video diffusion model on the Objaverse dataset for object-centric image-to-3D. It is well-established that the Objaverse dataset contains quite many low-quality 3D objects which may significantly degenerate the model performance . We, therefore, start by manually curating a subset with higher quality, comprising approximately 290k synthetic 3D triangle meshes. We then construct a $360\degree$ orbit video dataset by rendering synthetic 3D objects in this filtered subset on $N=18$ fixed camera poses. Specifically, for each object, we normalize the mesh to unit length, fix the camera distance to 2, set the elevation to 0, and uniformly move the camera on the azimuth $N$ times (from 0 to $2\pi$) to generate a smooth orbit video at 512 resolution.

2.3 Training.

We follow SVD to adopt the commonly used EDM framework with the simplified diffusion loss for fine-tuning. For the noise distribution used in training, we inspired by , adopt a relatively large noise distribution with $P_{\text{mean}}=1.5$ and $P_{\text{std}}=2.0$. To enable classifier-free guidance, we randomly set the latent and the CLIP embedding of the front view to zero independently with a probability of 20%. To speed up training and save GPU VRAM, we preprocess and store the latent and CLIP embedding of all video frames beforehand. During training, we load these tensors directly from the disk instead of computing them on the fly. We fine-tuned the base diffusion model for 22.5k steps, taking about one day. More detailed training settings are provided in the supplemental material.

3 Robust 3D Reconstruction and Mesh Extraction

After obtaining dense views of an object using the fine-tuned video diffusion model, our next step is to reconstruct the underlying 3D object from these views. While the obtained multi-view images are of high resolution and satisfactory consistency, it’s challenging to ensure strict pixel-to-pixel alignment between different views. In this case, directly applying pixel-wise loss for 3D reconstruction often leads to artifacts such as floaters or blurry textures . In response, we propose using image-level perceptual loss to resolve inconsistencies among multiple input images. To support image-level loss, we opt for 3D Gaussian Splatting due to its fast training speed and great memory efficiency that enables rendering the full image with minimal cost. We adopt LPIPS as the perceptual loss and combine it with the original loss for 3DGS, i.e.

where $I$ and $I^{\text{gt}}$ represent the rendered images and the ground truth images (our generated views), $\lambda_{s}$ and $\lambda_{l}$ refer to the loss weights.

Initialization is crucial for 3D Gaussian Splatting to achieve promising results . Due to no sparse point cloud available in object-centric reconstruction, the original 3DGS initializes the Gaussians with Gaussian-blob and removes unnecessary Gaussians by resetting opacities. This requires many optimization iterations to eliminate the floaters, and artifacts may occur when the input images are not perfectly consistent. To accelerate the reconstruction, we propose to initialize the Gaussians by space carving . Concretely, we first segment the foreground of the generated frames with an off-the-shelf background removal model to obtain an object mask for each view. An occupancy grid is then established by projecting voxels onto image planes to determine whether the projected pixel belongs to the object and aggregating the results of all projections under all views. Finally, we exploit marching cubes to obtain a surface according to this occupancy grid and uniformly sample $N_{\text{init}}$ points on the surface as the initializing positions of the Gaussians.

3.2 Mesh Extraction.

For the demand of real-world applications, we also propose a mesh extraction pipeline for generated views. Similar to , we adopt NeuS with multi-resolution hash grid for fast surface reconstruction. While V3D can generate a considerable number of views, it remains modest compared to the typical use cases of NeuS, which commonly involve more than 40 posed images . Therefore we adopt a normal smooth loss and a sparse regularization loss to ensure a reasonable geometry. Besides, since our generated images are not perfectly consistent and it is not feasible for NeuS to exploit image-level loss due to the high render cost, the textures of the mesh extracted by NeuS are often blurry. To address this, we further refine the texture of the NeuS extracted mesh with generated multi-views using LPIPS loss while keeping the geometry unchanged, which greatly reduces the rendering cost and enhances the quality of final outputs. Owing to the great efficiency of differentiable mesh rendering , this refinement process can be completed within 15 seconds.

4 Scene-level Novel View Synthesis

In addition to object-centric generation, we further extend video diffusion to scene-level novel view synthesis. Unlike generating views of objects, novel view synthesis entails generating images along a given camera path, which requires precise control over camera poses and accommodating multiple input images.

Previous approaches, such as Zero-1-to-3 , incorporate camera poses into image diffusion by introducing camera embeddings. However, this method did not explicitly utilize 3D information, thus making it challenging to ensure consistency across multiple views and precise control over camera pose . To better incorporate 3D prior into video diffusion and precisely control the poses of generated frames, we draw inspiration from GeNVS and ReconFusion , integrating a PixelNeRF feature encoder into video diffusion models. Specifically, given a set of posed images as condition $\pi_{\text{cond}},I_{\text{cond}}=\{(I_{i},\pi_{i})\}_{i}$, we adopt a PixelNeRF to render the feature map of the target camera poses $\{\pi\}$ by:

where $\{\pi\}$ refers to the camera poses of the frames we want to generate. We then concatenate this feature map to the input of the U-Net to explicitly encode relative camera pose information. This approach seamlessly supports an arbitrary number of images as input. Other conditionings are similar to those of object-centric generation, except that the number of CLIP image embedding has been increased from one to multiple.

4.2 Data.

For scene-level novel view synthesis, we fine-tune the base video diffusion model on MVImgNet which consists of 219k posed real-world video clips. To make the videos from MVImgNet compatible with the input size of the video diffusion, we obtain the object masks with an off-the-shelf background removal tool, then crop, recenter the image, and adjust the principal points and focal lengths according to the foreground mask.

4.3 Training.

Except for the simplified diffusion loss used in video diffusion models, we follow ReconFusion to regulate the parameters of the PixelNeRF encoder with a down-sampled photometric loss:

where $f_{\text{RGB}}$ refers to another color branch of the PixelNeRF encoder that predicts a downsampled image and $x_{\downarrow}$ refers to the downsampled target frames. We experimentally found this loss crucial to avoid local minima, as illustrated in . We follow InstructPix2Pix , initializing the newly added input channels for the PixelNeRF feature in the first layer of the diffusion U-Net with zero to best keep the pre-trained priors. For input views conditioning, we randomly select 1 to 4 images as input and set the conditions to zeros independently with a probability of 20%. We present more training details of the scene-level model in the supplemental material.

Experiments

In this section, we present our experiments on validating the effectiveness of the proposed approach in both object-centric and scene-level 3D generation. Specifically, we quantitatively and qualitatively compare V3D with previous state-of-the-art methods in image-to-3D and novel view synthesis. Additionally, we conduct several ablation studies to evaluate the importance of model designs, including noise distribution, number of fine-tuning steps, pre-training prior, and camera conditioning. The detailed results are shown as follows.

We evaluate the performance of the proposed V3D on image-to-3D and compare the results with state-of-the-art methods. The upper part of Fig. 3 demonstrates qualitative comparisons between our approach with 3DGS-based methods TriplaneGaussian and LGM . Our approach demonstrates impressive quality improvement, while TriplaneGaussian and LGM tend to generate a much more blurry appearance due to the limited number of Gaussians they can produce. The lower part of Fig. 3 presents a comparison between state-of-the-art SDS-based methods, including Magic123 and ImageDream . V3D outperforms in both front-view alignment and fidelity. In contrast, objects generated by Magic123 exhibit inaccurate geometry and blurry backface, while ImageDream tends to generate assets with over-saturated textures. Importantly, our method achieves these results within 3 minutes, making significant acceleration compared to optimization-based methods which require more than 30 minutes. Fig. 4 illustrates comparisons on multi-view generation with SyncDreamer and Wonder3D , where our approach consistently achieves superior results with a higher resolution of 512 and much finer details. SyncDreamer tends to generate multi-view images that are simple in geometry and with over-saturated texture. We attribute this to the significant difference in structure between SyncDreamer and the original Stable Diffusion, thus failing to fully leverage the information from image pre-training. As for Wonder3D, our model triples the number of generated multi-views and doubles the resolution, achieving significant improvements in both texture quality and multi-view consistency.

1.2 Quantitative Comparisons.

For qualitative comparisons, we conduct a user study to evaluate the generated 3D assets with human ratings. Specifically, we ask 58 volunteers to evaluate objects generated by V3D and other methods under 30 condition images by watching the rendered $360\degree$ spiral videos, and choose the best method in (a) Alignment: how well does the 3D asset align with the condition image; (b) Fidelity: how realistic is the generated object. Tab. 2 demonstrates the win rate of each method in the two criteria. V3D is regarded as the most convincing model of all, and it significantly outperforms other competitors in both image alignment and fidelity.

2 Novel View Synthesis

For scene-level novel view synthesis, we test the performance of the proposed V3D on the 10-category subset of the CO3D dataset. To align with the settings of previous methods, we fine-tune V3D on videos in each category for only one epoch (denoted as “fine-tuned”). The results are shown in Tab. 3. Our approach achieves consistently better performance in terms of image metrics against previous state-of-the-art novel view synthesizers by a clear margin, demonstrating the effectiveness of utilizing pre-trained video diffusion models in scene-level novel view synthesis. Besides, V3D shows impressive generalization ability as the zero-shot version of V3D (only trained on MVImgNet) beats most of the competitors.

We show in Fig. 5 qualitative comparisons on generated multi-views between SparseFusion on the hydrant subset of the CO3D dataset . For better comparison, we use COLMAP to perform multi-view stereo reconstruction given camera poses and showcased, in Fig. 5, the number of points in the obtained point cloud and the Chamfer distance between the one reconstructed from ground truth images. The point clouds reconstructed from V3D generated images contain more points and are closer to the point clouds reconstructed from real images. This indicates a significant advantage of our method in terms of both reconstruction quality and multi-view consistency.

3 Ablations

To demonstrate the impact of large-scale video pre-training, we trained a variant of the model from scratch on the constructed Objaverse $360\degree$ orbit video for 45k steps, which, apart from the initialized weights of the diffusion U-Net, is identical to the full model. As shown in Fig. 6, this from-scratched variant completely failed to generate multi-view images even with doubled training steps compared to the full model, demonstrating the significant importance of adopting large-scale video pre-training.

3.2 Number of Fine-tuning Steps.

To illustrate the impact of the Objaverse dataset on the pre-trained model, we fine-tune the base model with different training steps, as shown in the top part of Fig. 6. Due to the considerably lower complexity of renders from Objaverse objects, excessive fine-tuning steps can yield overly simplistic textures in the generated frames, while an inadequate number of fine-tuning steps may lead to strange and inaccurate geometries.

3.3 Noise Distribution.

To validate the necessity of adopting a large noise distribution during training, we fine-tune the base diffusion model with a smaller noise schedule ($P_{\text{mean}}=0.7$ and $P_{\text{std}}=1.6$), which is used in the image-to-video pre-training in Stable Video Diffusion. As demonstrated in the lower part of Fig. 6, the model trained with a smaller noise distribution tends to generate geometric structures that are unreasonable (the pistol) or degenerated (the cartoon-style astronaut). We attribute this phenomenon to the strong image signal we exploited and the relatively simple nature of the synthetic Objaverse dataset, indicating the need to shift the noise distribution towards stronger noise.

3.4 Camera Conditioning.

To assess the impact of the approach of camera conditioning, we introduce two variants that condition on camera poses by commonly used learnable camera embedding or the Plücker ray embedding. For a fair comparison, for both variants, we additionally concatenate the multi-view images used as conditions into the input of the diffusion U-Net and similarly set the weights of added channels to zero to minimize the gap. As shown in Tab. 3, either camera embedding or Plücker embedding provides accurate 3D information and results in a significant degeneration in model performance.

Limitations and Conclusion

Although achieves state-of-the-art performance in 3D generation, V3D would produce unsatisfying results for some complex objects or scenes, such as inconsistency among multiple views or unreasonable geometries. Concrete failure cases and analyses are discussed in the supplemental material.

0.2 Conclusion.

In this paper, we propose V3D, a novel method for generating consistent multi-view images with image-to-video diffusion models. By fine-tuning the base video diffusion model on 3D datasets, we extend video diffusion models to effective 3D generators. Specifically, for object-centric 3D generation, we fine-tune the video diffusion on synthesizing $360\degree$ videos of 3D objects to predict dense views given a single image. To obtain the underlying 3D asset from generated views, we propose a tailored reconstruction pipeline with designed initialization and texture refinement, enabling the reconstruction of high-quality 3D Gaussians or delicate textured meshes within 3 minutes. We further extend our framework to scene-level novel view synthesis, achieving precise control over the camera path with great multi-view consistency. We conduct extensive experiments to validate the effectiveness of the proposed approach, illustrating its remarkable performance in generating consistent multi-views and generalization ability. We hope our method can serve as an efficient and powerful approach for high-quality 3D generation and could pave the way for more extensive applications of video diffusion models in 3D tasks.

References