Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning

Introduction

Large text-to-image models trained on web-scale image-text pairs generate diverse and high quality images. While these models can be further adapted for text-to-video (T2V) generation by using video-text pairs, video generation still lags behind image generation in terms of quality and diversity. Compared to image generation, video generation is more challenging as it requires modeling a higher dimensional spatiotemporal output space while still being conditioned only on a text prompt. Moreover, video-text datasets are typically an order of magnitude smaller than image-text datasets .

The dominant paradigm in video generation uses diffusion models to generate all video frames at once. In stark contrast, in NLP, long sequence generation is formulated as an autoregressive problem : predicting one word conditioned on previously predicted words. Thus, the conditioning signal for each subsequent prediction progressively gets stronger. We hypothesize that strengthening the conditioning signal is also important for high quality video generation, which is inherently a time-series. However, autoregressive decoding with diffusion models is challenging since generating a single frame from such models itself requires many iterations.

We propose Emu Video to strengthen the conditioning for diffusion based text-to-video generation with an explicit intermediate image generation step. Specifically, we factorize text-to-video generation into two subproblems: (1) generating an image from an input text prompt; (2) generating a video based on the stronger conditioning from the image and the text. Intuitively, giving the model a starting image and text makes video generation easier since the model only needs to predict how the image will evolve in the future.

Since video-text datasets are much smaller than image-text datasets, we also initialize our factorized text-to-video model using a pretrained text-to-image (T2I) model whose weights are kept frozen. We identify critical design decisions–changes to the diffusion noise schedule and multi-stage training–to directly generate videos at a high resolution of $512$px. Unlike direct T2V methods , at inference, our factorized approach explicitly generates an image, which allows us to easily retain the visual diversity, style, and quality of the text-to-image model (examples in Fig. 1). This allows Emu Video to outperform direct T2V methods, even when accounting for the same amount of training data, compute, and trainable parameters.

Contributions. We show that text-to-video (T2V) generation quality can be greatly improved by factorizing the generation into first generating an image and using the generated image and text to generate a video. Our multi-stage training enables us to directly generate videos at a high resolution of $512$px, bypassing the need for a deep cascade of models used in prior work . We design a robust human evaluation scheme–JUICE–where we ask evaluators to justify their choices when making the selection in the pairwise comparisons. As shown in Fig. 2, Emu Video significantly surpasses all prior work including commercial solutions with an average win rate of $91.8\%$ for quality and $86.6\%$ for text faithfulness. Beyond T2V, Emu Video can be used out-of-the-box for image-to-video where the model generates a video based on a user-supplied image and a text prompt. In this setting, Emu Video’s generations are preferred $96\%$ of the times over VideoComposer .

Related Work

Text-to-Image (T2I) diffusion models. Diffusion models are a state-of-the-art approach for T2I generation, and out-perform prior GAN or auto-regressive methods . Diffusion models learn a data distribution by gradually denoising a normally distributed variable, often called ‘noise’, to generate the output. Prior work either denoises in the pixel space with pixel diffusion models , or in a lower-dimensional latent space with latent diffusion models . In this work, we leverage latent diffusion models for video generation.

Video generation/prediction. Many prior works target the constrained settings of unconditional generation, or video prediction . These approaches include training VAEs , auto-regressive models , masked prediction , LSTMs , or GANs . However, these approaches are trained/evaluated on limited domains. In this work, we target the broad task of open-set T2V generation.

Text-to-Video (T2V) generation. Most prior works tackle T2V generation by leveraging T2I models. Several works take a training-free approach for zero-shot T2V generation by injecting motion information in the T2I models. Tune-A-Video targets one-shot T2V generation by fine-tuning a T2I model with a single video. While these methods require no or limited training, the quality and diversity of the generated videos is limited.

Many prior works instead improve T2V generation by learning a direct mapping from the text condition to the generated videos by introducing temporal parameters to a T2I model . Make-A-Video utilizes a pre-trained T2I model and the prior network of to train T2V generation without paired video-text data. Imagen Video builds upon the Imagen T2I model with a cascade of diffusion models . To address the challenges of modeling the high-dimensional spatiotemporal space, several works instead train T2V diffusion models in a lower-dimensional latent space , by adapting latent diffusion T2I models. Blattmann et al. freeze the parameters of a pre-trained T2I model and train new temporal layers, whilst Ge et al. build on and design a noise prior tailored for T2V generation. The limitation of these approaches is that learning a direct mapping from text to the high dimensional video space is challenging. We instead strengthen our conditioning signal by taking a factorization approach. Unlike prior work that enhancing the conditions for T2V generation including leveraging large language models (LLMs) to improve textual description and understanding , or adding temporal information as conditions , our method does not require any models to generate the conditions as we use the first frame of a video as the image condition.

Factorized generation. The most similar works to Emu Video, in terms of factorization, is CogVideo and Make-A-Video . CogVideo builds upon the pretrained T2I model for T2V generation using auto-regressive Transformer. The auto-regressive nature is fundamentally different to our explicit image conditioning in both training and inference stages. Make-A-Video leverages the image embedding condition learnt from a shared image-text space. Our factorization leverage the first frame as is, which is a stronger condition. Moreover, Make-A-Video initializes from a pretrained T2I model but finetunes all the parameters so it cannot retain the visual quality and diversity of the T2I model as we do.

The goal of text-to-video (T2V) generation is to construct a model $f$ that takes as input a text prompt $\mathbf{p}$ to generate a video $\mathbf{V}$ consisting of $T$ RGB frames. Recent methods directly generate the $T$ video frames at once using text-only conditioning. Our approach builds on the hypothesis that stronger conditioning by way of both text and image can improve video generation (cf. § 3.2).

Conditional Diffusion Models are a class of generative models that are trained to generate the output using a conditional input $\mathbf{c}$ by iteratively denoising from gaussian noise. At training time, time-step $t\in[0,N]$ dependent gaussian noise $\bm{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{1})$ is added to the original input signal $\mathbf{X}$ to obtain a noisy input $\mathbf{X}_{t}=\alpha_{t}\mathbf{X}+\sqrt{1-\alpha_{t}}\bm{\epsilon}_{t}$. $\alpha_{t}$ defines the “noise schedule”, i.e., noise added at timestep $t$ and $N$ is the total number of diffusion steps. The diffusion model is trained to denoise $\mathbf{X}_{t}$ by predicting either $\bm{\epsilon}_{t}$, $\mathbf{X}$, or $v_{t}=\alpha_{t}\bm{\epsilon}_{t}-\sqrt{1-\alpha_{t}}\mathbf{X}$ (called v-prediction ). The signal-to-noise ratio (SNR) at timestep $t$ is given by $(\frac{\alpha_{t}}{1-\alpha_{t}})^{2}$ and decreases as $t\rightarrow N$. At inference, samples are generated by starting from pure noise $\mathbf{X}_{N}\sim\mathcal{N}(\mathbf{0},\mathbf{1})$ and denoising it. Note that at inference time $\mathbf{X}_{N}$ has no signal, i.e., zero SNR which has significant implications for video generation as we describe in § 3.2.

2 Emu Video

Image conditioning. We condition on the starting frame, $\mathbf{I}$, by concatenating it with the noise. Our design choice allows the model to use all the information in $\mathbf{I}$ unlike other choices that lose image information by using a semantic image embedding for conditioning. We represent $\mathbf{I}$ as a single-frame video, i.e., $T=1$ and zero-pad it to obtain a $T\times C\times H\times W$ tensor. We use a binary mask $\mathbf{m}$ of shape $T\times 1\times H\times W$ that is set to $1$ at the first temporal position to indicate the position of the starting frame, and zero otherwise. The mask $\mathbf{m}$, starting frame $\mathbf{I}$, and the noised video $\mathbf{X}_{t}$ are concatenated channel-wise as the input to the model.

Model. We initialize our latent diffusion model $\mathcal{F}$ using the pretrained T2I model . Like prior work , we add new learnable temporal parameters: a $1$D temporal convolution after every spatial convolution, and a $1$D temporal attention layer after every spatial attention layer. The original spatial convolution and attention layers are applied to each of the $T$ frames independently and are kept frozen. The pretrained T2I model is already text conditioned and combined with the image conditioning described above, $\mathcal{F}$ is conditioned on both text and image.

Zero terminal-SNR noise schedule. We found that the diffusion noise schedules used in prior work have a train-test discrepancy which prevents high quality video generation (reported for images in ). At training, the noise schedule leaves some residual signal, i.e., has non-zero signal-to-noise (SNR) ratio even at the terminal diffusion timestep $N$. This prevents the diffusion model from generalizing at test time when we sample from random gaussian noise with no signal about real data. The residual signal is higher for high resolution video frames, due to redundant pixels across both space and time. We resolve this issue by scaling the noise schedule and setting the final $\alpha_{N}=0$ , which leads to zero SNR at the terminal timestep $N$ during training too. We find that this design decision is critical for high resolution video generation.

Interpolation model. We use an interpolation model $\mathcal{I}$, architecturally the same as $\mathcal{F}$, to convert a low frame-rate video of $T$ frames into a high frame-rate video of $T_{p}$ frames. The interpolation model operates on $T_{p}\times C\times H\times W$ inputs/outputs. For frame conditioning, the input $T$ frames are zero-interleaved to produce $T_{p}$ frames, and a binary mask $\mathbf{m}$ indicating the presence of the $T$ frames are concatenated to the noised input (similar to the image conditioning for $\mathcal{F}$). The model is trained on video clips of $T_{p}$ frames of which $T$ frames are fed as input. For efficiency, we initialize $\mathcal{I}$ from $\mathcal{F}$ and only train the temporal parameters of the model $\mathcal{I}$ for the interpolation task.

Simplicity in implementation. Emu Video can be trained using standard video-text datasets, and does not require a deep cascade of models, e.g., $7$ models in , for generating high resolution videos. At inference, given a text prompt, we run $\mathcal{F}$ without the temporal layers to generate an image $\mathbf{I}$. We then use $\mathbf{I}$ and the text prompt as input to $\mathcal{F}$ to generate $T$ video frames, directly at high resolution. We can increase the fps of the video using $\mathcal{I}$. Since the spatial layers are initialized from a pretrained T2I model and kept frozen, our model retains the conceptual and stylistic diversity learned from large image-text datasets, and uses it to generate $\mathbf{I}$. This comes at no additional training cost unlike approaches that do joint finetuning on image and video data to maintain such style. Many direct T2V approaches also initialize from a pretrained T2I model and keep the spatial layers frozen. However, they do not employ our image-based factorization and thus do not retain the quality and diversity in the T2I model.

Robust human evaluation (JUICE). Similar to recent studies , we find that the automatic evaluation metrics do not reflect improvements in quality. We primarily use human evaluation to measure T2V generation performance on two orthogonal aspects - (a) video generation quality denoted as Quality (Q) and (b) the alignment or ‘faithfulness’ of the generated video to the text prompt, denoted as Faithfulness (F). We found that asking human evaluators to JUstify their choICE (JUICE) when picking a generation over the other significantly improves the inter-annotator agreement (details in Appendix C). The annotators select one or more pre-defined reasons to justify their choice. The reasons for picking one generation over the other for Quality are: pixel sharpness, motion smoothness, recognizable objects/scenes, frame consistency, and amount of motion. For Faithfulness we use two reasons: spatial text alignment, and temporal text alignment.

3 Implementation Details

We provide complete implementation details in the supplement Appendix A and highlight salient details next.

Architecture and initialization. We adapt the text-to-image U-Net architecture from for our model and initialize all the spatial parameters with the pretrained model. The pretrained model produces square $512$px images using an $8$ channel $64\times 64$ latent as the autoencoder downsamples spatially by $8\times$. The model uses both a frozen T5-XL and a frozen CLIP text encoder to extract features from the text prompt. Separate cross-attention layers in the U-Net attend to each of the text features. After initialization, our model contains $2.7$B spatial parameters which are kept frozen, and $1.7$B temporal parameters that are learned.

The temporal parameters are initialized as identity operations: identity kernels for convolution, and zeroing the final MLP layer of the temporal attention block. In our preliminary experiments, the identity initialization improved the model convergence by $2\times$. For the additional channels in the model input due to image conditioning, we add $C+1$ additional learnable channels (zero-initialized) to the kernel of the first spatial convolution layer. Our model produces $512$px square videos of $T=8$ or $16$ frames and is trained with square center-cropped video clips of $1$, $2$ or $4$ seconds sampled at $8$fps or $4$fps. We train all our models with a batch size of $512$ and describe the details next.

Efficient multi-stage multi-resolution training. To reduce the computational complexity, we train in two stages - (1) for majority of the training iterations ($70$K) we train for a simpler task: $256$px $8$fps $1$s videos, which reduces per-iteration time by $3.5\times$ due to the reduction in spatial resolution; (2) we then train the model at the desired $512$px resolution on $4$fps $2$s videos for $15$K iterations. The change in spatial resolution does not affect the $1$D temporal layers. Although the frozen spatial layers were pretrained at $512$px, changing the spatial resolution at inference to $256$px led to no loss in generation quality. We use the noise schedule from for $256$px training, and with zero terminal-SNR for $512$px training using the v-prediction objective with $N=1000$ steps for the diffusion training. We sample from our models using $250$ steps of DDIM . Optionally, to increase duration, we further train the model on $16$ frames from a $4$s video clip for $25$K iterations.

Finetuning for higher quality. Similar to the observation in image generation , we find that the motion of the generated videos can be improved by finetuning the model on a small subset of high motion and high quality videos. We automatically identify a small finetuning subset of $1.6$K videos from our training set which have high motion (computed using motion signals stored in H.264 encoded videos). We follow standard practice and also apply filtering based on aesthetic scores and CLIP similarity between the video’s text and first frame.

Interpolation model. We initialize the interpolation model from the video model $\mathcal{F}$. Our interpolation model takes $8$ frames as input and outputs $T_{p}\!=\!37$ frames at $16$fps. During training, we use noise augmentation where we add noise to the frame conditioning by randomly sampling timesteps $t\in\{0,...250\}$. At inference time, we noise augment the samples from $\mathcal{F}$ with $t=100$.

Dataset. We train Emu Video on a dataset of $34$M licensed video-text pairs Our videos are $5$-$60$ seconds long and cover a variety of natural world concepts. The videos were not curated for a particular task and were not filtered for text-frame similarity or aesthetics. Unless noted, we train the model on the full set, and do not use the $1.6$K high motion quality finetuning subset described in § 3.3.

Text prompt sets for human evaluation. We use the text prompt sets from prior work (cf. Appendix Tab. 10) to generate videos. The prompts cover a wide variety of categories that can test our model’s ability to generate natural and fantastical videos, and compose different visual concepts. We use our proposed JUICE evaluation scheme ( Sec. 3) for reliable human evaluation and use the majority vote from $5$ evaluators for each comparison.

We study the effects of our design decisions using the $8$ frame generation setting and report human evaluation results in Tab. 1 using pairwise comparisons on the $307$ prompt set of .

Factorized vs. Direct generation. We compare our factorized generation to a direct T2V generation model that generates videos from text condition only. We ensure that the pretrained T2I model, training data, number of training iterations, and trainable parameters are held constant for this comparison. As shown in LABEL:tab:ablate_keyframe_t2v_vs_t2i2v, the factorized generation model’s results are strongly preferred both in Quality and Faithfulness.The strong preference in Quality is because the direct generation model does not retain the style and quality of the text-to-image model despite frozen spatial parameters, while also being less temporally consistent (examples in Fig. 4).

Zero terminal-SNR noise schedule. We compare using zero terminal-SNR for the high resolution $512$px training against a model that is trained with the standard noise schedule. LABEL:tab:ablate_keyframe_zero_snr shows that generations using zero terminal-SNR are strongly preferred. This suggests that the zero terminal-SNR noise schedule’s effect of correcting the train-test discrepancy as described in § 3.2 is critical for high resolution video generation. We also found that zero terminal-SNR has a stronger benefit for our factorized generation compared to a direct T2V model possibly. Similar to images , in the direct T2V case, this decision primarily affects the color composition. For our factorized approach, this design choice was critical for object consistency and high quality as our qualitative results in Fig. 4 show.

Multi-stage multi-resolution training. We spend most training budget ($4\!\times\!$) on the $256$px $8$fps stage compared to the $3.5\!\times\!$ slower (due to increased resolution) $512$px $4$fps stage. We compare to a baseline that trains only the $512$px stage with the same training budget. LABEL:tab:ablate_keyframe_multi_stage shows that our multi-stage training yields significantly better results.

High quality finetuning. We study the effect of finetuning our model on automatically identified high quality videos in LABEL:tab:ablate_keyframe_finetuning. We found that this finetuning improves on both metrics. In particular, finetuning improves the model’s ability to respect the motion specified in the text prompt as reflected by the strong gain in Faithfulness.

Parameter freezing. We test if freezing the spatial parameters of our model affects performance. We compare against a model where all parameters are finetuned during the second $512$px training stage. For fair comparison, we use the same conditioning images $\mathbf{I}$ across our model and this baseline. LABEL:tab:ablate_keyframe_parameter_freezing suggests that freezing the spatial parameters produces better videos, while reducing training cost.

2 Comparison to prior work

We evaluate Emu Video against prior work and train $\mathcal{F}$ to produce $16$ frame $4$ second long videos and use the best design decisions from § 4.1, including high quality finetuning. We use the interpolation model $\mathcal{I}$ on our generations to get $16$fps videos. Please see Appendix A for details on how we interpolate 16-frame videos with $\mathcal{I}$.

Human evaluation of text-to-video generation. Since many recent prior methods in text-to-video generation are closed source , we use the publicly released examples from each of these methods. Note that the released videos per method are likely to be the ‘best’ representative samples from each method and may not capture their failure modes. For Make-A-Video, we obtained non cherry-picked generations through personal communication with the authors. For CogVideo , we perform T2V generation on the prompt set from using the open source models. We also benchmark against commercially engineered black-box text-to-video solutions, Gen2 and PikaLabs , for which we obtain generations through their respective websites using the prompts from . We do not cherry-pick or contrastively rerank our videos, and generate them using a deterministic random noise seed that is not optimized in any way.

Since each method generates videos at different resolutions, aspect-ratios, and frame-rates, we reduce annotator bias in human evaluations by postprocessing the videos for each comparison in Fig. 2 so that they match in these aspects. Full details on this postprocessing and the text prompts used are in Appendix D. As shown in Fig. 2, Emu Video’s generations significantly outperform all prior work, including commercial solutions, both in terms of Quality (by an average of $91.8\%$) and Faithfulness (by an average of $86.6\%$). We show some qualitative comparisons in Fig. 5 and some additional generations in Fig. 1. Emu Video generates videos with significantly higher quality, and overall faithfulness to both the objects and motion specified in the text. Since our factorized approach explicitly generates an image, we retain the visual diversity and styles of the T2I model, leading to far better videos on fantastical and stylized prompts. Additionally, Emu Video generates videos with far greater temporal consistency than prior work. We hypothesize that since we use stronger conditioning of image and text, our model is trained with a relatively easier task of predicting how an image evolves into the future, and thus is better able to model the temporal nature of videos. Please see Appendix E for more qualitative comparisons. We include human evaluations where videos are not postprocessed in the supplement Appendix D, where again Emu Video’s generations significantly outperform all prior work. The closest model in performance compared to ours is Imagen Video when measured on Faithfulness, where we outperform Imagen Video by $56\%$. Imagen Video’s released prompts ask for generating text characters, a known failure mode of latent diffusion models used in Emu Video.

We inspect the reasons that human evaluators prefer Emu Video generations over the two strongest competitors in Fig. 6. A more detailed inspection is provided in Appendix C. Emu Video generations are preferred due to their better pixel sharpness and motion smoothness. While being state-of-the-art, Emu Video is also simpler and has a two model cascade with a total of $6.0$B parameters ($2.7$B frozen parameters for spatial layers, and $1.7$B learnable temporal parameters each for $\mathcal{F}$ and $\mathcal{I}$), which is much simpler than methods like Imagen Video ($7$ model cascade, $11.6$B parameters), Make-A-Video ($5$ model cascade, $9.6$B parameters) trained using similar scale of data.

Automated metrics. In Tab. 2, we compare against prior work using the zero-shot T2V generation setting from on the UCF101 dataset . Emu Video achieves a comptetitive IS score and a higher FVD . Prior works suggest that the automated metrics are flawed and do not capture human preferences . We believe FVD penalizes our high quality generations that are different from the UCF101 videos, while IS is biased towards its training data . To confirm this, we use human evaluations to compare our generations to Make-A-Video. We use a subset of $303$ generated videos ($3$ random samples per UCF101 class) and find that our generations are strongly prefered ( Tab. 2 Right). Qualitative comparisons can be found in Appendix E.

Animating images. A benefit of our factorized generation is that the same model can be used out-of-the-box to ‘animate’ user-provided images by supplying them as the conditioning image $\mathbf{I}$. We compare Emu Video’s image animation with three methods, concurrent work and commercial image-to-video (I2V) solutions , on the prompts from and . All the methods are shown the same image generated using a different text-to-image model and expected to generate a video according to the text prompt. We use the API for in our comparisons since the official training data and model is not available. We report human evaluations in Tab. 3 and automated metrics in the supplement (cf. Appendix Tab. 9). Human evaluators strongly prefer Emu Video’s generations across all the baselines. These results demonstrate the superior image animation capabilities of Emu Video compared to methods specifically designed for the image-to-video task.

3 Analysis

Nearest neighbor baseline. We expect good and useful generative models to outperform a nearest neighbor retrieval baseline and create videos not in the training set. We construct a strong nearest neighbor baseline that retrieves videos from the full training set ($34$M videos) by using the text prompt’s CLIP feature similarity to the training prompts. When using the evaluation prompts from , human evaluators prefer Emu Video’s generations $81.1\%$ in Faithfulness over real videos confirming that Emu Video outperforms the strong retrieval baseline. We manually inspected and confirmed that Emu Video outperforms the baseline for prompts not in the training set.

Extending video length with longer text. Recall that our model conditions on the text prompt and a starting frame to generate a video. With a small architectural modification, we can also condition the model on $T$ frames and extend the video. Thus, we train a variant of Emu Video to generate the future $16$ frames conditioned on the ‘past’ $16$ frames. While extending the video, we use a future text prompt different from the one used for the original video and visualize results in Fig. 7. We find that the extended videos respect the original video as well as the future text prompt.

We presented Emu Video, a factorized approach to text-to-video generation that leverages strong image and text conditioning. Emu Video significantly outperforms all prior work including commercial solutions. There is a difference in the image conditioning used for our model at train and inference: at training, we use a video frame sampled from real videos, while at inference we use a generated image (using the spatial parameters of the model). In practice, this difference does not affect the quality of the generated video for most scenarios. However, in cases where the generated image used for conditioning at inference is not representative of the prompt, our model has no way to recover from this error. We believe that improving the models ability to recover from such errors is an important direction for future work. Strengthening the conditioning for video models using pure autoregressive decoding with diffusion models is not currently computationally attractive. However, further research may provide benefits for longer video generation.

We propose advancements in generative methods specifically to improve the generation of high dimensional video outputs. Generative methods can be applied to a large variety of different usecases which are beyond the scope of this work. A careful study of the data, model, its intended applications, safety, risk, bias, and societal impact is necessary before any real world application.

Acknowledgments. We are grateful for the support of multiple collaborators at Meta who helped us in this work. Baixue Zheng, Baishan Guo, Jeremy Teboul, Milan Zhou, Shenghao Lin, Kunal Pradhan, Jort Gemmeke, Jacob Xu, Dingkang Wang, Samyak Datta, Guan Pang, Symon Perriman, Vivek Pai, Shubho Sengupta for their help with the data and infra. We would like to thank Uriel Singer, Adam Polyak, Shelly Sheynin, Yaniv Taigman, Licheng Yu, Luxin Zhang, Yinan Zhao, David Yan, Emily Luo, Xiaoliang Dai, Zijian He, Peizhao Zhang, Peter Vajda, Roshan Sumbaly, Armen Aghajanyan, Michael Rabbat, and Michal Drozdzal for helpful discussions. We are also grateful to the help from Lauren Cohen, Mo Metanat, Lydia Baillergeau, Amanda Felix, Ana Paula Kirschner Mofarrej, Kelly Freed, Somya Jain. We thank Ahmad Al-Dahle and Manohar Paluri for their support.

Appendix A Implementation Details

In this section we include details on the architectures and hyper-parameters used for training the models in the main paper, and on the use of multiple conditionings for classifier-free guidance. For both our text-to-video ($\mathcal{F}$) and interpolation ($\mathcal{I}$) models we train with the same U-Net architecture. We share the exact model configuration for our U-Net in Tab. 4, and the configuration for our 8-channel autoencoder in Tab. 5.

Tab. 6 shares the training hyperparameters we used for various stages of our training – 256px training, 512px training, High Quality finetuning, and frame interpolation. For inference, we use the DDIM sampler with 250 diffusion steps. We use Classifier Free Guidance (CFG) with $w_{img}$ of 7.5 for image generation, and $w_{img}$ of 2.0 and $w_{txt}$ of 7.5 for both video generation and frame interpolation. We share more details about handling multiple conditionings for Classifier Free Guidance next.

Multiple Conditionings for CFG. For video generation, our model receives two conditioning signals (image $\mathbf{I}$, text prompt $\mathbf{p}$), which we use in conjunction for Classifier Free Guidance . Eq. 1 lists the combined CFG equation we use.

In Eq. 1 there is an ordering on the conditionings. We also considered alternate orderings in which we start with the text conditioning first instead of the image conditioning:

Eq. 2 did not lead to improvement over Eq. 1, but required significantly different values for $w_{i}$ and $w_{p}$ to work equally well. We also considered formulas without ordering between the two conditionings, for instance:

Similar to Eq. 2, those formulas did not improve over Eq. 1, and in addition miss the useful properties listed above.

Selecting CFG scales. Eq. 1 requires to find the guidance factor $w_{i}$ for image and $w_{p}$ for text. We found that these factors influence the motion in the generated videos. To quantify this, we measure a ‘motion score’ on the generated videos by computing the mean energy of the motion vectors in the resulting H.264 encoding. We found that the motion score was a good proxy for the amount of motion, but did not provide signal into consistency of the motion. Higher motion as computed through motion vectors does not necessarily translate to interesting movement, as it could be undesirable jitter, or reflect poor object consistency. Tab. 7 shows how the CFG scales directly influence the amount of motion in the generated videos.

After narrowing down a few CFG value combinations by looking at the resulting motion score, we identified the best values by visual inspection and human studies. Qualitatively, we found that the (1) higher $w_{i}$ for a fixed $w_{p}$, the more the model stays close to the initial image and favors camera motion; and (2) the higher $w_{p}$ for a fixed $w_{i}$, the more the model favors movement at the expense of object consistency.

Frame Interpolation Model. Here, we include extra details on the frame interpolation model, $\mathcal{I}$. First we explain our masked zero-interleaving strategy. Second we explain how we interpolate 16-frame 4fps videos from $\mathcal{F}$. § 3.3 in the main paper details how $\mathcal{I}$ is trained to take 8 zero-interleaved frames (generated from $\mathcal{F}$ at 4fps) as conditioning input and generate 37 frames at 16fps. One option for training an interpolation model that increases the fps by 4-fold is to generate 3 new frames between every pair of input frames (as in ). However, the downside to this approach is that the resulting interpolated video has a slightly shorter duration than the input video (since every input frame has 3 new generated frames after it, except the last input frame). We instead take the approach of using $\mathcal{I}$ to increase the duration of the input video, and we design a zero-interleaving scheme accordingly. Our interpolation model is trained to generate 3 new frames between every pair of frames, and also 4 new frames either side of the input video. As a result, during training $\mathcal{I}$ takes as conditioning input a 2s video, and generates a 2.3s video.

For interpolating 16-frame input videos from $\mathcal{F}$ (as described in § 4.2 in the main paper), we simply split the videos into two 8-frame videos and run interpolation on both independendly. In order to construct our final interpolated video, we discard the overlapping frames (the last 5 frames of the first interpolated video, and the first 4 of the second), and concatenate the two videos frame-wise. The resulting interpolated video is 65 frames long at 16fps (4.06 seconds in duration – we refer to these videos as 4 seconds long in the main paper for brevity).

Appendix B Additional experiments

We detail additional experiments, viz. (i) an investigation into the effect of the initial image on our video generations, (ii) a quantitative comparison to prior work in image animation with automated metrics, (iii) a joint investigation into the effect of the number of training steps and data, and finally (iv) an analysis into the effect of the amount of training data.

Image conditioning for commercial T2V systems. We study the effect of image conditioning on the commercial T2V solution from Gen2 in Tab. 8. The Gen2 API has two video generation variants: (1) A pure T2V API that accepts a text prompt as input and generates a video; and (2) an ”image + text” API, denoted as Gen2 I2V, that takes an image and a text prompt as input to generate a video. We use images generated from for the Gen2 I2V variant.

We observe that the Gen2 I2V variant outperforms the Gen2 API that only accepts a text prompt as input. We benchmark Emu Video against both variants of the API and observe that it outperforms Gen2 and the stronger Gen2 I2V API. In Tab. 3, we also compare Emu Video using the same images as Gen2 I2V for “image animation” and observe that Emu Video outperforms Gen2 I2V in that setting as well.

Automated metrics for image animation. We follow the setting from Tab. 3 and report automated metrics for comparison in Tab. 9. Following , we report Frame consistency (FC) and Text consistency (TC). We also report CLIP Image similarity (IC) to measure the fidelity of generated frames to the conditioned image. We use CLIP ViT-B/32 model for all the metrics. Compared to VideoComposer , Emu Video generates smoother motion, as measure by frame consistency, maintains a higher faithfulness to the conditioned image, as measured by the image score, while adhering to the text on both the prompt sets. Emu Video fares slightly lower compared to PikaLabs and Gen2 on all three metrics. Upon further inspection, Emu Video (motion score of 4.98) generates more motion compared to PikaLabs and Gen2 (motion scores of 0.63 and 3.29 respectively). Frame and image consistency favour static videos resulting in the lower scores of Emu Video on these metrics.

Effect of the number of training steps and data. In Fig. 8, we vary the number of training steps in the initial low-resolution high-FPS pretraining stage. Note that since we run one full epoch through the data during this training stage, reducing the steps correspondingly also reduces the amount of training data seen. We finetune each of these models at higher resolution/low FPS ($512$px, $4$fps) for the same (small) number of steps – $15$K. We compare the model trained with $100\%$ low-resolution pretraining with models with less low-resolution pretraining using human evaluations. We observe a gradual drop in performance as we reduce the low-resolution pretraining iterations to $75\%$, $50\%$ and $25\%$, indicating the importance of that stage.

Effect of the amount of training data. In Fig. 9, we vary the amount of training data, while keeping the training iterations fixed for both the training stages, and perform a similar comparison as in Fig. 8. Here we find a much smaller drop in performance as we reduce the amount of data. This suggests that Emu Video can be trained effectively with relatively much smaller datasets, as long as the model is trained long enough (in terms of training steps).

Appendix C Human evaluations

We rely on human evaluations for making quantitative comparisons to prior work. In Sec. 4 in the main paper, we introduce our method for robust human evaluations. We now give extra details on this method, termed JUICE, and analyse how it improves robustness, and explain how we ensure fairness in the evaluations. Additionally, in Tab. 10 we summarize the prompt datasets used for evaluations.

When comparing to prior work, we use human evaluations to compare the generations from pairs of models. Unlike the naive approach, where evaluators simply pick their choice from a pair of generations, we ask the evaluators to select a reason when making their choice. We call this approach JUICE, where evaluators are asked to ‘justify your choice’. We show an example of the templates used for human evaluations for both video quality and text faithfulness in Figure 10, where the different possible justifying reasons are shown. One challenge faced when asking evaluators to justify their choice is that human evaluators who are not experts in video generation may not understand what is meant by terms such as “Object/scene consistency” or “Temporal text alignment” or may have subjective interpretations, which would reduce the robustness of the evaluations. To alleviate this challenge, for each justifying option we show the human evaluators examples of generated video comparisons where each of the factors could be used is used in determining a winner. It is important that when giving human evaluators training examples such as these that we do not bias them towards Emu Video’s generations over those of prior work. Thus, to ensure fairness in the comparisons, we make sure that these training examples include cases where generated videos from different prior works are superior to Emu Video and vice-versa. As detailed in the main paper, for each comparison between two videos from two different models, we use the majority vote from 5 different human evaluators. To further reduce annotator bias we make sure that the relative positioning of the generated videos being shown to the human evaluators is randomized. For details on how we ensure fairness in human evaluations when comparing videos with different resolutions, see Appendix D.

Next, we analyze quantitatively how JUICE improves human evaluation reliability and robustness. To identify unbiased JUICE factors differentiating any two video generation models on Quality and Faithfulness, we made an initial pool of random video samples generated by a few models, and asked internal human raters to explicitly explain their reasoning for picking one model over another. We then categorized them into five reasons for Quality and two for Faithfulness as mentioned in Section 3.2.

Effect of JUICE on improving evaluation reliability and robustness of human evaluations. We measure the reliability of our human evaluations when evaluators are required to justify their choice. For each pair of videos which are compared, we look at the votes for model A vs. model B and call the agreement between annotators either ‘split’ ($2|3$ or $3|2$ votes), ‘partial’ ($4|1$ or $1|4$ votes), or ‘complete’ ($5|0$ or $0|5$ votes). We run human evaluations comparing our generations vs. Make-A-Video, first using a naive evaluation template and then with JUICE, and show the results in Fig. 11. We observe that the number of samples with ‘split’ agreement is decreased significantly by $28\%$, and the number of ‘complete’ agreements is increased by $24\%$.

Next, we use Fleiss’ kappa as a statistical measure for inter-rater reliability for a fixed number of raters. This metric stands for the amount by which the observed agreement exceeds the agreement by chance, i.e., when the evaluators made their choices completely randomly. Fleiss’ kappa works for any number of evaluators giving categorical ratings and we show the values in Fig. 12. The value of kappa is always in the range of $$, with positive kappa values representing an agreement. To better understand its behavior and range of scores in our evaluation setup, we perform an experiment on a simulated data representing our specific case of 304 tasks with two classes, model A-vs-B, and five evaluators per task. We begin with computing the kappa value when we have a ‘complete’ agreement among evaluators on all tasks, i.e. when all five evaluators choose either model A or model B in each task. This run receives a kappa value of 1 (blue dot in Fig. 12). We gradually decrease the number of samples with complete agreement by introducing samples with ‘partial’ agreement when four out of five evaluators picked model A or model B (green line in Fig. 12) Similarly, we decrease the number of samples with complete agreement by replacing them with samples where three out of the five evaluators picked model A or model B, illustrated with a red line. As shown in the plot, the kappa value ranges from$-0.2$(ratings always being ‘split’) to$1.0$ (ratings always having ‘complete’ agreement). Different proportions of samples with ‘complete’, ‘partial’ or ‘split’ agreements result in a kappa value in the shaded region. We compute and compare kappa values for the naive evaluation and JUICE evaluation–0.004 and 0.31, respectively–confirming the improvement in the inter-rater reliability of JUICE.

Analyzing human evaluations. To clearly understand the strengths of each model in our evaluations, we find the most contributing factors when Emu Video generations are preferred to each baseline in Figures 6, 14. A more detailed distribution of each reason and its co-occurrence with other factors is illustrated in Figure 13. We similarly, plot the percentage of each reason picked for the best three baseline generations preferred to Emu Video in Figure 15.

Appendix D Comparisons to Prior Work

In § 4.2 in the main paper, we conduct human evaluations comparing Emu Video to prior work. Here, we share further details and include human evaluation results using a different setup. Specifically, in § D.1 we outline the prompt datasets that are used in comparisons to prior work. In § D.2 we detail how we sampled from the commercial models that we compare to in the main paper. In § D.3 we give details on the postprocessing done for the human evaluations in Fig. 2 in the main paper. In § D.4 we include further human evaluations conducted without postprocessing the videos from Emu Video or prior work.

Since many of the methods that we compare to in Fig. 2 are closed source, we cannot generate samples from all of them with one unified prompt dataset, and instead must construct different datasets via each method’s repsective publicly released example generated videos. In total, we use 5 different prompt datasets. The human evaluations in Fig. 2 for Make-A-Video, Imagen Video, Align Your Latents, PYOCO, and Reuse & Diffuse were conducted using the prompt datasets from the respective papers (see Tab. 10 for details). Certain methods that we compare to are either open-source (CogVideo) or can be sampled from through an online interface (Gen2 and Pika Labs). For these, human evaluations are conducted using the prompt set from Align Your Latents.

D.2 Sampling from Commercial Models

The commercially engineered black-box text-to-video models that we compare to (Pika Labs and Gen2) can be sampled from through an online interface. Here we include details for how we sampled from these models. In both cases, these interfaces allow for certain hyper-parameters to be chosen which guide the generations.

We selected optimal parameters for each of the models by varying the parameters over multiple generations and choosing those that consistently resulted in the best generations. For Pika Labs, we use the arguments “-ar 1:1 -motion 2” for specifying the aspect ratio and motion. For Gen2, we use the “interpolate” and “upscale” arguments and a “General Motion” score of 5. All samples were generated on October 24th 2023.

D.3 Postprocessing Videos for Comparison

Our goal with our main human evaluations in Fig. 2 is to ensure fairness and reduce any human evaluator bias. To ensure this fairness, we postprocess the videos from each model being compared (as outlined in § 4.2 in the main paper). Here, we give further details on the motivation behind this decision, and explain how this postprocessing is done. Results for human evaluations conducted without any postprocessing are discussed in § D.4.

As outlined in Appendix C, our human evaluations are conducted by showing evaluators repeated comparisons of videos generated by two different models for the same prompt, and asking them which model they prefer in terms of the metric being evaluated. It is key for the fairness of the human evaluation that the evaluator treats each comparison independently. It is hence important that the evaluator does not know which model generated which video, otherwise they can become biased towards one model over the other. Since each method generates videos at different dimensions (see Tab. 11), conducting the human evaluations without postprocessing the videos would lead to this annotator bias. Hence we decide to postprocess the videos being compared such that they have the same aspect-ratios, dimensions and frame rates so that they are indistinguishable aside from their generated content. For each pair of models being compared, we downsample these dimensions to the minimum value between the two models (see Tab. 12 for details). Next, we detail how we postprocess the videos.

Aspect Ratio. Since Emu Video generates videos at a 1:1 aspect ratio, all videos are postprocessed to a 1:1 aspect ratio by centre cropping.

Spatial Dimension. The height and width of videos are adjusted using bilinear interpolation.

Video Duration. The duration of videos is reduced via temporal centre cropping.

Frame rate. The frame rate is adjusted using torchvision. The number of frames is selected according to the desired frame rate and video duration.

Next we discuss human evaluation results where videos are compared without any postprocessing.

D.4 Prior Work at Original Dimensions

In this Section, we include further human evaluation results between Emu Video and prior work where we do not perform any postprocessing on the videos and conduct the evaluations with the original dimensions (as detailed in Tab. 11). In this system-level comparison, human evaluators are comparing between videos that may have very different aspect ratios, durations, and frame rates, and in turn may become biased towards one model over another after seeing repeated comparisons. We note that since the dimensions of the videos here are so large, we must scale the height of each video so that both compared videos can fit on one screen for human evaluators. All other dimensions remain as in the original sampled videos. The results are in Tab. 13. Similar to the human evaluations conducted with postprocessed videos in Fig. 2 in the main paper, Emu Video significantly outperforms prior work in terms of both text faithfulness and video quality. Even when comparing Emu Video’s generated videos to generated videos with longer durations (including PYOCO, Imagen Video), wider aspect ratios (incliding Gen2, Align Your Latents), or higher frame rates (including Pika, Gen2), human evaluators still prefer Emu Video’s generated videos in both metrics. We hypothesize that the vastly improved frame quality and temporal consistency of Emu Video still outweighs any benefits that come from any larger dimensions in the prior work’s videos.

Interestingly, Emu Video wins by larger margins here than in the postprocessed setting (an average win rate of 93.8% in quality and 93.1% in faithfulness here, vs. $91.8\%$ and $86.6\%$ in the postprocessed comparison). We conjecture that this improvement in win rates for Emu Video may be due to the potential evaluator bias introduced in this evaluation setting. This introduced bias tends to favor Emu Video since our video generations are on average superior in terms of quality and faithfulness than those of prior work. Hence in this paper we primarily report and refer to the human evaluation scores from the fairer postprocessed setting.

Appendix E Qualitative Results

In this Section, we include additional qualitative results from Emu Video (in § E.1), and further qualitative comparisons between Emu Video and prior work (in § E.2)

Examples of Emu Video’s T2V generations are shown in Fig. 16, and Emu Video’s I2V generations are shown in Fig. 17. As shown, Emu Video generates high quality video generations that are faithful to the text in T2V and to both the image and the text in I2V. The videos have high pixel sharpness, motion smoothness and object consistency, and are visually compelling. Emu Video generates high quality videos for both natural prompts and fantastical prompts. We hypothesize that this is because Emu Video is effectively able to retain the wide range of styles and diversity of the T2I model due to the factorized approach.

E.2 Qualitative Comparisons to Prior Work

We include further qualitative comparisons to prior work in Figs. 18, LABEL:, 19, LABEL:, LABEL:, 20, LABEL:, LABEL:, 21, LABEL:, LABEL:, 22, LABEL: and 23. This Section complements § 4.2 in the main paper where we quantatively demonstrate via human evaluation that Emu Video significantly outperforms the prior work in both video quality and text faithfulness. Emu Video consistently generates videos that are significantly more text faithful (see Figs. 19, LABEL: and 21), with greater motion smoothness and consistency (see Figs. 20, LABEL: and 22), far higher pixel sharpess (see Fig. 23), and that are overall more visually compelling (see Fig. 18) than the prior work.