CVPR 2023 Text Guided Video Editing Competition

Introduction

Leveraging AI for video editing has the potential to unleash creativity for artists of all skill levels. In the last year, numerous papers have been written on Text Guided Video Editing (TGVE), including Dreamix , Tune-A-Video , Gen-1 , among others . While the qualitative results are getting more and more impressive, there is no standard quantitative benchmark in this field. Without a standard benchmark, and with many papers being closed-source, it is impossible to know which model is the state-of-the-art or to understand the strengths and weaknesses of different models.

To address this, we have made these contributions:

Released an open-source dataset of 76 videos, each with 4 prompts for text guided video editing

Organized a competition workshop at CVPR 2023 to evaluate TGVE models

Conducted human evaluation and automated evaluations to rank 8 TGVE models

In this paper, we will describe the dataset, the evaluation methodology, and the findings from the competition. We will also describe the model that won the competition.

Related Work

The development of text-to-image models such as Stable Diffusion has led to rapid process in computer-generated art, including text-to-3D , text-to-audio , to text-to-video . While text-to-video is an incredibly challenging problem for today’s generative AI technology, it is our view that generative AI will become adept at editing existing videos over the next couple of years. Five years ago, using AI to edit a video in any complex way (e.g. anything more complicated than a simple style-transfer) was nearly unthinkable. However, building on top of advances in text-to-image models, works such as Dreamix , Tune-A-Video , and Gen-1 are able to take a video and a text prompt and edit the video to match the prompt.

In terms of algorithms, there are many options for how to design a TGVE model. Tune-A-Video is the first open-source video diffusion model for TGVE. It inflates an image diffusion model into a video model with cross-frame attention, and finetunes it on a single video to generate videos with related motion. Based on it, Edit-A-Video , Video-P2P , vid2vid-zero exploit Null-Text Inversion for precise inversion to preserve the unedited region. Dreamix instead finetunes a video foundation model pretrained on large-scale video data, and establishes superior visual quality in TGVE.

Some recent models use zero-shot techniques that don’t need to be finetuned on each input video. To preserve structural and motion information in source video, FateZero merges attention features pre and post-editing with the editing masks produced by Prompt2Prompt . Text2Video-Zero converts the latent to directly emulate motions, while Pix2Video aligns the latent of the current frame with previous frames via cross-frame attention. To further enhance pixel-level temporal consistency, Rerender A Video propagates the edits using temporal-aware patch matching and frame blending. TokenFlow extracts inter-frame feature correspondences using a nearest-neighbor search and propagates the edited tokens throughout the entire video flow.

Now, what is a good way to evaluate these models? In Table 1, we summarize the evaluation datasets used in several TGVE papers. While there isn’t a standardized dataset in these papers, there are common themes in the datasets. The typical setup is 10 to 100 videos, with 2 to 4 prompts per video. Videos are short (under 10 seconds) and low resolution (often 480x480). Common video sources include YouTube and DAVIS .

The TGVE 2023 Dataset

Now we describe our approach for creating the competition dataset. We begun by collecting 100 videos from DAVIS, YouTube, and Videvo. All of the videos have Creative Commons or other open licenses that allow us to modify and redistribute the videos. We filtered it down to 76 videos total. We made the dataset small so that human-evaluation is affordable.

Most of today’s AI video-editing methods are only able to handle short, low-resolution videos. So, instead of using the whole video, we reduce each video to 32 frames (for DAVIS and Videvo) or 128 frames (for YouTube). We crop and downscale the videos to 480x480. In the future, as these methods improve, we may develop a version of the dataset with larger and longer videos.

Each video has a ground-truth caption. To create the ground-truth captions, we started with the provided caption (e.g. from YouTube), and we manually improved the caption to describe precisely what is happening in the short video clip.

Each video has 4 “edit captions" that describe how we want the TGVE models to edit the videos. Specifically, each video has 4 types of edit captions: style-change (e.g. neon lights style), background-change (e.g on the mars), object-change (e.g. change human to panda), or multiple-changes (at least 2 types of edits). To create the edit captions, we initially asked ChatGPT to take the 76 ground-truth captions and produce edits to the style, background, or objects. However, the ChatGPT captions were a bit boring – turning dogs into cats, sunrises into sunsets, and lakes into oceans. We manually edited many of the captions to make them more creative – turning dogs into kangaroos, sunrises into abstract vector-art, and lakes into underwater coral reefs. We show examples in Figures 1 and 2

Most papers on text-guided video editing include both automated metrics (e.g. CLIP score or FVD score ) and human metrics (based on people labeling data). In our conversations with several researchers in this field, everyone said that automated metrics are quite noisy, and human evaluation is more trustworthy. This matches our own experience.

We used human evaluation to select the winner of the competition. Specifically, we developed a mechanical turk interface where labelers are shown 3 videos: the input video, the baseline edited video (the baseline is Tune-A-Video ), and the video edited by the proposed model. After the labeler has watched 3 videos, they need to answer 3 questions:

Text alignment: Which video better matches the caption?

Structure: Which video better preserves the structure of the input video?

Quality: Aesthetically, which video is better?

For each submission to the challenge, we used this approach to compare the submitted videos to the Tune-A-Video baseline.

Competition Results

We received 5 submissions to the competition, and we now summarize the method used in each submission.

Team PAIR used the image-based diffusion models ControlNet and InstructPix2Pix . Optical flow is used to help with temporal stability.

Team RewardT2VE used Tune-A-Video and Make-A-Protagonist . With these methods, the RewardT2VE team generated many videos and used ImageReward to measure aesthetic quality and CLIPScore to measure alignment with the prompt. The team submitted the best videos based on these metrics.

Team Noah Wukong used a Video Diffusion Model with 3D convolutions.

Team CAMP won the competition with their method, Text-Based Two-Stage Video Editing, which is described later in this paper.

In Table 2, we show the results from all submissions. We include automated metrics (CLIPScore and PickScore ) and human evaluation from Mechanical Turk. For human evaluation, when we report the number 0.689, it means the human evaluators preferred the proposed method over the baseline method 68.9% of the time.

We use human evaluation to choose the winners of the challenge, with Team CAMP beating the Tune-A-Video 59.1% of the time. It’s interesting to note that the automatic metrics are uncorrelated (perhaps even anti-correlated) with the human evaluation results. The contest organizers also reviewed the videos and found that we agreed with human evaluators, and to our eye the CAMP videos were significantly better than the others.

Team CAMP’s winning method: Two-Stage Video Editing (2SVE)

This section was written by the competition winners from Tencent Holdings Ltd and Tsinghua University. Figures 3–16 were created by the competition winners.

We propose a two-stage video-to-video editing method using both text and image guidance based on diffusion models, called Two-Stage Video Editing (2SVE). The method can edit the foreground, background, and style of the input video according to the given prompt. The model structure is shown in Figure 3. In the first stage, we divide the current four tasks: style, object, background, and multiple, into two types for future processing. For the style transfer tasks without too many changes in texture or structure, according to the given prompt, we use ControlNet together with either LoRA or a reference-only image for the corresponding style to guide the generation (Figure 10). For other tasks that need to make changes to the texture or structure of the video, we finetuned the diffusion model for image generation using the video frames with their frame prompt to get our Stage 1 model. More specifically, each frame prompt contains the timestamp as well as the video name to make it unique, and the added unique identifier is called Video Prompt Anchor (VPA) (Figure 3-a, b), see Section 5.1. In the second stage, we further finetuned the model with both the original data and the frames generated by the Stage 1 model, hence to get our Stage 2 model. Since the output frames of Stage 1 have a highly structural alignment with the target prompts compared with the original video frames, we call these data "highly aligned prompts and images" (Figure 3-c), see Section 5.3 for more detail. During the generation process of the two stages, we use ControlNet to manipulate the frame generation. At the same time, by using both the Segment Anything Model (SAM) and OpenCLIP , combined with traditional computer vision skills including erode and dilate, we created a method called Target Segment Process (TSP) to automatically process the input frames for ControlNet (Figure 5), see Section 5.2. By using the Video Prompt Anchor, Multi-frame renderinghttps://xanthius.itch.io/multi-frame-rendering-for-stablediffusion, and ControlNet, we improve the coherency and consistency of video generation, see Section 5.4. After going through the two-stage text-guided video editing framework (2SVE), given an input video $X=\{x_{0},x_{1},x_{2}....x_{N}\}$, where $x_{i}$ represents the $i$-th input frame of the video seperating by 0.1 seconds, we finally compared the results of multiple versions from the two stages: $Y_{1}=\{y_{10},y_{11},y_{12}....y_{1N}\}$ and $Y_{2}=\{y_{20},y_{21},y_{22}....y_{2N}\}$, and select the best output as the final competition submission. Finally, by using VPA for the text control, TSP and ControlNet for the image control, multi-frame rendering for continuity control, and the highly aligned prompt and images, we ensured the alignment of the generated video and text, and preserved the coherency, consistency, and structure of the generated video.

Finetune: Many video generation models are created based on text-to-image generation models, which generate videos by adding additional time sequence modules and improving the generation capabilities along the time axis. At the same time, in order to maintain the original generation ability of the text-to-image model, fewer parameters are selected to be updated from the original model. Pix2Video uses a pre-trained structure-guided image diffusion model to perform text-guided edits on an anchor frame. Tune-A-Video leverages the pre-trained T2I diffusion models for T2V generation by updating spatiotemporal attention (ST-Attn). Video ControlNet does not require any training or fine-tuning of the diffusion models. Our method finetuned the entire UNet with the VAE and CLIP frozen. Finally, in order to reduce overfitting and improve the generation diversity, we merged the finetuned model with the base model with weights of 0.5 each as well. As shown in Figure 4, our method does not use ControlNet during the training phase. The base model for finetuning is chosen to be a weighted merge of DreamShaperhttps://huggingface.co/Lykon/DreamShaper and MeinaMixhttps://civitai.com/models/7240/meinamix, with a weight of 0.5 each.

The main advantage of our method is that the frame-by-frame video editing process can reduce the usage of computing resources and can support higher resolution and arbitrary duration video editing tasks. By adding methods such as TSP and highly aligned prompts, we provide greater control over video editing capabilities and achieve some more substantial and difficult video editing tasks. We have also tried to improve the consistency of video editing through certain methods. However, because our method is still frame-by-frame video editing, how to better improve the coherence of the generated part in the video in actual generation tasks requires further research.

There are already some works such as DreamBooth that represent a given subject with rare token identifiers during fine-tuning a pre-trained diffusion model. Inspired by this, we added a unique identifier text based on the video prompt, namely "CVPR2023-TGVE", to represent the style and the distribution of the competition dataset, we call this the Video Prompt Anchor (VPA). By adding VPA in the generation stage of the image-based video editing, the generated results have achieved stronger stability and continuity. In the training phase, we also added video names to improve the differences in prompt text embedding between videos, but in the generation phase, we did not add this part of the information, because the video name contains object information (eg. goldfish-video), and thus will interfere with video editing results (Figure 3-a, b). By adding the Video Prompt Anchor in the generation stage, it can better generate videos that retain the original video structure including color and object texture, and reduce the randomness of single frame generation together with the help from ControlNet. The comparison of the generation effects of various methods is shown in Figure 11. The advantages of Video Prompt Anchor in maintaining the coherence and consistency of video will be introduced in Section 5.4. Figure 15 for more effects of using only VPA+ControlNet to obtain video consistency and coherence at different stages with various generation methods, and Figure 16 shows specifically the usage of ControlNet.

2 Target Segment Process

In the video editing task, some methods usually use the input video as a reference during generation , but there are also other methods that do not rely on the input video after fully finetuned the model on the input video . Target Segment Process (TSP) is our method that comprehensively processes the input image for the ControlNet. We combine models such as Segment Anything Model, OpenCLIP, and ControlNet, together with our finetuned model, with the help of traditional image processing techniques including dilate, erode, to transfer input video frames $X=\{x_{0},x_{1},x_{2}....x_{N}\}$ into $\bar{X}=\{\bar{x}_{0},\bar{x}_{1},\bar{x}_{2}...\bar{x}_{N}\}$, as the input of the ControlNet. Thereby reducing the difficulty introduced by the large changes in texture and structure before and after the generation process. Figure 5 introduces the pipeline of TSP. We extract the embeddings from both the automatically extracted prompt text changes before and after editing (eg. goldfish) and the SAM segmented mask through OpenCLIP’s text-encoder and image-encoder respectively, and then we calculate the similarity distance between the text and the image embedding, so as to automatically determine the target mask that needs to be processed. After getting the mask, we will try a variety of processing methods in different schemes as the input and reference for further generation, as shown in Figure 12.

3 Highly Aligned Prompts and Images

In order to improve the model’s ability to edit video frames, we add the generated data from the first stage together with the original training data to fine-tuning of the second stage model. Figure 3-c shows the original data of the goldfish video and the output of Stage 1. We think that this part of the data is highly aligned in terms of prompt and frame from the original videos, compared to other datasets (MSVD ) introduced additionally. At the same time, in order to prevent the newly added generated data from affecting the original training data, we filtered these generated data and only added the frame ID to the prompt for training, and did not use the Video Prompt Anchor as usual. Figure 14 shows some examples. Figure 6 shows the comparison of the generation effect between the first stage and the second stage.

4 The Preservation for Video Consistency

Many existing video processing methods align the time sequence in multiple frames, or they add the time sequence-based attention structure into the image-based diffusion model to obtain sequential information . In Section 5.1, we introduced the original image generation effect of Video Prompt Anchor (VPA). Using VPA in combination with ControlNet, the model can effectively restore the original image, and thus can ensure the coherence and consistency of the original image. Based on this idea, our basic scheme adopts the method of first freezing our generated random seed, combining with using the output frame prompt (Figure 3-b) and putting the original video frame into the pre-trained ControlNet to guide the generation of the entire video. Because the structures of the input and output frame prompts are similar, the styles, textures, and hues between the generated frames based on VPA+ControlNet can ensure a high degree of coherence and consistency, as shown in Figure 15. In order to further improve stability and consistency, we propose Multi-frame Video rendering for StableDiffusion (namely MVSD for short). This method takes the generation result of the first frame and the previous frame of the video together with the original image of the current frame as the input of stable diffusion impainting, which refers to the characteristics of other positions on the canvas, namely the generated first and previous frames, to assimilate the style to the current frame. Based on this idea, we added the operation of the target segment process, and processed the inpainting mask more finely, which not only refers to the style of the surrounding frames, but also reduces the interference from the adjacent reference frames, as shown in Figure 15. We show the difference between VPA-MVSD and VPA-MVSD-TSP in the end-to-end system in Figure 16.

Conclusions

Text-guided video editing (TGVE) is a rapidly-improving field of Generative AI research. However, there has been no standard benchmark for TGVE, and many TGVE models are closed-source, so it is difficult to determine which TGVE methods are the best. To address this, we introduced the TGVE 2023 dataset, and we organized a TGVE competition at CVPR 2023. Our competition received 5 submissions, and human evaluation showed that 2 of the submissions outperformed the baseline method. The competition was won by Team CAMP, which developed a novel Two-Stage Video Editing (2SVE) pipeline that incorporates ControlNet, Segment Anything Model (SAM), OpenCLIP, and the MSVD dataset. In human evalations, Team CAMP’s 2SVE method outperformed the baseline 59.1% of the time.

References

Appendix A Further details on Team CAMP’s 2SVE method