A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Anyi Rao, Jiaze Wang, Linning Xu, Xuekun Jiang, Qingqiu Huang, Bolei Zhou, Dahua Lin

Introduction

In 1900, film pioneer George Albert Smith firstly introduced shot type transitions into videos, which revolutionized traditional narrative thought and this technique remains widely used in today’s video editing . Shot, a series of visual continuous frames, plays an important role in presenting the story. It can be recognized from multiple attributes, such as scale and movement. As illustrated in Fig. 1, five scale types and four movement types of shots are widely adopted in video editing, serving for different scenes and emotional expressions.

We are in the era of web 2.0 where user-generated videos proliferate, the techniques to analyze shot types have seen increasing demand in real-world applications: 1) With the capability of recognizing shot types, the videos shared online can be automatically classified or organized not only by their content, e.g. object categories, but also by shot types. Thus the system will be able to respond to queries like “long shots over a city” etc. 2) By analyzing the sequence of shots in movies, we may provide a data-driven view of how professional movies are constructed. Such insight can help ordinary users to make videos that look more professional – one can even build softwares to guide video production for amateurs.

Despite the potential value of shot type analysis, it is true that most previous works in computer vision primarily focus on objective content understanding. For example in video analysis, we focus on classifying and localizing the actions , while shot type analysis has been rarely investigated and lacks appropriate benchmark. Existing datasets on shot type classification are either too small or not publicly available. However, we believe that the analysis of cinematic techniques (e.g. shot types) are also equally important.

To facilitate researches along this direction, we construct a new dataset MovieShots, which composes of over $46K$ shots collected from public movie trailers, with annotations on five scale types and four movement types. We select out scale and movement from many other shot attributes, as they are the two most common and distinguishable attributes that can uniquely characterize a shot in video, where the scale type is decided by the amount of subjects within the frame, and the movement type is determined by the camera motion .

We further propose a novel subject centric framework, namely Subject Guidance Network (SGNet), to classify the scale and movement type of a shot. The key point here is to find out the dominant subject in a given shot, then we can decide its scale according to the portion it takes, and differentiate between the camera movement and subject movement to determine the movement type. SGNet successfully separates the subject and background in a shot and takes them to guide the full images to predict the labels for scale and movement type.

The contributions of this work are as follows: 1) We construct MovieShots, a large-scale $46K$ shot dataset with professionally annotated scale and movement attributes for each shot. 2) SGNet is proposed to classify scale and movement type simultaneously based on the subject centric lens. Our experiments show that this framework greatly improves the classification performance comparing to traditional methods and conventional deep networks TSN and I3D .

Related Work

Shot Type Classification Datasets. Traditional shot type classifications mainly focus on sports videos . Sports video is a special kind of video that contains many clips such as video replays or comments, which is hard to transfer to general video scenarios. Previous movie shot type researches are limited on their evaluation benchmarks. They collect no more than twenty films with about one-thousand shots. There is no public available dataset to test the functionality of these methods. It is noticed that these datasets annotate either scale or movement attribute only, lacking a comprehensive description for a shot. In order to solve these limitations, we collect a $10\times\sim 100\times$ larger dataset, with $46K$ video shots annotated with both scale and movement attributes from more than $7K$ public movie trailers.

Shot Type Classification Methods. Conventional methods for shot scale classification use SVM with dominate color region , low-level texture features , or optical flow . Decision tree method sets up fixed rules to classify the scale type of a shot. Scene depth is applied to infer the scale but is limited to the depth approximation accuracy and lacks generalization ability. For movement type classification, traditional approaches rely on the manual design of a motion descriptor. e.g. design motion vectors CAMHID and 2DMH to capture the camera movement. leverage optical flow to find an alternative of the motion vectors. However, all these methods heavily depend on hand-crafted features that are not applicable to general cases. Our SGNet separates the subject from image and considers both the spatial and temporal configurations of a given shot, achieving much improved performance with better generalization ability.

Video Analysis and Understanding in One Shot. Most previous single shot video understanding tasks are about action recognition and temporal action localization . Video object detection , video object segmentation , video person recognition , video-text retrieval and some low-level vision tasks, e.g. video inpainting video super-resolution are also applied in single shot videos. However, research on video shot type is rarely explored, despite of its huge potential for video understanding. We set up a benchmark with our MovieShots dataset and conduct a detailed study on it.

MovieShots Dataset

To facilitate the shot type analysis in videos, we collect MovieShots, a large-scale shot type annotation set that contains $46K$ shots from $7858$ movies. The details of this dataset are specified as follows.

Following previous definition on shot type , shot scale is defined by the amount of subject figure that is included within the frame, while shot movement is determined by the camera movement or the lens change.

Shot scale has five categories: 1) long shot (LS) is taken from a long distance, sometimes as far as a quarter of a mile away; 2) full shot (FS) barely includes the human body in full; 3) medium shot (MS) contains a figure from the knees or waist up; 4) close-up shot (CS) concentrates on a relatively small object, showing the face or the hand of a person; (5) extreme close-up shot (ECS) shows even smaller parts such as the image of an eye or a mouth.

Shot movement has four categories: 1) in static shot, the camera is fixed but the subject is flexible to move; 2) for motion shot, the camera moves or rotates; 3) the camera zooms in for push shot, and 4) zooms out for pull shot. While all the four movement types are widely used in movies, the use of push and pull shots only takes a very small portion. The usage of different shots usually depends on the movie genres and the preferences of the filmmakers.

2 Dataset Statistics

MovieShots consists of $46,857$ shots from $7,858$ movie trailers, covering a wide variety of movie genres to ensure the inclusion of all scale and movement types of shot. Table 2 compares MovieShots with existing private shot type datasets, noting that none of them are publicly available. MovieShots is significantly larger than others in terms of the shot number and the video coverage, with a more comprehensive annotation covering both the scale type and the movement type for each shot.

3 Annotation Procedure

Building a large-scale shot dataset is challenging in two aspects: appropriate data collections and accurate data annotations. We firstly crawled more than $10K$ movie trailers online. Because movie trailers usually contain advertisements and big subtitles displaying the actor/director information, we firstly cleaned them with auto advertisement/big text detection and went through a second round of manual check. Noting that shot detection is a well solved problem and we used an off-the-shelf approach to cut shots in these trailers and filtered out failure cases with manual check. All our annotators are cinematic professionals in film industries or cinematic arts majors, who provide high quality labels. We also set up annotation prototypes for these well defined criterion of shot types, as illustrated in Fig. 2. Additionally, three rounds annotation procedures have been done to ensure the high consistency. We finally achieve $95\%$ annotation consistency, with those inconsistent shots being filtered out in our experiments.

SGNet: Subject Guidance Network

In this section, we introduce our Subject Guidance Network (SGNet) for scale and movement type classification. The overall framework is shown in Fig. 3.

We firstly divide a shot into $N$ clips to capture the contextual variations along the temporal dimension. Each clip passes through a two-branch classification network and outputs a feature vector. The feature vectors coming from the $N$ clips are pooled together and pass through a fully-connected layer to get the final prediction.

It is noticed that, the separation of subject and background information is critical for both two tasks. While the scale type depends on the portion of the subject in the shot, the movement type relies on the background motion rather than the subject motion, as the changes of the background information are closely related to the camera motion. To reduce the burden of the whole pipeline, a light-weight subject map generator is designed to separate the subject and backgroundBackground image is equal to the whole image minus the subject part. in both image and flow in an effective way. The subject map here is a saliency map sharing the same width and height as the original image with values in range of zero to one. We use subject map to guide the whole image to predict the scale, and use the background map to guide the whole image to predict the movement with the help of an obtained variance map.

In the following two subsections, we will first introduce the subject map guidance for shot type classification in Section 4.1, and elaborate on our subject map generation in Section 4.2.

As discussed before, the separation of subject and background is crucial to shot type classification. In this section, we firstly explain how the obtained subject map guides scale and movement classification respectively.

For scale type classification, the subject map $[1\times W\times H]$ element-wise multiplies with the whole image $[3\times W\times H]$ to get an subject image $[3\times W\times H]$ . Then we take the whole image and the obtained subject image as two input pathways. Each of them is sent into a ResNet50 network. To apply subject guidance, we fues the subject feature map from different stages of feature representations into the whole image pathway. Specifically, these fusion connections are applied right after pool1, res2, res3, and res4 in the ResNet50 backbone. The fusion is conducted by lateral connections .

For movement type classification, the background guidance is applied in a similar way to the scale type prediction. Additionally, a variance map module is further introduced, inspired by the fact that the changes of appearance along time is a cue for movement classification. For example, in a static shot, the appearances of background among different clips are almost the same, while the background appearances changes significantly as time changes in motion shots, push shots and pull shots. We calculate one variance map $\mathbf{V}_{m}\in\mathcal{R}^{N\times N}$ for each shot among its different clips ( $N=8$ clips in our experiments) at different stages of the backbone ResNet50. Specifically, these stages include those after pool1, res2, res3, and res4, the same as those used in the previous fusions. We apply inner product between the two normalized feature maps $\mathbf{F}_{m,i},\mathbf{F}_{m,j}\in\mathcal{R}^{h_{m}\times w_{m}\times c_{m}}$ from two clips $i,j$ at stage $m$ to get $\mathbf{V}_{m,(i,j)}$ . The inner product here is equivalent to calculating the cosine similarity, which captures the similarities between different clips. $\mathbf{V}_{m}$ is achieved by concatenating all possible clips pair $\mathbf{V}_{m,(i,j)}$ . Finally, all $\mathbf{V}=\{\mathbf{V}_{m}\}$ among $M$ stages are concatenated along the channel-wise dimension and are fed into a two-layer FC for classification. The classification results using variance maps are fused with the image classification results for the final prediction.

2 Subject Map Generation

Now we elaborate on how we separate the subject from the background with our light-weight subject map generator.

Conventional saliency/attention map methods employ hand-crafted visual features or heuristic priors , which are incapable of capturing high-level semantic knowledge, making the predicted map unsatisfactory. Pre-trained state-of-the-art deep networks are usually very large with more than $50\sim 100$ layers and are not easy to be taken as a submodule in the designed networks to fine tune, considering the high computational costs. From another perspective, to train a randomly initialized subject network from scratch with only shot type label is impractical, since the supervision signal is too weak and the network is unable to converge, considering the subject map is a pixel-wise prediction but the annotation is a video-level label.

To strike a balance between the performance and the computational efficiency, we resort to knowledge distillation (KD) , considering that it is easy and flexible to learn, and achieves state-of-the-art performance on classification problems . A light-weight student generator (a 6-layer CNN) learns from its teacher network (a MSRA10K pre-trained $R^{3}$ Net ). Note that a naive KD using $\mathcal{L}_{2}$ loss is usually suboptimal because it is difficult to learn the true data distribution from the teacher and may result in missing generation details. Therefore, an additional adversarial loss with the help of Generative Adversarial Networks (GAN) is adopted. In all, the student generator is trained by minimizing the following three-term loss,

Experiments

All the experiments are conducted on MovieShots. The whole dataset is split into Train, Val, and Test sets with a ratio 7:1:2, as shown in Table 2.

Implementation Details.

We take cross-entropy loss for the classification result. The shot is evenly split into 3 clips in training and 25 clips in testing. The fusing function from the subject/background map to whole image is implemented by concatenating the output from the two branches. Image and flow are set up as two inputs and their classification score are fused to get the final results. We train these models for $60$ epochs with mini-batch SGD, where the batch size is set to 128 and the momentum is set to 0.9. The initial learning rate is $0.001$ and the learning rate will be divided by $10$ at the $20th$ and $40th$ epoch.

Evaluation Metrics.

We take the commonly used Top-1 accuracy as the evaluation metric. Specifically, in our experiment, we denote AccS for scale classification performance and AccM for movement classification performance.

2 Overall Results

We reproduce DCR , CAMHID and 2DMH according to their papers. DCR clusters dominant color sets and predicts shot type based on the ratio of different color sets. CAMHID , 2DMH are based on motion vectors. CAMHID takes SVD to get the dominant components. 2DMH disentangles the magnitude and orientation of motion vectors. TSN and I3D are experimented using authors’ code repositories. SGNet adopts ResNet50 as the backbone. All the network weights are initialized with pre-trained models from ImageNet unless specially stated.

Overall Results Analysis. 1) Traditional Methods. The overall results are shown in Table 3. The performances of DCR , CAMHID and 2DMH are restricted by their poor representations.

2) 3D Networks. For movement classification, I3D-ResNet50 achieves better result than TSN-ResNet50 (img + flow) since it captures more temporal relationships. With Kinetics400 pre-trained, I3D-ResNet50 gets 4.8 boost on $Acc_{M}$ . But in scale classification, I3D-ResNet50 performs worse than TSN-ResNet50 (img). The reason might be that I3D-ResNet50 is not good at capturing the spatial configuration of frames in predicting the shot scale. The performance of I3D-ResNet101 is similar to I3D-ResNet50 since deeper 3D networks needs more data to improve the performance.

From another perspective, 3D CNNs are much more computational expensive and need dense samples from videos, which causes the low speed for training and inference. We choose 2D TSN-ResNet50 as our backbone. The results prove that this 2D network can achieve better results than 3D networks with our careful designs. Deep 2D network TSN using image (TSN img) achieves $\sim 30$ % raise on AccS and AccM than traditional methods, as it captures high-level semantic information such as the subject contours and the temporal relationship in a shot.

3) Deeper Backbones. To show that the improvement does not come from the increase of model parameters, we compare SGNet w/ Var (img + flow) (use ResNet50 backbone) with TSN-ResNet152 (img + flow). SGNet w/ Var (img + flow) outperforms TSN-ResNet152 by a margin of 2.62 on $Acc_{S}$ and 3.84 on $Acc_{M}$ , with 15% fewer parameters and 19% fewer GFLOPs.

4) 2D Networks and Kinetics Pre-training. Our full model SGNet w/ Var (img + flow) which includes subject map guidance, motion information flow, and variance map, improves $3.49$ (relatively $4.12\%$ ) on $Acc_{S}$ and $11.40$ (relatively $16.11\%$ ) on $Acc_{M}$ compared to TSN (img), and $3.47$ (relatively $4.15\%$ ) on $Acc_{S}$ and $4.73$ (relatively $6.13\%$ ) on $Acc_{M}$ compared to TSN (img + flow). The full model get further improvements by 0.2 on $Acc_{S}$ and 1.8 $Acc_{M}$ with Kinetics pre-trained. This result shows that action recognition dataset can bring more help to shot movement predictions.

Analysis of Our Framework. Based on TSN (img), SGNet (img) takes the advantage of subject map guidance and improves the scale and movement results by $3.13$ and $0.96$ respectively, which shows the usefulness of subject guidance especially for scale type prediction. With the help of variance map, SGNet w/ Var (img) raise the movement classification performance from $70.46$ to $80.57$ (relatively $14.35\%$ ). Similarly, flow (SGNet img+ flow) helps the model to improve the movement results from $70.46$ to $80.65$ (relatively $14.46\%$ ). These results show that variance map and flow both capture the movement information and contribute to the great performance on movement type classification. As for scale type classification, variance map and flow bring slight improvements ( $0.2\sim 0.3$ ), which shows that the movement information captured by variance map and flow provide a weak assistance to the scale type classification. Finally, combining variance map and flow, SGNet w/ Var (img + flow) further gains improvement on scale ( $87.57$ ) and on movement ( $81.86$ ) classification and achieves the best performance among all (without Kinetics pre-trained).

3 Ablation Studies

We conduct ablation studies on the following designs to verify their effectiveness: 1) subject map guidance, 2) subject map generation, and 3) joint training.

The Effects of Different Subject and Background Map Guidances. In the first block of Table 4, we take TSN model using image and flow with variance map (TSN w/ Var img+flow) as our baseline to test the effects of different subject map guidances. It takes a single-branch ResNet50 as backbone and two models for image and flow respectively, and fuses their scores at the end. We observe that using only subject or background information is inferior to the performance of using the whole image and flow, with $\sim 5$ / $\sim 2$ drop on $Acc_{S}$ / $Acc_{M}$ .

Setting 4,5 in Table 4 are two branches setting with either subject or background guidance. In these experiments, we take a two-branch ResNet50 as backbone for image and flow model, one branch for subject/background and the other one for the whole image/flow. The output obtained from the first branch is concatenated with the output of second branch (+Subj and +Back) as guidance, and send to following networks. Generally, subject guidance achieves $0.4\sim 0.8$ performance gain on scale classification and background guidance outperforms subject guidance on movement prediction with $0.8\sim 1.0$ better results.

Setting 6 (Base+Subj+Back) in Table 4 use a four-branch ResNet50. Two branches are for subject guidance, and the rest two are for background guidance. With more information, the performance drops a little since the subject and background information might be mutually exclusive to each other.

The Influence of Different Subject Map Generations. As discussed above, a subject map generation module is needed to guide the network prediction. This module has many alternatives. Table 5 shows the comparisons between our approaches and self-attention SBS (Saliency-Based Sampling Layer) , and fine-tuned/fixed models $R^{3}$ Net-ResNet18/50 . We take TSN model using image and flow with variance map (TSN w/ Var img+flow) as our baseline to test the influence of different subject map generations.

Self-attention generation method SBS (Saliency-Based Sampling Layer) does not bring improvement compared with the baseline. The reason might be that self-attention is hard to learn from these weak labels, i.e. shot types. The pre-trained fixed networks (settings 3,4) bring gains to the performance, and the performance increases as the network becomes deeper. Moreover, when we fine tune these networks on our tasks (settings 5,6), the performance improves further with $\sim 2$ gains on both AccS and AccM.

Our light-weight subject map generation module is driven by two losses besides the classification cross entropy loss. The performance of using $\mathcal{L}_{2}$ loss (setting 7) is worse than fine-tuned $R^{3}$ Net-ResNet50. With the help of both $\mathcal{L}_{2}$ loss and adversarial loss $\mathcal{L}_{adv}$ (setting 8), student generator is on par with $R^{3}$ Net-ResNet50. However, compared with $R^{3}$ Net-ResNet50, our light-weight subject map student generator has 99.8% fewer parameters and 89.4% fewer GFLOPs (shown in Table 7), which largely speeds up the training and inference processes.

Two-task Joint Training. To investigate the relationship between the scale and movement classification, we conduct the joint training experiments on these two tasks, as shown in Table 7. We take our full model SGNet w/ Var (img + flow) as the baseline. We testify the coupling of scale and movement type by sharing the same modules from bottom to top gradually. As lined out in the second row, sharing the subject map generation (SMG) module is helpful to the performance, where $Acc_{S}$ and $Acc_{M}$ raises $0.55$ and $0.33$ respectively. However, when we further share these two tasks classification till ResNet50’s res1 and res4 modules, we observe that joint training is harmful to the performance when they share more branches. These prove that both scale and movement benefit from the subject guidance. The spatial layout learnt from scale and the camera motion learnt from the movement contribute complementaily to the subject map generation. While the subject guidance is shared by both tasks, the distinct goals of the two tasks still require task-specific designs in the later part to learn better representations.

4 Qualitative Results

In this section, we show the qualitative results of subject map generation and the variance map computation.

Subject Map. Fig. 4(a) compares our generated subject map with those generated by $R^{3}$ Net-ResNet50 in fixed setting. Our generated subject map achieves much better generation result that are consistent with our human judgment. The first row in the figure is an over-the-shoulder static close-up shot, where both methods successfully predict the subject woman rather than the man with back head. But our method outputs much less noise. The second row is a full shot. Our method successfully detects both two people and does not include the background stone into the subject map. In the third and fourth row cases, $R^{3}$ Net-ResNet50-fixed outputs blurred area around the contours of two people while our method obtains a sharp shape of the subject.

Variance Map. The variance map is important for predicting shot movement. We divide a shot video into 8 clips, plot the variance map for each movement type in the test set and average these variance maps in Fig. 4(b). The variance map is of size $8\times 8$ , and these gray scale blocks show the similarity among clips in the variance map. As noted from the plot, the variance map of the static shot is nearly an all-one matrix, meaning that there is no significant change between the eight clips. The near identity matrix shape of motion shots reveal that it has the least similarities between consecutive clips.

Application

Shot type analysis has a wide range of potential applications. In this section, we illustrate one such application of realizing automatic video editing with the help of shot type classification.

Automatic Video Editing: Shot Type Changing. Video editors usually try out different shot types to convey emotions and stories, which consumes a lot of time and resources. In many cases, the change of shot type changes the semantic of a movie and effects audience’s emotions, revealing the intent of the directors.

The model we propose in this paper could classify the shot type of any given video shot. Fig. 5 shows a shot clip is adopted here to cut shots from the film. from the famous film Titanic, demonstrating how our model can be applied to changing shot scales to achieve a desired artistic expression. The original shot is a medium shot. Suppose we want to emphasize the role of speaker in this dialogue scene, we may want to use some close-up shot to emphasize the speaker. Firstly, we propose many cropping regions randomly depend on the position of the speaker and generate the corresponding candidate shots. Secondly, we use our shot classification model to classify these candidate shots and assign them with confidence scores. Note that the original shot is a single shot. We divide it into four shots depending on the active-speaker . In Shot 1, Rose and Jack walk on the deck of the ship; Shot 2, Rose talks; Shot 3, they stop and Rose looks at Jack; Shot 4, Jack talks, as illustrated in Fig. 5. We change the style of the original shot by selecting parts from the divided four shots and replace them with the candidates with high scores and the desired scale types, as shown in Fig. 6. After these changes, the emotion of this clip turns to be more intense and the speaking cast is being emphasized after changing from a middle shot to a close shot. One more result on DAVIS dataset is also shown in Fig. 6. These results demonstrate the importance of shot type in videos, especially for their emotion and aesthetic analysis.More results and their corresponding videos are shown in the supplementary videos.

Conclusion

In this work, we construct a large-scale dataset MovieShots for shot analysis, which containing $46K$ shots from $7K$ movie trailers with professionally annotated scale and movement attributes. We propose a Subject Guidance Network (SGNet) to capture the contextual information and the spatial and temporal configuration of a shot for our shot type classification task. Experiments show that this network is very effective and achieves better results than existing methods. All the studies in this paper together show that shot type analysis is a promising direction for edited video analysis which deserves further research efforts.

Acknowledgement: This work is partially supported by the SenseTime Collaborative Grant on Large-scale Multi-modality Analysis (CUHK Agreement No. TS1610626 & No. TS1712093), the General Research Fund (GRF) of Hong Kong (No. 14203518 & No. 14205719), and Innovation and Technology Support Program (ITSP) Tier 2, ITS/431/18F.