On the effectiveness of task granularity for transfer learning

Introduction

Common-sense video understanding entails fine-grained recognition of actions, objects, spatial temporal relations, and physical interactions, arguably well beyond the capabilities of current techniques. A general framework will need to discriminate myriad variations of actions and interactions. For example, we need to be able to discriminate similar actions that differ in subtle ways, e.g., ’putting a pen beside the cup’, ’putting the pen in the cup’, or ’pretending to put the pen on the table’. Similarly, one must be able to cope with diverse actors and object classes. Such generality and fine-grained discrimination are key challenges to video understanding. In contrast to current approaches to action recognition and captioning on relatively small corpora, with coarse-grained actions, this paper considers fine-grained classification and captioning tasks on large-scale video corpora. Training is performed on the Something-Something dataset (Goyal et al. (2017)), with $174$ fine-grained action categories and thousands of different objects, all under significant variations in lighting, viewpoint, background clutter, and occlusion.

We describe a DNN architecture comprising a 2-channel convolutional net and an LSTM recurrent network for video encoding. A common encoding is shared for classification and caption generation. The resulting network performs better than baseline action classification results in Goyal et al. (2017), and it gives impressive results on an extremely challenging fine-grained captioning task. We also demonstrate the quality of learned features through transfer learning from Something-Something features to a kitchen action dataset. The main contributions in the paper include:

1. Explore the link between label granularity and feature quality: We exploit 3 levels of granularity in sth-sth-v2, namely, action groups, action categories, and captions. Experiments show that more fine-grained labels yield richer features (see Table 3 and Fig. 4).

2. Baselines for captioning on sth-sth-v2 data: We note that the captioning task is new for this dataset; the original dataset did not provide captions.

3. Captioning as a source task for transfer learning: We show that models trained to jointly perform classification and captioning learn features that transfer better to other tasks (e.g., see Fig. 4). To the best of our knowledge, captioning has, to date, been used as a target task. Our results suggest that captioning is a powerful source task.

4. 20bn-kitchenware: We introduce a new dataset, ostensibly for video transfer learning.

Video Data

Video action classification and captioning have received significant attention for several years, but progress has been somewhat slow in part because of a lack of large-scale corpora. Using web sources (e.g., YouTube, Vimeo and Flickr) and human annotators, larger datasets have been collected (e.g., Kay et al. (2017); Monfort et al. (2018)), but they lack control over pose variations, motion and other scene properties that might be important for learning fine-grained models. More recently, crowd-sourced/crowd-acted data has emerged. This allows targeted video domains, action classes, and control over subtle differences between fine-grained action classes. The first version of Goyal et al. (2017) has $100,000$ videos of human-object interactions, comprising $50$ coarse-grained action groups, decomposed further into $174$ related action categories. The videos exhibit significant diversity in viewing and lighting, objects and backgrounds, and the ways in which actors perform actions. Baseline performance in Goyal et al. (2017) was a correct action classification rate of $11.5\%$, and $36.2\%$ on action groups. Zhou et al. (2017) report $42.01\%$ classification accuracy on Something-Something-V1 action categories. With our architecture we obtain validation accuracy of 38.8%.

Something-Something-V2 is larger, with $220,847$ videos of the same $174$ action categories. In addition, each video includes a caption that was authored and uploaded by the crowd actor. These captions incorporate the action class as well as descriptions of the objects involved. That is, the captions mirror the action template, but with the generic placeholder Something replaced by the object(s) chosen by the actor. As an example, a video with template action ’Putting [something] on [something]’, might have a caption ’Putting a red cup on a blue plastic box’. In a nutshell, this dataset provides different levels of granularity: $50$ coarse-grained action groups, $174$ fine-grained action categories, and even more fine-grained labeling via video captions.

Video Classification and Captioning Tasks

Due to the prevalence of datasets like UCF-101 (Soomro et al. (2012)), sports1M (Karpathy et al. (2014)) and more recently Kinetics (Kay et al. (2017)), most research on action classification has focused on models for coarse-grained action classification. In extreme cases, action classes can be can be discriminated from a glimpse of the scene, often encoded in isolated frames; e.g., inferring “soccer play” from the presence of a green field. Even when motion is essential to the action, many existing approaches do well by encoding rough statistics over velocities, directions, and motion positions. Little work has been devoted to the task of representing details of object interactions or how configurations change over time.

Image and video captioning have received increasing attention since the release of captioning data, notably, Microsoft COCO (Chen et al. (2015)) and MSR-VTT (Xu et al. (2016)). One problem with existing captioning approaches is that many datasets implicitly allow models to “cheat”, e.g., by generating phrases that are grammatically and semantically coherent, but only loosely related to the fine-grained visual structure. It has been shown, for example, that a language model trained on unordered, individual words (e.g., object-nouns) predicted by a separate NN can compete with captioning model trained on the actual task end-to-end (e.g., Yao et al. (2015); Heuer et al. (2016)). Similarly, nearest neighbor methods have been surprisingly effective (Devlin et al. (2015)).

Captioning tasks, if designed appropriately, should capture detailed scene properties. Labels with more subtle and fine-grained distinctions would directly expose the ability (or inability) of a network to correctly infer the scene properties encoded in the captions. The captions provided with the Something-Something dataset are designed to be sufficiently fine-grained that attention to details is needed to yield high prediction accuracy. For example, to generate correct captions, networks need not only infer the correct actions, but must also learn to recognize different objects and their properties, as well as object interactions.

Approach

We use a modified encoder-decoder architecture with an action classifier applied to the video encoding (see Fig. 1). The decoder or classifier can be switched off, leaving pure classification or captioning models. One can also jointly train classification and captioning models. We train our two-channel architecture to solve four different tasks with different granularity levels: - Coarse-grained classification on action groups (CG actions) - Fine-grained classification on 174 action categories (FG actions) - Captioning on simplified objects (SO captions) - Fine-grained Captioning with full object placeholders (Captions) Finally, we investigate the effectiveness of the learned features for transfer learning.

The video encoder receives the input video $v$, and maps it to an embedded representation $h$. Conditioned on $h$, a caption decoder generates the caption $c$, and a classifier predicts the action category. The encoder processes the video with a two-channel convolutional architecture (Fig. 2). A spatial 2D-CNN and a spatiotemporal 3D-CNN are applied in parallel. The 3D-CNN features are used in lieu of a separate module to compute motion (e.g., optical flow) features. The basic building block of each channel is a $3\times 3\times 3$ ($3\times 3$ in 2D-CNN channel) convolution filter with batchnorm and ReLU activation. To aggregate features across time, feature vectors from each channel are combined and fed to a 2-layer bidirectional LSTM. We average these features to get the video encoding, $h$.

The action classifier applies a fully-connected layer to the encoding $h$, followed by a softmax layer. Training uses a cross-entropy loss over action categories. The caption decoder is a two-layer LSTM. Much like conventional encoder-decoder methods for video captioning (Venugopalan et al. (2014); Donahue et al. (2015); Kaufman et al. (2016) and Sutskever et al. (2014)), our decoder generates captions using a softmax over vocabulary words, conditioned on previously generated words. The loss is the usual negative log-probability of the word sequence:

where $w^{i}$ denotes the $i^{\text{th}}$ word of the caption, $h$ is the video encoding, and $\theta$ denotes model parameters. To optimize speed and memory usage during training, the length of captions generated by the decoder is fixed at 14 wordsLess than $1\%$ of Something-Something captions have more than 14 words.. As is common for encoder-decoder networks, we train with teacher-forcing (Williams & Zipser (1989)). At test time, the input to the decoder at each time-step is the token generated at the previous time-step (i.e., no teacher forcing). The model is trained end-to-end for classification and captioning with a weighted sum of the classification and captioning losses, i.e.,

With $\lambda=1$ or $\lambda=0$, we end up with pure classification and captioning tasks respectively. For other values of $\lambda$, they are trained jointly. The encoder is shared by the action classifier and the caption decoder. The experiments below also compare this joint training regime with models for which the encoder is trained on the classification loss or the captioning loss alone.

Existing action classification methods differ in the way they aggregate information over time. Many approaches rely on spatial features with CNNs applied to individual frames (e.g., see Karpathy et al. (2014)). While such features have been successful, they fail to exploit temporal information in video. Other approaches use spatial and temporal information (e.g., Karpathy et al. (2014); Tran et al. (2015); Ji et al. (2013)). Recent methods involve Imagenet-pretrained 2D-CNNs that are inflated into 3D (Carreira & Zisserman (2017); Hara et al. (2017)). Our video encoder is most closely related to approaches that perform temporal reasoning via a recurrent convolutional architecture (Molchanov et al. (2016); Baccouche et al. (2011); Li et al. (2017)). It is also related to TwoStream architectures (Simonyan & Zisserman (2014a)); but our model uses generic 3D CNN features rather than optical flow.

Experiments

We train four two-channel models that are trained to solve coarse-grained and fine-grained classification and captioning tasks. In what follows we discuss different tasks in more details.

Something-Something provides coarse-grained categories (action groups), each comprising a set of fine-grained actions. We trained a classification model on coarse-grained action groups, using the M($256$-$256$) architecture, with accuracy of $57.6\%$ (see Table 2 (top-left)). Table 1 reports the performance of our model for fine-grained classification. For the pure classification task (with $\lambda=1$) we consider different numbers of features produced by the 2D (spatial) CNN and the 3D (spatio-temporal) CNN. The total number of features is $512$ in all cases. The results show that the model benefits from 2D and 3D features. The even split M($256$-$256$) provides a good trade-off between performance and model complexity. We therefore use this model below, unless otherwise specified.

We can perform coarse-grained classification by mapping predictions from the fine-grained classifier onto the action groups. To this end we sum the probabilities of fine-grained actions belonging to each action group. Interestingly, the resulting accuracy on coarse-grained action groups increases to $62.5\%$. This improvement suggests that fine-trained training provides higher quality features. We also examine to what extent coarse-grained performance accounts for fine-grained accuracy; i.e, how better fine-grained performs compared to chance when conditioned on coarse-grain predictions. For example, conditioned on a predicted action group, if we select the most frequent action within the action group, fine-grained test accuracy would be $23.8\%$. One can also fix the coarse-grained model and train a linear classifier on top of its penultimate features. This yields test accuracy of $41.7\%$, still $8.7\%$ below test performance for the corresponding architecture trained on the fine-grained task, further supporting our contention that training on fine-grained details yields richer features.

2 Classification Baselines.

As a baseline, we use ImageNet-pretrained models (Simonyan & Zisserman (2014b); He et al. (2015)) on individual frames, to which we add layers. First, we use just the middle frame of the video, with a 2-layer MLP with $1024$ hidden units. We also consider a baseline in which we apply this approach to all $48$ frames, and then average frame by frame predictions. We experiment with aggregating information over time using a LSTM layer with 1024 units. We report results in Table 3. There is a marked improvement with the LSTM, confirming that this task improves with temporal analysis. Our best baseline was achieved with a VGG architecture, and the test accuracy is close to the best architecture reported to date on Something-Something (e.g., Zhou et al. (2017)).

3 Captioning with simplified object placeholders.

The ground truth object placeholders in Something-Something video captions (i.e., the object descriptions provided by crowd actors) are not highly constrained. Crowd actors have the option to type in the objects they used, along with multiple descriptive or quantitative adjectives, elaborating shape, color, material or the number of objects involved. Accordingly, it is not surprising that the distribution over object placeholders is extremely heavy-tailed, with many words occurring rarely. To facilitate training we therefore replaced all words that occurred $5$ times or less by [Something]. After removing rare words, we are left with $2880$ words comprising around $30,000$ distinct object placeholders (i.e., different combination of nouns, adjectives, etc).

We consider a simplified task in which we modify the ground truth captions so they only contain one word per placeholder, by keeping the last noun, removing all other words from the placeholders. By substituting the pre-processed placeholders into the action category, we obtain a simplified caption. Table 5 shows an example of the process. The result is a reduced vocabulary with $2055$ words. In the spectrum of granularity, captioning with simplified objects can be considered as a middle ground between fine-grained action classification and captioning with full labels.

We train different variations of our two-channel architecture on captions with simplified objects. Table 6 summarizes our results. We observe that the model with an equal number of 2D- and 3D-channels performs best (albeit by a fairly small margin). Also the best captioning model performs best on the classification task. We also evaluate the models using standard captioning metrics: BLEU (Papineni et al. (2002)), ROUGE-L (Lin (2004)) and METEOR (Denkowski & Lavie (2014)).

We also train networks on the full object placeholders. This constitutes the finest level of action granularity. The experimental results are shown in Table 8. They show that, again, the best captioning model also yields the highest corresponding classification accuracy. The Exact-Match accuracy is significantly lower than for the simplified object placeholders, as it has to account for a much wider variety of phrases. The captioning models produce impressive qualitative results with a high degree of approximate action and object accuracy. Some examples are shown in Figure 9(n). More examples can be found in the appendix.

To the best of our knowledge there are no baselines for the Something-Something captioning task. To quantify the performance of our captioning models, we count the percentage of generated captions that match ground truth word by word. We refer to this as “Exact-Match Accuracy”. This is a challenging metric as the model is deemed correct only if it generates the entire caption correctly. If we use the action category predicted by model M($256$-$256$), trained for classification, and replace all occurrences of [something] with the most likely object string conditioned on that action class, the Exact-Match accuracy is $3.15\%$. The same baseline for simplified object placeholders is $5.69\%$. We also implemented a conventional encoder-decoder model for captioning 4.

In all our experiments we use frame rate of $12fps$. During training we randomly pick $48$ consecutive frames. For videos with less than $48$ frames, we replicate the first and last frames to achieve the intended length. We resize the frames to $128\times 128$, and then use random cropping of size $96\times 96$. For validation and testing, we use $96\times 96$ center cropping. We optimize all models using Adam, with an initial learning rate of $0.001$.

While the captioning task theoretically entails action classification, we found that our two-channel networks optimized on the pure captioning task do not perform as well as models trained jointly on classification and captioning (see 7). By coarsely tuning the hyper-parameter $\lambda$ empirically, we found $\lambda=0.1$ to work well and fix it at this value for the captioning experiments below. More specifically, we first train with a pure classification loss, by setting $\lambda=1$, and subsequently introduce the captioning loss by gradually decreasing $\lambda$ to 0.1.

Transfer Learning

One astonishing property of neural networks is their ability to learn representations transfer well to other tasks (e.g., Donahue et al. (2013); Sharif Razavian et al. (2014)). A distinguishing feature of the ImageNet task, which likely contributes to its potential for transfer learning, is the dataset size and the variety of fine-grained discriminations required. In what follows we explore transfer learning performance as a function of course task granularity.

We introduce 20bn-kitchenware, a few-shot video classification dataset with 390 videos over 13 action categories (see Fig. 4). It contains video clips of manipulating a kitchen utensil for roughly 4 seconds and was designed to capture fine-grained actions with subtle differences. For each utensil $X\in\{\text{fork, spoon, knife, tongs}\}$, the target label belongs to one of 3 actions, “Using $X$”, “Pretending to use $X$” or “Trying but failing to use $X$”. In addition to these 12 action categories, we also include a fall-back class labeled “Doing other things”.

We further encourage the model to pay attention to visual details by including unused ‘negative’ objects. The last row of Fig. 4 shows one example; the target label indicates a manipulation of tongs, but the clip also contains a spoon with an egg in it. Given the limited amount of data available for training130 samples – the rest are used for validation and testing., the action granularity and the presence of negative objects, we hypothesize that only models that have some understanding of physical world properties will perform well. We will release 20bn-kitchenware upon publication of this paper.

Our transfer learning experiments considers four two-channel models that have been respectively pre-trained on coarse-grained labels (classification on action groups), on fine-grained labels (classification on 174 action categories), on simplified captions (captioning on fine-grained action categories expanded with single object descriptor) and on template captions (captioning on fine-grained action categories expanded with object descriptors). We also include two neural nets pre-trained on other datasets: a VGG16 network pre-trained on ImageNet, and an Inflated-ResNet34 pre-trained on Kineticshttps://github.com/kenshohara/3D-ResNets-PyTorch.

The overall training procedure remains the same for all models. We freeze each pre-trained model and then train a neural net on top of extracted penultimate features. Independent of the architecture used, we use the pre-trained model to produce $12$ feature vectors per second. To achieve this, where necessary we split the input video into smaller clips and apply the pre-trained network on each clip individuallyVGG16 applied to individual frames. For Inflated-ResNet34, video was split into clips of 16 frames.. In the simplest case, we pass the obtained features through a logistic regressor and average predictions over time. We also report results for which we classify pre-trained features using an MLP with 512 hidden units as well as a single bidirectional LSTM layer with 128 hidden states. This allows the network to perform some temporal analysis about the target domain.

For each pretrained model and classifier, we evaluate 1-shot, 5-shot and 10-shot performance, averaging scores over 10 runs. Fig. 5 shows the average scores with 95$\%$ confidence intervals. The most noticeable findings are:

1. Logistic Regression vs MLP vs BiLSTM: Using a recurrent network yields better performance.

2. Something-Something features vs others: Our models pre-trained on Something-Something outperform other external models. This is not surprising given the target domain; 20bn-kitchenware samples are, by design, closer to Something-Something samples than ImageNet or Kinetics ones. It is surprising that VGG16 features perform better on 20bn-kitchenware than Kinetics features.

3. Effect of the action granularity: Fig. 5 supports the contention that training on fine-grained tasks yields better features. The best model on this benchmark is that trained jointly on full captions and action categories. The only exception is the model trained to do pure captioning.

Conclusions

Pre-training neural networks on large labeled datasets has become a driving force in deep learning applications. Some might argue that it may be considered a serious competitor to unsupervised learning as a means to generate universal features for the visual world. Ever since ImageNet became popular as a generic feature extractor, a hypothesis been that it is dataset size, the amount of detail and the variety of labels that drive a network’s capability to learn useful features. To the degree that this hypothesis is true, generating visual features capable of transfer learning should involve source tasks that (i) are fine-grained and complex, and (ii) involve video not still images, because video is a much more fertile domain for defining complex tasks that represent aspects of the physical world.

This paper provides further evidence for that hypothesis, showing that task granularity has a strong influence on the quality of the learned features. We also show that captioning, which to the best of our knowledge has been used only as a target task in transfer learning, can be a powerful source task. Our work suggests that one gets substantial leverage by utilizing ever more fine-grained recognition tasks, represented in the form of captions, possibly in combination with question-answering. Unlike the current trend of engineering neural networks to perform bounding box generation, semantic segmentation, or tracking, the appeal of fine-grained textual labels is that they provide a simple homogeneous interface. More importantly, they may provide “just enough” localization and tracking capability to solve a wide variety of important tasks, without allocating valuable compute resources to satisfy intermediate goals at an accuracy that these tasks may not actually require.

References

Appendix

In this supplementary document, we provide:

Qualitative examples for our classification and captioning.

Visualization of our classification and captioning models using Grad-CAM.

The full list of action categories of 20bn-kitchenware.

Here we provide video examples and their ground truth action categories, along with model predictions for each. We use our M($256$-$256$) which is trained with $\lambda=0.1$. Interestingly, notice that even when the predicted actions are incorrect, e.g. row $4$ in Figure 6, they are, nevertheless, usually quite sensible.

Qualitative Examples of Captioning

Below are examples of videos. accompanied by their their ground truth caption and the caption generated by the model. We use model M($256$-$256$) in this section as well, which is also trained jointly for classification and captioning (with $\lambda=0.1$).

Visualization of classification model with Grad-CAM

To visualize regularities learned from data, we extracted temporally-sensitive saliency maps using Grad-CAM Selvaraju et al. (2016), for both classification and captioning task. To this end we extended the Grad-CAM implementation for video processing. Figure 10 shows saliency maps of examples from Something-Something obtained with model M($256$-) trained on fine-grained action categories, with $\lambda=1$ (i.e., the pure classification task).

Visualization of captioning model using Grad-CAM

To get saliency maps during the captioning process, we calculate the Grad-CAM once for each token, for which different regions of the video are highlighted. Figures 11,-13 shows saliency maps for the captioning model, jointly trained with $\lambda=0.1$. Notice how the attentional focus of the model changes qualitatively as we perform Grad-CAM for different tokens in the target caption.

bn-kitchenware action categories

Table 9 lists the full list of 20bn-kitchenware action categories.