Revisiting the "Video" in Video-Language Understanding

Introduction

Videos offer the promise of understanding not only what can be discerned from a single image (e.g. scenes, people, and objects), but also multi-frame event temporality, causality, and dynamics (Figure 1(a)). Correspondingly, there lies a central question at the heart of video research: What makes a video task uniquely suited for videos, beyond what can be understood from a single image?

As a field, video analysis has considered this question deeply in the context of action classification in videos . The emergence of strong convolutional models for image classification enabled researchers to better characterize the limits of single-frame understanding for recognizing actions . A key finding from this analysis was that, in many standard video datasets at the time, temporal understanding was simply not required to perform well on these benchmarks. For example, recognizing static scene context like the presence of a pool was sufficient to recognize the “diving” activity from a single frame . The impact of such analysis was tremendous: later datasets were designed to capture a richer distribution of temporal understanding with better disentanglement of such cues , and model designs evolved further to better capture the now necessary dynamics to address these improved tasks .

Meanwhile, the recent advent of self-supervised image-language models with competitive performance to standard image-classification models means that we have a unique opportunity to reconsider this fundamental question in the context of standard discriminative video-language tasks, such as video question answering and video-language retrieval . In particular, we can now build beyond prior (video-only) analysis work, largely constrained to recognition settings of limited atomic actions in relatively short clips, towards more complex (temporal, causal) event understanding in longer-horizon, multimodal settings where the expressivity of natural language can potentially describe a richer event space.

The primary motivation of our work is to analyze these existing video-language benchmarks by revisiting the video, and derive insights that can help guide the further development of the field. Our driving question is, to what extent can image-level understanding obtained from a single frame (well-chosen, without temporal context) address the current landscape of video-language tasks? To accomplish this, we make the following key contributions:

First, we introduce the atemporal probe (ATP) model to provide a stronger bound on the capabilities of image-level understanding in video-language settings than traditional random frame and mean pooling baselines . Here, we leverage a frozen self-supervised image-language model (e.g. CLIP ) to extract a set of image and language representations: our ATP model must then learn to select a single frozen representation corresponding to a single frame, and forward that to the downstream video-language task. Critically, our framework is constrained to not be capable of reasoning temporally, and its output is ultimately bottlenecked by what a frozen image-language model can discern from an individual, decontextualized video frame.

Second, we apply ATP to analyze a wide range of video-language datasets, focusing primarily on video question answering with extensions to text-to-video retrieval (per Figure 1(b)). To our surprise, we find that many standard and recent benchmarks can be potentially well-addressed with single-frame image understanding. In particular, while this was not our primary aim, we find that our learned ATP model is able to outperform recent state-of-the-art video-language models on standard vision-language benchmarks , despite its substantial bottleneck constraints on model capacity, capability, and inputs. We find that even recent benchmarks that explicitly design for temporal and causal understanding (e.g., ), can have a non-trivial subset of questions answerable by simple single-frame event recognition. As shown in Figure 1(c), while the question asking “why” an event occurred suggests causal understanding may be needed, our ATP model shows that in practice simple scene and object recognition can ascertain the correct answer from a single chosen frame.

Finally, we examine how ATP and the insights it provides can help with improving both dataset and video-level temporal modeling designs. As a case study, we closely examine the NExT-QA benchmark . We find that ATP is able to better identify collections of “causal” and “temporal” questions that cannot be well-addressed with single-frame understanding. In Figure 1(d), ATP struggles to answer this question since it necessitates multi-event reasoning across time. By improving the disentanglement of video- and image-level understanding in the benchmark data, we can better understand the progress of state-of-the-art video techniques leveraging motion features and event reasoning architectures over image-centric models, a result that is not as apparent in the original setting. We further validate our analysis by training a temporal video-level model on top of our ATP selectors, achieving a new state-of-the-art for this benchmark with improved efficiency. Taken together, our analysis suggest key avenues by which our ATP technique can guide continued development of video-language datasets and models in future work.

Background and Related Work

Our work is related to many different areas of vision and vision-language research, including video-specific and image-specific settings. In this section, we discuss the key relevant areas of prior work that motivate our contributions.

Video-language understanding (tasks). Understanding events in their multimodal vision-language context is a long-standing challenge for the computer vision community. Standard video-language tasks include both discriminative tasks, such as video question answering , text-to-video/moment retrieval , and generative tasks, such as video captioning and open-ended VQA . In context, we choose a representative subset of these video-language benchmarks well-suited to studying event temporality and causality. In particular, we choose to focus on discriminative tasks, since automatic metrics (without human-in-the-loop) for generative tasks with causal descriptions remains an open research challenge . Furthermore, many video-language tasks involve heavy reasoning over auxiliary text inputs, such as scripts . These exciting directions are complementary to our goal: we focus instead on revisiting event temporality in the real-world videos themselves.

Video-language understanding (approaches). Standard approaches for addressing these tasks often operate on a combination of image-derived appearance and video-derived motion features as input to an architecture that combines information across the temporal dimension for the final task. While these models are traditionally quite heavy, employing dense features extracted from many frames, recent work has suggested that enabling end-to-end training through sparsity can improve accuracy. Our proposed approach aims to complement these prior lines of work by taking a different approach: instead of focusing explicitly on improving state-of-the-art accuracies, we impose strong learnability and representation constraints to better analyze the degree to which full video-level understanding is truly necessitated by current benchmarks, to help guide future model and dataset designs for capturing deeper event understanding.

Temporality in videos (action recognition). Action and event recognition are fundamental tasks for video understanding, and the subject of recurring deep analysis regarding the role of temporality in action classification , with profound downstream impacts on dataset and subsequent model designs . We draw inspiration from this foundational prior work, while also aiming to broaden analysis beyond characterizing limited sets of atomic actions towards longer-horizon temporal and causal event understanding, which multimodal video-language contexts have the potential to better capture .

Image-language understanding. The advent of new self-supervised vision-language models trained at scale , where models learn a joint embedding space for vision and language without explicit low-level labels, has proven disruptive for image and image-language understanding tasks . We leverage these models, both vision and language components, as foundations for our analytical technique to better characterize the extent to which image-language understanding can address current video-language tasks. Our work is complementary to prior image-language analytical work which revealed unintended language bias: we aim to characterize the extent of unintended video-specific biases in this multimodal setting.

Efficient image-centric video modeling. Finally, we note that aspects of our technical approach draw inspiration from efficient image-centric video modeling literature, which aim to improve efficiency and for tasks like action recognition and localization by learning how to selectively process a sparse number of frames from the input video.

Technical Approach

In this section, we describe our technical approach for our atemporal probe (ATP), a new modeling tool for characterizing the boundary of image-constrained understanding in the context of standard discriminative video-language tasks.

We first briefly introduce the notation and discriminative video-language tasks we consider in this work, namely video question answering and text-to-video retrieval:

Video question answering. Our primary analysis setting is on video question answering: given a paired collection of videos $C_{V}$, and language questions and answers $C_{L}=\{C_{Q},C_{A}\}$, the goal is for each (video, question) pairing $(V,Q)$ to provide the correct answer in $A$.

Video-language retrieval. We also examine video-language retrieval, to assess the generality of our approach. In text-to-video retrieval, the objective is complementary: given a paired collection of videos $C_{V}$ and language descriptions $C_{L}$, the goal is to use the language $L$ to retrieve the specific video $V$ that it originally corresponded with.

We note that in both settings, there exist video $V$ and language $L$ ($=(Q,A)$) inputs common to each task. While our work ultimately analyzes performance on these downstream tasks with respect to their inputs and metrics, our core goal for this work is to provide an improved analytical tool for characterizing specific instantiations of these tasks.

2 Motivating a Stronger Image-Centric Baseline

Traditionally, video models and benchmarks establish their efficacy over image-level understanding by reporting results with a model based on a single (center-most, randomly, etc.) chosen video frame . Because videos can be considered noisy collections of frames, such baselines may not truly represent the bounds of what image-constrained understanding can achieve in video-language contexts (Figure 2). In particular, we seek to answer the question: if we can select a “good frame” from the video and only derive our understanding from that one frame, what video-language tasks are we capable of performing?

Intuitively, settings where only scene-level descriptions are being assessed should likely be addressable from a single frame, as should simple event recognition (per prior analysis in the domain of action recognition, Section 2). However, by the same intuition, questions/tasks that attempt to fully assess deeper event dynamics, causal, or temporal understanding should in principle be unanswerable from a single frame alone, requiring reasoning over multiple events which are not necessarily co-located in time. A compelling baseline that effectively bounds image-level understanding can thus potentially help distinguish between these settings.

3 Atemporal Probe (ATP) Model

Overview. With the motivating insight above, we propose an atemporal probe (ATP) model: a new, stronger analytical approach for characterizing the degree to which video-language tasks can be addressed exclusively with vision-language representations derived from image-only settings. The ATP model (Figure 3) is tasked with finding a single (frozen, image-derived) embedding from the video and forwarding this to the downstream video-language task. Our ATP model does not use any temporal information to perform this selection and is permutation-invariant, processing unordered frame embeddings with self-attention operations (without any sequence positional information). Further, we ensure that the learnable portion of ATP remains low capacity, with only a few, small layers and number of heads.

ATP (Context). We illustrate an overview of our ATP model in the larger video-language task context in Figure 3(a). For each video $V\in C_{V}$, we draw a random sparse (shuffled) subset of frames $F=\{v_{1},\dots,v_{n}\}\in V$, where usually $n<<|V|$, the length of the video. We also take as input to our task a pretrained, self-supervised image-language model $M=\{M_{I},M_{L}\}$, which consists of two components $M_{I}$ and $M_{L}$ for the vision and language components, respectively. These are used to encode all video $V$ and language $L$ inputs to the original video-language task.

We proceed to encode each of the frames with the pre-trained vision encoder $M_{I}(F)=\{x_{1},\dots,x_{n}\}$ to get vision embeddings $x_{i}$ corresponding to each frame $v_{i}$. Intuitively, because our encoder is completely frozen and never updated, $x_{i}$ is a representation of what an image-constrained visual encoder can discern; no additional information of the broader video is encoded here. Furthermore, our model treats the set $\{x_{1},\dots,x_{n}\}$ as an unordered set, without any temporal positional information.

where the goal is to select a single representation $x_{i}\in\{x_{1},\dots,x_{n}\}$ to pass to the final video-language task. Depending on the original video-language task formulation, ATP can take additional language inputs $M_{L}(L)$ (e.g. the encoded question for video question answering; Sec. 4.1).

ATP (Selection). In Figure 3(b), we illustrate a more detailed view of the ATP selection operation. Given the inputs provided by the frozen pre-trained image and language encoders, the ATP model must now perform embedding selection, passing one of these input visual embeddings, unmodified, to the downstream video-language task. To accomplish this, ATP first encodes the (unordered, shuffled) input image encoding sequence $\{x_{1},\dots,x_{n}\}$ with a learnable selector encoder $E_{s}$ as follows:

where $\{s_{1},\dots,s_{n}\}$ correspond to the original $\{x_{1},\dots,x_{n}\}$ and are only used for selection. We instantiate $E_{s}$ in our work as low-capacity transformer , with 3 or fewer layers and heads: we choose a self-attention based encoder here because it is conducive towards permutation invariant model design . Because our original embedding sequence $\{x_{1},\dots,x_{n}\}$ is unordered, and we provide no positional encodings (only learnable modality encodings to differentiate vision from language inputs), this operation is thus strictly atemporal.We include detailed experimental analysis and discussion of ATP atemporality (including relative vs. absolute encoder designs) in the supplement. These encodings $\{s_{1},\dots,s_{n}\}$ are input to a final multilayer perceptron (MLP) to obtain logits for the final selection operation:

Our final selection operation ($S(g)\mapsto x_{i}$) is discrete: ATP must select a single embedding $x_{i}$. To ensure learnability, we consider two versions of our selector $S$ during training, both operating on the logits $g$: the first employs a straight-through Gumbel-Softmax estimator , the second applies softmax and ensures entropy decreases over time . In either case, at final test-time inference, the operation is made fully discrete; see supplement for details.

Training. ATP is trained within the context of the overall video-language task framework, where the groundtruth answer or retrieval supervises the task loss, and gradients are backpropagated into the learnable ATP parameters. We re-iterate that no modifications are made to the frozen image-language encodings, and the final video-language task is performed directly on these frozen representations without any additional downstream learnable parameters. For both tasks, we optimize for the groundtruth similarity between the vision and language encodings. For video question answering, we consider a cross entropy loss over the answer set , and for retrieval our loss is based on the standard InfoNCE contrastive loss ; see supplement for details.

4 Improving Temporal Modeling with ATP

In the final part of our experiments (Section 4), we additionally consider how our learned ATP embedding selector models (in Section 3.3) can improve downstream temporal models (Figure 4). Intuitively, ATP learns to be an effective (language-conditional) event recognizer; building on this intuition, we propose a straightforward model that partitions the original video $V$ into $k$ partitions $V^{(1)},\dots,V^{(k)}$ and runs (a learned, now frozen) ATP model on each partition to obtain selected candidate embeddings $x_{i}^{(1)},\dots,x_{j}^{(k)}$ for the $k$ partitions. These per-partition outputs are then useful candidates for a separate, final learnable model $T$ that performs temporal reasoning and outputs a video-level embedding for the final video-language task. In Section 4.3 experiments, this downstream temporal model is a distinct transformer model, equipped to perform video-level reasoning on top of ATP’s output selections (for details, see supplement).

Experiments

Benchmarks. We consider three representative benchmarks for video question answering: NExT-QA , VALUE-How2QA , and MSR-VTT-MC . We also examine the generality of our ATP model for text-to-video retrieval on DiDeMo , MSR-VTT , and ActivityNet . For each benchmark, we follow standard protocols outlined by prior work for dataset processing, metrics, and settings; see supplement for details and analysis. We choose these benchmarks specifically to provide a broad coverage of durations, source video domains (general activities, instructional, etc.), and designs.

Implementation. We implement our ATP model with a few-layer, low-capacity transformer in PyTorch , and train all models using the Adam optimizer. Main paper results here reported on ViT-B-32 (CLIP) inputs for consistency . See supplementPlease see project website for supplementary material and code release. for more.

2 Analyzing Video-Language with ATP

Preliminary (upper bound) analysis. As a preliminary step, we first examine the performance of ATP under oracle conditions (with respect to the downstream video-language task) to establish a kind of upper bound for ATP on the set of benchmarks. In this analysis, we sample $n$ input frames from the video, varying $n$, and encode them with a pretrained model. In this oracle setting only, ATP then selects a frozen embedding from this set that maximizes the downstream groundtruth accuracy on the video-language task. Note that in this analysis, the oracle empowered ATP is still bottlenecked by what the image-level representation is able to capture. We repeat this analysis for multiple samples (dependent on the video lengths), and report the average. As shown in Figure 5, we observe that upper bound accuracies are competitive with state-of-the-art video models, even with relatively small $n$ sample sizes, suggesting the promise for analyzing these datasets with a learnable ATP.

ATP analysis (video QA). We apply a learnable ATP model to analyze a suite of standard video-language benchmarks. We first center our analysis discussion on video question-answering (video QA) benchmarks, since we find these benchmarks provide strong potential for deep multi-event understanding. Per Section 4.1, we focus on three representative benchmarks for analysis: NExT-QA , VALUE-How2QA , and MSR-VTT-MC . We re-iterate that our primary goal with ATP is one of analysis: to better characterize these instantiations of the video-language task. In Tables 1, 2, and 3, we report results for each benchmark.

On MSR-VTT-MC (Table 1), our learned ATP model outperforms recent state-of-the-art video-language models , when considering relatively few frames at inference and despite its substantial (single-frame) bottleneck constraints on model capacity, capability, and inputs. Critically, ATP substantially improves over standard atemporal baselines, including random single-frame and mean-pooling with CLIP , offering a stronger bound.

On VALUE-How2QA (Table 2), we find that our learned ATP model offers significantly stronger accuracies than prior state-of-the-art models. Note that the HERO baselines here also use the same input CLIP embeddings, and no auxiliary text inputs, for fair comparison. One takeaway finding from our analysis of this benchmark was that counting questions, often designed to track state over the course of a video, were in fact often addressable by a single well-chosen frame that showed sufficient number of the items.

Finally, on NExT-QA (Table 3), we find that even this recent benchmark, which is explicitly designed for temporal and causal understanding, can have a non-trivial subset of questions answerable by simple single-frame event recognition. In Figure 6, we show two different “causal-how” questions, which aim to assess both causality and dynamics. In the case of Figure 6(a) specifically, we observe that as long as the ATP model is able to select the informative frame with a clear depiction of the child on the cycle, the answer is readily apparent without deep video-level understanding. Quantitatively, our ATP model provides a stronger bound than standard image-level baselines; we also augment the the HGA baseline with CLIP features for fair comparison.

ATP analysis (retrieval). We also apply our learnable ATP model on standard retrieval benchmarks: DiDeMo , MSR-VTT , and ActivityNet . In Table 4, we observe that our technique generalizes well to other discriminative video-language settings, establishing stronger bounds on image-centric performance and showing competitive accuracies with recent state-of-the-art methods. Our ATP model’s performance on paragraph retrieval settings, like ActivityNet, highlights an area of improvement for image-bottleneck understanding: because paragraphs describe multiple dense events in long videos, it can be difficult to use a single frame embedding to capture this description well. We provide an extended discussion of prior work comparisons, limitations, and potential future directions for text-to-video retrieval as part of our supplement.

3 Improving Dataset and Model Design with ATP

Finally, we consider how to leverage our ATP model and its insights to improve both dataset and model design. For this section, we choose to focus on the NExT-QA benchmark as a case study, since it is a key recent effort towards improving the field’s focus on causal and temporal understanding in video-language tasks.

Improving dataset design with ATP. From our initial analysis of the NExT-QA benchmark in Section 4.2, we found that ATP provides a surprising degree of accuracy on causal and temporal questions, despite its strong image-centric bottleneck. Because ATP provides a stronger bound on the capability of image-level understanding for these questions, it can help better disentangle questions that necessitate full video-level understanding (such questions will be largely unanswerable for the ATP model) from ones that do not.

We accomplish this by considering an ensemble of ATP models on the dataset, and leveraging their confidences and agreement to determine a subset of ATPhard questions. We determine any heuristics through k-fold cross validation on the training set. In parallel, we manually annotate a subset of the validation set for (video, question) pairs that predicate video-level understanding (see supplement for procedure details and limitations of our ATP technique). The results of our final analysis are shown in Figure 7. We find that our ATP based technique maintains the recall of the video-level understanding questions on both the causal and temporal dataset splits, while simultaneously improving upon their precision (by filtering out “easy” questions).

Furthermore, we can also show how this ATPhard subset better benchmarks progress on video-level causal and temporal understanding (in Table 3) that may have been otherwise obscured. While ATP nearly matches the other models on the main dataset due to the inclusion of “easier” questions, this harder subset reveals a substantial gap relative to the state-of-the-art temporal reasoning model.

Together, these results suggest ATP in-the-loop can be an effective tool during future dataset design and creation.

Improving model design with ATP. As described in Section 3.4, we can leverage ATP to provide candidate frame embeddings for a downstream temporal model. As a first step towards improving temporal modeling (and efficiency), we introduce this model (denoted Temp[ATP] in Table 3) and benchmark it on the NExT-QA dataset. This model achieves a new state-of-the-art accuracy, outperforming the HGA (and HGA + CLIP) baselines on the main NExT-QA dataset, while operating at significantly reduced processing cost due to ATP (see supplement for efficiency discussion).

We make two additional observations: first, on the ATPhard subset, we find that this temporal model recovers much of the performance gap between ATP and the HGA model (we attribute the remaining gap to HGA’s incorporation of additional motion features, which can aid in addressing some challenging dynamics questions), further verifying the potential dataset design contribution of ATP. Second, we observe that the aggregated confidence scores of the ATP ensemble provides a clear disentanglement signal on hard vs. easy problems, without access to groundtruth. Setting a heuristic threshold with k-fold cross validation on training, we use this signal to smartly ensemble ATP and Temp[ATP] further. For questions ATP can address, we do not need to consider additional (potentially noisy) frames, and we can skip the full temporal model. For ones ATP is less confident, the temporal model is run. This ensemble (denoted Temp[ATP] + ATP in Table 3) achieves a significant further accuracy and efficiency increase on NExT-QA.

Conclusion

In this work, we revisit a fundamental question of video understanding (what makes a video task uniquely suited for videos, beyond what can be understood from a single image?), building beyond prior analyses in action recognition towards video-language settings with more complex events. First, we propose an atemporal probe (ATP) model to provide a stronger bound on how much of video-language understanding can be addressed from image-language understanding only. Second, we use ATP to characterize both the limitations and potential of current video-language benchmarks for video question answering and video-language retrieval. Surprisingly, we find that single frame understanding can often achieve strong performance, even in settings intended for complex multi-frame event understanding and compared with recent large-scale video models. Third, we show how ATP can be leveraged to improve designs for both video-language datasets (disentangling unintentional atemporal biases) and video-level models (improving efficiency and accuracy). Going forward, we envision ATP as joining a broader, standard toolkit for video-language researchers and practitioners, revealing insights into complementary, video-specific sources of bias in multimodal video settings.

Broader Impacts and Limitations. We provide a detailed discussion of limitations and implications for broader impacts of our proposed ATP and analysis in our supplement.

Acknowledgements. This work is supported in part by Toyota Research Institute (TRI), the Stanford Institute for Human-Centered AI (HAI), Samsung, Salesforce, and an NDSEG Fellowship (for S.B.). This article reflects the authors’ opinions and conclusions, and not any other entity. We also thank our colleagues in the Stanford Vision and Learning Lab (including Jim Fan, Ajay Mandlekar, Andrey Kurenkov, Sanjana Srivastava, Boxiao Pan, Ehsan Adeli) and the Stanford NLP Group (including Alex Tamkin, Percy Liang, Siddharth Karamcheti) for valuable discussions and support. We also thank Linjie Li and the authors of benchmarks examined here for their help towards enabling more direct comparisons with prior work, as well as our anonymous reviewers for helpful suggestions and feedback.

References