Learning Video Object Segmentation from Unlabeled Videos

Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu-Wing Tai, David Crandall, Steven C. H. Hoi

Introduction

Video object segmentation (VOS) has two common settings, zero-shot and one-shot. Zero-shot VOS (Z-VOS)Some conventions also use ‘unsupervised VOS’ and ‘semi-supervised VOS’ to name the Z-VOS and O-VOS settings . In this work, for notational clarity, the terms ‘supervised’, ‘weakly supervised’ and ‘unsupervised’ are only used to address the different learning paradigms. is to automatically segment out the primary foreground objects, without any test-time human supervision; whereas one-shot VOS (O-VOS) focuses on extracting the human determined foreground objects, typically assuming the first-frame annotations are given ahead inference 1. Current leading methods for both Z-VOS and O-VOS are supervised deep learning models that require extensive amounts of elaborately annotated data to improve the performance and avoid over-fitting. However, obtaining pixel-wise segmentation labels is labor-intensive and expensive (Fig. 1 (a)).

It is thus attractive to design VOS models that can learn from unlabeled videos. With this aim in mind, we develop a unified, unsupervised/weakly supervised VOS method that mines multi-granularity cues to facilitate video object pattern learning (Fig. 1 (b)). This allows us to take advantage of nearly infinite amounts of video data. Below we give a more formal description of our problem setup and main idea.

Problem Setup and Main Idea. Let $\mathcal{X}$ and $\mathcal{Y}$ denote the input video space and output VOS space, respectively. Deep learning based VOS solutions seek to learn a differentiable, ideal video-to-segment mapping $g^{*}\!\!\!:\!$ $\mathcal{X}<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>↦</mo></mrow><annotation encoding="application/x-tex">\mapsto</annotation></semantics></math>↦\mathcal{Y}$ . To approximate $g^{*\!}$ , recent leading VOS models typically work in a supervised learning manner, requiring $N$ input samples and their desired outputs $y_{n}\!\!:=\!g^{*\!}(x_{n})$ , where $\{(x_{n},y_{n})\}_{n\!}\!\subset<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi mathvariant="script">X</mi></mrow><annotation encoding="application/x-tex">\mathcal{X}</annotation></semantics></math>X\times$ $\mathcal{Y}$ . In contrast, we address the problem in settings with much less supervision: (1) the unsupervised case, when we only have samples drawn from $\mathcal{X}$ , $\{x_{n}\}_{n\!}\!\subset$ $\mathcal{X}$ , and want to approximate $g^{*}$ , and (2) the weakly supervised learning setting, in which we have annotations for $\mathcal{K}$ , which is a related output domain for which obtaining annotations is easier than $\mathcal{Y}$ , and we approximate $g^{*}$ using samples from $\mathcal{X}<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>×</mo></mrow><annotation encoding="application/x-tex">\times</annotation></semantics></math>×\mathcal{K}$ .

The standard way of evaluating learning outcomes follows an empirical risk/loss minimization formulation :

In our method, the prior knowledge $\mathcal{Z}$ , in the unsupervised learning setting, is built upon several heuristics and intrinsic properties of VOS itself, while in the weakly supervised learning setting, it additionally considers a related, easily-annotated output domain $\mathcal{K}$ . For example, part of the fore-background knowledge could be from a saliency model (Fig. 1 (b)), or in a form of CAM maps from a pre-trained image classifier (i.e., a related image classification domain $\mathcal{K}$ )Note that any unsupervised or weakly supervised object segmentation/saliency model can be used; saliency , and CAM are just chosen due to their popularity and relatively high performance.. Exploring VOS in an unsupervised or weakly supervised setting is appealing not only because it alleviates the annotation burden of $\mathcal{Y}$ , but also because it inspires an in-depth understanding of the nature of VOS by exploring $\mathcal{Z}$ . Specifically, we analyze several different types of cues at multiple granularities, which are crucial for video object pattern modeling:

At the frame granularity, we leverage information from an unsupervised saliency method or CAM activation maps to enhance the foreground and background discriminability of our intra-frame representation.

At the short-term granularity, we impose the local consistency within the representations of short video clips, to describe the continuous and coherent visual patterns within a few seconds.

At the long-range granularity, we address semantic correspondence among distant frames, which makes the cross-frame representations robust to local occlusions, appearance variations and shape deformations.

At the whole-video granularity, we encourage the video representation to capture global and compact video content, by learning to aggregate multi-frame information and be discriminative to other videos’ representations.

All these constraints are formulated under a unified, multi-granularity VOS (MuG) framework, which is fully differentiable and allows unsupervised/weakly supervised video object pattern learning, from unlabeled videos. Our extensive experiments over various VOS settings, i.e., object-level Z-VOS, instance-level Z-VOS, and O-VOS, show that MuG outperforms other unsupervised and weakly supervised methods by a large margin, and continuously improves its performance with more unlabeled data.

Related Work

Z-VOS. As there is no indication for objects to be segmented, conventional ZVOS methods resorted to certain heuristics, such as saliency , object proposals , and discriminative motion patterns . Recent advances have been driven by deep learning techniques. Various effective networks have been explored, from some early, relatively simple architectures, such as recurrent network , and two-stream network , to recent, more powerful designs, such as teacher-student adaption , neural co-attention and graph neural network .

O-VOS. As the annotations for the first frame are assumed available at the test phase, O-VOS focuses on how to accurately propagate the initial labels to subsequent frames. Traditional methods typically used optical flow based propagation strategy . Now, deep learning based solutions become the main stream, which can be broadly classified into three categories, i.e., online learning, propagation and matching based methods. Online learning based methods fine-tune the segmentation network for each test video on the first-frame annotations. Propagation based methods rely on the segments of the previous frames and work in a frame-by-frame manner. Matching based methods segment each frame according to its correspondence/matching relation to the first frame.

Typically, current deep learning based VOS solutions, under either Z-VOS or O-VOS setting, are trained using a large amount of elaborately annotated data for supervised learning. In contrast, the proposed method trains a VOS network from scratch using unlabeled videos. This is essential for understanding how visual recognition works in VOS and for narrowing down the annotation budget.

2 VOS with Unlabeled Training Videos

Learning VOS from unlabeled videos is an essential, yet rarely touched avenue. Among a few efforts, represents an early attempt in this direction, which uses a modified, purely unsupervised version of to generate proxy masks as pseudo annotations. With a similar spirit, some methods use heuristic segmentation masks or weakly supervised location maps as supervisory signals. With a broader view, some works capitalized on untrimmed videos tagged with semantic labels. In addition to increased annotation efforts, they are hard to handle such a class-agnostic VOS setting. Recently, self-supervised video learning has been applied for O-VOS , which imposes the learned features to capture certain constraints on local coherence, such as cross-frame color consistency and temporal cycle-correspondence .

Our method is distinctive for two aspects. First, it explores various intrinsic properties of videos as well as class-agnostic fore-background knowledge in a unified, multi-granularity framework, bringing a more comprehensive understanding of visual patterns in VOS. Second, it shows strong video object representation learning ability and, for the first time, it is applied to diverse VOS settings after only being trained once. This gives a new glimpse into the connections between the two most influential VOS settings.

Proposed Algorithm

Frame Granularity Analysis: Fore-background Knowledge Understanding. As $\varphi$ is VOS-aware, basic fore-background knowledge is desired to be encoded. In our method, such knowledge (Fig. 1 (b)) is initially from a background prior based saliency model (in an unsupervised learning setting), or in a form of CAM maps (in a weakly supervised learning setting).

Formally, for each frame $X_{t}$ , let us denote its corresponding initial fore-background mask as $Q_{t}\!\in\!\{0,1\}^{W\!\times\!H}$ (i.e., a binarized saliency or CAM activation map). In our frame granularity analysis, the learning of $\varphi$ is guided by the supervision signals of $\{Q_{t}\}_{t=1}^{T}$ , i.e., utilizing the intra-frame information $\textbf{{x}}_{t\!}\!=_{\!}\!\varphi(X_{t})$ to regress $Q_{t}$ :

Short-Term Granularity Analysis: Intra-Clip Coherence Modeling. Short-term coherence is an essential property in videos, as temporally close frames typically exhibit continuous visual content changes . To capture this property, we apply a forward-backward patch tracking mechanism . It learns $\varphi$ by tracking a sampled patch forwards in a few successive frames and then backwards until the start frame, and penalizing the distance between the initial and final backwards tracked positions of that patch.

where ${}_{\!}$ $S_{\Rightarrow\!}$ is a sigmoid-normalized response map whose size is rescaled into ${}_{\!}$ $(H_{\!},W)$ . The new location of $p$ in ${}_{\!}$ $X_{t+1\!}$ is then inferred according to the peak value on ${}_{\!}$ $S_{\Rightarrow}$ . After obtaining the forward tracked patch $p^{\prime}$ in ${}_{\!}$ $X_{t+1}$ , we backward track ${}_{\!}$ $p^{\prime}_{\!}$ to ${}_{\!}$ $X_{t\!}$ and ${}_{\!}$ get ${}_{\!}$ a ${}_{\!}$ backward tracking response map ${}_{\!}$ $S_{\Leftarrow}$ :

Ideally, the peak of $S_{\Leftarrow\!}$ should correspond to the location of $p$ in the initial frame $X_{t}$ . Thus we build a consistency loss that measures the alignment error between the initial and forward-backward tracked positions of $p$ :

where $G_{p}\!\in\!^{W\!\times\!H}$ is a $(H,W)$ -dimensional Gaussian-shape map with the same center of $p$ and variance proportional to the size of $p$ . As in , the above forward-backward tracking mechanism is extended to a multi-frame setting (Fig. 3). Specifically, after obtaining the forward tracked patch $p{{}^{\prime}\!}$ in $X_{t+1}$ , $p{{}^{\prime}\!}$ is further tracked to the next frame ${}_{\!}$ $X_{t+2}$ , and ${}_{\!}$ a ${}_{\!}$ new ${}_{\!}$ tracked ${}_{\!}$ patch $p^{\prime\prime}$ is obtained. Then $p^{\prime\prime}\!$ is reversely tracked ${}_{\!}$ to ${}_{\!}$ $X_{t+1\!}$ and ${}_{\!}$ further ${}_{\!}$ to ${}_{\!}$ the ${}_{\!}$ initial ${}_{\!}$ frame $X_{t}$ , and the local consistency loss in Eq. 5 is computed. Moreover, during training, we first random sample a short video clip consisting of six successive frames. Then we perform above forward-backward tracking based learning strategy over three frames random drawn from the six-frame video clip. With above designs, $\varphi$ captures the spatiotemporally local correspondence and is content-discriminative (due to its cross-frame target re-identification nature).

Long-Term Granularity Analysis: Cross-Frame Semantic Matching. In addition to the local consistency among adjacent frames, there also exist strong semantic correlations among distant frames, as frames from the same video typically contain similar content . Capturing this property is essential for $\varphi$ , as it makes $\varphi$ robust to many challenges, such as appearance variants, shape deformations, object occlusions, etc. To address this issue, we conduct a long-term granularity analysis, which casts cross-frame correspondence learning as a dual-frame semantic matching problem (Fig. 4). Specifically, given a training pair of two disordered frames $(X_{i},X_{j})$ randomly sampled from $\mathcal{X}$ , we compute a similarity affinity $\textbf{{A}}_{i,j}$ between their embeddings: $(\varphi(X_{i}),\varphi(X_{j}))$ by a co-attention operation :

where $\textbf{{m}}_{i}$ is a 2-D spatial coordinate of $X_{i}$ , and $\textbf{{m}}_{j}$ the corresponding sampling coordinates in $X_{j}$ . Using $\tau_{i,j}$ , we can warp $X_{i}$ to $X_{j}$ . Similarly, we can also compute $\tau_{j,i}$ , i.e., a 2-D warping from $X_{j}$ to $X_{i}$ . Let us consider two sampling coordinates $\textbf{{m}}_{i}$ and $\textbf{{n}}_{j}$ in $X_{j}$ and $X_{i}$ , respectively, we introduce a semantic matching loss :

where $\Omega$ refers to the image lattice, $\textbf{{A}}_{i,j}(\textbf{{m}}_{i},\textbf{{o}}_{j})\!\in_{\!}\!$ gives the similarity value between the positions $\textbf{{m}}_{i}$ and $\textbf{{o}}_{j}$ in $X_{i}$ and $X_{j}$ , and $\iota(\textbf{{m}}_{i},\textbf{{o}}_{j})$ determines if the correspondence between $\textbf{{m}}_{i}$ and $\textbf{{o}}_{j}$ is geometrically consistent. ${}_{\!}$ If ${}_{\!}$ $||\textbf{{m}}_{i},\tau_{i,j}(\textbf{{o}}_{j})||_{\!}\!\leq_{\!}\!1$ , $\iota_{\!}\!=_{\!}\!1$ ; otherwise $\iota_{\!}\!=_{\!}\!0$ .

Video Granularity Analysis: Global and Discriminative Representation Learning. So far, we have used the pairwise cross-frame information in local and long terms to boost the learning of $\varphi$ . $\varphi$ is also desired to learn a compact and globally discriminative video representation. To achieve this, with a global information aggregation module, we perform a video granularity analysis within an unsupervised video embedding learning framework , which leverages supervision signals from different videos.

Starting ${}_{\!}$ with ${}_{\!}$ our ${}_{\!}$ global ${}_{\!}$ information ${}_{\!}$ aggregation ${}_{\!}$ module, we split $\mathcal{X}_{\!}\!=_{\!}\!\{_{\!}X_{t\!}\}_{t=1\!}^{T}$ into $K$ segments of equal durations: ${}_{\!}$ $\mathcal{X}_{\!}\!=\!\cup^{K}_{k=1}\mathcal{X}_{k}$ . For each segment $\mathcal{X}_{k}$ , we randomly sample a single frame, resulting in a $K_{\!}$ -frame abstract $\mathcal{X}^{\prime}_{\!}\!\!=_{\!}\!\{X_{t_{k}}\}_{k=1\!}^{K}$ of $\mathcal{X}$ . $\mathcal{X}^{\prime}_{\!}$ reduces the redundancy among successive frames while preserving global information.

With a similar spirit of key-value retrieval networks , for each $X_{t_{k}}\!\in\!\mathcal{X}^{\prime}$ , we set it as a query and the rest frames $\mathcal{X}^{\prime}/X_{t_{k}}$ as reference. Then we compute the normalized cross-correlation between the query and reference:

Our global information aggregation module gathers information from the reference set by a correlation-based feature summarization procedure. For the query frame $X_{t_{k}}$ , we obtain its global information augmented representation by:

During training, the video granularity analysis essentially discriminates between a set of surrogate video classes . Specifically, given $N$ training videos, we randomly sample a single frame from each video, leading to $N$ training instances: $\{X^{n}\}_{n=1}^{N}$ . The core idea is that, for a query frame $X^{n}_{t_{k}}$ in the $n$ -th video, its global feature embedding is close to the instance $X^{n}$ from the same $n$ -th video, and far from other unrelated instances $\{X^{n^{\prime}}\}_{n^{\prime}\neq n}$ (from the other ${}_{\!}$ $N_{\!}\!-_{\!}\!1$ videos). We ${}_{\!}$ solve ${}_{\!}$ this ${}_{\!}$ as ${}_{\!}$ a ${}_{\!}$ binary ${}_{\!}$ classification problem via maximum likelihood estimation (MLE). In particular, for $X^{n}_{t_{k}}$ , instance $X^{n}$ should be classified into $n$ , while other instances $\{X^{n^{\prime}}\}_{n^{\prime}\neq n}$ shouldn’t be. The probability of $X^{n}$ being recognized as instance $n$ is:

where ‘GAP’ stands for global average pooling. Similarly, given $X^{n}_{t_{k}}$ , the probability of other instances $X^{n^{\prime}}$ be recognized as instance $n$ is:

Correspondingly, the probability of $X^{n^{\prime}\!}$ not being recognized as instance ${}_{\!}$ $n$ is ${}_{\!}$ $1_{\!}\!-_{\!}\!P(n|X^{n^{\prime}\!})$ . The ${}_{\!}$ joint ${}_{\!}$ probability of ${}_{\!}$ $X^{n\!}$ being ${}_{\!}$ recognized ${}_{\!}$ as ${}_{\!}$ instance ${}_{\!}$ $n_{\!}$ and ${}_{\!}$ $X^{n^{\prime}\!}$ not ${}_{\!}$ being is: $P(n|X^{n})\prod_{n^{\prime}\neq n}(1\!-\!P(n|X^{n^{\prime}\!}))$ , under the assumption that different instances being recognized as $n$ are independent.

Then the loss function is defined as the negative log likelihood over $N$ query frames from $N$ videos:

Next we will describe the network architecture during the training and inference phases. An appealing advantage of our multi-granularity VOS network is that, after being trained in a unified mode, it can be directly applied to both Z-VOS and O-VOS settings with only slight adaption.

Network Architecture. Our whole module is end-to-end trainable. The video representation space $\varphi$ is learned by a fully convolutional network, whose design is inspired by ResNet-50 . In particular, the first four groups of convolutional layers in ResNet are preserved and dilated convolutional layer is used to maintain enough spatial details as well as ensure a large receptive field, resulting in a 512-channel feature representation x whose spatial dimensions are $1/4$ of an input video frame $X$ .

During training, we use a mini-batch of $N_{\!}\!=_{\!}\!16$ videos and scale all the training frames into $256_{\!}\!\times_{\!}\!256$ pixels. For frame granularity analysis, all the frames access to the supervision signal from the loss $\mathcal{L}_{\text{frame}}$ in Eq. 2.

Iterative Training by Bootstrapping. As seen in Fig. 1 (b), the fore-background knowledge from the saliency or CAM is ambiguous and noisy. Inspired by Bootstrapping , we apply an iterative training strategy: after training with the initial fore-background maps, we use our trained model to re-label the training data. With each iteration, the learner bootstraps itself by mining better fore-background knowledge and then leading a better model. Specifically, for each training frame $X$ , given the initial fore-background mask $Q\!\in\!\{0,1\}^{64\!\times\!64}$ and current prediction $\bar{P}^{i}\!\in\!\{0,1\}^{64\!\times\!64}$ of the model in $i$ -th training iteration, the loss in Eq. 2 in $(i\!+\!1)$ -th iteration is formulated in a bootstrapping format:

where $\alpha\!=\!0.05$ and $Q_{\textbf{{m}}}$ gives the value in position m. In such a design, the ‘confident’ fore-background knowledge is generated as a convex combination of the initial fore-background information $Q$ and model prediction $P$ .

In the $i$ -th training iteration, the overall loss to optimize the whole network parameters is the combination of the losses in Eq. 15, 4, 8 and 14:

where $\beta$ s are coefficients: $\beta_{1}\!=\!0.1,\beta_{2}\!=\!0.02$ and $\beta_{3}\!=\!0.5$ .

The above designs enable a unified un-/weakly supervised feature learning framework. Once the model is trained, the learned representations $\varphi$ can be used for Z-VOS and O-VOS, with slight modifications. In practice, we find that our model can perform well after being trained with 2 iterations; please see §4.2 for related experiments.

3 Inference Modes for Z-VOS and O-VOS

Now we detail our inference modes for object-level Z-VOS, instance-level Z-VOS, and O-VOS settings.

Instance-Level Z-VOS Setting. Our model can also be adapted for the instance-level Z-VOS setting, in which different object instances must be discriminated, in addition to separating the primary video objects from the background without test-time human supervision. For each test frame, we first apply mask-RCNN to produce a set of category agnostic object proposals.Then we apply our trained model for producing a binary foreground-background mask per frame. After combining object bounding-box proposals with binary object-level segmentation masks, we can filter out the background proposals and obtain pixel-wise, instance-level object candidates for each frame. Finally, to link those object candidates across different frames, similar to , we use overlap ratio and optical flow as the cross-frame candidate-association metric. Note that, mask-RCNN can be replaced with non-learning Edgebox and GrabCut, resulting a purely unsupervised/weakly-supervised protocol.

O-VOS Setting. In O-VOS, for each test video sequence, instance-level annotations regarding multiple general foreground objects in the first frame are given. In such a setting, our trained network works in a per-frame matching based mask propagation fashion. Concretely, assume there are a total of $L$ object instances (including the background) in the first-frame annotation, each spatial position $\textbf{{n}}_{\!}\!\in_{\!}\!\Omega$ will be associated with a one-hot class vector $\hat{\textbf{{y}}}_{\textbf{{n}}\!}\!\in_{\!}\!\{0,1\}^{L}$ , whose element $\hat{y}^{l}_{\textbf{{n}}\!}$ indicates whether pixel $\textbf{{n}}_{\!}$ belong to $l$ -th object instance. Starting from the second frame, we use both the last segmented frame $X_{t-1}$ as well as current under-segmented frame $X_{t}$ to build an input pair for our model. Then we compute their similarity affinity $\textbf{{A}}_{t-1,t\!}\!\in_{\!}\!^{(64\!\times\!64)\!\times\!(64\!\times\!64)}$ in the feature space: $\textbf{{A}}_{t-1,t\!}\!=_{\!}\!\text{softmax}({\textbf{{x}}_{t-1}}^{\!\!\!\top}\textbf{{x}}_{t})$ . After that, for each pixel m in $X_{t}$ , we compute its probability distribution $\textbf{{v}}_{\textbf{{m}}}\!\in\!^{L}$ over the $L$ object instances as:

where $\textbf{{A}}_{t-1,t}(\textbf{{n}},\textbf{{m}})\!\in_{\!}\!_{\!}$ is the affinity value between pixel n in $X_{t-1}$ and m in $X_{t}$ . For m, it is assigned to $l^{*\!}$ -th instance: $l^{*\!}\!=_{\!}\!\mathop{\arg\max}_{l}(\{v^{l}_{\textbf{{m}}}\}_{l=1}^{L})$ , where $\textbf{{v}}_{\textbf{{m}}\!}\!=\![v^{l}_{\textbf{{m}}}]_{l=1}^{L}$ . Then we get its label vector $\hat{\textbf{{y}}}_{\textbf{{m}}}$ . In this way, from the segmented frame $X_{t}$ , we move to the next input frame pair $(X_{t},X_{t+1})$ and get the segmentation result for $X_{t+1}$ . As our method does not use any first-frame fine-tuning or online learning technique, it is fast for inference.

Experiment

Implementation Details. We train the whole network from scratch on the OxUvA tracking dataset, as in . OxUvA comprises 366 video sequences with more than 1.5 million frames in total. We train our model with SGD optimizer. For our bootstrapping based iterative training, two iterations are used and each takes about 8 hours.

Configuration and Reproducibility. MuG is implemented on PyTorch. All experiments are conducted on an Nvidia TITAN Xp GPU and an Intel (R) Xeon E5 CPU. All our implementations, trained models, and segmentation results will be released to provide the full details of our approach.

2 Diagnostic Experiments

A series of ablation studies are performed for assessing the effectiveness of each essential component of MuG.

Initial Fore-Background Knowledge. Baselines Heuristic Saliency and CAM give the scores of initial fore-background knowledge, based on their CRF-binarized outputs. As seen, with the low-quality initial knowledge, our MuG gains huge performance improvements ( $+20.8\%$ and $+15.9\%$ promotions), showing the significance of our multi-granularity video object pattern learning scheme.

Multi-Granularity Analysis. Next we investigate the contributions of multi-granularity cues in depth. As shown in Table 1, the intrinsic, multi-granularity properties are indeed meaningful, as disabling any granularity analysis component causes performance to erode. For instance, removing the frame granularity analysis during learning hurts performance (mean $\mathcal{J}$ : $\!58.0\!\rightarrow\!40.2$ , $\!61.2\!\rightarrow\!40.2$ ), due to the lack of fore-/background information. Similarly, performance drops when excluding short- or long-term granularity analysis, suggesting the importance of capturing local consistency and semantic correspondence. Moreover, considering video granularity information also improves the final performance, proving the meaning of comprehensive video content understanding in video object pattern modeling.

Iterative Training Strategy. From Table 1, we can see that with more iterations of our bootstrapping training strategy ( $1\!\rightarrow\!2$ ), better performance can be obtained. However, further iterations ( $2\!\rightarrow\!4$ ) give only marginal performance change. We thus use two iterations in all the experiments.

More Training Data. To show the potential of our unsupervised/weakly supervised VOS learning scheme, we probe the upper bound by training on additional videos. With more training data (1400 videos) from LaSOT dataset , performance boosts can be observed in both two settings.

3 Performance for Object-Level Z-VOS

Datasets. Experiments are conducted on two famous Z-VOS datasets: DAVIS and Youtube-Objects , which have pixel-wise, object-level annotations. DAVIS16 has 50 videos (3,455 frames), covering a wide range of challenges, such as fast motion, occlusion, dynamic background, etc. It is split into a train set (30 videos) and a val set (20 videos). Youtube-Objects contains 126 video sequences that belong to 10 categories (such as cat, dog, etc.) and has 25,673 frames in total. The val set of DAVIS16 and whole Youtube-Objects are used for evaluation.

Evaluation Criteria. For fair comparison, we follow the official evaluation protocols of each dataset. For DAVIS16, we report region similarity $\mathcal{J}$ , boundary accuracy $\mathcal{F}$ and time stability $\mathcal{T}$ . For Youtube-Objects, the performance is evaluated in terms of region similarity $\mathcal{J}$ .

Post-processing. Following the common protocol in this area , the final segmentation results are optimized by CRF (about 0.3s per frame).

Quantitative Results. Table 2 presents the comparison results with several non-learning, unsupervised or weakly supervised learning competitors in DAVIS16 dataset. In particular, MuG exceeds current leading unsupervised learning-based methods (i.e., Motion Masks and TSN ) in large margins (58.0 vs 48.9 and 58.0 vs 31.2). MuG also outperforms classical weakly-supervised Z-VOS method COSEG , and all the previous heuristic methods. Table 3 summarizes comparison results on Youtube-Objects dataset, showing again our superior performance in both unsupervised and weakly supervised learning settings.

Runtime Comparison. The inference time of MuG is about 0.6s per frame, which is faster than most deep learning based competitors (e.g., MotionMask (1.1s), TSN (0.9s)). This is because, except CRF , there is no other pre-/post-processing step (e.g., superpixel , optical flow , etc.) and online fine-tuning .

4 Performance for Instance-Level Z-VOS

Datasets. We test the performance for instance-level Z-VOS on DAVIS17 dataset, which has 120 videos and 8,502 frames in total. It has three subsets, namely, train, val, and test-dev, containing 60, 30, and 30 video sequences, respectively. We use the ground-truth masks provided by the newest DAVIS challenge , as the original annotations are biased towards the O-VOS scenario.

Evaluation Criteria. Three standard evaluation metrics, provided by DAVIS17, are used, i.e., region similarity $\mathcal{J}$ , boundary accuracy $\mathcal{F}$ and the average value of $\mathcal{T}\&\mathcal{F}$ .

Quantitative Results. Three top-performing ZVOS methods from the DAVIS17 benchmark are included. As shown in Table 4, our model achieves comparable performance with the fully supervised methods (i.e., AGS and PDB ). Notably, it significantly outperforms recent RVOS (mean $\mathcal{T}\&\mathcal{F}$ : $+14.8\%$ and $+19.2\%$ in unsupervised and weakly-supervised learning setting, respectively).

Runtime Comparison. The processing time for each frame is about 0.7s which is comparable to AGS and PDB , and slightly slower than RVOS (0.3s).

5 Performance for O-VOS

Datasets. DAVIS16 and DAVIS17 datasets are used for performance evaluation under the O-VOS setting.

Evaluation Criteria. Three standard evaluation criteria are reported: region similarity $\mathcal{J}$ , boundary accuracy $\mathcal{F}$ and the average value of $\mathcal{T}\&\mathcal{F}$ . For DAVIS16 dataset, we further report the time stability $\mathcal{T}$ .

Quantitative Results. Table 5 and Table 6 give evaluation results on DAVIS16 and DAVIS17, respectively. Table 5 shows that our unsupervised method exceeds representative self-supervised methods (i.e., TimeCyle and CorrFlow ) and the best non-learning method (i.e., BVS ) across most metrics. In particular, with the learned CAM as supervision, our weakly supervised method further improves the performance, e.g., mean $\mathcal{J}$ of 65.7. Table 6 verifies again our method performs favorably against the current best unsupervised method, CorrFlow, according to mean $\mathcal{T}\&\mathcal{F}$ (54.3 vs 50.3). Note that CorrFlow and our method use the same training data. This demonstrates our MuG is able to learn more powerful video object patterns, compared to previous self-learning counterparts.

Runtime Comparison. In instance-level Z-VOS setting, MuG runs about 0.4s per frame. This is faster than matching based methods (e.g., SIFT Flow (5.1s) and mgPFF (1.3s)), and favorably against self-supervised learning methods, e.g., TimeCycle and CorrFlow .

6 Qualitative Results

Fig. 5 presents some visual results for object-level ZVOS (top row), instance-level Z-VOS (middle row) and O-VOS (bottom row). For blackswan in DAVIS16 , the primary objects undergo view changes and background clutter, but our MuG still generates accurate foreground segments. The effectiveness of instance-level Z-VOS can be observed in tram of DAVIS17 . In addition, MuG can produce high-quality results with the given first-frame annotations in O-VOS setting (see the results on the last row for scooter-black in DAVIS17 ), although the different instances suffer from fast motion and scale variation. More results can be found in supplementary materials.

Conclusion

We proposed MuG – an end-to-end trainable, unsupervised/weakly supervised learning approach for segmenting objects from the videos. Different from current popular supervised VOS solutions requiring extensive amounts of elaborately annotated training samples, our MuG models video object patterns by comprehensively exploring supervision signals from different granularities of unlabeled videos. Our model sets new state-of-the-arts over diverse VOS settings, including object-level Z-VOS, instance-level Z-VOS, and O-VOS. Our model opens up the probability of learning VOS from nearly infinite amount of unlabeled videos and unifying different VOS settings from a single view of video object pattern understanding.