RANet: Ranking Attention Network for Fast Video Object Segmentation

Ziqin Wang, Jun Xu, Li Liu, Fan Zhu, Ling Shao

Introduction

Semi-supervised Video Object Segmentation (VOS) aims to segment the object(s) of interests from the background throughout a video, in which only the annotated segmentation mask of the first frame is provided as the template frame at test phase. This challenging task is of great importance for large scale video processing and editing , and many video analysis applications such as video understanding and object tracking .

Early VOS methods mainly resort to online learning (OL) techniques which fine-tune a pre-trained classifier on its first frame. Matching or propagation based methods have also been proposed for VOS. Matching based methods segment pixels according to the pixel-level matching scores between the features of the first frame and of each subsequent frame (Fig. 1 (a)), while propagation based methods mainly rely on temporally deforming the annotated mask of the first frame via predictions of the previous frame (Fig. 1 (b)).

The respective benefits and drawbacks of these methods are clear. Specifically, OL based methods achieve accurate VOS at the expense of speed, requiring several seconds to segment each frame . On the contrary, simple matching or propagation based methods are faster, but with sub-optimal VOS accuracy. Matching based methods bear up the mismatching problem, i.e., violating the temporal consistency of the primary object with constantly changing appearance in the video. On the other hand, propagation based methods suffer from the drifting problem due to occlusions or fast motions between two sequential frames. In summary, most existing methods cannot tackle the VOS task with both satisfactory accuracy and speed, which are essential for practical applications. More efficient methods are still required to reach a better speed-accuracy trade-off for the VOS task.

With the above considerations, in this work, we develop a real-time network for fine-grained VOS performance. The developed network benefits from an encoder-decoder structure, and learns pixel-level matching, mask propagation, and segmentation in an end-to-end manner. Fig. 1 (c) shows a glimpse of the proposed network. A Siamese network is employed as the encoder to extract pixel-level matching features, and a pyramid-like decoder is used for simultaneous mask propagation and high-resolution segmentation.

A key problem in our framework is how to connect the pixel-level matching encoder and propagation based decoder in a meaningful manner. The encoder produces dynamic foreground and background similarity maps, which cannot be directly fed into the decoder. To this end, we propose a Ranking Attention Module (RAM, see Fig. 1 (c)) to reorganize (i.e., rank and select) the similarity maps according to their importance for fine-grained VOS performance. The proposed Ranking Attention Network (RANet) can better utilize the pixel-level similarity maps for fine-grained VOS, greatly alleviating the drawbacks of previous matching or propagation based methods. Experiments on DAVIS16 and DAVIS17 datasets demonstrate that the proposed RANet outperforms previous VOS methods in terms of speed and accuracy, e.g., achieving $\mathcal{J}\&\mathcal{F}=85.5\%$ at a speed of $30$ FPS on DAVIS16.

The contributions of this work are three-fold:

We integrate the benefits of matching and propagation frameworks in an end-to-end manner and develop a real-time network for the semi-supervised VOS task.

We propose a novel Ranking Attention Module to rank and select conformable feature maps according to their importance for fine-grained VOS performance.

Experiments on DAVIS ${}_{16}/_{17}$ datasets show that the proposed RANet achieves competitive or even better performance than previous VOS methods, at real-time speed. The proposed RANet achieves accurate VOS results even been trained only with static images.

Related Works

Online learning based methods. OL based methods fine-tune on the first frame of a video to extract the primary object(s), and then segment the video frame-by-frame. OSVOS uses a pre-trained object segmentation network, and fine-tunes it on the first frame of the test video. OnAVOS extends OSVOS with an online adaptation mechanism, and OSVOS-S utilizes semantic information from an instance segmentation network. LucidTracker introduces a data augmentation mechanism for online fine-tuning. DyeNet integrates instance re-identification and temporal propagation, and uses OL to boost the performance. PReMVOS integrates techniques from instance segmentation , optical flow , refinement, and re-identification together with extensive fine-tuning, and achieves satisfactory performance. In summary, OL is very effective for the VOS task. Therefore, subsequent methods regard OL as a conventional technique to boost VOS performance. However, OL based methods are computationally expensive for practical applications. In this work, we solve the VOS problem with a very fast network that obtains a competitive accuracy at a speed of $30$ FPS on DAVIS16, $130\sim 400$ times faster than previous OL based methods .

Propagation or matching based methods. Propagation based methods additionally resort to the previous frame(s) for better VOS performance. Masktrack tackles VOS by combining the image and segmentation mask of the previous frame as the input. This strategy is also used in CINM , OSMN and RGMP . RGMP stacks the first, previous and current frames’ features during propagation through a Siamese architecture network. In this work, we also utilize the Siamese network, but use a pixel-level matching technique instead of simply stacking, and feed the previous frame’s mask into the decoder, instead of the encoder as in RGMP . OSMN introduces a modulator to manipulate the intermediate layers of the segmentation network, by using visual and spatial guidance. Optical flow is also used to guide the propagation process in many methods . However, it fails to distinguish non-rigid objects from motionless sections of the background. All these strategies are effective, but still, suffer from the drifting problem. MaskTrack embraces OL to remember the target object, which eliminates this problem and improves VOS performance. However, since OL is time-consuming, we employ more efficient matching techniques to handle this drifting problem.

Matching based methods are very efficient. They first calculate pixel-level matching between the features of the template frame and the current frame in videos, and then segment each pixel of the current frame directly from the matching results. Pixel-Wise Metric Learning predicts each pixel by nearest neighbor matching in pixel space to the template frame. However, the point-to-point correspondence strategy often results in noisy predictions. To ease this problem, we apply a decoder to utilize the matching results as guidance. Hu et al. proposed a soft matching mechanism in VideoMatch , which performs soft segmentation upon the averaged similarity score maps of matching features to generate smooth predictions. However, due to the lack of temporal information, they still suffer from the mismatching problem. In this work, we employ both the strategies of point-to-point correspondence matching for pixel-level object location and temporal propagation, to handle the mismatching and drifting problem. FEELVOS employs global and local matching for more stable pixel-level matching, but only calculates extreme value maps for final segmentation, losing major information of the similarity maps. Our RAM can better utilize the similarity information. Moreover, for faster speed, we use a light-weight decoder and employ a standard ResNet pre-trained on ImageNet as the backbone, instead of the time-consuming semantic segmentation networks used in previous methods .

Proposed Method

In this section, we first provide an overview of the developed Ranking Attention Network (RANet) in §3.1. In §3.2, we describe the proposed Ranking Attention Module (RAM), and extend it for multi-object VOS in §3.3. Finally, we present the implementation details and training strategies for RANet in §3.4 and §3.5, respectively.

Our RANet consists of three seamless parts: an encoder for feature extraction, an integration of correlation and RAM, and a decoder for feature merging and final segmentation. An illustration of our RANet is shown in Fig. 2.

Siamese Encoder. To obtain correlation information for accurate VOS, we employ Siamese networks (with shared weights) as the encoder to extract features of the first frame and the current frame. Then we extract pixel-level features from the first frame, reshape it into a conformable shape, as the template features for correlation calculation.

Correlation and RAM for Matching. Correlation is widely used in object tracking. In SiamFC , correlation is used to locate the position of the object using similarity maps. In our RANet, to locate each pixel of the object(s) for segmentation, we need pixel-level similarity maps by calculating the correlation between each pixel-level feature of the template and current frames. Note that there is one similarity map for each pixel-level template feature. The detailed formulation of correlation will be described in §3.2. We then utilize the mask of the first frame to select foreground (FG) or background (BG) similarity maps as FG or BG features for segmentation. Since the number of FG or BG pixels varies in different videos, the number of FG or BG similarity maps is dynamic, and hence the decoder has to deal with FG or BG similarity features with dynamic channel sizes. To handle this dynamic channel-size problem, we propose a RAM module to rank and select the most important similarity maps and organize them in conformable shape. This part will also be exhaustively explained in §3.2. The RAM module provides abundant and ordered features for segmentation, and leads to better performance, as will be shown in the ablation study in §4.3. For simplicity, here we only consider the single-object VOS in §3.2. Extension of our RANet for multi-object VOS will be described in §3.3.

Propagation. Here we utilize the simple mask propagation method , while other propagation or local-matching methods would potentially improve our RANet. We feed the predicted mask of the previous frame, together with the selected features of FG (or BG) by the proposed RAM, into the subsequent decoder. In this way, our RANet utilizes both matching and propagation techniques.

Light-weight Decoder. This part contains a merge module and a pyramid-like network, which are described in the Supplementary File. The merge module refines the two streams of ranked similarity maps, and then concatenates these maps with previous frame’s mask. In the merge module, the two streams of the network share the same parameters. A pyramid-like network is employed to obtain the final segmentation, with skip-connections to utilize multi-scale features of different layers.

2 Correlation and Ranking Attention Module

In Fig. 4, we present some examples of the similarity maps. Each similarity map is associated with a certain pixel in template frame, whose new position in the current frame is at the maximum (i.e., brightest point) of the similarity map. Additionally, in contrast with SiamFC , since we obtain these maps in a weakly-supervised manner, the contours of the bear, which are essentially preserved for segmentation, are maintained. On the right side of Fig. 4, we show some output features of the merging module. The object can be distinguished after the merging networks.

Ranking Attention Module (RAM). We first utilize the mask of the first frame to filter FG and BG similarity maps. Then we design a FG path and a BG path network to process the similarity features. Since the number of the FG or BG pixels varies in different videos, the number of FG or BG similarity maps changes dynamically. However, regular CNNs require input features with a fixed number of channels. To tackle this issue, we propose a Ranking Attention Module (RAM) to rank and select important features. That is, we learn a scoring scheme for the similarity maps, and then rank and select these maps according to their scores.

Finally, we rank the similarity maps in $\mathbf{S}^{1}$ according to the corresponding scores in $\mathbf{r}^{1}$ from largest to smallest:

If the number of the FG similarity maps $\overline{\mathbf{S}}^{1}$ is less than the target channel size (set as 256), we pad the ranked feature with zero maps; and if the number is larger than the target channel size, the redundant features are discarded, such that the channel size can be fixed. The BG tensor $\mathbf{\hat{S}}^{0}$ are similarly processed. An illustration of the proposed ranking mechanism is shown in Fig. 3.

3 Extension for Multi-object VOS

A trivial extension of single-object VOS methods to perform multi-object VOS is to deal with the multiple objects in videos one-by-one. But this strategy would be inefficient when there are many objects. To make the proposed RANet efficient for multi-object VOS, we share the features extracted by the encoder and also the similarity maps $\mathbf{S}$ computed by correlation for all the $N$ objects. Then, for each object $i$ ( $i=1,...,N$ ), we generate its FG and the corresponding BG masks, and segment the FG (or BG) independently using the light-weight decoder. Finally, we use a softmax function to compute the final results on VOS.

4 Implementation Details

Here, we briefly describe the encoder and decoder, and present the detailed network structure in Supplemental File.

Decoder. The decoder is a three-level pyramid-like network with skip connection. The multi-scale features of current frame extracted by encoder are fed into the decoder. However, using all the features in the decoder would bring huge computational costs. To speed up our RANet, we first reduce the channel sizes of the multi-scale features using convolutional layers, and then feed them into the decoder.

5 Network Training

We train our network using the Adam with an initial learning rate of $10^{-5}$ , to optimize a binary cross-entropy loss. During training and test, the input image is resized into $480\times 864$ . We use random Thin Plate Splines (TPS) transformations, rotations ( $-30^{\circ}<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>∼</mo></mrow><annotation encoding="application/x-tex">\sim</annotation></semantics></math>∼30^{\circ}$ ), scaling ( $0.75<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>∼</mo></mrow><annotation encoding="application/x-tex">\sim</annotation></semantics></math>∼1.25$ ), and random cropping for data augmentation, just as . The random TPS transformations are performed by setting $16$ control points and randomly shifting the points within a $15\%$ margin of the image size.

Pre-train on static images. Following , we pre-train the proposed RANet using static images. To train our RANet for single-object VOS, we use the images from the MSRA10K , ECSSD , and HKU-IS datasets in the saliency community . To train RANet for multi-object VOS, we add the SOC and ILSO datasets containing multi-object images. Fig. 5 (a) shows a pair of generated static images. As will be shown in §4.2 and §4.3, the proposed RANet achieves competitive results when been trained only with static images.

Video fine-tuning. Though our RANet can achieve satisfactory results when been trained only with static images, we further exploit its performance by performing video fine-tuning on benchmark datasets. To fine-tune our RANet for specific single-object VOS task, we then fine-tune the network on the training set of the DAVIS16 dataset . During training, we randomly select two frames with data transformations from one video as the template and current frames, and randomly select the mask of a frame near the current frame (we set the maximum interval as 5). We fine-tune our RANet for specific multi-object VOS task on the training set of the DAVIS17 dataset . Fig. 5 (b) shows an example of paired video training images.

Experiments

In this section, we first describe our experimental protocol (§4.1), and then compare the proposed ranking attention network (RANet) with the state-of-the-art VOS methods (§4.2). We next perform a comprehensive ablation study to gain deeper insights into the proposed RANet, especially on the effectiveness of the ranking attention module (§4.3). Finally, we present the visual results to show the robustness of RANet against challenging scenarios (§4.4). More results are provided in the Supplementary File.

Training datasets. We evaluate the proposed RANet on the DAVIS16 and DAVIS17 datasets. The DAVIS16 dataset contains 50 videos ( $480$ p), annotated with pixel-level object masks (one per sequence) densely on the total 3455 frames, and it is divided into a training set ( $30$ videos), and a validation set ( $20$ videos). The DAVIS17 dataset , that contains videos with multiple objects, is an extension of DAVIS16, and it contains a training set with $60$ videos, a validation set with $30$ videos, and a test-dev set with $30$ videos. In all datasets, there is no overlap among the training, validation, and test sets.

Testing phase. Similar to SiamFC , we crop the first frame and extract the features as the template features ( $\mathcal{K}$ in §3.2), then compute the similarity maps between the features of template frame and of the test frames one-by-one, and finally segment the current test frame. The video data used are in different goals: 1) to evaluate our RANet for single-object VOS, we test it on the validation set ( $20$ videos) of ; 2) to judge the effectiveness of our RANet trained only on static images, we evaluate it on the $50$ videos of the whole DAVIS16 dataset; 3) to assess our RANet for multi-object VOS, we evaluate it on the validation and test sets of , which respectively contain $30$ videos. To compare with OL based methods, we follow , fine-tuning on the first frame with data augmentation for each video. We use the same training strategy as pre-training on static images, but the learning rate is $10^{-6}$ .

Evaluation metrics. We use seven standard metrics suggested by : three region similarity metrics $\mathcal{J}$ Mean, $\mathcal{J}$ Recall, $\mathcal{J}$ Decay; three boundary accuracy metrics $\mathcal{F}$ Mean, $\mathcal{F}$ Recall, $\mathcal{F}$ Decay; and $\mathcal{J}\&\mathcal{F}$ Mean, which is the average of $\mathcal{J}$ Mean and $\mathcal{F}$ Mean.

2 Comparison to the state of the art

Comparison Methods. For single object VOS, we compare our RANet with $6$ state-of-the-art OL based and $11$ offline methods in Table 1, including OSVOS-S , PReMVOS , RGMP , FEELVOS , etc. To evaluate our RANet trained with static images, we compare it with some methods without using DAVIS training set. For multi-object VOS, we compare with some state-of-the-art offline methods , and also list results of some OL based methods for reference.

Results on DAVIS16-val. As shown in Table 1, without online learning (OL) technique, our RANet still achieves a $\mathcal{J}\&\mathcal{F}$ Mean of $85.5\%$ at a speed of $33$ milliseconds ( $30$ FPS). For RANet, its metric results are higher than all the methods without OL techniques, while its speed is higher than all the compared methods, except SiamMask . But please note that SiamMask performs badly on objective metrics, e.g., $70.0\%$ at $\mathcal{J}\&\mathcal{F}$ , $15.5$ points lower than our RANet. Even when compared with the state-of-the-art OL based methods such as OSVOS-S and OnAVOS , our offline RANet achieves comparative results. The RANet can be improved by OL techniques. The OL boosted RANet, denoted as RANet+, achieves a $\mathcal{J}\&\mathcal{F}$ Mean of $87.1\%$ , outperforming all OL based VOS methods.

Results on DAVIS16-trainval. We also evaluate the performance of our RANet trained only with static images (i.e., without video fine-tuning). MaskTrack has the most similar setting as our RANet in this case, since it also uses only static images to train its networks. Contrast to MaskTrack, our RANet does not rely on OL techniques, speeding up for nearly a hundred times faster. In Table 2, we list the results of different methods that do not require fine-tuning/training on video data. Again, our RANet outperforms all the other methods by a clear margin.

Results on DAVIS17 dataset. The DAVIS17 dataset is challenging due to multi-object scenarios. To evaluate our RANet on DAVIS17-val and DAVIS17-test sets, we use the RANet trained on multi-instance static images and the DAVIS17-train dataset, as described in §3.5. In Table 3, we show the comparison of our RANet with state-of-the-art VOS methods. It can be seen that on the DAVIS17-val dataset, our RANet achieves higher metric results than the w/o OL methods. Furthermore, on the more challenging DAVIS17-testdev dataset, our RANet even outperforms the OL based method OnAVOS in terms of $\mathcal{J}$ Mean.

Speed. Here, we evaluate the speed-accuracy performance of different methods on DAVIS16-val set. Our RANet runs on a TITAN Xp GPU. In Table 1, we list the average time of different methods processing a frame of 480p resolution. Note that the proposed RANet spends 33 milliseconds on each frame, much faster than most of the previous methods. As shown in Fig. 6. The recently proposed method SiamMask is a little faster than our RANet but at expenses of much lower results on $\mathcal{J}\&\mathcal{F}$ Mean than ours.

3 Validation of the Proposed RANet

We now conduct a more detailed examination of our proposed RANet on the VOS task. We assess 1) the contribution of the proposed ranking attention module (RAM) to RANet; 2) the importance of correlation layer (CL) to RANet; 3) the influence of propagating previous frame’s mask (PM) on RANet; 4) the effect of static image pre-train (IP) and video fine-tuning (VF) on RANet; and 5) the impact of online learning (OL) technique to RANet.

The comparison of RANet w/ RAM, w/o Ranking, and Maximum is listed in Table 4. It can be seen that, the RANet w/ RAM achieves $3.6\%$ and $4.4\%$ higher than the baselines w/o Ranking and Maximum, respectively. The RANet w/o Ranking organizes the similarity maps based on the spacial information of the template frame, while the RANet with Maximum losses most useful information in similarity maps by only extracting the maximum values.

2. How important is the correlation and RAM to our RANet? To evaluate the importance of correlation layer in our RANet, we remove the correlation layer, and simply concatenate the features extracted by the encoder, as RGMP does. The following RAM module is also meaningless and is removed. Thus we have a new variant of RANet: -CL. However, as shown in Table 5, the performance of this variant is very bad (67.5% on $\mathcal{J}$ Mean). Thus, the correlation layer is important to our RANet, and serves as the base for the proposed RAM module.

3. How does the previous frame’s mask (PM) influence our RANet? We study how PM influences our RANet. To this end, we set all the pixels of the PM as zero, and re-train our RANet. Thus we have a baseline of -PM. Results in Table 5 shows that, the variant -PM of RANet will drop $\mathcal{J}$ Mean by $4.1$ points. This indicates that the temporal information propagated by PM is very useful for our RANet.

4. What are the effects of pre-training on static images and video fine-tuning in our RANet? To answer this question, we study how each training strategy affects the performance of RANet. We first train RANet only on video data and have a baseline: -IP. We then train RANet only on static images and have the second baseline: -VF. The results of $\mathcal{J}$ Mean by the variants -IP and -VF on DAVIS16-val dataset are listed in Table 5. As can be seen, both baselines drop significantly on $\mathcal{J}$ Mean when compared to the original RANet. Specifically, static image pre-train (IP) improves the $\mathcal{J}$ Mean from $73.2\%$ to $85.5\%$ , while video fine-tuning (VF) improves the $\mathcal{J}$ Mean by $5.6$ points. The performance drops (from 85.5% to 73.2%) of removing IP is mainly due to the over-fitting of RANet on the DAVIS16-training set, which only contains 30 single-object videos.

5. The trade-off between performance and speed using online learning. In Table 6, we also show the performance and run-time of RANet with or without OL technique. One can see that, as the number of iterations increases in OL, the results of our RANet on $\mathcal{J}\&\mathcal{F}$ Mean are continuously improved with different extents, while at a cost of speed.

4 Qualitative Results

In Fig. 7, we show some qualitative visual results of the proposed RANet on the DAVIS16 and DAVIS17 datasets. It can be seen that, the RANet is very robust against many challenging scenarios, such as appearance changes ( $1$ -st row), fast motion ( $2$ -nd row), occlusions ( $3$ -th row), and multi-objects ( $4$ -rd and $5$ -th rows), etc.

Conclusion

In this work, we proposed a real-time and accurate VOS network, which runs at 30 FPS on a single Titan Xp GPU. The proposed ranking attention network (RANet) end-to-end learned the pixel-level feature matching and mask propagation for VOS. A ranking attention module was proposed to better utilize the similarity features for fine-grained VOS performance. The network treated the point-to-point matching feature as a guidance instead of the final results, to avoid noisy predictions. Experiments on DAVIS ${}_{16}/_{17}$ datasets demonstrate that our RANet achieves state-of-the-art performance on both segmentation accuracy and speed.

This work can be further extended. First, the proposed ranking attention module can be applied to other applications such as object tracking and stereo vision . Second, better propagation or local matching techniques can be employed for better VOS performance.

Acknowledgements. We thank Dr. Song Bai on the initial discussion of this project.