Self-Regularized Prototypical Network for Few-Shot Semantic Segmentation

Introduction

Deep learning has achieved enormous success in segmentation , thanks to large-scale segmentation datasets and powerful convolutional neural network (CNN)-based architectures . However, it is expensive and laborious to obtain large datasets with pixel-wise annotations, which makes it impractical when dealing with real-world problems with numerous categories . In addition to the difficulty of getting finely-labeled training data, poor generalization capability is witnessed when transferring the knowledge learned from training data to unseen classes, or when the training and testing images vary significantly in appearance. To address the aforementioned challenges, a rising task, the few-shot segmentation is proposed . The few-shot segmentation is defined as: giving a small set of examples termed support images and their annotation masks, to segment unseen images termed query images based on the feature extraction of the support images. Specifically, in one-shot segmentation, each query image has reference to only one support image.

Existing few-shot segmentation approaches can be divided into either prototypical methods or parametric methods , based on how they establish linkages between support set and query set. In prototypical methods, the query mask is obtained by calculating the similarity between query features and the prototypes extracted from the support features by masked average pooling, where a prototype is a feature vector that contains class-related representations. Whereas, in parametric methods, the knowledge extraction and query segmentation are combined together, e.g., by applying convolution over the concatenation of query features and support features. In this work, the prototypical manner is adopted for its characteristics of lightweight, compact, and robust to over-fitting. Moreover, based on the architecture of CNN, we propose a self-regularized prototype network (SRPNet) that enhances the few-shot segmentation through supervised prototype generation, improved similarity measuring, and iteratively optimized segmentation.

We analyze the occurrence of unsuccessful mask predictions and attribute them to two challenges: 1) poor prototype that happens when the prototypes generated by masked average pooling over support features are not representative and can hardly restore even the mask of support set itself, and 2) poor generalization, which is usually induced by less-effective similarity measures, diverse appearance of objects in support set and query set, and the inherent difficulty of transferring learned knowledge to unknown classes. Examples of the both challenges are collected from experiments on PASCAL-5i and shown in Figure 1. Despite extensive efforts devoted to improving poor generalization , the poor prototype remains an overlooked problem with plenty of room for improvement. We, however, consider the prototype generation a process most worth exploring and optimizing, since if the prototype obtained from the support feature is not an apposite representative, it can hardly achieve satisfying performance on the query set. Here, we evaluate the generated prototypes by applying them reversely onto the support set itself and try to restore the support masks using the same prediction method as for the query images. In fact, we do have observed in experiments that the generated prototypes cannot restore the support mask well - losing many details and sometimes even not consistent (see the first column in Figure 1). Such results indicate that the generated prototypes lack discriminative representative and explicit supervision on the prototype generation is required. Prototype extraction (by masked average pooling) is a relatively fixed process and heavily depends on feature extraction. Without effective supervision during the end-to-end training, it always happens that the embedded feature map cannot deliver a prototype that comprehensively outlines the object. Some previous works share a similar spirit and introduce supervision onto the interaction of support and query features, concretely through the prototype alignment regularization or the cross-reference module . However, they neither directly evaluate the quality of generated prototypes nor give explicit supervision on the prototype generation process.

Here, we propose a more direct yet effective supervision module termed self-regularized prototype (SRP) generation to evaluate and enhance the prototypes. Firstly, we generate prototypes by masked-average-pooling of support masks over embedded support features. The generated prototype is then applied back to the support feature to restore the support mask. The quality of the generated prototype is measured by its predictive accuracy on the support image, i.e., how well the prototype can restore the support mask. We raise that the quality of a prototype can be evaluated by the extent to which it restores the support mask. Thus, we apply the prototype back to the support feature to perform segmentation. The “extent” is quantified through certain evaluation metrics (i.e., mIoU and Binary-IoU). Poor prototype examples in Figure 1) indicate that the prototypes cannot well recover support mask due to a lack of distinction and representativeness. Explicit supervision is desired for prototype generation. To facilitate the network to learn better prototypes, we introduce an additional loss over the restored support mask, and support mask ground truth is used for regularizing the prototype generation in reverse. The model is motivated to produce prototypes with enhanced consistency and comprehensiveness for support and query sets, offering improved segmentation performance. The proposed regularization is only imposed on the support features, avoiding repeated interaction between the support and query set. The computational cost for regularization occurs only in training, leaving inference free. Once the feature maps are extracted, no extra learnable parameters are introduced and thus it is less prone to over-fitting. To retain more details, we adopt a pyramid structure, where feature maps at various levels of the backbone extractor are concatenated and down-sampled to proper channel size, forming a final feature map with sufficient global scenery representation.

Although our prototypes are well learned, plain generalization will make such improvement less pronounced on query set. For solving this, we adopt an iterative query inference (IQI) module where a collection of prototypes are used for segmentation. Each prototype included is modified based on the initial prototype learned by SRP. Besides, we adopt a new metric - the fidelity - for measuring the similarities. Both fidelity and cosine similarity measure the angle rather than the absolute distance between two separate vectors. However, unlike cosine similarity that has a symmetrical value space about the origin while the negative part is meaningless in image segmentation, our fidelity distributes only between 0 and 1 and exhibits more evident distinctions for vectors in different directions.

In short, the main contributions of our work are:

We propose a direct yet effective self-regularization module. Prototypes are generated, evaluated, and regularized under the supervision of support masks, which differs from existing works.

We adopt fidelity as the distance metric in prototype generation for the first time, which provides a more evident distinction for vectors.

We adopt an iterative query inference module, which uses a collection of prototypes for segmentation and improves the generalization ability for query inference.

We achieve new state-of-the-art performance on two few-shot segmentation benchmarks.

Related Work

Semantic segmentation is a task that labels each pixel in an image by the most appropriate semantic category from the predefined ones . Most recent methods since the first Fully Convolutional Network (FCN) by Long et al. are constructed by deep convolutional neural networks and achieved remarkable improvements in segmentation performance. However, FCN lacks strategies to dig global context and spatial details. Dilated convolutions are advantageous in remaining spatial resolution while enlarging the receptive fields. In addition, pyramid structure, e.g., feature pyramid network or pyramid spatial pooling , is widely employed to capture multi-scale features. In this work, we perform dense prediction following the FCN. Dilated convolutions and pyramid structure are adopted to improve the feature extraction capabilities.

2 Few-shot Classification

The target of few-shot learning is to capture new concepts by just a few examples . Existing work can be divided into three mainstreams, which are based respectively on metric learning , optimization learning , and graph neural network . It was first proposed by Oriol et al. to encode the input into embedded features and perform weighted nearest neighbor matching for classification. Snell et al. proposed a prototypical network that uses a feature vector called prototype to represent each class. Recently, the concept of few-shot has been extended from classification to the challenging semantic segmentation, aiming to segment objects of unseen categories with limited annotated images. Our work applies the prototypical network to few-shot segmentation, in which features at each pixel is classified by its distance to class prototypes.

3 Few-shot Segmentation

As an extension of few-shot classification, the few-shot segmentation aims to generalize the segmentation capability to new categories, while being supported by only a few annotated examples . As the first to apply the spirit of few-shot classification to segmentation tasks, Shaban et al. proposed a two-branch architecture, where a conditioning branch generates free parameters that are used to modulate the FCN-based segmentation branch. Such parametric modules fuse the features extracted from the support set for the execution of segmentation. Similarly, Rakelly et al. realized segmentation by decoding the concatenation of support and query features.

Prototype methods are popular in few-shot segmentation. For example, Dong et al. proposed a dense prototype learning and adopted Euclidean distance as the metric. Wang et al. adopted a simpler design, in which prototypes are incorporated with query features to generate annotation masks directly. Recently, PPNet and PMMs , which are also a prototype networks, learns to use the distance to each class prototype to perform metric space classification. In PPNet, they proposed part-aware prototypes to highlight the fine-grained features, and try to enhance the prototypes based on both labeled and unlabeled images. In PMMs, the prototype represents a part of an object instead of a class of samples, and the prototype network does not involve a single sample or a mixed prototype of a class of samples. PFENet generates training-free prior masks, and enriches query features with support features and the prior masks.

In most prototypical networks , cosine similarity is used to weight features for the foreground predictions. In our paper, a new distance metric, fidelity is used for generating guidance map. The fidelity measures the distance between the respective density matrices of the query feature and the prototype and has a more evident distinction for vectors in different directions, as well as improved segmentation performance.

The interaction between query and support branches has drawn attention recently. For example, Wang et al. proposed a reverse segmentation that uses the query image with its predicted annotated mask to segment the original support images. Their routine was to perform the few-shot segmentation in a reverse direction, using the query image with its predicted mask to segment the original support images. In this way, they claimed to be able to generate more consistent prototypes between support and query, offering better generalization performance. A similar method is seen in Liu et al. , where the query set and the support set are concatenated to produce foreground predictions for the purpose of training supervision. In our work, we propose a most direct yet effective prototype regularization. Concretely, the generated prototypes are applied reversely on the support features to restore the support masks. We ensure that the prototypes best recovers support features as a prior of its generalization to query features. The segmentation masks of both the support and query sets are adopted as training supervisions.

Method

Few-shot segmentation aims at segmenting objects based on the support information from just a few annotated training images. Each few-shot segmentation task $\mathcal{T}$ (also named as an episode $\mathcal{T}$) consists of a support set $S$ supplied with ground-truth masks and a query set $\mathcal{Q}$. The support set $\mathcal{S}=\{I,M\}$ contains only a few image - mask pairs. The query set $Q$ contains $N_{q}$ image-mask pairs. Specifically in a $C$-way $K$-shot segmentation task where there are $C$ classes and $K$ support samples for each of the $C$ classes, we denote the support set as $\mathcal{S}=\{(I_{c,k},M_{c,k})\}$, where $k\in\{1,\cdots,K\}$ and $c\in\{1,\cdots,C\}$, indicating that $K$ image-mask pairs per semantic class from the $C$ classes are included. For example, for the class $c$, where $c\in\{1,...,c,...,C\}$, there are $K$ image-mask pairs as support samples, i.e., $\{(I_{c,1},M_{c,1}),(I_{c,2},M_{c,2})...,(I_{c,K},M_{c,K})\}$. The semantic classes are consistent in the support set and the query set, i.e., the query images are segmented to the $C$ classes (and the background class) provided in the support set. The goal is summarized as, to generate a model that, when given a support set $S$, predicts the annotation masks for the query images $Q$.

The overall classes are divided into two non-overlapping sets of classes $C_{seen}$ and $C_{unseen}$, used for training and testing respectively. The training set $D_{train}$ is constructed from images of classes $C_{seen}$ and the testing set $D_{test}$ is constructed from images of classes $C_{unseen}$. Both the training set $D_{train}$ and the testing set $D_{test}$ contains lots of episodes $\mathcal{T}$. The training set $D_{train}$ is composed of all image-mask pairs that contain at least one pixel in the segmentation mask from training classes. The testing set is randomly sampled. The testing classes $C_{unseen}$ are different from the training classes $C_{seen}$. Notably, for images in $D_{train}$, annotations of $C_{unseen}$ objects are excluded from the $D_{train}$ by labeling them as background, while the images are included as long as there is an object from $C_{seen}$ present. The features are embedded from images by initializing CNN with pre-trained weights. During training, the model gains knowledge from the support set and then applies the knowledge to segment the query set. As each episode contains different semantic classes, the model can be well generalized after training. After obtaining the segmentation model $L$ from the training set $D_{train}$, we evaluate its few-shot segmentation performance on the test set $D_{test}$.

2 Architecture Overview

As opposed to the parametric way that fuses the support and query features to generate segmentation masks, we aim to learn an accurate and robust class-specifical prototype representation. The learned prototype is good enough to represent the corresponding class and can deal with different kinds of objects/images, which performs segmentation via non-parametric metric learning. The overall architecture of the proposed training model is shown in Figure 2. During each episode, images from the support and query set are embedded indiscriminately into deep features by a shared backbone. The masked average pooling (MAP) is then applied over the support features to produce prototypes. MAP is also useful in filtering background noise. The query images are segmented by assigning the class with the most similar prototype to each pixel. A self-regularization is applied to the prototypes, by segmenting the support images using their own prototypes. This operation encourages the few-shot model to generate more accurate prototypes as a prior to improving the consistency between support and query set.

Following , we adopt ResNet-50 as the backbone network. Besides, prototypes generated by the high-level feature maps of support images are usually abstract. To retain more details, we adopt a pyramid structure, where feature maps at various levels of the backbone extractor are aggregated to provide sufficient global scenery representation. A pyramid feature aggregation is applied to the convolutional blocks to generate feature maps at different semantic levels. As shown in Figure 2, these pyramid features are up-sampled to the same size and their concatenations are processed by a $1\!\times\!1$ Conv to 512 channels. No extra free parameters will be introduced then. We aim to optimize the above-mentioned backbone to learn a consistent embedding space and train the model end-to-end.

3 Prototype Generation

Our model learns representative and well-separated prototype representation, a feature vector, for each semantic class, including the background, based on the prototypical network. We produce the foreground objects and background prototypes by applying the support masks over the feature maps separately. Given a support set $S=\{(I_{c,k},M_{c,k})\}$, where $c$, $k$ indexes the class and the shot respectively, let $F_{c,k}$ be the support feature map from the backbone and corresponds to the image $I_{c,k}$, the prototype of class $c$ is computed by masked average pooling:

where $bg$ represents background. These prototypes are optimized non-parametrically during the end-to-end training as described in the following two sections.

4 Non-Parametric Distance Fidelity

Here, we describe how the segmentation is performed using these prototypes and introduce our non-parametric distance fidelity. We first calculate the similarity between each generated prototype over semantic classes (including background) and the feature vector $F_{q}^{(x,y)}$ at each position $(x,y)$ in the query feature. The softmax function is applied over the similarities to produce a probability map $\hat{M}_{q}$. Formally, given a distance function $\mathcal{D}$, let $\mathcal{P}=\{p_{1},...,p_{C}\}\cup\{p_{bg}\}$ denote the prototype collection of prototype $p_{c}$ for class $c$ and $p_{bg}$ for background. For each $p_{j}\in\mathcal{P}$ (where $j=1,...,C\!+\!1$) and the query feature map $F_{q}$ we have

where $\alpha$ is an amplification factor. The predicted segmentation mask is derived by

The commonly adopted distance metric $\mathcal{D}$ is cosine distance and squared Euclidean distance . Here we introduce another distance metric, fidelity, which is defined as the distance between the respective density matrices of query features and prototypes. The density matrix as mentioned is defined formally as the outer product of a coordinate vector and its conjugate transpose. For each $F^{(x,y)}_{q}$ from query features and $p_{j}\in\mathcal{P}$ that both are reshaped to $1_{height}\times 1_{width}\times N_{channel}$, we first normalize them respectively by:

where $p_{j}^{T}$ and $(F^{(x,y)}_{q})^{T}$ is the transpose of $p_{j}$ and $F^{(x,y)}_{q}$, respectively. Both fidelity and cosine similarity measures distance from the direction rather than absolute value. Cosine similarity has a symmetrical value space about the origin, where the negative part is meaningless in the image segmentation. Unlike cosine similarity, the fidelity distributes only between 0 and 1 and thereby has a more evident distinction for vectors with different directions.

Learning proceeds by minimizing the negative log-probability

where $N$ is the total number of spatial pixels, and $M_{q}$ is the ground truth mask of the query image.

5 Self-Regularized Prototype (SRP)

In most previous works, the training process is only driven by the loss between predicted masks and ground truth masks of query images. Attempts (e.g., prototype alignment ) were devoted to improving the generalization of knowledge from support images to query images, which encourages the consistency of mutual generalizability between support and query images. However, in addition to the prototype alignment, it is also worth verifying the quality of the prototypes generated by the support set, i.e., the extent to which these prototypes can restore the support masks. We consider the feature extraction a process worth exploring and optimizing since if the prototype obtained from non-parametric average pooling on the support features is not a good representative, there is no way it will get a good performance on the query set. In fact, we have observed in experiments that the generated prototype did not restore the support images well - losing many details and sometimes even not consistent when using the same prediction method as for the query images (see Figure 1). This requires supervision on the prototype generation process from support features.

To address these issues, we propose a Self-Regularized Prototype (SRP) module that can not only evaluate but also improve the quality of prototypes by incorporating the loss between the prototype-restored support masks and the original support masks. In this module, we adopt a more direct and effective method, which is to apply the prototypes to the supporting features to form support score maps. When the score maps are applied to pixel-wise feature vectors of support images themselves, supporting masks will be generated. The additional loss of the prototype-restored support masks and ground truth support masks are used for regularizing the prototype generation reversely. The prototype generation is therefore encouraged to retain specific details and enhance the distinction between classes. Meanwhile, considering that the generalization on support images is undoubtedly the best, the evaluation of the generated prototypes on support images will impose an upper bound of segmentation. The performance on query set will never exceed this upper bound set regardless of generalization. The consistency and generality between support and query can also be reasonably quantified in this way.

The proposed SRP is illustrated in Figure 2. The prototypes obtained from support set are employed to generate segmentation masks for not only query set but also support set, following the method explained in section 3.4. For each $p_{j}\in\mathcal{P}$ and each position’s feature in support set $F_{s}^{(x,y)}$, we have

The predicted segmentation mask for support set is then given by

Similarly the loss between support predictions and annotations

where $w_{s}$ and $w_{q}$ are used to balance the contribution of support loss and query loss. By SRP, we provide a quantification of the goodness of the learned prototypes, and meanwhile impel the network to learn more accurate and comprehensive prototypes and to generate more consistent prototypes for support and query sets, offering better segmentation performance.

6 Iterative Query Inference (IQI)

A single prototype, though fine-tuned by end-to-end training, is not sufficiently explicit to segment an object due to the limited details it can retain and the large variation in object appearance within the same category. Usually, the prediction merely indicates the rough position and shape of the objects. To solve this issue, we propose an iterative collection of prototypes in query inference to further enhance the representative and generalization of prototypes according to their evaluation on support set. It’s not necessary to include iterative optimization of prototypes in the training, because few commonality is observed in the detailed expression among different objects. The structure of our Iterative Query Inference (IQI) is shown in Figure 3, in which the prototypes are collected iteratively by:

where $\eta$ is the reference rate and $\partial\mathcal{L}_{sup}/\partial p_{j;n-1}$ is the gradients, $n\in\{1,\cdots,N\}$ and $N$ is the amount of collected prototypes. We fuse the predicted score maps $\{\hat{M}_{q;j}^{n}:n=1,\cdots,N\}$ to generate the final segmentation $\hat{M}_{q}$ by

where $\rho_{n}=IoU(\hat{M}_{s},M_{s})$ is the intersection-over-union metric of evaluating the support set.

Our few-shot segmentation model adopts prototype embedding. Predictions are performed on the computed feature maps, thereby requiring no extra passes through the network. In addition, the proposed regularization is only imposed on the support features, avoiding repeated interaction between the support and query set. The computational cost for regularization occurs only in training, leaving inference free. Once the feature maps are extracted, no extra learnable parameters is introduced and thus is less prone to over-fitting.

Experiments

We evaluate our approach on PASCAL-5i and MS COCO . PASCAL-5i is created from PASCAL VOC 2012 by Shaban et al. . The 20 categories included are divided evenly into 4 splits, each containing 5 categories. The training is executed in a cross-validation fashion, where 3 splits are used for training while the rest one is used for evaluation. Experimental results are reported on each of the 4 splits in testing. The categories in each split are as follows: split 1: $\{$bottle, boat, bird, bicycle, aeroplane$\}$; split 2: $\{$cow, chair, cat, car, bus$\}$; split 3: $\{$person, motorbike, horse, dog, dining-table$\}$; split 4: $\{$tv/monitor, train, sofa, sheep, potted-plant$\}$. Categories are grouped according to the alphabetical order of category names so that the difficulty levels of different splits could be very large. Therefore, all methods have large performance difference among different splits. The testing results are reported by the average of 5 independent runs, each containing 1000 episodes, and the random seed is different. Our models are also evaluated on MS COCO , with its 80 object classes evenly divided into 4 splits. $N_{query}=1$ is applied for all experiments. The per-class Intersection over Union ($IoU$) is defined as $\frac{tp}{tp+fp+fn}$, where the $tp$, $fp$ and $fn$ is the count of true positives, false positives and false negatives, respectively. We use the $mean{\text{-}}IoU$ over all classes as the evaluation metric.

2 Implementation Details

The ResNet-50 is adopted as our feature extractor, and shared weights are applied to support and query sets. We reserve the first 5 convolutional blocks, while discarding other layers. Input images are resized to (417, 417) and augmented using random horizontal flipping. You may notice later in qualitative results that the details of some objects in the support mask are too fine to be recognized due to the resizing process, but our predictions are not affected. The batch size is 1. To maintain large spatial resolution, the last two down-sampling operations are discarded, i.e., the output stride is 8. Following , dilated convolutions are used in the last two blocks to enlarge the receptive fields. The model is trained end-to-end by SGD for 30,000 iterations. The momentum is set to 0.9, the learning rate is initialized to 0.001 and decreased by 0.1 every 10,000 iterations, and the weight decay is set to 0.0005.

3 Ablation Studies

Ablation studies are conducted to verify the effect of each of our contributions on performance. The experiments in Table 1 include: Cos - a baseline prototype network with cosine similarity; F - a baseline prototype network with fidelity as the distance metric; F+SRP - extends the baseline model with the prototype regularization on support set; F+IQI - extends the baseline model with iterative query inference; F+SRP+IQI - represents our full approach. Our baseline is Cos in Table 1, i.e., ResNet-50 based prototype network with cosine similarity. We choose this one because cosine similarity is commonly used in prototypical methods and we aim to enhance the prototypical way by analyzing the weaknesses of the prototypical methods and proposing SRPNet to address these weaknesses. We also apply the F+SRP on support set in each split. The inference of supporting images is only related to the quality of the prototype generation and has nothing to do with the generalizability of the model. Meanwhile, the improvement will be less pronounced due to the generalization from support set to query set - which is consistent with our results when we compute and compare the difference between mean-IoU of F+SRP and F, on support set and query set respectively. The IQI is applied for improving the generalizability and mitigating such performance loss, given that the prototypes are trained sufficiently discriminative for classes. In other words, the improvements observed on query set come from two aspects, one is the optimized prototype extraction, and the other is the knowledge combination from versatile prototypes during inference. Specifically, the number of iteratively optimized prototypes in inference is set to 5.

Table 1 shows the ablations with the aforementioned variants in the one-shot setting on PASCAL-5i. The ablation studies clearly show the contributors to our performance improvement. The performance gain brought by our F (fidelity) vs. cosine distance metric is 1.8%. Compared to F, our most important contribution of SRP improves the $mean{\text{-}}IoU$ by 4.3%. However, the upper limit of the SRP contribution can be as high as 7.5%, when applying the same prototypes on support set itself. F+IQI imposes an improvement of 2.6% compared to F. Combining SRP and IQI improves the overall performance by 6.1%, which is higher than applying either of SRP and IQI separately, but slightly lower than the sum of their independent improvements (which is $\sim$6.9%). While SRP optimizes the prototype generation (revealed by that it adapts much better to support set), the IQI effectively generalizes the learned prototypes to new classes in testing.

In Figure 4, we visualize the tSNE of features by cosine distance (left) and fidelity distance (right), respectively. Both fidelity and cosine similarity measure differences in angles rather than absolute distances. Cosine similarity has a symmetrical value space with respect to the origin, where the negative part has no meaning in image segmentation. The value space for fidelity is positive. Compared to cosine similarity, there is a more pronounced difference in the fidelity of vectors in different directions. As shown in Figure 4, compared to cosine distance, the fidelity distance has a more distinct inter-class feature distribution (e.g., green vs. blue) and a more compact intra-class feature distribution (e.g., green), which supports the superior of fidelity over cosine distance.

4 Results on PASCAL-5i

We compare our model with previous few-shot segmentation methods under the metric of $mean{\text{-}}IoU$. The performances of 1-shot and 5-shot segmentation are shown respectively in Table 2. Our proposed SRPNet outperforms the state-of-the-art methods, e.g., PFENet . We report standard deviations with respect to the random seed after running experiments multiple times. During testing, we average the results from 5 runs with different random seeds (thus different testing images), each run containing 1,000 episodes. The very small standard deviation values demonstrate the robustness of our proposed model.

We outperforms PFENet by 0.7%, which is statistically significant under the standard deviation of 0.15. However, PFENet uses prior information from the ImageNet pre-trained model and employs additional context modules, i.e., Pyramid Pooling Module (PPM) , to enhance their high-level feature modeling. More importantly, we adopt a different strategy from PFENet, i.e., we use the prototypical method while PFENet uses the parametric method (see Line 16-24). The prototypical way is more lightweight than the parametric way, but parametric methods achieve better performance than prototypical methods recently. We analyze the weaknesses of the prototypical way and propose SRPNet to address these issues, resulting in significant performance gain compared to our prototypical baseline and better results than PFENet, the state-of-the-art parametric method.

To better evaluate the performance of our proposed approach, we use Dice as another evaluation metric to report our results in Table 3. It is difficult to segment objects by a few support images. Few-shot segmentation is challenging while valuable for practical applications. Our proposed approach has achieved 61.5% mean-IoU and 76.2% Dice under the most challenging 1-shot segmentation setting, which demonstrates the effectiveness of the proposed SRPNet.

Qualitative results of 1-way 1-shot segmentation on PASCAL-5i are shown in Figure 5. None of the shown case is trivial due to reasons like the presence of multiple objects (e.g., bicycles, cow1 and motorbike2), the huge variances in object size (e.g., bird1, aeroplane1 and cow2) and appearance (e.g., bird2), different viewing angles (motorbike1 and horses), as well as the requirements of retaining object details. Using only a single annotated supporting image, satisfactory segmentation results can be obtained on unseen classes, which demonstrates the powerful learning and generalization capabilities. Our prototype regularization - F+SRP - significantly and intuitively improves the expression of details when applying extracted prototypes onto the support images themselves, as shown in Figure 6. For example, in the segmentation of horse, PANet and PFENet get confused by the eye of the horse when generating prototype, resulting in a disordered prediction on the query image. Whereas, given that our method benefits from supervision of prototype generation using support ground-truth, it obtains relatively complete predictions, retaining details such as horse legs and tails. The advantages of prototype supervision are also observed when transferring knowledge between objects of different sizes. In the case of airplane, the support image contains a normal-sized airplane, while the query image is two tiny airplanes. PANet and PFENet fail in identifying the details and depicts roughly the overall contours of the supporting aeroplane, which results in only blurry encircling objects when predicting the query. Some of the circles are even meaningless, perhaps due to the failure to effectively identify and eliminate human objects in the supporting image. Our approach, as opposed, narrowed down the prediction to a more precise range, restoring the details of the tiny aeroplane, including the wings.

We still fail some challenging cases as shown in Figure 7. Two types of errors are listed. We speculate that one is caused by the model’s simple interpretation of the segmentation task as removing the background or retaining all targets except for the monotonous background (e.g., removing the lake in support as well as the ground in query, in the bird case), and the other is caused by the similarity of objects (misidentifying horses as cows, in the cow case).

5 Results on MS COCO

We test our full approach F+SRP+IQI on MS COCO, which contains 80 object categories. Experimentally, a subset of the original large-scale is evaluated, which contain 39,107 (40 classes) training samples, 5,895 validation samples (20 classes) and 9,763 testing samples (20 classes). Our SRP achieves $mean{\text{-}}IoU$ improvement of 2.7% in the 1-shot setting and 6.0% in 5-shot setting, compared to PFENet and as shown in Table 4.

MS COCO has more classes than PASCAL-5i, resulting in more confusion among classes. A better prototype that contains discriminative clues against visual-similar classes helps enhance the segmentation performance. Our proposed Self-Regularized Prototypical Networks optimizes the prototype generation and collects a set of iteratively refined prototypes, producing more representative and generalizable prototypes and better supporting segmentation on MS COCO. Additionally, there are more training samples in MS COCO, which also contributes to the optimization of prototype generation via self-regularization.

Conclusion

In this work, we present a self-regularized prototypical network for few-shot semantic segmentation. A direct yet effective regularization is proposed by evaluating prototypes on support set reversely, based on the observations that the generated prototype cannot consistently describe support set itself. Our SRPNet exploits the in-depth knowledge from the support set, and offers high-quality prototypes that can well represent each semantic category while distinguishing it from other categories. An iterative process is adopted in query inference, which fuses multiple prototypes based on the regularized prototype to generate the final segmentation result. Besides, we adopt fidelity as the distance metric for the first time, which achieves more evident distinctions between prototypes and feature maps. Our proposed SRPNet leads to new state-of-art performance on 1-shot and 5-shot segmentation benchmarks.

References