SPGNet: Semantic Prediction Guidance for Scene Parsing

Introduction

Semantic segmentation , as a step towards scene understanding , is a challenging problem in computer vision. It refers to the task of assigning semantic labels, such as person and sky, to every pixel within an image. Recently, Deep Convolutional Neural Networks (DCNNs) have significantly improved the performance of semantic segmentation systems.

In particular, DCNNs, deployed in a fully convolutional manner , have attained remarkable results on several semantic segmentation benchmarks . We observe two key design components shared among state-of-the-art semantic segmentation systems. First, multi-scale context module, exploiting the large spatial information, enriches the local features. Typical examples include DeepLab which adopts several parallel atrous convolutions with different rates and PSPNet which performs pooling operations at different grid scales. Recently, SENets and GENets employ the ‘squeeze-and-excite’ (Figure 1 (a)) or more general ‘gather-and-excite’ framework (Figure 1 (b)) and obtain remarkable results on image classification task. Motivated by this, we propose a simple yet effective attention module, called Semantic Prediction Guidance (SPG), which learns to re-weight the local feature map values via the guidance from pixel-wise semantic prediction. Unlike the ‘gather-and-excite’ module (where context information is gathered from a large spatial extent and local features are excited accordingly), our SPG module adopts the ‘supervise-and-excite’ framework (Figure 1 (c)). Specifically, we inject the semantic supervision to the feature maps followed by a simple $1\times 1$ convolution with sigmoid activation function (i.e., ‘supervise’ step). The resulting feature maps, called “Guided Attention”, are used as a guidance to re-weight the other transformed feature maps correspondingly (i.e., ‘excite’ step). We further add an ‘identity’ mapping in the module, similar to the residual block . Additionally, our learned “Guided Attention” allows us to visually explain the “re-weighting” mechanism in our SPG module.

Another important design component is the encoder-decoder structure, where high-level semantic information is captured in the encoder path while the detailed low-level boundary information is recovered in the decoder path. The systems , employing the single-stage encoder-decoder structure (i.e., the encoder-decoder structure is stacked only once), have demonstrated outstanding performance on several semantic segmentation benchmarks. On the other hand, the multi-stage encoder-decoder models , also known as stacked hourglass networks , refine the keypoint estimation iteratively by propagating information across stages for the task of human pose estimation. Interestingly, we observe that the multi-stage encoder-decoder structure is seldom explored in the context of semantic segmentation, except . In this work, we revisit the multi-stage encoder-decoder networks on the Cityscapes dataset . We find that by carefully selecting features across stages, a two-stage encoder-decoder network coupled with our proposed SPG module can significantly outperform its one-stage counterpart with similar parameters and computations.

On Cityscapes test set , our proposed SPGNet outperforms the strong baseline DenseASPP when only exploiting the ‘fine’ annotations. Our overall mIoU is slightly behind the concurrent work DANet but detailed class-wise mIoU reveals that our model is better than DANet in 14 out of 19 semantic classes. Furthermore, our SPGNet requires only $22.7\%$ computation of DANet .

We propose a simple yet effective attention module, called SPG, which adopts a ‘supervise-and-excite’ framework.

We explore multi-stage encoder-decoder networks on semantic segmentation task. Incorporating our proposed SPG module to the multi-stage encoder-decoder networks further improves the performance.

We demonstrate the effectiveness of our SPGNet on the challenging Cityscapes dataset. Our model outperforms the strong baseline DenseASPP , and is better than DANet in 14 out of 19 semantic classes. Our SPGNet strikes a better accuracy/speed trade-off, requiring only $22.7\%$ computation of DANet.

We provide detailed ablation studies along with the visualization of our learned attention maps. We also discuss the effectiveness of employing multi-stage encoder-decoder networks on semantic segmentation.

Related Works

Semantic Segmentation: Most state-of-the-art semantic segmentation models are based on FCN . The detailed object boundary information is usually missing due to the pooling or convolutions with striding operations within the network. To alleviate the problem, one could apply the atrous convolution to extract dense feature maps. However, it is computationally expensive to extract output feature maps that are 8 or even 4 times smaller than the input resolution using state-of-the-art network backbones . On the other hand, the encoder-decoder structures capture the context information in the encoder path and recover high resolution features in the decoder path. Additionally, contextual information has also been explored. ParseNet exploits the global context information, while PSPNet uses spatial pyramid pooling at several grid scales. DeepLab uses several parallel atrous convolution with different rates in the Atrous Spatial Pyramid Pooling module, while DPC applies neural architecture search for the context module. Finally, our proposed Semantic Prediction Guidance (SPG) bears a similarity to the Layer Cascade method which treats each pixel differently. Instead of classifying easy pixels in the early stages within the network, our SPG module weights each pixel according to the predictions in the first stage of our stacked network.

Multi-Stage Networks: Multi-stage networks have been widely used and explored in human pose estimation. Multi-stage networks aim to iteratively refine estimation. To maximally utilize the capacity of each stage, CPM and Stacked Hourglass propagate not only features to the next stage, but also remap predicted heatmaps into feature space by a 1x1 convolution and concatenate with feature maps. MSPN further optimizes feature flow across stages by propagating intermediate features of encoder-decoder of previous stage to the next stage. MSPN demonstrates superior performance over single stage counterpart with similar parameters and computations. On the other hand, Stacked Deconvolutional Network uses multiple deconvolution networks for semantic segmentation. However, it only passes features across stages and neglects predictions of every stage. Additionally, Zhou et al. propose a cascade segmentation module. In this work, we find predictions can be served as a special attention to propagate useful features across stages.

Attention Module: Attention mechanism has been widely used recently in multiple computer vision tasks. Chen et al. learn an attention module to merge multi-scale features. Kong and Fowlkes propose a gating module that adaptively selects features pooled with different field sizes. Recently, the self-attention module has been explored by several works for computer vision tasks. In contrast, our proposed SPG module is more similar to the other works that employ the ‘squeeze-and-excite’ or ‘gather-and-excite’ framework. In particular, Squeeze-and-Excitation Networks (SENets) squeeze the features across spatial dimensions to aggregate the information for re-weighting feature channels. Hu et al. generalize SENets with ‘gather-and-excite’ operations where long-range spatial information is gathered to re-weight (or ‘excite’) the local features. Motivated by this, our proposed SPG module employs the ‘supervise-and-excite’ framework, where our local features are guided by the semantic supervision. Additionally, EncNet also adds supervision to their global feature. However, our supervision is pixel-level instead of image-level.

Methods

Figure 2 shows our proposed SPGNet, which consists of multiple stages and each stage is based on an encoder-decoder architecture: encoder produces dense feature maps at multiple scales and also an image-level feature vector using global average pooling (GAP). Decoder starts with this feature vector and gradually recovers spatial resolution by combining corresponding encoder feature map using an upsample module, described in Sec 3.3.

Our SPGNet stacks multiple stages, where earlier decoder output is fed into a semantic prediction guidance (SPG) module (detailed in Sec 3.4) to generate input feature for the next stage. In addition, we employ Cross Stage Feature Aggregation to enhance latter stage encoders by taking advantage of earlier stage encoder / decoder features, as shown in Figure 2(c). The decoder output in the final stage is bilinearly upsampled to input image resolution, generating per-pixel semantic prediction results.

The multi-stage design of SPGNet is inspired by Stacked Hourglass for human pose estimation. Our method differs from Stacked Hourglass in 1) we carefully design the encoder-decoder architecture in each stage instead of using a symmetric hourglass network, and 2) latter stage input is generated from SPG module rather than simply passing the features combined with predictions from previous stage.

2 Encoder / Decoder Design

Hourglass networks assign equal computation to both encoder and decoder, making it unavailable to use pre-trained weights on ImageNet . In contrast, Feature Pyramid Networks (FPN) use well-designed classification networks for encoder and design a simple decoder consisting of only nearest-neighbor interpolations to upsample decoder feature maps. Our encoder-decoder design principles follow FPN (e.g., all the feature maps in the decoder contain 256 channels), but we employ two more components to make it more efficient and effective. First, we incorporate a global average pooling after the output of encoder to generate the $1\times 1$ image-level features followed by another $1\times 1$ convolution to transform its feature channels to 256. Second, instead of using a single nearest-neighbor interpolation, we design an efficient upsample module, as described in the next section.

3 Upsample module

As illustrated in Figure 2(a, b), our decoder adopts upsample module to recover feature map resolution step-by-step. Specifically, each module in the decoder takes two input feature maps, one from encoder and one from previous layer output. The input from encoder is first transformed by a residual block to reduce the dimension to output channel of the decoder. Then, the input from previous layer output is bilinear upsampled and added to the transformed encoder output. Instead of passing this merged feature directly to next upsample module, we further add another residual block to better fuse features from two different sources.

4 Semantic Prediction Guidance

Our proposed SPG module differs from SENets and GENets on using supervised semantic predictions to guide the ‘excite’ step. We further verified having explicit supervision improves model performance. The benefit of our proposed SPG module are twofold: the ‘gather’ step is implicitly folded into the encoder-decoder architecture, which allows SPG module to be computationally efficient (about $1\%$ increase in FLOPs) and have a small memory footprint ($2.3\%$ higher peak memory usage). Meanwhile, using semantic prediction makes SPG module more explainable. See Section 4.5 for visualization.

Experiments

We perform experiments on the Cityscapes dataset , which contains 19 classes. There are 5,000 images with high quality annotation (called “fine”), divided into 2,975/500/1,525 images for training, validation and testing. We only use the “fine” annotation in this paper.

2 Implementation Details

Networks. We employ ResNet in the encoder module. The “Stem” in Figure 2 consists of a $7\times 7$ convolution with stride $=2$ followed by a $3\times 3$ max pooling with stride $=2$. We replace BatchNorm layers with synchronized Inplace-ABN , and adopt bilinear interpolation in all the upsampling operations.

Training settings. We use mini-batch SGD momentum optimizer with batch size 8, initial learning rate 0.01, momentum 0.9 and weight decay 0.0001. Following prior works , we use the “poly” learning rate schedule where the learning rate is scaled by $(1-\frac{\text{iter}}{\text{iter}_{\text{max}}})^{0.9}$. For data augmentation, we employ random scale between [0.5, 2.0] with a step size of 0.25, random flip and random crop. We train the model for $80,000$ iterations on “train”set for ablation study. To evaluate our model on the “test” set, we train the model on the concatenation of “train” and “val” set.

3 Comparison with State-of-the-Arts

In Table 1, we report our Cityscape “test” set result. We only use “fine” annotations and thus compare with the other state-of-art models that adopt the same setting in the table. Similar to other models, we use the multi-scale inputs (scales = $\{0.75,1.0,1.25,1.5,1.75,2.0\}$) during inference. We also report the model parameters and computation FLOPs (w.r.t., a single $1024\times 2048$ input size).

Our best SPGNet model variant employs a 2-stage encoder-decoder structures with ResNet-50 as encoder backbone and decoder channels $=256$. Our model outperforms most top-performing approaches on Cityscapes with much less computation. Notably, most state-of-the-art methods are mainly based on systems using atrous convolutions to preserve feature maps resolution, which however requires a large amount of computation (as indicated by #FLOPs in Table 1). On the contrary, our proposed SPGNet, built on top of an efficient encoder-decoder structure, strikes a better trade-off between accuracy and speed.

To be concrete, the computation of our SPGNet is almost half of DenseASPP, the previous published state-of-the-art model using only fine annotations, but our performance is $0.5\%$ mIoU better. We also compare our SPGNet with another concurrent work DANet . Our computation is around $22.7\%$ of DANet with only $0.4$ mIoU degradation.

We further compare per-class results with the top-2 performing approaches in Table 2. Surprisingly, our SPGNet outperforms DenseASPP in 15 out of 19 classes and DANet in 14 out of 19 classes. The main degradation of our overall mIoU comes from the “truck” class which is $10.7$ IoU worse than DenseASPP and $9.5$ IoU worse than DANet. We think it is because there are only few “truck” annotations in Cityscapes and our SPGNet requires supervision for learning the guided attention.

4 Ablation Studies

Here, we provide ablation studies on Cityscapes val set.

We perform ablation studies on the SPG design in Table 3. The baseline is a simple 2-stage encoder-decoder network by directly passing the 1st stage decoder features to the 2nd stage encoder. This baseline model uses the Cross-Stage Feature Aggregation (CSFA) which is slightly better than the case without CSFA by 0.18%. We first verify whether passing semantic prediction together with the decoder features to next stage is helpful. We transform the predictions from the 1st stage decoder output by applying a $1\times 1$ convolution. The sum of the transformed predictions and the 1st stage decoder output is passed to the next stage (denoted as SPG (sum)). It achieves $76.96\%$ mIoU which is $0.65\%$ mIoU better than the baseline. Additionally, our proposed SPG module uses the transformed semantic predictions to ‘excite’ the decoder features. We explore two ways for excitation: one is by applying softmax on the spatial dimension $\text{H}\times\text{W}$ (SPG (softmax)) and the other is using sigmoid (SPG(sigmoid)). The SPG (softmax) scheme improves the baseline by $0.86\%$ mIoU while the SPG (sigmoid) scheme achieves the best mIoU of $77.67\%$ ($1.36\%$ mIoU better than the baseline). Comparing results of SPG (sigmoid) scheme ($77.67\%$ mIoU) with SPG (sum) scheme ($76.96\%$ mIoU), it shows the importance of using ‘Excite’ to re-weight features. Finally, we investigate the effect of adding the identity mapping path and the supervision in SPG module. Dropping the identity mapping path in Figure 3 degrades the performance from $77.67\%$ to $77.24\%$, while removing the supervision on learning the guided attention decreases the performance to $77.12\%$ in which our SPG module degenerates to a special case of ‘gather-and-excite’ (where the features are ‘gathered‘ from the 1st-stage decoder output).

To demonstrate the gain of SPG module comes from supervision, we compare SPG module with its unsupervised counterpart, i.e. SE and GE modules. Using SE and GE modules achieves 77.09 mIoU and 77.22 mIoU respectively, both results are better than the baseline 76.31 mIoU and using GE is slightly better than SE which is consistent with the findings in . However, they are still worse than using our proposed supervise-and-excite (i.e. SGP with 77.67 mIoU). The additional gain mainly comes from adding supervision in supervise-and-excite.

We experiment the effect of using more stages and the results are shown in Table 5. Similar to the situation in pose estimation that performance gets saturated as the number of stages increases. But in our case the performance saturates very quickly and achieves optimal with 2 stages. It is possible that by carefully balancing the loss weights among stages the performance might be better for models with more than 2 stages. However, for simplicity, we focus on models with only 2 stages in this paper.

Our two-stage SPGNet could potentially employ two different backbones in each encoder module. As shown in Table 6, although employing ResNet-$18$+ResNet-$50$ (i.e., ResNet-18 in the 1st encoder and ResNet-50 in the 2nd encoder) and ResNet-$50$+ResNet-$18$ have similar parameters and computation, using deeper model in the first stage outperforms the other one. We think it is crucial to “encode” the features in the early stage with a stronger backbone. Adopting R-$50$+R-$50$ achieves the best performance. For simplicity, we only adopt the same network backbones in all the encoder modules in this paper.

In Table 7, we study the effect of adopting different backbones in the encoder module(s). We observe that using deeper encoder improves the result and using ResNet-50 in a 2-stage SPGNet achieves a good trade-off between #Params, #FLOPs and performance.

We study the effects of on-line hard example (or pixel) mining (OHEM) in Table 8. We apply OHEM to all stages (i.e., the decoder output in each stage) in our SPGNet. As shown in the table, using OHEM consistently improves the performance.

We experiment on the effect of decoder channels in Table 9. Employing ResNet-50 as the encoder backbone and decoder channels $=256$ achieves the best validation mIoU.

Flip and multi-scale test. We further add flip and multi-scale test to the best model (ResNet-50 with 2 stages, in Table 9). By adding scales = $\{0.75,1.0,1.25,1.5,1.75,2.0\}$, the performance further improves from 80.91 to 81.86.

5 Visualization of Guided Attention

In this section, we visualize the learned Guided Attention in our best model variant (a stack of two encoder-decoder structures with ResNet-50 as encoder backbone). The Guided Attention maps (with 256 channels) is obtained by applying a $1\times 1$ convolution with sigmoid activation to the prediction in the 1st stage decoder output. Therefore, we have a convolution weight matrix with size $C\times 256$ (Figure 4 top-right), where $C$ is the number of semantic classes on the dataset. To visualize the attention for class $c$, we would like to know which channels among the 256 channels in the Guided Attention map that the class $c$ contributes most. Therefore, for class $c$, we extract the corresponding $1\times 256$ convolution weight vector (Figure 4 red row in matrix) from the $C\times 256$ matrix. In the vector, we then select the indexes of the top $15$ largest weights (Figure 4 yellow elements in vector), which is used to index the corresponding channels in the Guided Attention maps (Figure 4 yellow slices from the purple Guided Attention maps), i.e., those channels in the Guided Attention maps have the largest responses for the class $c$. Then, we visualize the attention by taking $l_{2}$ norm of the selected channels.

General classes. We visualize the learned Guided Attention for four representative classes in Figure 5. ‘Car’ and ‘Person’ are most common ‘thing’ classes in the Cityscapes dataset. ‘Building’ is a common ‘stuff’ class and ‘Pole’ is a common thin ‘stuff’ in Cityscapes. The activations are normalized between 0 (blue color) and 1 (red color).

From Figure 5, we observe several interesting behaviors:

The guided attention learns localization of objects. The activations for ‘thing’ align quite well with the actual position of those objects.

Guided attention focus on object co-occurrence. For example, ‘Car’ and ‘Person’ objects are usually on the road and the attentions for these classes learn to focus on both corresponding instances and road.

Guided attention can find small objects. For example, there are multiple thin ‘Poles’ in the third row of Figure 5 and guided attention can find most of them.

Semantically similar classes. We find guided attention is also capable of differentiating semantically similar classes. In Figure 6, we visualize the attention for two semantically similar classes: ‘Person’ and ‘Rider’. The attention for ‘Rider’ mainly fires for the rider instance on the right, and it does not fire for the two person instances on the left of the image. Our guided attention makes the features, passed to the next stage, more discriminative to semantically similar classes through the injected supervision, allowing our SPGNet to achieve better results on both ‘Person’ and ‘Rider’ classes than other state-of-the-art models, as shown in Table. 2.

Failure cases. Our SPGNet confuses among ‘Truck’, ‘Bus’ and ‘Train’. We visualize the attentions for these classes in Figure 7. We observe that the Guided Attention maps for these classes usually activate together on the same object. It potentially produces features that are less discriminative to those classes, resulting in our worse performance on ‘Truck’, ‘Bus’ and ‘Train’, as shown in Table. 2.

6 Generalization to Other Datasets

To demonstrate that our model can be generalized to other datasets, we perform more experiments on the PASCAL VOC 2012 and PASCAL Person-Part . For both datasets, we follow the settings in to train the model with a crop size of $513\times 513$, batch size of 28 for 30,000 iterations.

PASCAL VOC 2012: The SPGNet with a stack of 2 ResNet-50 achieves 77.33 mIoU. The performance of SPGNet is comparable with the current state-of-the-art ResNet-101 DeepLabV3+ which achieves 77.37 mIoU with encoder stride=32 for a fair comparison.

PASCAL Person-Part: Table 10 shows comparison with state-of-the-art results on Pascal Person-Part. Our SPGNet with a stack of 2 ResNet-50 achieves 67.23 mIoU with a single scale input, and 68.36 mIoU with multi-scale inputs. Note that our SPGNet does not require extra MPII training data , as used in .

Conclusion

We have proposed the SPGNet which demonstrates state-of-the-art performance for semantic segmentation on Cityscapes. Our proposed SPG module employs the ‘supervise-and-excite’ framework, where the local features are reweighted via the guidance from semantic prediction. The Guided Attention maps within the SPG module allows us to visually interpret the corresponding reweighting mechanism. Our experimental results show that a two-stage encoder-decoder network paired with our SPG module can significantly outperform its one-stage counterpart with similar parameters and computations. Finally, we plan to explore a more computationally efficient encoder-decoder structure for semantic segmentation in the future.

This work is in part supported by IBM-Illinois Center for Cognitive Computing Systems Research (C3SR) - a research collaboration as part of the IBM AI Horizons Network and Intelligence Advanced Research Projects Activity (IARPA) via contract D17PC00341, ARC DECRA DE190101315. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government. The authors thank Samuel Rota Bulò and Peter Kontschieder for the valuable discussion about the global pooling kernel size.

Appendix A Extra Ablation Studies

We provide extra ablation studies on Cityscapes val set.

We perform experiments to demonstrate the effectiveness of our proposed upsample module. We compare the decoder equipped with our proposed upsample module against the one using FPN-style upsample module (i.e., bilinear upsample + residual blocks vs. nearest-neighbor upsample + single convolutions). In these experiments, we use ResNet-18 for encoder and we do not use global average pooling. For a fair comparison, we follow to implement FPN decoder module and only use the largest resolution feature maps for prediction. We also add synchronized Inplace-ABN after all convolutions in our FPN implementation. Decoder channels are set to $128$ for both cases. Results are shown in Table 11. FPN-style upsample module and our proposed module have similar parameters but our upsample module requires $10$B fewer FLOPs than the FPN-style module, thanks to the bottleneck design in residual blocks. Furthermore, using our upsample module, the performance is almost $1.5$ mIoU better than the FPN-style upsample module.

We experiment with the effect of Global Average Pooling (GAP) by using a single-stage encoder-decoder with ResNet-18 as encoder backbone. The GAP operation is deployed after the encoder features. The decoder module uses 128 channels.

We compare three strategies during inference:

GAP: Use global average pooling during inference on the $1024\times 2048$ image .

TILED: Crop overlapping patches within the image that have the same size as training crop size (e.g. $769\times 769$), and use $\frac{1}{3}$ overlap between patches (e.g., overlap with 256 pixels) .

AP: Replace global average pooling with an average pooling whose kernel size is the same as training crop size divided by the stride of that feature maps .

As shown in Table 12, we observe that using global average pooling (GAP) only improves the performance slightly by 1.3% due to the asymmetric setting during training and inference (i.e., train with crop size $769\times 769$ but inference with image size $1024\times 2048$). The TILED strategy resolves this problem by employing the same pooling kernel size during training and inference. However, it introduces extra computation since it requires processing redundant pixels within the overlapped regions among patches. Furthermore, it requires some heuristics to resolve the conflicts within the overlapped regions (e.g., average the predictions in the overlapped regions), which may lead to sub-optimal merging. On the other hand, the AP strategy is more efficient than the TILED strategy and performs slightly better, since no overlapped regions are processed.

References