CASENet: Deep Category-Aware Semantic Edge Detection

Introduction

Figure 1 shows an image of a road scene from Cityscapes dataset with several object categories such as building, ground, sky, and car. In particular, we study the problem of simultaneously detecting edge pixels and classifying them based on association to one or more of the object categories . For example, an edge pixel lying on the contour separating building and pole can be associated with both of these object categories. In Figure 1, we visualize the boundaries and list the colors of typical category combinations such as “building+pole” and “road+sidewalk”. In our problem, every edge pixel is denoted by a vector whose individual elements denote the strength of pixel’s association with different semantic classes. While most edge pixels will be associated with only two object categories, in the case of junctions one may expect the edge pixel to be associated with three or even more. We therefore do not restrict the number of object categories a pixel can be associated with, and formulate our task as a multi-label learning problem. In this paper, we propose CASENet, a deep network able to detect category-aware semantic edges. Given $K$ defined semantic categories, the network essentially produces $K$ separate edge maps where each map indicates the edge probability of a certain category. An example of separately visualized edge maps on a test image is given in Figure 2.

The problem of edge detection has been shown to be useful for a number of computer vision tasks such as segmentation , object proposal , 3d shape recovery , and 3d reconstruction . By getting a better understanding of the edge classes and using them as prior or constraints, it is reasonable to expect some improvement in these tasks. With a little extrapolation, it is not difficult to see that a near-perfect semantic edge, without any additional information, can solve semantic segmentation, depth estimation , image-based localization , and object detection . We believe that it is important to improve the accuracy of semantic edge detection to a certain level for moving towards a holistic scene interpretation.

Early work tends to treat edge information as low-level cues to enhance other applications. However, the availability of large training data and the progress in deep learning methods have allowed one to make significant progress for the edge detection problem in the last few years. In particular, there have been newer data sets . The availability of large-scale semantic segmentation data sets can also be easily processed to obtain semantic edge data set as these two problems can be seen as dual problems.

The definition of boundary or edge detection has evolved over time from low-level to high-level features: simple edge filters , depth edges , object boundaries , and semantic contours . In some sense, the evolution of edge detection algorithms captures the progress in computer vision from simple convolutional filters such as Sobel or Canny to fully developed deep neural networks.

Early edge detection methods used simple convolutional filters such as Sobel or Canny .

Depth edges

Some previous work focuses on labeling contours into convex, concave, and occluding ones from synthetic line drawings and real world images under restricted settings . Indoor layout estimation can also be seen as the identification of concave boundaries (lines folding walls, ceilings, and ground) . By recovering occluding boundaries , it was shown that the depth ordering of different layers in the scene can be obtained.

Perceptual edges

A wide variety of methods are driven towards the extraction of perceptual boundaries . Dollar et al. used boosted decision trees on different patches to extract edge maps. Lim et al. computed sketch tokens which are object boundary patches using random forests. Several other edge detection methods include statistical edges , multi-scale boundary detection , and point-wise mutual information (PMI) detector . More recently, Dollar and Zitnick proposed a realtime fast edge detection method using structured random forests. Latest methods using deep neural networks have pushed the detection performance to state-of-the-art.

Semantic edges

The origin of semantic edge detection can be possibly pinpointed to . As a high level task, it has also been used implicitly or explicitly in many problems related to segmentation and reconstruction . In some sense, all semantic segmentation methods can be loosely seen as semantic edge detection since one can easily obtain edges, although not necessarily an accurate one, from the segmentation results. There are papers that specifically formulate the problem statement as binary or category-aware semantic edge detection . Hariharan et al. introduced the Semantic Boundaries Dataset (SBD) and proposed inverse detector which combines both bottom-up edge and top-down detector information to detect category-aware semantic edges. HFL first uses VGG to locate binary semantic edges and then uses deep semantic segmentation networks such as FCN and DeepLab to obtain category labels. The framework, however, is not end-to-end trainable due to the separated prediction process.

DNNs for edge detection

Deep neural networks recently became popular for edge detection. Related work includes SCT based on sparse coding , $N^{4}$ fields , deep contour , deep edge , and CSCNN . One notable method is the holistically-nested edge detection (HED) which trains and predicts edges in an image-to-image fashion and performs end-to-end training.

2 Contributions

Our work is related to HED in adopting a nested architecture but we extend the work to the more difficult category-aware semantic edge detection problem. Our main contributions in this paper are summarized below:

To address edge categorization, we propose a multi-label learning framework which allows improved edge learning than traditional multi-class framework.

We propose a novel nested architecture without deep supervision on ResNet , where bottom features are only used to augment top classifications. We show that deep supervision may not be beneficial in our problem.

We outperform previous state-of-the-art methods by significant margins on SBD and Cityscapes datasets.

Problem Formulation

Given an input image, our goal is to compute the semantic edge maps corresponding to pre-defined categories. More formally, for an input image $\mathbf{I}$ and $K$ defined semantic categories, we are interested in obtaining $K$ edge maps $\{\mathbf{Y}_{1},\cdots,\mathbf{Y}_{K}\}$, each having the same size as $\mathbf{I}$. With a network having the parameters $\mathbf{W}$, we denote $\mathbf{Y}_{k}(\mathbf{p}|\mathbf{I},\mathbf{W})\in$ as the network output indicating the computed edge probability on the $k$-th semantic category at pixel $\mathbf{p}$.

Possibly driven by the multi-class nature of semantic segmentation, several related works on category-aware semantic edge detection have more or less looked into the problem from the multi-class learning perspective. Our intuition is that this problem by nature should allow one pixel belonging to multiple categories simultaneously, and should be addressed by a multi-label learning framework.

We therefore propose a multi-label loss. Suppose each image $\mathbf{I}$ has a set of label images $\{\mathbf{\bar{Y}}_{1},\cdots,\mathbf{\bar{Y}}_{K}\}$, where $\mathbf{\bar{Y}}_{k}$ is a binary image indicating the ground truth of the $k$-th class semantic edge. The multi-label loss is formulated as:

where $\beta$ is the percentage of non-edge pixels in the image to account for skewness of sample numbers, similar to .

Network Architecture

We propose CASENet, an end-to-end trainable convolutional neural network (CNN) architecture (shown in Fig. 3(c)) to address category-aware semantic edge detection. Before describing CASENet, we first propose two alternative network architectures which one may come up with straightforwardly given the abundant previous literature on edge detection and semantic segmentation. Although both architectures can also address our task, we will analyze issues associated with them, and address these issues by proposing the CASENet architecture.

We address the edge detection problem under the fully convolutional network framework. We adopt ResNet-101 by removing the original average pooling and fully connected layer, and keep the bottom convolution blocks. We further modify the base network in order to better preserve low-level edge information. We change the stride of the first and fifth convolution blocks (“res1” and “res5” in Fig. 3) in ResNet-101 from 2 to 1. We also introduce dilation factors to subsequent convolution layers to maintain the same receptive field sizes as the original ResNet, similar to .

2 Basic architecture

A very natural architecture one may come up with is the Basic architecture shown in Fig. 3(a). On top of the base network, we add a classification module (Fig. 3(d)) as a $1\times 1$ convolution layer, followed by bilinear up-sampling (implemented by a $K$-grouped deconvolution layer) to produce a set of $K$ activation maps $\{\mathbf{A}_{1},\cdots,\mathbf{A}_{K}\}$, each having the same size as the image. We then model the probability of a pixel belonging to the $k$-th class edge using the sigmoid unit given by $\mathbf{Y}_{k}(\mathbf{p})=\sigma(\mathbf{A}_{k}(\mathbf{p}))$, which is presented in the Eq. (1). Note that $\mathbf{Y}_{k}(\mathbf{p})$ is not mutually exclusive.

3 Deeply supervised architecture

One of the distinguishing features of the holistically-nested edge detection (HED) network is the nested architecture with deep supervision . The basic idea is to impose losses to bottom convolution sides besides the top network loss. In addition, a fused edge map is obtained by supervising the linear combination of side activations.

Note that HED only performs binary edge detection. We extended this architecture to handle $K$ channels for side outputs and $K$ channels for the final output. We refer to this as deeply supervised network (DSN), as depicted in Fig. 3(b). In this network, we connect an above-mentioned classification module to the output of each stack of residual blocks, producing 5 side classification activation maps $\{\mathbf{A}^{(1)},\dots,\mathbf{A}^{(5)}\}$, where each of them has $K$-channels. We then fuse these 5 activation maps through a sliced concatenation layer (the color denotes the channel index in Fig. 3(g)) to produce a $5K$-channel activation map:

$\mathbf{A}^{f}$ is fed into our fused classification layer which performs $K$-grouped $1\times 1$ convolution (Fig. 3(f)) to produce a $K$-channel activation map $\mathbf{A}^{(6)}$. Finally, 6 loss functions are computed on $\{\mathbf{A}^{(1)},\dots,\mathbf{A}^{(6)}\}$ using the Equation 1 to provide deep supervision to this network.

Note that the reason we perform sliced concatenation in conjunction with grouped convolution instead of the corresponding conventional operations is as follows. Since the 5 side activations are supervised, we implicitly constrain each channel of those side activations to carry information that is most relevant to the corresponding class.

With sliced concatenation and grouped convolution, the fused activation for a pixel $\mathbf{p}$ is given by:

This essentially integrates corresponding class-specific activations from different scales as the finally fused activations. Our experiments empirically support this design choice.

4 CASENet architecture

Upon reviewing the Basic and DSN architectures, we notice several potential associated issues in the category-aware semantic edge detection task:

First, the receptive field of the bottom side is limited. As a result it may be unreasonable to require the network to perform semantic classification at an early stage, given that context information plays an important role in semantic classification. We believe that semantic classification should rather happen on top where features are encoded with high-level information.

Second, bottom side features are helpful in augmenting top classifications, suppressing non-edge pixels and providing detailed edge localization and structure information. Hence, they should be taken into account in edge detection.

Our proposed CASENet architecture (Fig. 3(c)) is motivated by addressing the above issues. The network adopts a nested architecture which to some extent shares similarity to DSN but also contains several key improvements. We summarize these improvements below:

Replace the classification modules at bottom sides to the feature extraction modules.

Put the classification module and impose supervision only at the top of the network.

Perform shared concatenation (Fig. 3(h)) instead of sliced concatenation.

The difference between side feature extraction and side classification is that the former only outputs a single channel feature map $\mathbf{F}^{(j)}$ rather than $K$ class activations. The shared concatenation replicates the bottom features $\mathbf{F}=\{\mathbf{F}^{(1)},\mathbf{F}^{(2)},\mathbf{F}^{(3)}\}$ from Side-1-3 to separately concatenate with each of the $K$ top activations:

The resulting concatenated activation map is again fed into the fused classification layer with $K$-grouped convolution to produce a $K$-channel activation map $\mathbf{A}^{(6)}$.

In general, CASENet can be thought of as a joint edge detection and classification network by letting lower level features participating and augmenting higher level semantic classification through a skip-layer architecture.

Experiments

In this paper, we compare CASENetSource code available at: http://www.merl.com/research/license#CASENet. with previous state-of-the-art methods, including InvDet , HFL , weakly supervised object boundaries , as well as several baseline network architectures.

We evaluate the methods on SBD , a standard dataset for benchmarking semantic edge detection. Besides SBD, we also extend our evaluation to Cityscapes , a popular semantic segmentation dataset with pixel-level high quality annotations and challenging street view scenarios. To the best of our knowledge, our paper is the first work to formally report semantic edge detection results on this dataset.

The dataset consists of 11355 images from the PASCAL VOC2011 trainval set, divided into 8498 training and 2857 test imagesThere has been a clean up of the dataset with a slightly changed image number. We also report the accordingly updated InvDet results.. This dataset has semantic boundaries labeled with one of 20 Pascal VOC classes.

Cityscapes

The dataset contains 5000 images divided into 2975 training, 500 validation and 1525 test images. Since the labels of test images are currently not available, we treat the validation images as test set in our experiment.

2 Evaluation protocol

On both SBD and Cityscapes, the edge detection accuracy for each class is evaluated using the official benchmark code and ground truth from . We keep all settings and parameters as default, and report the maximum F-measure (MF) at optimal dataset scale (ODS), and average precision (AP) for each class. Note that for Citiscapes, we follow exactly to generate ground truth boundaries with single pixel width for evaluation, and reduce the sizes of both ground truth and predicted edge maps to half along each dimension considering the speed of evaluation.

3 Implementation details

We trained and tested CASENet, HED , and the proposed baseline architectures using the Caffe library .

Considering the misalignment between human annotations and true edges, and the label ambiguity of pixels near boundaries, we generate slightly thicker ground truth edges for network training. This can be done by looking into neighbors of a pixel and seeking any difference in segmentation labels. The pixel is regarded as an edge pixel if such difference exists. In our paper, we set the maximum range of neighborhood to be $2$. Under the multi-label framework, edges from different classes may overlap.

Baselines

Since several main comparing methods such as HFL and HED use VGG or VGG based architectures for edge detection and categorization, we also adopt the CASENet and other baseline architectures on VGG (denoted as CASENet-VGG). In particular, we remove the max pooling layers after conv4, and keep the resolutions of conv5, fc6 and fc7 the same as conv4 (1/8 of input). Similar to , both fc6 and fc7 are treated as convolution layers with $3\times 3$ and $1\times 1$ convolution and dimensions set to 1024. Dilation factors of 2 and 4 are applied to conv5 and fc6.

To compare our multi-label framework with multi-class, we generate ground truth with non-overlapping edges of each class, reweight the softmax loss similar to our paper, and replace the top with a 21-class reweighted softmax loss.

Initialization

In our experiment, we initialize the convolution blocks of ResNet/VGG in CASENet and all comparing baselines with models pre-trained on MS COCO .

Hyper-parameters

We unify the hyper-parameters for all comparing methods with the same base network, and set most of them following HED. In particular, we perform SGD with iteration size of 10, and fix loss weight to be 1, momentum 0.9, and weight decay 0.0005. For methods with ResNet, we set the learning rate, step size, gamma and crop size to $1e-7$ / $5e-8$, $10000$ / $20000$, $0.1$ / $0.2$ and $352\times 352$ / $472\times 472$ respectively for SBD and Cityscapes. For VGG, the learning rate is set to $1e-8$ while others remain the same as ResNet on SBD. For baselines with softmax loss, the learning rate is set to 0.01 while other parameters remain the same. The iteration numbers on SBD and Cityscapes are empirically set to $22000$ and $40000$.

Data augmentation

During training, we enable random mirroring and cropping on both SBD and Cityscapes. We additionally augment the SBD data by resizing each image with scaling factors {0.5, 0.75, 1.0, 1.25, 1.5}, while no such augmentation is performed on Cityscapes.

4 Results on SBD

Table 4.4 shows the MF scores of different methods performing category-wise edge detection on SBD, where CASENet outperforms previous methods. Upon using the benchmark code from , one thing we notice is that the recall scores of the curves are not monotonically increasing, mainly due to the fact that post-processing is taken after thresholding in measuring the precision and recall rates. This is reasonable since we have not taken any postprocessing operations on the obtained raw edge maps. We only report the MF on SBD since AP is not well defined under such situation. The readers may kindly refer to supplementary materials for class-wise precision recall curves.

We compare the proposed multi-label loss with the reweighted softmax loss under the Basic architecture. One could see that using softmax leads to significant performance degradation on both VGG and ResNet, supporting our motivation in formulating the task as a multi-label learning problem, in contrast to the well accepted concept which addresses it in a multi-class way.

Is deep supervision necessary?

We compare CASENet with baselines network architectures including Basic and DSN depicted in Fig. 3. The result empirically supports our intuition that deep supervision on bottom sides may not be necessary. In particular, CASENet wins frequently on per-class MF as well as the final mean MF score. Our observation is that the annotation quality to some extent influenced the network learning behavior and evaluation, leading to less performance distinctions across different methods. Such distinction becomes more obvious on Cityscapes.

Is top supervision necessary?

One might further question the necessity of imposing supervision on Side-5 activation in CASENet. We use CASENet- to denote the same CASENet architecture without Side-5 supervision during training. The improvement upon adding Side-5 supervision indicates that a supervision on higher level side activation is helpful. Our intuition is that Side-5 supervision helps Side-5 focusing more on the classification of semantic classes with less influence from interacting with bottom layers.

Visualizing side activations

We visualize the results of CASENet, CASENet- and DSN on a test image in Fig. 5. Overall, CASENet achieves better detection compared to the other two. We further show the side activations of this testing example in Fig. 6, from which one can see that the activations of DSN on Side-1, Side-2 and Side-3 are more blurry than CASENet features. This may be caused by imposing classification requirements on those layers, which seems a bit aggressive given limited receptive field and information and may caused performance degradation. Also one may notice the differences in “Side5-Person” and “Side5-Boat” between CASENet- and CASENet, where CASENet’s activations overall contain sharper edges, again showing the benefit of Side-5 supervision.

From ResNet to VGG

CASENet-VGG in Table 4.4 shows comparable performance to HFL-FC8. HFL-CRF performs slightly better with the help of CRF postprocessing. The results to some extent shows the effectiveness our learning framework, given HFL uses two VGG networks separately for edge localization and classification. Our method also significantly outperforms the HED baselines from , which gives $44$ / $41$ on MF / AP, and $49$ / $45$ with detection.

Other variants

We also investigated several other architectures. For example, we kept the stride of 2 in “res1”. This downgrades the performance for lower input resolution. Another variant is to use the same CASENet architecture but impose binary edge losses (where a pixel is considered lying on an edge as long as it belongs to the edge of at least one class) on Side-1-3 (denoted as CASENet-edge in Fig. 4). However we found that such supervision seems to be a divergence to the semantic classification at Side-5.

In this paper, we proposed an end-to-end deep network for category-aware semantic edge detection. We show that the proposed nested architecture, CASENet, shows improvements over some existing architectures popular in edge detection and segmentation. We also show that the proposed multi-label learning framework leads to better learning behaviors on edge detection. Our proposed method improves over previous state-of-the-art methods with significant margins. In the future, we plan to apply our method to other tasks such as stereo and semantic segmentation.

Appendix A Multi-label Edge Visualization

In order to effectively visualize the prediction quality of multi-label semantic edges, the following color coding protocol is used to generate results in Fig. 1, Fig. 4, and Fig. 7. First, we associate each of the $K$ semantic object class a unique value of Hue, denoted as $\mathsf{H}\triangleq[\mathsf{H}_{0},\mathsf{H}_{1},\cdots,\mathsf{H}_{K-1}]$. Given a $K$-channel output $\mathbf{Y}$ from our CASENet’s fused classification module, where each element $\mathbf{Y}_{k}(\mathbf{p})\in$ denotes the pixel $\mathbf{p}$’s predicted confidence of belonging to the $k$-th class, we return an HSV value for that pixel based on the following equations:

which is also how the ground truth color codes are computed (by using $\mathbf{\hat{Y}}$ instead). Note that the edge response maps of testing results are thresholded with 0.5, with the two classes having the strongest responses selected to compute hue based on Eq. (5).

For Cityscapes, we manually choose the following hue values to encode the 19 semantic classes so that the mixed Hue values highlight different multi-label edge types:

The colors and their corresponding class names are illustrated in following Table 3. The way Hue is mixed in equation 5 indicates that any strong false positive response or incorrect response strength can lead to hue values shifted from ground truth. This helps to visualize false prediction.

Appendix B Additional Results on SBD

Fig. 8 shows the losses of different tested network configurations between iteration 100-500. Note that for Fig. 1, loss curves between iteration 0-8000 is not available due to the large averaging kernel size. One can see CASENet’s fused loss is initially larger than its side5 loss. It later drops faster and soon become consistently lower than the side5 loss (see Fig. 1).

B.2 Class-wise prediction examples

We illustrate 20 typical examples of the class-wise edge predictions of different comparing methods in Fig. 4.4 and 4.4, with each example corresponding to one of the SBD semantic category. One can observe that the proposed CASENet slightly but consistently outperforms ResNets with the basic and DSN architectures, by overall showing sharper edges and often having stronger responses on difficult edges.

Meanwhile, Fig. 4.4 shows several difficult or failure cases on the SBD Datasets. Interestingly, while the ground truth says there is no “aeroplane” in the first row and “dining table” in the second, the network is doing decently by giving certain level of edge responses, particularly in the “dining table” example. The third row shows an example of the false positive mistakes often made by the networks on small objects. The networks falsely think there is a sheep while it is in fact a rock. When objects become smaller and lose details, such mistakes in general happen more frequently.

B.3 Class-wise precision-recall curves

Fig. 4.4 shows the precision-recall curves of each semantic class on the SBD Dataset. Note that while post-processing edge refinement may further boost the prediction performance , we evaluate only on the raw network predictions to better illustrate the network performance without introducing other factors. The evaluation is conducted fully based on the same benchmark code and ground truth files released by . Results indicate that CASENet slightly but consistently outperforms the baselines.

B.4 Performance at different iterations

We evaluate the Basic, DSN, CASENet on SBD for every 2000 iterations between 16000-30000, with the MF score shown in Fig. B.4. We found that the performance do not change significantly, and CASENet consistently outperforms Basic and DSN.

Considering that many datasets adopts the training + validation + test data split, we also randomly divided the SBD training set into a smaller training set and a new validation set with $1000$ images. We used the average loss on validation set to select the optimal iteration number separately for both Basic and CASENet. Their corresponding MFs on the test set are $71.22\%$ and $71.79\%$, respectively.

For more qualitative results, the readers may kindly refer to our released videos on Cityscapes validation set, as well as additional demo videos.

C.2 Class-wise precision-recall curves

Fig. 4.4 shows the precision-recall curves of each semantic class on the Cityscapes Dataset. Again the evaluation is conducted only on the raw network predictions. Since evaluating the results at original scale ($1024\times 2048$) is extremely slow and is not necessary, we bilinearly downsample both the edge responses and ground truths to $512\times 1024$. Results indicate that CASENet consistently outperforms the ResNet with the DSN architecture.