Fast detection of multiple objects in traffic scenes with a common detection framework

I Introduction

Vision-based traffic scene perception (TSP) is one of many fast-emerging areas in the intelligent transportation system. This field of research has been actively studied over the past decade . TSP involves three phases: detection, recognition and tracking of various objects of interest. Since recognition and tracking often rely on the results from detection, the ability to detect objects of interest effectively plays a crucial role in TSP. In this paper, we focus on three important classes of objects: traffic signs, cars, and cyclists. Fig. 1 shows a typical on-road traffic scene with the detected objects of interest and illustrates some positive examples from the three mentioned classes.

The aim of traffic sign detection is to alert the driver of the changed traffic conditions. The task is to accurately localize and recognize road signs in various traffic environments. Prior approaches use color and shape information. However, these approaches are not adaptive under severe weather and lighting conditions. Additionally, appearance of traffic signs can physically change over time, due to the weather and damage caused by accidents. Instead of using color and shape features, most recent approaches employ texture or gradient features, such as local binary patterns (LBP) and histogram of oriented gradients (HOG) . These features are partially invariant to image distortion and illumination change, but they are still unable to handle severe deformations.

Car detection is a more challenging problem compared to traffic sign detection due to its large intra-class variation caused by different viewpoints and occlusions. Fig. 2 shows a set of different cars with a large intra-class variation. Although sliding-window based detection methods have shown promising results in face and human detection , they often fail to detect cars due to a large variation of viewpoints. Recently the deformable parts model (DPM) , which has gained a lot of attention in generic object detection, has been adapted successfully for car detection . In addition to the DPM, visual subcategorization based approaches have been applied to improve the generalization performance.

Cyclist detection is a new attractive application in the domain of TSP. At present, only few methods are designed purposely for cyclist detection. Many existing pedestrian detection approaches can be adapted for cyclist detection because appearances of pedestrians are very similar to appearances of cyclists along the road. Compared to pedestrian detection, the new problem is more difficult because the various appearances and viewpoints increase the diversity of the cyclists. Therefore, existing pedestrian detection approaches hardly achieve the acceptable detection performance.

Most previous methods have designed specific detectors using different features for each of these objects. The approach we claim here differs from these existing approaches in that we propose a single learning based detection framework to detect all the three important classes of objects. The proposed framework consists of a dense feature extractor and detectors of these three classes. Once the dense features have been extracted, these features are shared with all detectors. The advantage of using one common framework is that the detection speed is much faster, since all dense features need only to be evaluated once in the testing phase. The proposed framework introduces spatially pooled features as a part of aggregated channel features to enhance the feature robustness to noises and image deformations. In order to further improve the generalization performance, we propose an object subcategorization method as a means of capturing intra-class variation of objects.

The remainder of this paper is organized as follows: we briefly review related works in Section II. The structure of the proposed detection framework will be discussed in Section III. Experimental settings and results of all three applications are given in Section IV. Section V summaries this paper and points the direction of future work.

II Related works

Object detection is a challenging but important application in the computer vision community. It has achieved successful outcomes in many practical applications such as face detection and pedestrian detection . Complete survey of object detection can be found in . This section briefly reviews several generic object detection methods.

One classical object detector is the detection framework of Viola and Jones which uses a sliding-window search with cascaded classifiers to achieve accurate location and efficient classification . The other commonly used framework is using a linear support vector machine (SVM) classifier with histogram of oriented gradients (HOG), which has been applied successfully in pedestrian detection . These frameworks achieve excellent detection results on rigid object classes. However, for object classes with a large intra-class variation, their detection performance falls down dramatically .

In order to deal with appearance variations in object detection, a deformable parts model (DPM) based method has been proposed in . This method relies on a variant of HOG features and window template matching, but explicitly models deformations using a latent SVM classifier. It has been applied successfully in many object detection applications . In addition to the DPM, visual subcategorization is another common approach to improve the generalization performance of detection model. It divides the entire object class into multiple subclasses such that objects with similar visual appearance are grouped together. A sub-detector is trained for each subclass and detection results from all sub-detectors are merged to generate the final results. Recently, a new detection framework which uses aggregated channel features (ACF) and a cascaded AdaBoost classifier has been proposed in . This framework uses exhaustive sliding-window search to detect objects at multi-scales. It has been adopted successfully for many practical applications .

II-B Traffic sign detection

Many traffic sign detectors have been proposed over the last decade with newly created challenging benchmarks. Interested reader should see which provides a detailed analysis on the recent progress in the field of traffic sign detection. Most existing traffic sign detectors are appearance-based detectors. These detectors generally fall into one of four categories, namely, color-based approaches, shape-based approaches, texture-based approaches, and hybrid approaches.

Color-based approaches usually employ a two-stage strategy. First, segmentation is done by a thresholding operation in one specific color space. Subsequently, shape detection is implemented and is applied only to the segmented regions. Since RGB color space is very sensitive to illumination change, some approaches convert RGB space to HSI space which is insensitive to light change. Other approaches implement segmentation in the normalized RGB space which is shown to outperform the HSI space . Both HSI and normalized RGB space can alleviate the negative effect of illumination change, but still fail on some severe situations.

Shape-based approaches detect edges or corners from raw images using canny edge detector or its variants. Then, edges and corners will be connected to regular polygons or circles by using Hough-like voting scheme. These detectors are invariant to illumination change, but the memory and computational requirement is quite high for large images. In , the genetic algorithm is adopted to detect circles and is invariant to projective deformation, but the expensive computational requirement limits its application.

Texture-based approaches firstly extract hand-crafted features computed from texture of images, and then use these extracted features to train a classifier. Popular hand-crafted features include HOG, LBP, ACF, etc . Some approaches use the HOG features with a SVM, others use the ACF features with an Adaboost classifier. Besides the above approaches, a convolutional neural network (CNN) has been adopted for traffic sign detection and achieved excellent results in .

Hybrid approaches are a combination of the aforementioned approaches. Usually, the initial step is the segmentation to narrow the search space, which is same as the color-based approaches. Instead of only using edges features or texture-based features, these methods use them together to improve the detection performance.

One standard benchmark for traffic sign detection is the German traffic sign detection benchmark (GTSDB) which collects three important categories of road signs (prohibitory, danger, and mandatory) from various traffic scenes. All traffic signs have been fully annotated with the rectangular regions of interest (ROIs). Researchers can conveniently compare their work based on this benchmark.

II-C Car detection

Many existing car detectors are vision-based detectors. Interested reader should see which discusses different approaches for vehicle detection using mono, stereo, and other vision-sensors. We focus on vision-based car detectors using monocular information in this paper. These detectors can be divided into three categories: DPM-based approaches, subcategorization-based approaches and motion-based approaches.

DPM-based approaches are built on the deformable parts model (DPM) which has been successfully adopted in car detection . In , a variant of DPM discretizes the number of car orientations and each component of the mixture model corresponds to one orientation. The authors of train a variant of DPM to detect cars under severe occlusions and clutters. In , occlusion patterns are used as training data to train a DPM which can reason the relationships between cars and obstacles for detection.

Visual subcategorization which learns subcategories within an object class is a common approach to improve the model generalization in car detection . It usually consists of two phases: feature extraction and clustering. Samples with similar visual features are grouped together by applying clustering algorithm on extracted feature space. Subcategorization-based methods are commonly used with DPM to detect cars from multiple viewpoints. In , subcategories of cars corresponding to car orientation are learned by using locally linear embedding method with HOG features. In , cars with similar viewpoints, occlusions, and truncation scenarios are grouped in the same subcategory by using a semi-supervised clustering method with ACF features.

Motion-based approaches often use appearance cues in monocular vision since monocular images do not provide any 3D and depth information. In , adaptive background model is used to detect cars based on motion that differentiated them from the background. The authors of propose an adaptive background model to model the area where overtaking cars tend to appear in the camera’s field of view. Optical flow , which is a popular tool in machine vision, has been used for monocular car detection. In , a combination of optical flow and symmetry tracking is used for car detection. Optical flow is also used in conjunction with appearance-based techniques in .

The KITTI vision benchmark (KITTI) is a novel challenging benchmark for the tasks of monocular, stereo, optical flow, visual odometry, and 3D object detection. The KITTI dataset provides a wide range of images from various traffic scenes with fully annotated objects. Objects in the KITTI dataset includes pedestrians, cyclists, and vehicles.

II-D Cyclist detection

Many existing cyclist detection approaches use pedestrian detection techniques since pedestrians are very similar to cyclists along the road. In , corner feature extraction, motion matching, and object classification are combined to detect pedestrians and cyclists simultaneously. In , a stereo vision based approach is proposed for pedestrian and cyclist detection. It uses the shape features and matching criterion of partial Hausdorff distance to detect pedestrians and cyclists. However, these approaches cannot distinguish cyclists from pedestrians. Besides the above approaches, the authors of proposed a cyclist detector by using a fixed camera to detect two wheels of bicycle on road, but this approach is limited to detect crossing cyclists. Moreover, all above approaches are designed for traffic monitoring using fixed camera and cannot be used for on-road detection which aims at intelligent driving.

III Our approach

Despite several important techniques have been proposed on object detection, the conventional sliding-window based method of Viola and Jones is still the most successful and practical object detector. The VJ framework consists of two main components: a dense feature extractor and an AdaBoost classifier. In this paper, we build a common object detection framework for traffic scene perception based on the VJ framework, but our framework can employ a number of Adaboost classifiers to detect target objects of different classes. Apart from basic components of VJ framework, we propose an object subcategorization method to improve the generalization performance and employ spatially pooled features to enhance the robustness and effectiveness.

Fig. 3 shows an overview of our framework. In the training phase, we firstly check the intra-class variation of the input object class with respect to object properties, e.g. size, orientation, aspect ratio, and occlusion. If the variation is considerable large, we apply the object subcategorization method to categorize the training data into multiple subcategories and train one sub-detector for each subcategory. Otherwise, we train a single detector for the entire training data. In the testing phase, raw detection results from all sub-detectors need to be calibrated before merging them together. Non-maximum suppression is used to eliminate redundant bounding boxes. If the framework employs detectors of different classes, detection results need to be carefully merged together.

For object classes with a large intra-class variation like cars, the appearance and shape of cars change significantly as the viewpoint changes. In order to deal with these variations that cannot be tackled by the conventional VJ framework, we present an object subcategorization method which aims to cluster the training data into visually homogeneous subcategories. The proposed subcategorization method applies an unsupervised clustering algorithm on one specific feature space of the training data to generate multiple subcategories. This method simplifies the original learning problem by dividing it into multiple sub-problems and improves model generalization performance.

A variety of hand-designed features can be used to perform the clustering algorithm, such as HOG and ACF . HOG is successful at capturing the shapes of objects while does not consider color information. ACF combines both color information and gradient information, which is shown to outperform HOG . In our experiments, a total of 10 channels of features are used for clustering: LUV color channels (3 channels), histogram of oriented gradients at 6 bins (6 channels), and normalized gradient magnitude (1 channel). To extract features from the training data, all positive samples are resized to the median object size.

III-A2 Geometrical Features

Besides the visual features, geometrical information of objects can be extracted from traffic scenes using a variety of sensors and methods. In the KITTI dataset, objects in images from a velodyne laser scanner were annotated with 3D bounding boxes and 3D orientations. Ohn-Bar et al. proposed an analysis of different types of geometrical features, which showed that the geometrical features outperform the visual features for clustering, even the CNN features. We use the following set of geometrical features to represent the object instances in our experiments.

3D orientation The appearance and shape of objects change significantly as the viewpoint changes. We include the 3D orientation (relative orientation between the object and camera) in clustering, aiming at grouping objects with similar visual appearance together.

Aspect-ratio The aspect-ratio (width/height) of objects is strongly correlated with the geometry of objects being detected. We use this feature because learning models at different aspect-ratios significantly improve the generalization performance.

Truncation level The truncation level refers to the percentage of the object outside of the image boundaries. This feature strongly affects the appearance of objects.

Occlusion index Instead of using subtle occlusion patterns defined in , we use an occlusion index to indicate whether an object is not occluded, partially occluded, largely occluded or an unknown situation. We simplify the occlusion patterns because some occlusion features cannot be defined for each object, such as occlusion level, related orientation, and relative 3D point. The above features can only be defined when the obstacle is an annotated object. However, in the KITTI dataset, many obstacles are unlabelled objects.

III-A3 Clustering

A clustering algorithm is used to generate a predefined number of clusters on a specific feature space. Traditional clustering schemes, such as k-means or single linkage, suffer from the cluster degeneration which means that a few clusters claim most data samples . The cluster degeneration problem can be alleviated by using spectral clustering. Spectral clustering followed by k-means often outperforms the traditional schemes. We implement the normalized spectral clustering using the algorithm proposed in . The quality of clustering results is very sensitive to the predefined number of clusters. Unfortunately, how to determine the appropriate number of centroids is still an open question. We experimentally determine the number of clusters for each application.

III-B Feature extraction

The proposed framework introduces spatially pooled features as a part of the aggregated channel features and employs them as dense features in the training phase. All feature channels are aggregated in $4\times 4$ blocks in order to produce fast pixel lookup features.

Given an input image $I$, a channel $C$ of $I$ is a feature map, where the output pixels are computed from corresponding pixels of the input image. Aggregated channel features are extracted from multiple image channels using pixel lookups method. Many image channels are available for extracting features. For example, a trivial channel of a grayscale image is the image itself. For a color image, each color channel can be used as a channel. Other channels can be computed using various transformations of $I$. In order to accelerate the speed of feature extraction, all transformations are required to be translational invariant. It means that the transformation need only to be evaluated once on the entire image rather than separately for each overlapping detection window.

ACF uses the same channel features as ChnFtrs : LUV color channels (3 channels), histogram of oriented gradients (6 channels), and normalized gradient magnitude (1 channel). ACF combines the richness and diversity of statistics from these channels, which is shown to outperform HOG . Prior to computing these 10 channels, we smooth the input image $I$ to suppress fine scale structures as well as noises.

LUV color channels LUV color space contains 3 channels, L channel describes the lightness of the object, U channel and V channel represent the chromaticity of the object. Compared to RGB space, LUV space is able to partially invariant to illumination change. So the proposed detector can work under different light conditions. Images can be converted to LUV space by using a specific transformation.

Gradient magnitude channel A normalized gradient magnitude is used to measure the edge strength. Gradient magnitude $M(x,y)$ at location $(x,y)$ is computed by $\sqrt{I_{x}^{2}+I_{y}^{2}}$, where $I_{x}$ and $I_{y}$ are first intensity derivatives along the $x$-axis and $y$-axis, respectively. Since the gradient magnitude is computed on 3 LUV channels independently, only the maximum response is used as the gradient magnitude channel.

Gradient histogram channels A histogram of oriented gradients is a weighted histogram where bin index is determined by gradient orientation and weighted by gradient magnitude . The histogram of oriented gradients at location $(x,y)$ is computed by $M(x,y)\cdot\mathds{1}{[\Theta(x,y)=\theta]}$, where $\mathds{1}$ is the indicator function, $M(x,y)$ and $\Theta(x,y)$ are the gradient magnitude and discrete gradient orientation, respectively. ACF quantizes the orientation space to 6 orientations and compute one gradient histogram channel for each orientation.

III-B2 Spatially pooled features

Spatial pooling is used to combine multiple visual descriptors obtained at nearby locations into a lower dimensional descriptor over the pooling region. We follow the work of which is shown that pooling can enhance the robustness of two hand-crafted low-level features, covariance features and LBP .

Covariance matrix A covariance matrix is a positive semidefinite matrix which provides a measure of the relationship between multiple sets of variates. The diagonal elements of a covariance matrix represent the variance of each feature and non-diagonal elements represent the correlation between different features. In order to compute the covariance matrix, we use the following variates proposed in :

where $x$ and $y$ indicate the pixel location. $I_{x}$ and $I_{y}$ are first intensity derivatives along the horizontal-axis and vertical-axis respectively. Similarly, $I_{xx}$ and $I_{yy}$ are second intensity derivatives, respectively. $M$ is the gradient magnitude $\sqrt{I_{x}^{2}+I_{y}^{2}}$. $O_{1}$ is the edge orientation $\textrm{arctan}(|I_{x}|/|I_{y}|)$ and $O_{2}$ is an additional edge orientation in which,

where the atan2 function is defined in terms of the arctan in the following:

The covariance descriptor of a region is a $9\times 9$ covariance matrix which can be computed efficiently because the computational cost is independent of the size of the region. We also exclude the variance of pixel locations (x and y coordinates) and the correlation coefficient between pixel locations (x and y coordinates), since these features do not capture discriminative information. Due to the symmetry, each covariance descriptor finally contains $42$ different values.

Spatially pooled covariance The spatial invariance and robustness of the covariance descriptors can be improved by applying pooling method. There are two common pooling methods in this context: average pooling and max pooling. Max pooling is used in our framework as it has been shown to outperform average pooling in image classification . Max-pooling uses the maximum value of a pooling region to represent the pooled features in the region. It aims to retain the most salient information and discard irrelevant details and noises over the pooling region. The image window is divided into multiple dense patches (refer to Fig. 4). Covariance features are computed over pixels within each patch. Then, we perform max pooling over a fixed-size pooling region and use the pooled features to represent the covariance features in the pooling region. In fact, multiple covariance matrices within each pooling region are summarized into a single matrix which has better invariance to image deformation and translation. The pooled features extracted from each pooling region is called the spatially pooled covariance (sp-Cov) features in .

Implementation To expand the richness of our feature representation, we extract sp-Cov features using multi-scale patches with the following sizes: $4\times 4$, $8\times 8$ and $16\times 16$ pixels. Each scale will generate an independent set of visual descriptors. In our experiments, the patch step-size is set to be 1 pixel, the pooling region is set to be $4\times 4$ pixels, and the pooling spacing stride is set to be $4$ pixels.

Local Binary Pattern (LBP) LBP is a texture descriptor which uses a histogram to represent the binary code of each image patch . The original LBP is generated by thresholding the $3\times 3$-neighbourhood of each pixel with the value of centre pixel. All binary results are concatenated to form an 8-bit length binary sequence with $2^{8}$ different labels. The histogram of these 256 different labels can represent a texture descriptor. By following the work of , we convert the input image from the RGB space to LUV space, and extract the uniform LBP from the luminance (L) channel. The uniform LBP, which is an extension of the original LBP, can better filter out noises.

Spatially pooled LBP Similar to the sp-Cov features, the image window is divided into multiple dense patches and LBP histogram is computed over pixels within each patch. In order to enhance the invariance to image deformation and translation, we perform max pooling over a fixed-size pooling region and use the pooled features to represent the LBP histogram in the pooling region. The pooled features extracted from each pooling region is called the spatially pooled LBP (sp-LBP) features in .

Implementation To extract LBP, we apply the LBP operator on the 33-neighbourhood at each pixel. The LBP histogram is extracted from a $4\times 4$ pixels patch. We extract the 58-dimension LBP histogram using a C-MEX implementation of . In our experiments, the patch step-size, the pooling region, and the pooling spacing stride are set to 1 pixel, $8\times 8$ pixels, and 4 pixels, respectively. Instead of extracting LBP histograms from multi-scale patches, the sp-LBP and LBP are combined as channel features.

III-C Supervised learning

Once dense features have been extracted, we are in a position to train a classifier. Instead of training a standard AdaBoost, we use a shrinkage version of AdaBoost as the strong classifier and use decision trees as weak learners. To train the classifier, the procedure known as bootstrapping is applied, which collects the hard negative samples and re-trains the classifier. If the object subcategorization is applied to the training data, we train one classifier for each sub-detector. The pseudo code of the learning algorithm is presented in Algorithm 1.

Shrinkage The accuracy of AdaBoost can be further improved by applying a weighting coefficient known as shrinkage . The shrinkage version of AdaBoost can be viewed as a form of regularization for boosting. At each iteration, the coefficient of weak learner is updated by

Here $h_{t}(\cdot)$ is a weak learner of AdaBoost at the $t$-th round and $w_{t}$ is the coefficient of the weak learner. $\nu{}\in(0,1]$ is a learning rate which controls the trade-off between overall accuracy and training time. The smaller the value of $\nu$, the higher the overall accuracy as long as the number of weak learners is sufficiently large. Compared to the standard AdaBoost, shrinkage often produces better generalization performance .

Bootstrapping To improve the performance of the learned classifier, we perform three bootstrapping iterations in addition to the original training phase. The initial training phase randomly sample negative samples from training images with positive regions cropped out, and further bootstrapping iterations add more hard negatives to the training set. The learning process consists of 4 training iterations with increasing numbers of weak learners and the final model consists of 2048 weak learners.

III-D Post-processing

Raw detection results are generated by applying the trained detectors on test images, but these results often contain some noises and redundant information. To improve detection performance, some techniques are used to post-process raw detection results.

If we have multiple sub-detectors and apply them on test data, detection results of each sub-detector are required to merge together to generate the integrated results. However, the classifier of each sub-detector is learned with different training data, confidence scores of raw detection results output by individual classifiers need to be calibrated appropriately to suppress noises before merging them together. We address this problem by transforming the output of each classifier by a sigmoid regression to generate comparable score distributions . For sample $i$ in subcategory $k$, its confidence score is the output of the ensemble classifier which is defined as

where $A_{k}$, $B_{k}$ are the learned parameters for the $k$-th subcategory of the following regularized maximum likelihood problem:

The $g_{i}^{k}$ in equation 4 can be cancelled by reformulation:

$N_{k}$ is the total number of training examples for the $k$-th subcategory-specific classifier, $N_{+}$ is the number of positive examples, and $N_{-}$ is the number of negative examples.

III-D2 Non-maximum suppression (NMS)

NMS aims to suppress redundant overlaps among the raw detection results. When multiple bounding boxes overlap, NMS will eliminate the lower-scored detections and retain the highest-scored detection. Pascal overlap score is used to determine the overlap ratio $a_{0}$ between two bounding boxes. The overlap ratio $a_{0}$ is defined as

where $B_{1}$ and $B_{2}$ are two different bounding boxes. If the overlap ratio $a_{0}$ exceeds a predefined threshold, bounding box with the lower confidence score is discarded.

III-D3 Fusion of detection results

The proposed framework can detect multiple objects using detectors or sub-detectors of different classes. Suppose we have detection results generated from different detectors, there are probably some redundant detections on the results since different detectors may generate some overlapped bounding boxes. NMS is usually used to delete redundant bounding boxes. However, NMS is not suitable for all cases. Assume that a car is occluded by a cyclist, both the car and the cyclist are detected in the results. If their overlap ratio exceeds the threshold, NMS will simply delete the lower-scored detection, and retain the higher-scored detection. One true positive detection is removed in this case.

To solve the above problem, we merge all detection results in two steps. In the first step, we merge detection results which belong to the same class using the NMS. It means that we apply NMS to bounding boxes generated by either a single detector of one class or multiple sub-detectors of one class exclusively. Objects of the same class are easily detected redundantly by multiple sub-detectors or a single detector at different scales. NMS is used to remove these redundant detections. In the second step, we merge all remaining detection results of different classes without using NMS to generate the final bounding boxes.

IV Experiments

In this section, we conduct an experiment on traffic sign detection and evaluate our detector on the German Traffic Sign Detection Benchmark (GTSDB) .

The GTSDB dataset contains 600 images for training and 300 images for testing. Images are captured from various scenes (highway, urban, rural) and various time slots (morning, afternoon, dusk, etc.). The dataset contains more than 1000 traffic signs from different categories. Three main categories of traffic signs (Prohibitory, Danger, Mandatory) are selected as the target classes in the IJCNN 2013 competition and in our experiments. The resolutions of traffic signs vary from $16\times 16$ pixels to $128\times 128$ pixels. Fig. 5 shows traffic signs from three main categories on the GTSDB.

IV-A2 Evaluation criteria

Pascal overlap score is used to find the best match between each predicted bounding box and each ground truth. The minimum overlap ratio $a_{0}$ is set to be 60% on the GTSDB. Only the bounding box with the highest confidence score is counted as true positive if multiple bounding boxes satisfy the overlap criterion, the others are ignored. To compare the performance of different detectors, we follow the evaluation metric of the GTSDB which uses the area under the precision-recall curve (AUC) as a final score.

IV-A3 Parameter selection

To alleviate the effect of the illumination change, we apply the automatic color equalization algorithm (ACE) to globally normalize all images. The resolution of the traffic sign model is set to $20\times 20$ pixels and the dimension of model padding is set to $30\times 30$ pixels. This border provides an additional amount of context that helps improve the detection performance . Additionally, we increase the number of positive samples by adding jittered versions of the original samples, which significantly improves the detection performance. For prohibitory and danger signs, flipped versions are added to the training set. For mandatory signs, samples are randomly perturbed in translation ($$pixels), in scale ($[0.8,1]$ratio), in rotation ($$ degrees), and flipping. We demonstrate the performance gain on the test set in table I. Negative samples are collected from the GTSDB training images with the corresponding traffic sign regions cropped out.

IV-A4 Experimental design

We investigate the experimental design of the proposed detector on traffic sign detection. Since traffic signs are divided into three subcategories, we train one sub-detector for each subcategory. We train all detectors on the GTSDB training set and evaluate them on the GTSDB test set. All experiments are carried out using combined features (ACF+sp-Cov+sp-LBP) as dense features, Adaboost with shrinkage value of 0.1 as the strong classifier, and depth3-decision trees as weak learners (if not specified otherwise).

Shrinkage We evaluate the performance of AdaBoost with 4 different shrinkage values from $\{0.05,0.1,0.2,0.5\}$. We decrease the reject threshold of soft cascade by a factor of $\nu$ as coefficients of weak learners have been diminished by a factor of $\nu$. The area under precision-recall curve of different detectors are shown in Table II. We observe that applying a small shrinkage value often improves the detection performance and the best performance is achieved by setting $\nu=0.1$. However, without increasing the number of weak learners, setting the shrinkage value to be too small ($\nu=0.05$) can degrade the performance as the boosting cannot converge with a limited number of boosting iterations.

Depth of decision trees We trained 4 different traffic sign detectors with decision trees of depth 1 to depth 4. Table III shows the detection performance of different detectors. We observe that increase the depth of decision trees provides a performance gain, especially for the mandatory category. However, the depth-3 decision trees achieve better generalization performance and are faster to train than depth-4 decision trees.

Combination of features To compare the discriminative power of different feature representations, we evaluate the performance of various feature combinations. The results are shown in Table IV. We observe that the combination of the sp-Cov features and LUV outperforms the ACF features and combining more features can further improve the detection performance. The best result is achieved by using the combination of all features (sp-Cov+sp-LBP+ACF).

IV-A5 Comparison with state-of-the-art detectors

Detection performance of various detectors on the GTSDB test set are shown in Table V. The proposed detector achieves the comparable results with state-of-the-art detectors despite its simplicity. These detectors that offer better performance employ multi-scale models in detection. The authors of trained multiple subcategory-specific classifiers for each type of mandatory signs to achieve the best performance.

IV-B Car detection on UIUC dataset

Next, we conduct an experiment on car detection and compare detection performance of different detectors on the UIUC dataset . The UIUC dataset captures images of side views of cars with a resolution $40\times 100$ pixels. The training set contains 550 positive samples and 500 negative samples. The test set is divided into two sets: 170 single-scale test images, containing 200 cars at roughly the same scale as in the training images, and 108 multi-scale test images, containing 139 cars at various scales.

We follow the evaluation protocol provided along with the UIUC dataset. A bounding box is counted as true positive if it lies within 25% of the ground truth dimension in each direction. Only the bounding box with the highest confidence score is counted as true positive if multiple bounding boxes satisfy the criterion, the others are counted as false positives. In the dataset, three criteria are adopted to evaluate the performance: $F_{1}$-score, detection rate, and the number of false positives. $F_{1}$-score is the weighted harmonic mean of precision and recall.

The dimension of UIUC car model is set to $40\times 100$ pixels without marginal padding as the car images are clipped to the same size. We expand the positive samples by flipping car images along the vertical axis. Since viewpoints of cars in the UIUC dataset are limited to side-views, we train a single detector without applying subcategorization and bootstrapping. Table VI shows the results of different detectors on the multi-scale test subset. We observe that our detector achieves the best detection rate with slight more false positives on this dataset.

IV-C Car detection on KITTI dataset

To further demonstrate the effectiveness and robustness of the proposed detector on car detection, we evaluate our detector on a more challenging object detection benchmark, KITTI dataset

The KITTI dataset is a recently proposed challenging dataset which consists of 7481 training images and 7518 test images, comprising more than 80 thousands of annotated objects in traffic scenes. Table VII provides a summary of existing car datasets. We observe that the KITTI dataset provides a large number of cars with different sizes, viewpoints, occlusion patterns, and truncation scenarios. Due to the diversity of these objects, the dataset has three subsets (Easy, Moderate, Hard) with respect to the difficulty of object size, occlusion and truncation. Since the detection performance are ranked based on the moderately difficult results, we use the moderate subset as the training data in our experiments. The moderate subset contains 15710 cars, with the heights of the cars vary from 25 pixels to 270 pixels and the aspect ratios vary between 0.9 and 4.0. Since annotations of test data are not provided by the KITTI benchmark, we split the KITTI training images into training set (first 4000 images) and validation set (remaining 3481 images).

IV-C2 Evaluation criteria

We follow the provided protocol for evaluation. Pascal overlap score is used to find the best match and the minimum overlap ratio $a_{0}$ is set to be 70%. Only the bounding box with the highest confidence score is kept if multiple bounding boxes satisfy the overlap criterion, the others are counted as false positives. Instead of using AUC, average precision (AP) is used to evaluate the detection performance. The AP summaries the shape of the precision-recall curve, and is defined as the mean precision at a set of evenly spaced recall levels.

IV-C3 Parameter selection

We apply the proposed subcategorization method to categorize the training data into multiple subcategories. To find the model dimensions of each subcategory, we set the base height of each model to 52 pixels. From the base height, the width of each model can be obtained by taking the median aspect ratios of cars in the corresponding subcategory. Each model includes additional 4 pixels of margin on all sides. Using a model with suitable aspect ratio can significantly improves the detection performance due to better localization. We expand the positive training samples by randomly perturbing original car patches in translation ($$pixels), and in rotation ($$ degrees). Negative samples are collected from the KITTI training images with vehicles regions cropped out.

IV-C4 Experimental design

We investigate the experimental design of the proposed detector on car detection. We train car detectors on the training set and evaluate them on the validation set. All experiments are carried out using ACF as dense features, Adaboost with shrinkage value of 0.1 as the strong classifier, depth4-decision trees as weak learners, and $K=25$ in the subcategorization method (if not specified otherwise).

Number of subcategories To investigate the effect of different numbers of clusters in our subcategorization method, we set the number from {1, 4, 8, 12, 16, 20, 25}. Fig. 6(a) and Fig. 6(b) shows the effect of increasing the number of subcategories on geometrical feature space and visual feature space, respectively. We observe that the geometrical features outperform the ACF features in the spectral clustering. We also observe that the detection performance improves as we increase the number of subcategories with $K=25$ provides the best performance.

Depth of decision trees We trained 4 different car detectors with decision trees of depth 2 to depth 5. Table VIII shows the average precisions of different detectors. We observe that depth-4 decision trees provide the best generalization performance.

Combination of features We evaluate the performance of various feature combinations on car detection. The results are shown in table IX. We observe that the detection performance improves as we add more features and the best performance is achieved by using the combination of all features (sp-Cov+sp-LBP+ACF). The combination of sp-LBP features and ACF features also achieves the similar performance and is five times faster than the combination of all features. We use the combination of sp-LBP features and ACF features as dense features in the testing since it gives a better trade-off between detection performance and runtime.

IV-C5 Comparison with state-of-the-art detectors

Table X shows the performance comparison of state-of-the-art detectors on the KITTI testing set. Experimental results show that the proposed detector is of not only better performance than all DPM-based methods but also less runtime. More significantly, our detector outperforms the SubCat which employs a similar object subcategorization method and the Regionlets which employs a similar pooling strategy. We conjecture that the additional performance gain is provided by the spatially pooled features.

IV-D Cyclist detection on KITTI dataset

In this section, we conduct an experiment on cyclist detection and evaluate our detector on the KITTI dataset.

The KITTI dataset contains annotated cyclist objects which are captured from various traffic scenes. Similar to cars, cyclists are divided into three subsets (Easy, Moderate, Hard) and the moderate subset is used as the training data in our experiments. The moderate subset contains 1098 cyclists, with the heights of the cyclists vary from 25 pixels to 275 pixels and the aspect ratios vary between 0.3 and 1.5.

IV-D2 Evaluation criteria

The KITTI cyclist detection uses the same evaluation protocol with the car detection expect that the minimum overlap ratio is relaxed to 50%.

IV-D3 Parameter selection

The proposed subcategorization method is applied to cyclist detection. We define the dimensions of each cyclist model using the similar method in car detection. We set the base height of each model to 56 pixels, and the width of each model is derived from the median aspect ratios of cyclists in the corresponding subcategory. Each model includes additional 4 pixels of margin on all sides. We expand the positive training samples by randomly perturbing the original cyclists in translation ($$pixels), in rotation ($$ degrees). Negative patches are collected from the KITTI training images with cyclist regions cropped out.

IV-D4 Experimental design

We investigate the experimental design of our detector on cyclist detection. We train cyclist detectors on the training set and evaluate them on the validation set. All experiments are carried out using ACF as dense features, Adaboost with shrinkage value of 0.1 as the strong classifier, depth4-decision trees as weak learners, and $K=4$ in the subcategorization method (if not specified otherwise).

Number of subcategories We set the number of clusters from {1, 2, 3, 4, 5, 6} in our subcategorization method. Since the minority of cyclists are occluded and truncated, clustering on all geometrical features leads to a cluster degeneration problem. We carefully select the aspect-ratios of cyclists as the feature space to avoid the above problem. Fig. 6(c) shows the effect of increasing the number of subcategories. We observe that the detection performance improves as we increase the number of subcategories when $K<=4$. Since the number of cyclists is much less than cars, the number of cyclists in each subcategory becomes very small when we have too many subcategories, which results in an imbalanced learning problem and degrading the detection performance.

Depth of decision trees We trained 4 cyclist detectors with decision trees of depth 2 to depth 5. Average precisions of different detectors are shown in Table XI. We observe that depth-4 decision trees offer the best generalization performance, as similar in the car detection.

Combination of features We evaluate the performance of various feature combinations on cyclist detection. The results are shown in Table XII. We observe that the best performance is achieved by using the combination of sp-LBP features and ACF features. The performance declines when we add the sp-Cov features as a part of aggregated channel features. The reason may be due to the lack of enough cyclist training data. We use the combination of sp-LBP features and ACF features as the dense features in the testing.

IV-D5 Comparison with state-of-the-art detectors

Table XIII shows the performance comparison with state-of-the-art approaches. As shown in Table XIII, our detector outperforms all other methods on the test set. Specifically, our detector outperforms the best DPM-based method DPM-VOC+VP on all the three subsets by 16.29%, 14.95%, and 12.35%, respectively. Our detector also performs slightly better than the Regionlets .

IV-E An evaluation of the overall runtime

We conduct an experiment on evaluation of the overall runtime with various feature combinations on the KITTI dataset. All experiments are carried out on a computer with an octa-core Intel Xeon 2.50GHz processor. The average runtime of each component of our detection framework can be seen in Table XIV. For feature extraction, we observe that the ACF features can be extracted very quickly within 0.1s. When we add the sp-LBP features, the runtime increases moderately, but this features give an obvious performance gain in all three applications. When the sp-Cov features are employed, the runtime of feature extraction increases rapidly and dominates the total runtime of the system. For object detection, we observe that the car detector costs the most time in this framework since it has 25 sub-detectors. The traffic sign detector uses the least time since it has only 3 sub-detectors. We also observe that the runtime of detection increases as we add more features in this framework. According to observe the detection results of three applications, we conjecture that using a combination of ACF features and sp-LBP features can provide a better trade-off between detection performance and system runtime.

V Conclusion

In this paper, we propose a common framework for detecting three important classes of objects in traffic scenes. The proposed framework introduces spatially pooled features as a part of the aggregated channel features to enhance the robustness and employs detectors of three important classes to detect target objects. The detection speed of the framework is fast since dense features need only to be evaluated once rather than individually for each detector. To overcome the weakness of the VJ framework for object classes with large intra-class variations, we propose an object subcategorization method to capture the variations and improves the generalization performance. We demonstrated that our detector achieves the competitive results with state-of-the-art detectors in traffic sign detection, car detection, and cyclist detection. Future work could include that contextual information can be used to facilitate object detection in traffic scenes and convolutional neural network can be used to generate more discriminative feature representations.

Acknowledgements This work was in part supported by Australia’s Information and Communications Technology (ICT) Research Centre of Excellence.

References