KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
Yiyi Liao, Jun Xie, Andreas Geiger
I Introduction
One of the pioneering works in computer vision can be traced back to Larry Roberts’ “Blocks World” in the 1960s , which aimed at identifying individual objects and inferring the 3D structure of simple shapes from 2D images. With the goal of understanding a scene from visual cues, computer vision was viewed as a comparably easy first step towards solving higher-level reasoning tasks in robotics at that time (e.g., the MIT copy demo ). Albeit being seemingly easy for humans, robustly perceiving geometry and semantics from images proved hard for machines due to the high complexity of real-world environments. Thus, in the 1980s, computer vision and robotics evolved into their own, largely independent research fields. Only recently, the communities have realized that it is impossible to solve one without the other, e.g., in the context of self-driving . Similarly, computer vision’s interaction with computer graphics emerged in the 1990s and has gained traction over the last decade, in particular in areas such as neural and image-based rendering . These advances can in turn benefit robotics as simulation will be crucial for training and validating the next generation of robotic systems.
The converging trend of vision, graphics, and robotics motivates us to create a new dataset, KITTI-360, that addresses tasks at the intersection of these fields with a focus on autonomous driving. While the KITTI dataset has pushed the state-of-the-art in computer vision algorithms forward, it does not contain dense and complete semantic labels. Thus, many interesting interdisciplinary tasks, e.g., synthesizing novel view images jointly with semantics or reconstructing large-scale semantic maps, cannot be evaluated on KITTI. Moreover, the captured perspective front images provide only a partial view of the scene and the 3D information provided by the LiDAR sensor is very sparse. The GPS localization of KITTI is reliable but does not reach sub-pixel accuracy when fusing multiple frames. With KITTI-360 we address these shortcomings by providing a new dataset with more comprehensive semantic/instance labels in 2D and 3D, richer 360∘ sensory information (fisheye images and pushbroom laser scans), very accurate and geo-localized vehicle and camera poses, and a series of new challenging benchmarks, see Fig. 1 for an overview.
A key challenge towards building such a dataset is to obtain coherent dense and comprehensive semantics in 2D and 3D. Many existing datasets are annotated in the 2D image domain where pixel-wise labeling requires up to minutes per image for a human annotator. Other datasets are annotated in 3D space while ignoring information in the 2D image domain. A few datasets offer labels in both 2D and 3D. However, annotation is conducted independently, thus duplicating the labeling effort.
In this paper, we propose an alternative approach that leverages coarse 3D annotations to significantly simplify the dense annotation task and establish coherent labels in both 2D and 3D space. Moreover, this yields a unique instance index for each object in the scene across all 2D video frames. Specifically, we build a WebGL-based annotation tool that allows for annotating both static and dynamic scene elements directly in 3D using simple primitives. This approach has several advantages over labeling in 2D: First, objects often project into several video frames, thus lowering annotation efforts considerably. Further, the obtained 2D instance annotations are temporally coherent as they are associated with a single physical 3D object. Finally, our 3D annotations covering the full 3D scene are useful on their own, e.g., for reasoning in 3D or to enrich 2D annotations with approximate 3D geometry.
However, obtaining dense and accurate pixel-wise 2D labels and point-wise 3D labels from sparse, noisy point clouds and coarse 3D annotations is a challenging task. Towards solving this problem, we propose a non-local multi-field CRF model which reasons jointly about semantic and instance labels of all 3D points and 2D pixels. Our approach also leverages learning-based methods to provide dense semantic and instance priors in the 2D image domain. As evidenced by our experiments, our method outperforms label propagation methods operating purely in 2D as well as pure learning-based approaches. Furthermore, the probabilistic nature of our model allows for estimating label uncertainties which can be used to increase label accuracy when only a subset of the pixels require a label.
From the annotated dataset, we derive several benchmarks and baselines with novel and challenging tasks at the intersection of vision, graphics and robotics which we believe are crucial for making progress towards the grand challenge of fully autonomous driving. Our semantic scene understanding benchmark includes tasks for 2D/3D recognition and semantic scene completion. The former requires predicting a semantic/instance label for the visible part of the scene, while the latter aims for joint geometric completion and semantic perception that can benefit higher-level reasoning, e.g., control and planning. In our novel view synthesis benchmark, we establish a challenging task that requires synthesis of both RGB appearance and semantic labels at a given novel viewpoint, aiming to foster research on building fully labeled simulation environments from real-world images. Lastly, our semantic SLAM benchmark evaluates vehicle localization as well as geometric and semantic 3D reconstruction over long sequences.
We summarize the contributions of this paper as follows:
We present a novel georegistered dataset of suburban scenes recorded by a moving platform. The dataset comprises over k images and k laser scans.
We create and release a WebGL-based annotation tool that allows for labeling street scenes in 3D space. Exploiting our annotation tool, we obtain 3D annotations for all static and dynamic scene elements.
We propose a method which transfers these labels from 3D into 2D, yielding pixel-wise semantic instance annotations. We validate our approach in ablation studies and demonstrate its potential with respect to several 2D and 3D baselines.
Enabled by our dense and coherent semantic instance annotations in both 2D and 3D as well as accurate vehicle and camera poses, we establish an online benchmark with novel and challenging tasks at the intersection of computer vision, graphics and robotics. We believe that our dataset and benchmarks will complement existing datasets and foster novel research towards solving the grand goal of full autonomy.
This journal paper is an extension of a conference paper published at CVPR 2016 . In comparison to , we 1) extend our annotation tool and update our inference algorithm to support the annotation of dynamic objects; 2) provide a detailed description of the annotation tool and process; 3) establish new online benchmarks with held-out test data on a set of challenging tasks; 4) propose and evaluate several baselines to bootstrap the leaderboards and assess the difficulties of the tasks. We make our dataset http://www.cvlibs.net/datasets/kitti-360, utility scripts https://github.com/autonomousvision/kitti360scripts and annotation tool https://github.com/autonomousvision/kitti360labeltool publicly available.
II Related Work
In this section, we first discuss existing datasets in the context of autonomous driving, followed by a review of current methods for efficient (semi-automatic) label annotation.
Indoor Video Datasets: Several datasets provide annotations for video sequences captured in indoor scenes . The SUN RGB-D dataset provides labeled 2D polygons as well as 3D cuboids for k indoor RGB-D images. In a closely related work, ScanNet is annotated in 3D with its 2D labels directly obtained from 3D-to-2D projection based on the dense depth from RGB-D sensors. In this work, we focus on outdoor street scenes where 3D observations are much more sparse, posing a challenging task for 3D-to-2D label transfer.
Outdoor Datasets: A number of outdoor datasets of driving scenes have been released in the literature . We summarize the most related ones in Table I, categorized by whether they offer labels in the 2D image domain or in 3D space.
For datasets focusing on 2D labels, CamVid is the first dataset for semantic segmentation in the context of self-driving. However, CamVid does not provide instance labels and only a very limited number of frames. Both Cityscapes and Mapillary Vistas release thousands of manually annotated 2D images. However, they do not offer temporally coherent semantic instance annotations. Recently, Cityscape-VPS extends Cityscapes by providing semantic instance labels for every 5 frames. Furthermore, KITTI-STEP offers spatially and temporally dense semantic instance annotations for the KITTI tracking dataset . While aforementioned works focus on perspective images, WoodScape releases semantic instance annotations of fisheye images. Our dataset differs from the above in that we provide not only temporally coherent semantic instance annotations for perspective images, but also omnidirectional imagery, 3D laser scans, and 3D annotations which are useful for 3D reasoning. While focuses on inner-city scenes, our dataset comprises mainly suburban areas, thus both datasets complement each other (we use the same label definition to facilitate research).
Another line of works provides labels in 3D space. Toronto-3D , Paris-Lille-3D and DublinCity offer annotated point clouds collected from urban environments. Semantic3D.net presents a large-scale dataset with billion points, labeled with semantic categories. SemanticKITTI provides semantic labels for raw laser scans in KITTI, resulting in billion labeled 3D points in classes. Instead of focusing on point cloud semantic classification, Argoverse , Lyft , Waymo , and A*3D offer 2D/3D bounding boxes and establish benchmarks for 2D/3D detection and tracking We refer to 2D annotations as pixel-level annotations in Table I. 2D bounding boxes are not included.. In contrast to KITTI-360, the aforementioned datasets either lack dense annotations in images or they do not have per-point 3D annotations of stuff classes.
Our dataset provides labels for both 2D images and corresponding 3D points. Within this category, KITTI provides dense semantic information on images and k 3D bounding boxes. However, KITTI does not provide dense (per-point) 3D labels on the point cloud. Closely related to our work, ApolloScape annotates static scene elements in the 3D space The 3D annotation has not been released yet. and projects them to the 2D image space, followed by manual annotation of dynamic objects in images. In this work, we annotate both static and dynamic objects in 3D, providing coherent annotations for dynamic objects both in 2D and 3D. More recently, nuScenes and A2D2 released labels in both 2D and 3D. However, the labels of nuScenes are manually and independently annotated in 2D and 3D, and not every pixel is labeled in 2D. In contrast, we propose to leverage labels in the 3D space to infer dense labels in the image domain, thus providing consistent labels across 2D and 3D space. A2D2 labels 2D images and maps 2D labels to 3D to obtain per-point 3D labels. Thus, the 3D labels are limited to a small FoV of the cameras in the azimuthal direction. While A2D2 also provides 3D bounding boxes, all of them are within the FoV of the forward-facing camera. We instead offer per-point 3D labels and 3D bounding boxes within an azimuthal FoV of 360∘. A concurrent work, SemKITTI-DVPS provides labels in both 2D and 3D by projecting the 3D labels of SemanticKITTI to images. Compared to the projected sparse 2D labels of SemKITTI-DVPS, KITTI-360 offers dense pixel-wise labels and additionally provides 3D bounding boxes.
There also exist several synthetic urban datasets . However, there still exists a significant perceptual gap between the virtual and real domains , making synthetic-to-real generalization difficult.
Benchmarks: Recently, evaluation benchmarks have been widely recognized by the community. Some of the previously mentioned datasets also provide online evaluation benchmarks and held-out test data for different tasks. For instance, Cityscapes offers a benchmark suite for pixel and instance-level semantic segmentation as well as 3D vehicle detection. SemanticKITTI hosts lidar segmentation challenges to predict the category of every point. For datasets including both 2D and 3D annotations, KITTI , nuScenes , and ApolloScape provide benchmarks on a set of vision tasks including detection, stereo, localization, multi-object tracking, and segmentation in both 2D and 3D, etc. Moving beyond the established tasks, KITTI-360 provides novel benchmarks and will hold new challenges, e.g., on novel view semantic synthesis and semantic SLAM, to foster new progress towards full autonomy.
II-B Methods
Efficient Annotation: Many works have attempted to reduce the per pixel annotation time of individual images, including classical methods and learning based methods . While all of these methods focus on annotating images individually, we are interested in annotating 2D video sequences as well as 3D scenes. There is also a growing interest in autolabeling 3D shapes or 3D bounding boxes . These methods are only applicable to a specific class, e.g., vehicles. We instead annotate the full 3D scene and aim to obtain coherent per-pixel 2D annotations and per-point 3D annotations.
2D Label Propagation: Compared to annotating individual images, video sequences offer the advantage of temporal coherence between adjacent frames. Label propagation techniques exploit this fact by transferring labels from a sparse set of annotated keyframes to all unlabeled frames based on color and motion information. While in some works a single foreground object is assumed , here we focus on methods that can handle multiple object categories. Towards this goal, and propose a coupled Bayesian network based on video epitomes and semantic regions to propagate label information between two annotated keyframes. proposes a joint propagation strategy with synthesized training samples. To better account for errors in label propagation, proposes a hierarchy of local classifiers for this task and leverages a mixture-of-tree model for temporal association. The work of leverages label propagation as a data augmentation scheme and demonstrate improved performance on semantic segmentation. Optical flow is also commonly used for semantic video label transfer. uses optical flow of adjacent frames to warp network representations across time and thus propagates labels from previous frames to the current one. proposes to run a convolutional sub-network only on sparse keyframes and propagate the deep feature maps to other frames via flow fields. In the indoor scenario where dense geometry is available, proposes a method on RGB-D video propagating labels on super-pixel.
In contrast to the aforementioned methods which propagate labels in 2D, in this paper we propose to annotate both semantic and instance labels directly in 3D and then project these annotations into the 2D domain. While this approach requires a source of 3D information (e.g., SfM, stereo, laser), it is able to produce more accurate semantic and temporally consistent instance annotations for tracking purposes. Further, our experiments indicate that annotation in 3D is more time-efficient than labeling in 2D as scene elements can be separated more easily and often project into many images of the input video sequence while being only annotated once.
3D-to-2D Label Propagation: There are a few existing works on 3D-to-2D label transfer. Chen et al. leverage annotations from KITTI as well as 3D car models to infer separate figure-ground segmentation for all vehicles in the image. In comparison, our approach reasons jointly about all objects in the scene and also handles categories for which CAD models or 3D point measurements are unavailable (e.g., “Tree”, “Sky”). Huang et al. also applies 3D to 2D label transfer for generating the ApolloScape dataset. In this work, labels are transferred from 3D point clouds to images with simple splatting and projection. However, 3D points are too sparse compared with image pixels, thus, setting the splatting range is not trivial. Similarly, in , semantic labels annotated in the reconstructed scene are projected into each frame but not all 2D pixels are covered due to missing geometry. In addition, the two aforementioned works are limited to static scenes.
In the context of street view image segmentation, exploit the interaction between image pixels and 3D points to improve classification performance or efficiency. In comparison, our goal is to transfer ambiguous 3D primitive labels to every pixel in the image.
III Annotation
In this section, we describe our data collection efforts, data preprocessing, the annotation tool, and annotation details.
For data collection, we equipped a station wagon with one fisheye camera to each side and a perspective stereo camera (baseline cm) to the front. Furthermore, we mounted a Velodyne HDL-64E and a SICK LMS 200 laser scanning unit in pushbroom configuration on top of the roof. This setup is similar to the one used in KITTI , except that we gain a wider field of view with the additional fisheye cameras and the pushbroom laser scanner while KITTI only provides perspective images and Velodyne laser scans with a vertical field of view. Compared to omnidirectional camera systems , our setup benefits from increased resolution of the 3D reconstruction. Localization is provided by IMU and GPS which we fuse with visual features. Fig. 1 (top left) illustrates our setup.
Using this setup, we recorded several suburbs of a mid-size city corresponding to over k images and k laser scans, covering a driving distance of 73.7km. We estimate all vehicle and camera poses using structure-from-motion . More specifically, we minimize 3D reprojection errors based on all feature matches while regularizing against the GPS location. We further add loop-closures detected from LiDAR scans as regularization to complement image feature matching (which might fail on opposite-facing frames). This results in accurate georegistered camera poses. Fig. 2 illustrates the camera poses overlaid on OpenStreetMap http://www.openstreetmap.org/. We also plot the camera poses of the KITTI dataset for reference. KITTI-360 follows KITTI’s forward facing camera configuration, but has minimal overlap with KITTI in terms of trajectories. This allows us to split training and test data without conflicting with the KITTI dataset, e.g., avoiding the situation where a region is used for training in KITTI but testing in KITTI-360. Following KITTI, we use Mercator projection to convert geographic coordinates to a local Euclidean coordinate frame in order to facilitate usage of the dataset. The origin of the coordinate frame is chosen as the center of the map as illustrated in Fig. 2.
III-B Annotation Interface
To facilitate 3D annotation, we developed an online annotation tool based on WebGL. We release our annotation tool (see Fig. 3) as part of this project. It consists of three main components: a scene viewer (including 2D images and 3D scene), a semantic label selection panel, and controllers. Annotators are asked to insert 3D primitives with adjustable shapes and semantic labels into the 3D scene.
To annotate the data while limiting transfer bandwidth, we split the collected data into batches according to the accumulated driving distances. Specifically, a single batch contains observations within a driving distance of about 200 meters (240 frames on average) and there is an overlap of 10 meters between two consecutive batches. Within one batch, we accumulate 3D points observed from the Velodyne and SICK laser scanning unit as well as the stereo camera.
During annotation, the accumulated point clouds are downsampled to reduce data loading traffic and memory. However, downsampling makes it hard to precisely perceive dynamic objects whose 3D observations are distributed along a moving trajectory. To allow for accurate labeling of dynamic objects, we apply a simple heuristic to detect dynamic objects, see Appendix A-B. We then load all detected dynamic points at each frame into the annotation tool without down-sampling. To help the annotators efficiently identifying dynamic objects, we highlight dynamic objects using white color as illustrated in Fig. 3.
As auxiliary visualization to the 3D point clouds, we provide fisheye and perspective images (see “Side View” and “Front View” in Fig. 3) in order to allow annotators to select and perceive the scene from different camera views. We also visualize the pose of each camera, enabling annotators to quickly select informative viewpoints.
III-B2 Semantic Label Panel and Controllers
We show semantic labels with different colors in the label panel for users to choose from. To better assist annotators in placing the primitives accurately, we also offer easy-to-use controllers to interact with the 3D scene, including zoom, pan, rotation of the point cloud, switching data sources or camera views, and toggling annotations. We provide more details about the annotation interface in Appendix B.
III-C Annotation Details
We ask the annotators to annotate the 3D point clouds in the form of bounding primitives, i.e., place cuboids and ellipsoids to enclose objects in 3D and assign a semantic label to each of them. The 3D scene is annotated with 37 label classes, including 24 “instance” classes and 13 “stuff” classes. Labels are defined in accordance with the Cityscapes dataset label definition. More details about the label definition can be found in Appendix C. The annotations are categorized into static and dynamic objects, which are treated differently by our annotation tool.
Static labels can be further classified into two categories: “stuff” and “instance”. For instance classes, each object is constrained to be associated with only one cuboid primitive, representing both semantic and instance labels of this object. We ask the annotators to tightly enclose the point clouds with the bounding primitives. For stuff classes, which usually have irregular shapes, annotators are allowed to use multiple cuboids or ellipsoids to roughly enclose the 3D points of the target objects.
We also provide a “planar” annotation option for stuff categories on the ground such as “Road” and “Sidewalk”. Using this option, we allow annotators to draw a 2D polygon representing the ground object’s boundary in bird’s eye view. The interface then automatically estimates the height of the polygon based on the surrounding 3D geometry and extrudes the 2D polygon into 3D along the vertical direction to enclose corresponding 3D ground points. We provide more details in Appendix B-C.
III-C2 Dynamic Objects Annotation
Dynamic objects mainly comprise moving vehicle and pedestrian instances. In contrast to ApolloScape which annotates static objects in 3D and dynamic objects in 2D respectively, we annotate both static and dynamic objects in 3D space. However, compared to static objects, annotating dynamic objects in 3D outdoor scenes is more challenging as individual dynamic objects in the 3D reconstruction are hard to perceive and distinguish. Moreover, we need to label not only where the moving instance is, but also “when” the instance appears, requiring the annotation of moving 3D bounding boxes over time. A naïve solution is to place a 3D bounding box in every frame where the dynamic object is present. However, such an annotation process would be intractably slow. Thus, we instead implement a semi-automatic annotation scheme to reduce label time. Specifically, we minimize the effort required by annotators by making two assumptions: the size of the dynamic object is fixed over time and its trajectory is smooth. Under these assumptions, the required annotation is reduced to the size of a single 3D primitive and the pose of this primitive at several keyframes. Our annotation tool then automatically places the remaining primitives along the trajectory, see Appendix B-D for more details.
III-D Annotation Procedure
We annotated 379 batches in total, assigning one batch to one annotator. To control the annotation quality, we train and evaluate the annotators based on multiple pilot tasks until they have proven qualified for the full task. We also regularly verify their annotation quality and ask them for correction if necessary. We further identify a few annotators who consistently produced high-quality labels and ask them to cross-check other annotators’ quality. Our annotation interface simplifies the detection and correction of annotation errors compared to annotating image sequences, which requires corrections across multiple frames. Fig. 3 shows parts of an annotated batch via our web interface.
III-E Annotation Time
On average, annotating one full batch ( 240 frames) in 3D required about 3 hours. Thus, our annotators spend only 3 60 240 = 0.75 minutes for “annotating” one image. In comparison, 7 minutes are required for coarse annotation of semantic instance labels in the image domain, and 1.5 hours for pixel-accurate annotations as discussed by the creators of the Cityscapes dataset .
IV Label Transfer Method
In this section, we first provide an overview of our method for transferring the 3D annotation to semantic instance annotations in 2D. Next, we formally introduce the model and discuss parameter learning and inference.
Given 3D annotations, we are interested in generating dense semantic instance annotations for all images and all 3D points. To incorporate inductive biases about image formation and label smoothness, we explore a Conditional Random Field (CRF) model which reasons jointly about the labels of the 3D points and all pixels in the image. In practice, we apply the CRF at every timestamp independently to keep inference tractable. Despite independent inference, we are able to obtain consistent results over multiple frames thanks to the shared 3D annotations. We also experimented with inference over multiple adjacent frames but did not observe measurable improvements.
Let denote all 3D annotations available at timestamp . Here, and correspond to 3D bounding primitives of static and dynamic objects respectively, with and indexing each primitive. Note that a static primitive is used at all timestamps (if visible) whereas a dynamic primitive is only included in when it is labeled to appear at timestamp . Fig. 4(a) illustrates static and dynamic bounding primitives as well as their projection into the 2D image domain. With this design, our framework allows for annotating the same object using a unique instance ID across the entire sequence as well as across 2D and 3D.
Let denote the set of image pixels at timestamp and denote the visible 3D points at the same timestamp. The CRF model is defined over all elements in and . To obtain more complete 3D information, fuses stereo and laser scans over multiple frames. We first fuse points covering static parts of the scene, and then accumulate points of each dynamic object according to its bounding primitives and insert them into the static scene depending on the location of . We provide more details regarding the accumulation of static and dynamic 3D points in Appendix D-A.
IV-B Model
We now formalize the CRF model applied at every frame as illustrated in Fig. 4(b). Note that our 3D annotations are sparse and noisy, i.e., 3D points can carry none, one or multiple labels due to overlapping bounding primitives in 3D. The algorithm described in this section is designed to resolve these situations and infers marginal estimates for all 3D points and pixels in the image.
As the CRF model is applied at every frame independently, we drop the dependency on timestamp of and for simplicity. For each pixel and each 3D point , we specify random variables and taking values from the set of semantic (or instance) labels , where denotes the number of classes. For instance inference, we assign a unique ID to each object which projects into the image. Thus, semantic and instance inference can be treated equally under our model and we will refer to both as “semantic labels” in the following. Note that there is no need to distinguish static or dynamic objects in the single frame-based CRF model. Still, we are able to retrieve whether a pixel or a 3D point belongs to a dynamic object or not according to its instance ID.
Let denote the set of semantic labels. Dropping all dependencies on the image and point cloud for clarity we specify our CRF in terms of the following Gibbs energy function:
with unary potentials and pairwise potentials . For notational clarity, we omit all conditional dependencies on the input images, 3D points and 3D annotations.
Pixel Unary Potentials: The pixel unary potentials encode the likelihood of pixel taking label
where and denote learned feature weights. Our first constraint determines the set of admissible labels and is obtained by projecting all 3D bounding primitives (which are an upper bound on the objects’ extent) into the image. We formulate the constraint via a binary feature which takes for pixel if its ray passes through a primitive of class , and otherwise.
In addition, we exploit a data-driven approach in order to obtain a per-pixel probability distribution over semantic labels . Specifically, we project all non-occluded and uniquely labeled sparse 3D points into the image plane, and use these sparse projections as supervision to train a semantic segmentation network (PSPNet ) on the entire dataset. The output of the network’s last layer is taken as the probability distribution. We also augment the training dataset using Cityscape images and labels to enable the model to learn accurate object boundaries which is difficult to learn based on the projection of sparse and noisy LiDAR point clouds. As the semantic segmentation model does not distinguish instances, we further adopt a state-of-the-art instance segmentation method to obtain instance hypotheses for “car”, “truck”, and “pedestrian”. Thus, we effectively exploit the inductive biases of modern neural network architectures and co-training on related labeled datasets. As demonstrated in Appendix D-D, this leads to a significant improvement at object boundaries.
3D Point Unary Potentials: The 3D point unary potentials encode the likelihood of 3D point taking label :
where denotes a feature which takes if the 3D point lies within a 3D primitive of class within , and otherwise. As the “sky” class can’t be modeled with primitives, we set to if takes the label “sky”. Additionally, we create “virtual sky points” at infinity for all pixels whose ray doesn’t intersect any 3D primitive. Note that these pixels must correspond to sky regions as we assume that the scene is densely annotated, hence each object is contained in one or several bounding 3D primitive(s).
Pixel Pairwise Potentials: Our dense pairwise term encourages semantic label coherence and connects all pixels in the image via Gaussian edge kernels following
where is the 2D location of pixel and denotes its color value. Further, and are learned pairwise feature weights and parameterizes the kernel width.
3D Pairwise Potentials: Similarly, we apply a Gaussian edge kernel to encourage label consistency between 3D points based on their 3D location and surface normals
where is the 3D location of point and denotes the vertical (up) component of its normal. We use the normal’s z-component as it is the most discriminative cue for label changes between horizontal (e.g., road, sidewalk) and vertical (e.g., side of car, wall) surfaces. We estimate the respective normals using principal component analysis in a local neighborhood around each 3D point.
2D/3D Pairwise Potentials: Finally, we encourage coherence between all 3D points and the image pixels
where denotes the projection of the 3D laser or stereo point onto the image plane. Importantly, we project only points into the image which are likely to be visible. We determine these points by meshing the 3D point cloud using the ball-pivoting method of Bernardini et al. , and considering only 3D points in front of the mesh. We also experimented with multi-view reconstruction approaches for mesh generation, but obtained better results using this simpler approach. As applying the meshing algorithm independently for every frame is time-consuming, we generate meshes on entire batches, processing the static part and dynamic objects independently. This allows us to reuse the mesh of the static part for all frames of a batch.
IV-C Learning and Inference
This section describes inference and parameter estimation in our label transfer model.
Inference: At test time, we are interested in estimating the marginal distribution of each semantic or instance label in under our model, specified by the Gibbs distribution defined in Eq. IV-B. A likely configuration can then be estimated by variable-wise maximization of these marginals. As our graphical model is loopy, exact inference in polynomial time is intractable. We thus resort to variational inference and approximate the probability distribution on by replacing it with a factorized mean field distribution . This mean field approximation can be computed efficiently using bilateral filtering . As our model comprises three sets of densely connected variables (namely , and ), we exploit the algorithm of which generalizes to multiple fields. Fig. 4(c) illustrates the inference result for a single frame, overlaid on the corresponding input image. Moreover, we obtain an uncertainty estimate for each pixel/3D point by computing the entropy over the respective marginal distribution. We will use this estimate in Section VI to weigh the evaluation metrics according to the confidence of our label estimates.
Learning: We employ empirical risk minimization in order to learn the parameters in our model, considering the univariate logistic loss, defined as where denotes the marginal distribution at the respective site. Let us subsume all model parameters into . We define our minimization objective as the regularized univariate logistic loss:
Here, is the number of training images, denotes the ground truth semantic label and the approximate marginal at pixel in image , calculated via mean field approximation. is a quadratic regularizer on the parameter vector . We whiten all features and use a single value which we select via cross-validation on the training set. For learning the instance segmentation parameters we exploit the same loss as for semantic segmentation, but assign unique labels to each individual object, e.g., different cars will be assigned different labels even if they occlude each other. In order to associate 2D ground truth instances with 3D instances we project all visible 3D points into the image and find a consensus via the majority vote which gave good results in practice. As the number of instances per semantic class varies between images, we learn intra- and inter-class pairwise potentials using parameter tying. We optimize the objective function using stochastic gradient descent and obtain using auto differentiation. We make use of the ADADELTA algorithm with decay parameter and , and randomly sample a batch of training images at each iteration for which all gradients can be computed in parallel.
V Label Transfer Evaluation
In this section, we first introduce the datasets we use for training and evaluating our label transfer method. Next, we evaluate our method in ablation studies and compare it against several label transfer baselines. Finally, we also show qualitative results of our method.
We manually annotate a set of images with pixel-wise ground truth to train and evaluate our label transfer method. The training set contains images selected from diverse scenarios such that a substantial amount of pixels are labeled within each class. These training images are different from those used in our conference version . We create this new training set following the label definition of CityScapes as considers fewer classes. To enable comparison to 2D label transfer methods which require images with large overlapping regions, we additionally annotate adjacent frames from different suburbs in equidistant steps of frames in the 2D image domain for evaluation. The evaluation set has no spatial overlapping with the training set, allowing us to assess the generalization ability of our method. We evaluate our label transfer method on static and dynamic objects separately. Following , the performance of static objects is evaluated on densely labeled frames from suburbs containing the most frequently occurring classes. The remaining frames are sampled from different suburbs which contain dynamic objects. For these frames we label the dynamic objects while leaving the static region unannotated. We consider common dynamic objects, see Appendix E-A for details.
V-B Quantitative Evaluation
This section presents our quantitative evaluation on semantic and instance segmentation. We compare our method with several label transfer baselines and conduct ablation studies.
For evaluating semantic segmentation transfer performance, we measure overall performance by the mean intersection over union (mIoU) and the average pixel accuracy (Acc). While evaluates the weighted mean IoU which is biased by object occurrences, we follow Cityscapes and measure the mean IoU without weighting. For all experiments, we provide results for individual classes in Appendix E-A.
Baselines: We compare our method to several 2D to 2D label transfer methods on both static and dynamic objects in Table II. Here, the task is to predict the center frame from two annotated images ( frames corresponding to seconds of driving or meters travel distance). Our first baseline (“Label Prop.”) is the label transfer approach presented in . To ensure that all baselines have access to the same information, we do not select frames actively but use equidistantly spaced frames for all methods. We construct a second baseline (“Sparse Track. + GC”) using the feature tracking approach of to propagate semantic labels from the two closest labeled frames to the target frame. To densify the label map, we apply graph cuts (GC) with contrast sensitive edge potentials . In order to evaluate the value of 3D information, we implemented a third baseline (“3D Prop. + GC”) which works similar to the previous one, but replaces the sparse tracking part with correspondences obtained by transferring pixels of the two closest labeled frames to the target image via the visible vertices of our 3D mesh followed by graph cuts propagation.
While all aforementioned baselines require labeled adjacent frames as input at inference time, we consider two more methods that generalize to arbitrary frames. First, we train the segmentation model of Krähenbühl et al. (“Fully Conn. CRF”) which was also used in and which uses a similar inference algorithm as our label transfer method on all annotated adjacent frames of the test sequence. Finally, we evaluate the deep semantic segmentation network (“PSPNet”) that also provides dense unary information for our method. As discussed in Section IV-B, this model is trained on non-occluded sparse 3D projections combined with the CityScapes training set . Note that neither PSPNet nor our method has access to adjacent annotated frames for training or inference.
We further consider several 3D to 2D label transfer baselines that exploit our 3D annotations without requiring equidistantly labeled 2D annotations. Specifically, we project 3D primitives, meshes or visible 3D points into the 2D image domain, followed by graph cut inference (“3D Primitives + GC”; “3D Mesh + GC”; “3D Points + GC”).
Static Objects: Table III (left) shows the comparison on consecutive images of static objects. The results differ slightly from those presented in as 1) we updated the ground truth labels to be consistent with the extended label definition and 2) we measure mIoU following Cityscapes while reports weighted mIoU. From the 2D label transfer baselines shown at the top, the mesh transfer method which uses projected 3D information performs best in terms of mIoU. Furthermore, and maybe surprisingly, the sequence-specific fully connected CRF model performs on par or even better than special purpose label transfer methods. This is caused by the fact that optical flow (as used in ) often fails for street scenes like ours due to large displacements, perspective distortions, textureless regions and challenging lighting conditions. Interestingly, PSPNet achieves the best accuracy while performing worse on mIoU. Despite obtaining superior results on large objects (e.g., “Building”), it struggles with less-occurring classes such as “Trailer” and “Gate”.
The bottom half of Table II (left) compares the proposed method with respect to the 3D to 2D label transfer baselines. As evidenced by our results, simply projecting 3D primitives or meshes into the image and smoothing via GC does not perform well due to the crude approximation of the geometry. Better results are obtained when projecting the visible 3D points followed by spatial propagation. Finally, we observe that all baselines are outperformed by the proposed method (last row). Note that we also map the semantic labels of our 3D annotations to the most common categories considered in the static evaluation images (see Appendix C) for all 3D to 2D label transfer methods.
Dynamic Objects: We evaluate our method on dynamic objects against 2D label transfer baselines. Here, we consider all static regions as a single background class during evaluation. Note that we neglect “Fully Conn. CRF” and “PSPNet” as both methods address semantic segmentation and thus cannot distinguish static and dynamic objects within the same class. Table II (right) shows that our method also outperforms all 2D label transfer baselines on dynamic objects. While the mIoU is calculated over a different set of classes, the average performance of our method on dynamic objects is slightly degraded compared to our result on static objects. Labeling of dynamic objects is more challenging in our annotation pipeline for two reasons: Since we accumulate 3D points of dynamic points according to the annotated bounding primitives over multiple frames, slight misalignments of the primitives may lead to inaccurate accumulation and thus erroneous 3D cues. Furthermore, the accumulation of deformable objects (“Rider”, “Person”) leads to noisy 3D point clouds. Despite these challenges, our method achieves satisfying performance on all dynamic objects.
Annotation Time Comparison: While all 2D methods require every 10th frame to be labeled, our method (as well as the other 3D baselines) requires 3D annotations in the form of 3D primitives. Assuming minutes annotation time per image, this amounts to hours of annotation time per batch of frames when labeling one 2D image every th frame, while the respective 3D annotations for this scene can be obtained in about hours. This gain multiplies with the frame rate and the number of cameras (our setup has four).
Ablation Study: We validate the importance of the individual components of our model on semantic segmentation in Table III (upper left), evaluated on the densely labeled images of static objects. Starting with the appearance classifier trained on the projected sparse 3D points (“LA”), we incrementally add the terms , related to the 3D points (“3D”), the semantic pairwise term between pixels (“PW”), the 3D primitive constraints (“CO”) and finally the 3D pairwise constraints as specified in Eq. IV-B. We note that each component is able to increase performance. We obtain the largest improvement by reasoning about the relationship between points in 3D and pixels in the image.
Label Uncertainty: Here, we leverage our model’s awareness of label uncertainty to demonstrate that higher accuracy can be achieved in confident regions. To quantify uncertainty, we measure the entropy of the label marginal distribution at every pixel, see Fig. 5 (last row). Sorting all pixels according to their entropy allows us to predict the most certain regions in the image. Table III (bottom) shows our results on static objects when predicting only those parts of the image. Note how this helps to boost our performance to mIoU and accuracy when predicting at pixel density, demonstrating that our uncertainty estimates are well calibrated. In contrast, uncertainty is not directly accessible in most baseline models as they are deterministic or rely on MAP estimates. In the benchmarks introduced in Section VI where our inferred labels are considered as pseudo-ground truth, we adopt confidence weighted evaluation metrics leveraging the uncertainty to take into account the ambiguity in our automatically generated annotations.
V-B2 Instance Segmentation Transfer
As time consistent 2D instance ground truth is hard to obtain, most existing 2D label transfer methods focus on the semantic segmentation problem. Therefore, we chose to evaluate instance segmentation performance in an ablation study. We annotated the classes ‘‘Building’’, ‘‘Car’’, ‘‘Trailer’’, ‘‘Caravan’’ and ‘‘Box’’ with instances in our 2D ground truth While uses two sets of parameters for semantic and instance segmentation, we train a single model for instance segmentation and read semantic labels directly from the instance maps. Therefore, our predictions on classes without instance labels are the same in both semantic and instance segmentation maps.. For evaluation, we exploit the mIoU metric defined on instances following . Specifically, we first match the ground truth instances to the predicted instances. A pixel is then classified as true positive only when its predicted instance index matches the ground truth. Table III (right) shows our results. Note how the instance segmentation results are on par with the semantic segmentation, demonstrating our model’s intra-class separation ability. Moreover, we also observe higher instance segmentation accuracy when filtering uncertain predictions.
V-C Qualitative Evaluation
Fig. 5 illustrates our dense inference results qualitatively for 3 different scenes in terms of semantic instance segmentation on both static and dynamic objects. The first two rows illustrate both semantic and instance labels, where semantic information is color-coded and instances are separated by boundaries. The last row shows the confidence maps. While the proposed method is able to delineate most object boundaries satisfyingly, some challenges remain. Errors occur in regions where 3D points are absent due to far distance (1st & 3rd scene: far building). Another source of errors is inherent label ambiguities that occur for porous objects such as fences or trees (3rd scene: tree boundary) where even 2D ground truth annotation is a hard and ambiguous task. Finally, 3D points of dynamic objects are accumulated over multiple frames (2nd & 3rd scene), providing dense but less accurate 3D cues to the CRF model. However, note that our probabilistic inference algorithm is able to successfully identify those uncertain regions as demonstrated in the last row, where far buildings and object boundaries are predicted as less certain compared to other image regions.
VI Dataset & Benchmarks
We apply the proposed label transfer method to all frames captured by perspective cameras, resulting in 278k 2D semantic/instance segmentation maps, 1.0B 3D semantic points and 172.4M 3D instance points. We provide a statistical analysis of the 2D & 3D labels in Appendix F-A. We further deploy an online evaluation server and establish benchmarks on a set of challenging tasks relevant to autonomous driving. For all tasks, we split the data at the batch-level into disjoint training, validation and held-out test sets as specified in Appendix F-B. Specifically, we leverage KITTI-360 to address tasks at the intersection of vision, graphics and robotics which are commonly viewed as relevant towards achieving full autonomy, including tasks within the scope of semantic scene understanding, novel view synthesis and semantic SLAM. We now describe each task and the corresponding evaluation protocol in detail. Furthermore, we introduce initial baselines for each task.
In this section, we establish scene perception benchmarks in both 2D image space and 3D domain. We first implement benchmarks for the traditional tasks of 2D semantic segmentation and 2D instance segmentation on perspective images, using the inferred semantic/instance segmentation maps as pseudo ground-truth. While not the main focus of this work, we establish these standard 2D benchmarks to investigate whether there is a performance gap between methods operating in 2D and 3D. Furthermore, as our label definition is compatible with Cityscapes, this benchmark opens up the possibility for studying domain adaption across datasets in future work. Next, we establish benchmarks in the 3D domain, including bounding box detection and semantic/instance segmentation. Moreover, we consider a semantic scene completion task where the goal is to simultaneously complete the scene and infer corresponding semantic labels given limited observations. This task allows autonomous vehicles to hallucinate future possibilities and thus can benefit downstream tasks, e.g., predictive control.
2D Semantic Segmentation: We train and evaluate 2D segmentation baselines on the densely labeled images in KITTI-360. We consider two well-known methods, Fully Convolutional Neural Network (FCN) and Pyramid Scene Parsing Network (PSPNet) , as a reference. Following Cityscapes , we adopt mean intersection over union (mIoU) at two semantic granularities, i.e., classes and categories, where 19 classes are grouped into 7 coarse-grained categories. To account for label uncertainty, the mIoU is weighted by the confidence of our pseudo-ground truth labels. A formal definition of our metrics and a detailed definition of the classes and categories can be found in Appendix G-A. Table IV(a) shows that, unsurprisingly, PSPNet outperforms the naïve FCN on the test set.
2D Instance Segmentation: We use the established Mask R-CNN framework with different backbones as our baselines, see Table IV(b). We measure the Average Precision (AP) weighted by the label confidence over 10 thresholds, ranging from 0.5 to 0.95 with a step size of 0.05. The mean AP is then calculated over 7 classes that contain instance labels. We also compare mean AP50 given a threshold of 0.5. Both Table IV(b) and Fig. 6 suggest that Mask R-CNN with a deeper backbone leads to better performance. Note that we provide instance segmentation labels of “Buildings” which are not available for other outdoor datasets . This information allows future works to explore scene compositionality in real-world street scenes.
3D Bounding Box Detection: In this benchmark we measure the mean AP over two classes, “Building” and “Car”, since it is particularly challenging for learning-based algorithms to generalize well to other classes with fewer training samples. Following , the mean AP is calculated at a threshold of 0.25 and 0.5, respectively. We consider VoteNet and its simplified version BoxNet as baseline methods. Both methods require 3D point locations as input and output 3D bounding boxes and their semantic labels. Table IV(c) suggests that VoteNet can make reasonable predictions for both building and cars while it fails to predict 3D bounding boxes with high IoU values, see Fig. 7(f).
3D Semantic Segmentation: We establish a 3D semantic segmentation benchmark on the accumulated point clouds, where PointNet and PointNet++ are trained and evaluated as baselines. Both methods take as input point locations and colors to predict a semantic label for each 3D point. Following the 2D semantic segmentation task, we measure mIoU weighted by label confidence over classes and categories, respectively. Table IV(d) shows the quantitative comparison and Fig. 7(d) illustrates the performance of PointNet++. Interestingly, comparing Table IV(a) and Table IV(d) shows that the 3D semantic segmentation baselines’ overall performances are inferior compared to the 2D semantic segmentation methods, suggesting that parsing the semantic meaning of irregularly structured 3D point clouds remains more challenging and requires further work.
3D Instance Segmentation: We evaluate 3D instance segmentation results for “Building” and “Car”. Specifically, we measure the mean AP over a set of thresholds ranging from 0.5 to 0.95 with a step size of 0.05 and AP at a threshold of 0.25 and 0.5. As a first simple baseline, we naïvely cluster semantically labeled points into instances. We use PointNet++ for semantic segmentation and DBSCAN for clustering. We further evaluate PointGroup as a state-of-the-art method for 3D instance segmentation which takes as input point locations and colors. Table IV(e) demonstrates that PointGroup outperforms the naïve clustering-based method. The qualitative result of PointGroup is shown in Fig. 7(e). While the 2D and 3D results in Table IV(b) and Table IV(e) are defined over different sets of classes, we provide detailed results on each class in Appendix G-E. Interestingly, 3D instance segmentation methods achieve better performance on “Car” than 2D methods while performing worse on “Building”. We hypothesize that unlike in 2D where occlusions strongly impact the results (e.g. Fig. 6 left, pedestrian standing in front of a car), cars can be more easily separated in 3D. As for buildings, many instances are spatially connected (e.g., Fig. 6 right), making the instance segmentation task harder in 3D where boundaries are harder to detect on the sparse point cloud.
Semantic Scene Completion: While standard scene perception tasks aim to predict a semantic label for each observed scene point, the semantic scene completion task additionally requires predicting geometry and semantics in unobserved regions. Given a single LiDAR scan as input, this task requires semantic scene completion within a corridor of 30m around the vehicle poses of a 100m trajectory. For evaluation, we measure reconstruction quality and semantic prediction accuracy. The former measures geometric accuracy independent of semantics, using completeness and accuracy over a range of distance thresholds following common practice . We consider a threshold of 20cm as the main metric. As our ground truth reconstruction may not be complete, we evaluate accuracy only in observed regions. We further measure the F1 score as the harmonic mean of the completeness and the accuracy. The semantic prediction quality is conditioned on the geometric reconstruction. Specifically, we measure the confidence weighted mIoU over the same set of thresholds where a true positive prediction is made when 1) a ground truth point is classified as complete at the given threshold and 2) its closest reconstructed point has the correct label. See Appendix G-F for more details of the ground truth construction and evaluation metrics.
We consider two baselines for this task, both taking a single raw LiDAR frame as input. For calibration, we implement a naïve baseline which returns the input as output. The second baseline is a learning-based approach where we use an encoder-decoder architecture to predict the complete scene structure from the raw LiDAR scan. More details about this baseline can be found in Appendix G-F. Table IV(f) and Fig. 8 illustrate the results. As expected, the raw LiDAR scans are accurate but incomplete. The learning-based approach instead achieves higher completeness but the predictions are less accurate. For the learning-based approach we also predict a semantic label at each 3D point. Fig. 8 shows that the model is able to correctly predict semantic labels at a coarse level but struggles to predict smaller objects like cars.
Discussion: Our results show that 3D semantic segmentation is harder than 2D semantic segmentation. In contrast, our conclusions for instance segmentation vary for different classes. Some classes, e.g., cars, are easier to segment in 3D, suggesting further works can explore 3D information to enhance 2D instance segmentation. 3D bounding box detection remains challenging, especially when a high IoU is desired. Lastly, while inferring dense geometry and semantics from raw sparse observations can benefit autonomous driving, completing the scene and predicting semantics jointly is a difficult task that requires further research.
VI-B Novel View Synthesis
Simulation is an essential tool for training and evaluating autonomous vehicles. While existing methods trained in simulated scenes struggle to generalize to real scenes, creating a simulation environment based on real-world images is a promising direction to close the gap between real-world scenarios and synthetic environments . We thus establish challenging benchmarks towards this goal, including novel view appearance synthesis and novel view semantic synthesis.
Novel View Appearance Synthesis: In this benchmark, we are interested in novel view RGB image synthesis for driving scenarios. While we evaluate on a set of held-out perspective images, the benchmark participant can choose from a set of input modalities The used input modalities will be indicated on the leaderboard., including posed perspective/fisheye images or accumulated point clouds. For perspective and fisheye images, we release approximately of the frames for training and use the remaining for testing. In addition, the evaluation server also provides a harder setting with a drop rate. See appendix for details. The point cloud is accumulated over all frames where each point fuses colors from different viewpoints. We adopt three standard evaluation metrics for this benchmark: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and perceptual metric (LPIPS) .
We evaluate two sets of baselines on two different input modalities. We first consider a naïve baseline using the accumulated point cloud (PCL) as input. Specifically, we project non-occluded colored points to test viewpoints, followed by nearest neighbor interpolation to fill in the missing values. As there are no 3D points in the sky, we heuristically assign a mean blue color to the sky region. We further consider several state-of-the-art baselines for image-based novel view synthesis, including methods based on Neural Radiance Fields or per-view depth maps . Results in Table V (left) and Fig. 9 (1st row) reveal the challenges of this benchmark. We observe that NeRF shows promising results but struggles to synthesize fine structure. FVS performs better in rendering fine details (e.g., license plate) but exhibits noticeable artifacts due to the inaccurate underlying geometry (e.g., left car). Interestingly, FVS/PBNR performs better in LPIPS but has a lower PSNR compared to NeRF-based methods, suggesting that LPIPS is more sensitive to fine detail than larger regional errors.
Novel View Semantic Synthesis: An important property of simulation environments like CARLA is that they provide not only RGB images but also auxiliary information like semantic label maps. Towards a real-world simulator with the same capability, we therefore consider a novel benchmark that requires joint novel view and semantic synthesis. The input data for this task is the same as for the novel view synthesis task, while the methods are tasked to predict both an RGB image and a semantic segmentation map at a given target camera pose. Therefore, the evaluation metric of this task additionally comprises mIoU for semantic segmentation as shown in Table V (right). As no prior work has addressed this problem yet, we consider a naïve two-stage solution as baseline to bootstrap this benchmark, i.e., we apply an existing semantic segmentation model (PSPNet ) on the synthesized images. For comparison, we also evaluate the semantic segmentation performance on the original ground truth images (GT Image). Note that the artifacts in the synthesized images lead to a significant performance drop for semantic segmentation. As illustrated in Fig. 9, the fence is misclassified as building when the synthesized images are taken as input to PSPNet, despite that the fence is still visible in these images. It is also interesting to note that semantic segmentation performance is aligned with the LPIPS metric, as both apply pre-trained networks on synthesized images.
Discussion: Our baselines reveal the different challenges in novel view appearance synthesis with different input modalities. While point clouds provide a good representation of 3D geometry, it is not easy to model view dependency. When instead taking a sparse set of multi-view images as input, the task is similarly difficult despite little variation in the camera orientation. We believe that future works should explore the combination of different input modalities to improve image fidelity further. Moreover, given the low performance of our simple baselines on the novel view semantic synthesis task, there is a large potential for future improvements, i.e., by learning view and semantic synthesis jointly.
VI-C Semantic SLAM
We further establish a semantic SLAM benchmark at the intersection of robotics and computer vision. Here, the goal is to simultaneously estimate poses and reconstruct a semantic map from monocular/stereo images and/or LiDAR scans. While there is a growing interest in evaluating indoor semantic reconstructions of SLAM algorithms at room-level , existing works on outdoor semantic SLAM typically evaluate only pose estimation while ignoring the quality of the semantic reconstruction . Considering that the semantic reconstruction is valuable on its own for down-stream tasks, e.g. planning , we thus additionally evaluate geometric and semantic reconstructions where the latter is enabled by the dense semantic annotations of KITTI-360. For this benchmark, the test sequences are separated from those used for 3D scene perception such that the accumulated point cloud is held out from the public.
Localization: Given an estimated trajectory, we adopt the standard Absolute Pose Error (APE) and Relative Pose Error (RPE) as metrics for evaluating pose estimation. We consider four test sequences for this task and report the evaluation results on each test sequence without averaging.
We evaluate two baseline methods, ORB-SLAM2 and SUMA++ , where the former takes stereo images as input and the latter is applied on LiDAR scans. Table VI(a) compares the localization results of ORB-SLAM2 and SUMA++. For both methods, the APE exceeds 2 meters and the RPE is around 2% in general. ORB-SLAM2 achieves better overall performance compared to SUMA++, suggesting that the stereo images of our dataset contain rich features for the purpose of localization. One possibility for improving localization accuracy is to exploit the 3D bounding boxes and instance labels available in our dataset .
Geometric and Semantic Mapping: We measure the quality of the geometric and semantic mapping using the same metrics considered in the semantic scene completion benchmark. As richer input observations are available in this task, we adopt a smaller distance threshold of 10cm as the main metric. Specifically, we first measure geometry accuracy using completeness and accuracy, and then evaluate semantics on the completed ground truth points via the confidence weighted mIoU. As the mapping accuracy is highly correlated with the APE, we compare ground truth and estimated reconstruction in local windows to minimize the impact of pose drifts. Each local window consists of consecutive frames and is aligned to the ground truth based on the trajectory, see Appendix I-B for more details.
We use the same baselines considered in the localization benchmark. As ORB-SLAM2 does not provide dense reconstruction nor semantic information, we obtain dense semantic reconstruction by unprojecting 2D semantic segmentations (PSPNet ) using depth maps from semi-global matching (SGM) . SUMA++ aims for semantic SLAM and estimates poses and a semantic surfel map from LiDAR scans. We experimentally observe that it is sufficient to take the center of the surfels as the reconstructed points.
Table VI(b) and Fig. 10 show the reconstruction and semantic prediction results. We observed that both baselines produce good reconstructions on the ground region. For regions above the ground, SUMA++ is less complete as it only uses LiDAR scans and thus the maximum height is limited. ORB-SLAM2 + SGM results in higher completeness but worse accuracy. In terms of semantic predictions, SUMA++ produces reasonable results on the LiDAR scans but struggles to achieve good overall performance due to its low completeness. In contrast, ORB-SLAM2 + PSPNet contains more flying points due to the outliers of stereo matching (e.g., sky points colored blue). Exploring semantic information to remove sky points may further improve the performance of this baseline.
Discussion: We evaluate localization accuracy of existing SLAM methods and suggest exploring 3D instance-level information in further works. Further, reconstructing accurate geometry and semantics remains a challenging task. Our benchmark allows to investigate important questions towards solving this challenging task, e.g., which input modality is better suited for this task, whether semantic prediction and geometric reconstruction can benefit each other and if joint optimization is desirable.
VII Conclusion
We present KITTI-360, a large scale 3D video dataset comprising 300k images and laser point clouds with consistent semantics in both 2D and 3D. We create a WebGL-based annotation tool and annotate both static and dynamic objects in 3D. We propose a method to obtain dense semantic instance labels from annotated 3D primitives. In the presence of 3D data, our method yields better results compared to several 2D label transfer baselines while lowering the annotation time.
Furthermore, we establish novel online benchmarks for several challenging tasks at the intersection of computer vision, graphics and robotics. We evaluate several baselines for each benchmark. Our results show that existing methods achieve satisfactory results on well-established benchmarks, e.g., 2D/3D segmentation, where inference is directly performed on given observations. However, it is much harder to solve tasks that require jointly recovering the geometry, appearance and estimating the semantics as in the newly introduced tasks for semantic scene completion, novel view appearance/semantic synthesis and semantic SLAM. We hope that our dataset, online benchmarks and annotation tools will fertilize new research across communities, fostering progress towards the grand goal of full autonomy.
Acknowledgments
The authors thank Siyuan Peng, Bernhard Jaeger, Shrisha Bharadwaj, Apratim Bhattacharyya, Paul Henderson, and Zehao Yu for their help in implementing the baselines, Kashyap Chitta, Katja Schwarz, and Yue Wang for proofreading, and SurfingTech for annotating parts of the dataset. Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645. Yiyi Liao was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A.
References
Appendix A Annotation Data Preparation
To facilitate annotation, we accumulate all laser measurements in a common world coordinate system and augment them with 3D points from stereo matching . To reduce outliers of stereo matching, we consider only points up to m distance, and apply left-right as well as forward-backward consistency checks over 5 frames. We fuse all 3D points sequentially and ignore a point closer than cm to its nearest neighbor in the fused point cloud. This downsampling operation allows for reducing the data loading traffic and memory of the web based annotation tool.
A-B Dense Point Cloud on Dynamic Objects
Our simple dynamic object detection consists of two steps. In the first step, we apply volumetric fusion over a sequential of laser scans and search for 3D points located in (mostly) free regions. As a dynamic object always moves in the free space, the voxels along its moving trajectory are occupied occasionally and thus the expected status over time should be free. Hence the dynamic points can be detected by finding points inside of all “free” voxels. Specifically, we build a 3D occupancy volumetric grid for each batch by fusing Velodyne observations using Octomap . Given a set of measurements from frame to , the occupancy probability of each voxel is updated as follow:
where is the log-odds of the probability, and are the upper bound and lower bound of the log-odds respectively. Note that we clamp a voxel to be free if there is sufficient evidence from previous frames to support its free status. We denote the set of free voxels as .
Due to the noise in measurements and poses, the free voxels may also contain many 3D points of static objects as shown in Fig. 11(a). It is hard to avoid these false positive detections as each voxel is classified as free or occupied independently. Therefore, we consider a second step to filter out clusterings of noisy detections. Specifically, we segment the original accumulated point cloud into a branch of clusters using the Region Growing algorithm https://pcl.readthedocs.io/projects/tutorials/en/latest/region˙growing˙segmentation.html, and we calculate the occupancy probability of each cluster based on the detection in the first step:
where denotes a single point in the cluster, denotes number of points, and means that is spatially located within . A cluster is considered as dynamic if is larger than a given threshold. With the second step, we are able to filter out false positive detections as shown in Fig. 11(b). It is acceptable if a few false positive detections remains since the dense point cloud will be further labeled by our annotators.
Appendix B Annotation Interface
In this section, we demonstrate the annotation tool and process in detail.
Color Coding: Fig. 12 shows different color codings of 3D points that we provide in the annotation tool. Annotators can choose different color codings accordingly. For example, Fig. 12(c) helps annotators to identify the boundary between “Road” and “Sidewalk” and Fig. 12(d) allows annotators to check unannotated region (shown as white).
3D Viewpoint: To assist annotators to better visualize the scene, we also provide different viewports, namely, normal and orthographic viewports as shown in Fig. 13. The orthographic viewport helps annotators accurately identify stuff classes’ boundaries and annotate individual objects efficiently (see “fast object annotation mode” in Section B-B). Besides, annotators can also adjust point size and brightness to work with different levels of detail.
2D Camera View: For better perceiving the scene, we also show fisheye and perspective images as well as the pose of each camera, enabling annotators to select informative viewpoints efficiently, as shown in Fig. 14.
B-B Annotation Functions
We provide a few shortcuts and functions to facilitate the annotation process.
Bounding Primitive Manipulation: To best enclose the 3D object, each bounding primitive can be manipulated with translation, scaling, and rotation, resulting in degrees of freedom. In addition, annotators are also asked to assign an orientation to each bounding primitive to understand objects’ (especially instances) orientation. See Fig. 15 for each operation’s shortcuts.
Bounding Primitive Copy: We also provide a “copy” shortcut to allow annotators to quickly insert new annotations with the same label and similar pose to previously annotated objects. This is especially useful for objects appearing frequently with similar sizes such as building and car.
Fast Object Annotation: We support quickly annotating a subset of object classes by simply drawing a line along the object. This fast annotation mode is enabled under the orthographic view for a few pre-selected classes, where the annotator draws a line along the longest side in the middle. While this line only specifies the length of the object in one dimension and its orientation, we heuristically place a bigger bounding primitive centered at this line and iteratively shrink the bounding primitive until it touches any non-ground points. As shown in Fig. 16, this simple technique allows for efficient and accurate annotation.
Object-Centric Mode: To enable a clear observation of a single object from the accumulated point cloud, we also help annotators with the “object-centric” mode. With a single 3D bounding box selected, triggering the “object-centric” mode hides all the other bounding primitives as well as points far from the selected primitive. In addition, both front view and side views images are automatically switched to the ones in which the selected primitive is most visible.
Completeness Check: As illustrated in Fig. 12(d), the annotator can check the completeness level of the annotation by visualizing the point cloud based on existing bounding primitives. Specifically, we color each 3D point if it is enclosed by a bounding primitive and leave the unlabeled region as white. This helps the annotator easily identify any unlabeled 3D points.
B-C Ground Annotation
Ground bounding primitives are simply annotated as 2D polygons. The extruded height of the ground polygon is automatically determined as follows: for each vertex on the polygon, we first search the nearest camera of this given vertex, and assign the height of the camera to this vertex as its initial height . Then we search nearest neighbors of point in the 3D point cloud, and update height as the median height of these nearest neighbors. See Fig. 17 as an example for “Road” annotation. Annotators can also modify the 2D polygon anytime by dragging its control points.
B-D Dynamic Object Annotation
We implement a semi-automatic annotation scheme to label dynamic objects efficiently. Our semi-automatic annotation relies on two assumptions: the size of the dynamic object is fixed over time, and its trajectory is smooth. Therefore, the required annotation is reduced to adding posed 3D primitives at several keyframes. Our annotation tool then automatically places the remaining primitives along the trajectory. The smooth trajectory is obtained via spline interpolation based on the primitives at the keyframes. We annotate articulated dynamic objects, e.g., pedestrians, using the maximum extent bounding box.
As the speed of the dynamic object may not be constant, we place the remaining primitives based on the observed 3D points at each timestamp. Specifically, we first discretize the annotated 3D primitives into voxels and fuse the occupancy status of each voxel over all annotated primitives as shown in Fig. 18(a). A voxel is considered as occupied if it is occupied in any of the annotated 3D primitives, otherwise it is free. This fused occupancy status is considered an “occupancy template” for searching matching 3D primitive along the trajectory. Given a timestamp between the first and the last annotated timestamp, we slide the 3D primitive on the interpolated spline and calculate the occupancy status based on 3D points collected at the given timestamp. We choose the pose that provides the maximum overlap with the occupancy template, see Fig. 18(b).
To facilitate users’ interaction, we plot the interpolated spline in the annotation tool and allow the annotator to refine the spline by simply adjusting the 3D bounding primitives inserted at the keyframes. We also display the automatically generated 3D primitives to help the user check if they are accurate. The poses of each automatically generated bounding primitive can also be adjusted if necessary. Typically, it requires annotated keyframes to produce accurate annotations for the full moving trajectory. Fig. 19 illustrates the dynamic annotation at a given timestamp.
Appendix C Label Definition
Table VII shows the definition of the classes that we use for annotating 3D scenes. We adhere to the definition of Cityscapes as close as possible while a few inconsistent label definitions are inevitable as our annotations are performed in 3D. For example, the “Traffic Sign” in Cityscapes only includes the front side while we consider both the front and the back. We do not distinguish the back as each traffic sign is labeled by a single 3D bounding primitive and it might be observed in 2D from both sides. Thus, each traffic sign has a consistent label regardless of which side it is observed.
Appendix D More Details of Label Transfer Inference
In contrast to the point cloud accumulation for annotation as introduced in Appendix A, here we need to distinguish static and dynamic points for label transfer inference and thus determine visible points on every frame. Specifically, we consider a point static if it is not enclosed by any dynamic bounding primitive and accumulate all static points first. For each dynamic object, we retrieve all points inside the corresponding bounding primitives for every timestamp and accumulate them in the canonical object-centered coordinate system by taking the inverse transform of the object pose defined by (world-to-object transformation). Next, we insert the accumulated dynamic point clouds back to the world coordinate following the object pose (object-to-world transformation). This allows us to obtain dense 3D points during inference for both static and dynamic regions.
D-B Accumulation of Inferred 3D Label
Our inference is performed individually on each frame defined over the corresponding 2D pixels and visible 3D points. To obtain 3D labels on the accumulated point clouds, we thus fuse 3D labels obtained from each frame. Specifically, if a 3D point is visible in multiple frames, we take the majority of its inferred classes as its final label. The confidence of this 3D point is also averaged over confidence values of these points of the majority label. If a 3D point is not visible in any of the frames but is uniquely labeled by a single class, we assign this unique label to the 3D point and a confidence value of . In the remaining cases, we treat the 3D point’s label as unknown.
D-C Pixel Unary Potentials of Ground Objects
The first term of the pixel unary potential is a binary feature which indicates admissible labels. For non-planar object classes, is obtained based on the projection of 3D primitives, whereas planar object classes directly use projections of 2D polygons to obtain more accurate boundaries. As introduced in Appendix B-C, the ground bounding primitives are extruded to 3D to enclose the 3D points. This leads to oversized 2D projections, making it hard to determine the boundary between two adjacent ground object annotations (e.g., “Road” and “Sidewalk”). As opposed to our conference version , which exploits a geometric unary potential to address this problem, we instead directly project the 2D polygons of the ground objects before extruding them to 3D. This allows us to obtain accurate ground object boundaries from and avoid the complexity introduced by the additional geometric unary term.
D-D Instance Augmentation of Pixel Unary Potentials
We adopt a state-of-the-art panoptic segmentation method UPSNet pre-trained on Cityscapes to obtain instance hypotheses for “Car”, “Truck”, and “Pedestrian”. We first run UPSNet on our images to get probability maps of all instances. For each annotated instance within the aforementioned classes, we retrieve matched instances from the predictions of UPSNet. More specifically, given a set of 3D points annotated with one instance (e.g., one car) and a probability map of one instance predicted by UPSNet, we consider they match if more than of the 3D points fall into the high-probability region of the predicted instance. This allows for improving instance boundaries as shown in Fig. 20.
Appendix E More Results of Label Transfer Inference
Here, we show detailed quantitative comparisons for individual classes. Table VIII and Table IX show quantitative comparison to label transfer baselines on static and dynamic objects, respectively. We evaluate the intersection over union (IoU) of each class where the mIoU is the average over all classes. We further show detailed ablation study in Table X and Table XI for semantic and instance label transfer.
E-B Qualitative Comparison to Baselines
We compare our method qualitatively to several 2D-to-2D and 3D-to-2D label transfer baselines in Fig. 21. Note how the 2D-to-2D label transfer baselines fail in the presence of strong occlusions and large displacements.
E-C Qualitative Comparison of Ablation Study
Fig. 22 compares different variants of our label transfer model. Consistent with the quantitative analysis, our full model achieves the best performance.
Appendix F Dataset
Fig. 23 shows the distribution of the semantic labels in KITTI-360. Fig. 23(a) and Fig. 23(b) suggests that the semantic distribution of the 2D pixels and the 3D points are similar (except for the “Sky” class). We also show the distribution of our 3D bounding boxes in Fig. 23(c).
F-B Dataset Split
We split KITTI-360 into training and test sets without spatial overlapping as shown in Fig. 24. We maintain an online evaluation server and hold back the labels of the test set. Considering that different tasks involve different label modalities, the test set is further divided into two parts with different information released. Specifically, the first part of the test dataset is used for semantic scene understanding (except for semantic scene completion) and novel view synthesis, where we hold back the 2D semantic/instance segmentation maps and 3D pointwise labels. Note that the accumulated point clouds are released while their labels are removed. The other part is adopted for semantic scene completion and semantic SLAM where the accumulated point clouds are further removed. We release vehicle poses for both test sets.
Appendix G Semantic Scene Understanding Benchmark
We evaluate confidence weighted mIoU where both the intersection and the union are weighted (per-pixel) by the confidence of our pseudo-ground truth. More formally, let and denote the set of image pixels in the intersection and the union of one class label (or one category label), respectively. The weighted IoU of this class can be defined as follow:
where denotes the confidence value at pixel . In the standard evaluation for all pixels. The mIoU is then calculated as the mean of the weighted IoU over all class labels or category labels.
While we provide 19 classes for training following Cityscapes , we omit two classes, “Train” and “Bus” during evaluation since these two classes are rarely observed in the test region when we split the training and test sets according to the camera poses as shown in Fig. 24.
G-A2 Baselines
We train and evaluate two well-known methods, Fully Convolutional Neural Network (FCN) and Pyramid Scene Parsing Network (PSPNet) . For FCN, we adopt the ResNet-101 model provided by PyTorch as a backbone. The model is pre-trained on a subset of the Microsoft COCO dataset. As for PSPNet, we use the official PyTorch implementation https://github.com/hszhao/semseg which also uses ResNet-101 as backbone. The model is pre-trained on the ImageNet dataset.
G-A3 Additional Results
We show the IoU of each class in Table XII(a). We observe that PSPNet consistently outperforms FCN in most of the classes.
G-B Benchmark of 2D Instance Segmentation
Following , we measure the Average Precision (AP) over 10 IoU thresholds, ranging from 0.5 to 0.95 with a step size of 0.05. We calculate confidence weighted IoU per instance using Eq. 10. In this task, we consider 7 classes that contain instance labels, including “Building”, “Person”, “Rider”, “Car”, “Truck”, “Motorcycle” and “Bicycle”.
G-B2 Baselines
We evaluated two Mask R-CNN models with different backbones, i.e., ResNet-50 and ResNet-101, based on the official implementation https://github.com/facebookresearch/detectron2. Both backbones are pre-trained on the ImageNet dataset.
G-B3 Additional Results
Table XII(b) shows the AP of each individual class as well as the mean AP. We observe that performance of different backbones is similar in more frequently observed classes (e.g., “Building” and “Car”) while differs in less occurred classes.
G-C Benchmark of 3D Bounding Box Detection
We evaluate AP at a threshold of 0.5 and 0.25 for 3D bounding box detection. As it is particularly challenging for learning-based algorithms to generalize well to other classes with fewer training samples, we measure the mean AP over two classes: “Building” and “Car”.
G-C2 Baselines
We evaluate the state-of-the-art 3D bounding box detection method, VoteNet , and its simplified version, BoxNet as baselines. We adopt the official implementation https://github.com/facebookresearch/votenet for both methods.
G-C3 Additional Results
Table XII(c) shows the AP on each class as well as the mean AP. Both methods achieve reasonable performance at the IoU threshold of 0.25 while struggle at the higher threshold.
G-D Benchmark of 3D Semantic Segmentation
For 3D semantic segmentation, we also evaluate confidence weighted mIoU using Eq. 10. Here, the confidence of each 3D point is obtained by averaging the confidence of 3D points on multiple frames, as introduced in Appendix D-B. Similar to 2D semantic segmentation, we omit “Train” and “Bus” during evaluation. Note that there is no “Sky” point in 3D, thus it is also discarded. Moreover, since we train and evaluate only in static regions, we ignore “Rider” as it only appears as a dynamic object.
G-D2 Baselines
We train and evaluate two baselines, PointNet and PointNet++ . As the original implementations are built on Tensorflow, we adopt a faithful Pytorch reimplementation https://github.com/yanx27/Pointnet˙Pointnet2˙pytorch that contains both methods.
G-D3 Additional Results
Table XII(d) shows the detailed results of 3D semantic segmentation. As expected, PointNet++ achieves better performance on all classes compared to PointNet.
G-E Benchmark of 3D Instance Segmentation
In 3D instance segmentation, we also evaluate AP over 10 IoU thresholds, ranging from 0.5 to 0.95 with a step size of 0.05. The IoU of each 3D instance is weighted (per-point) by the confidence of our pseudo-ground truth. Here, we evaluate on “Building” and “Car” the same as the 3D box bounding detection benchmark.
G-E2 Baselines
We first consider a naïve baseline based on the results we obtained from 3D semantic segmentation using PointNet++ . Specifically, we first extract points of the same class label (“Building” or “Car”) based on the semantic segmentation results. Next, we group the extracted point cloud using DBSCAN http://www.open3d.org/docs/0.12.0/python˙api/open3d.geometry.PointCloud.html, where clusters with less than 500 points are ignored. Each valid cluster is then considered as an instance with a confidence score of . The second baseline is a state-of-the-art approach, PointGroup . We follow the official implementation https://github.com/dvlab-research/PointGroup to train and evaluate this baseline.
G-E3 Additional Results
Table XII(e) shows the detailed results of 3D instance segmentation. Interestingly, the 3D instance segmentation performance of “Car” is higher than the 2D baselines in Table XII(b). We hypothesize that unlike in 2D where occlusions strongly impact the results, cars can be more easily separated in 3D.
G-F Benchmark of Semantic Scene Completion
The ground truth of the semantic scene completion task is the accumulated point cloud within a corridor of 30m around the vehicle poses of a 100m trajectory (50m in each direction), see Fig. 25 for an illustration. The input to this task is a single LiDAR scan whose center is visualized by the blue star point. We first determine a set of neighboring vehicle poses close to the given center illustrated in Fig. 25(a), and then crop the accumulated point cloud using the union of circles located at those poses as shown in Fig. 25(b). To avoid evaluating in significantly occluded regions that typically occur when the vehicle turns a large angle, we also check the orientation of each pose as shown in Fig. 25(a). Specifically, if the forward direction of one pose deviates more than compared to the heading angle of the given center, it is eliminated from the neighboring poses.
G-F2 Evaluation Metric
In this task we evaluate geometric completion and semantic estimation, respectively. Geometric completion is evaluated via completeness and accuracy at a threshold of 20cm. Completeness is calculated as the fraction of ground truth points of which the distances to their closest reconstructed points are below the threshold. Accuracy instead measures the percentage of reconstructed points that are within a distance threshold to the ground truth points. As our ground truth reconstruction may not be complete, we prevent punishing reconstructed points by dividing the space into observed and unobserved regions, which are determined by the unobserved volume from a 3D occupancy map obtained using OctoMap . A reconstructed point is only evaluated when it falls into the observed region within the union of the neighboring circles shown in Fig. 25(b). We further measure the F1 score as the harmonic mean of the completeness and the accuracy. Note that SemanticKITTI also considers a semantic scene completion task, but considers voxel as representation and measures mIoU over voxels for both reconstruction and semantics. We instead avoid discretization and directly evaluate on point clouds using standard metrics to separately assess accuracy and completeness.
G-F3 Baselines
We implement two baselines for this task. For calibration, the first baseline returns the input LiDAR scan as output. The second baseline is a learning-based approach that adopts an encoder-decoder structure. Specifically, the encoder first learns features from the input point cloud. It then merges the point-wise features to voxels such that a 3D U-Net is applied to predict a volumetric reconstruction. The network is trained using a cross-entropy loss where the ground truth point cloud is also discretized into a volume. As our evaluation server requires submission in the form of point clouds, we uniformly and densely sample points from each occupied voxel as the final output.
G-F4 Additional Results
Table XII(f) shows detailed semantic estimation performance of the learning-based baseline. As can be seen, it is challenging to predict the geometry and the semantics jointly. The overall performance of this baseline is worse compared to baselines that directly perform 3D semantic segmentation in Table XII(d).
Appendix H Novel View Synthesis Benchmark
We adopt three standard metrics to evaluate novel view appearance synthesis: peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM), and perceptual metric (LPIPS) .
H-A2 Data Preparation
We select static scenes with a driving distance of meters each for evaluating NVS at a drop rate. We select one frame every 0.8 meters driving distance (corresponding to the overall average distance between frames) to avoid redundancy when the vehicle is slow. We release of the frames for training and retain for evaluation. Moreover, we select static scenes with a driving distance of meters each for evaluating NVS at a drop rate. On average, we select one frame every 4 meters driving distance in this setting. We release of the frames for training and retain for evaluation.
H-A3 Baselines
We evaluate two sets of baselines for this task. The first baseline (PCL) takes a colored point cloud as input. We project non-occluded points to the test viewpoint and interpolate the missing values to obtain the full image. To determine non-occluded points, we reconstruct a mesh using the ball-pivoting method on the accumulated point cloud. As there is no point in the sky region, we in-paint the sky using a constant blue color. The sky region is heuristically determined based on the projected 3D points, i.e., a large connected area in the upper half of the image without any 3D projections is considered as the sky.
The second set of baselines takes a set of images as input. For all NeRF-based methods , we train one model on each scene individually, using cascaded sampling with coarse samples and fine samples. We adopt the PyTorch reimplementation of NeRF https://github.com/yenchenlin/nerf-pytorch, the original implementation of mip-NeRF https://github.com/google/mipnerf and DS-NeRF https://github.com/dunbar12138/DSNeRF. As for Free View Synthesis (FVS) , we follow its original implementation https://github.com/isl-org/FreeViewSynthesis and use their released model trained on the Tanks and Temples dataset which generalizes well. We follow the original implementation https://gitlab.inria.fr/sibr/projects/pointbased˙neural˙rendering of PBNR that optimizes a set of attributes such as reprojected features or depth in each input view.
H-A4 Additional Results
We show additional qualitative results of all methods in Fig. 26 (left). The PCL baseline exhibits blocky artifacts due to interpolation. The vanilla NeRF shows promising performance but sometimes struggles due to the sparse input views. While mip-NeRF and DS-NeRF both improve the performance, the thin structures (e.g., fence) are still not well recovered. Interestingly, FVS and PBNR are better at preserving the fine details (e.g., license plate) but have lower PSNR. This could be due to small misalignments in the image space.
H-B Benchmark of Novel View Semantic Synthesis
We evaluate the confidence weighted mIoU using Eq. 10. Similar to the 2D semantic segmentation task, we omit “Train” and “Bus” during evaluation. We additionally omit “Truck”, “Person”’, “Rider”, “Bicycle” and “Traffic Light” as these classes do not appear in the static scenes for evaluating NVS.
H-B2 Baselines
As there is no existing research work on this new benchmark, we directly apply PSPNet used in the 2D semantic segmentation task to synthesized images for semantic label prediction.
H-B3 Additional Results
We show confidence weighted IoU on individual classes in Table XIII. Note that this naïve baseline leads to significantly degraded performance on most of the classes. As shown in Fig. 26 (right), small changes in the image space sometimes lead to sigficant changes in the semantic prediction.
Appendix I Semantic SLAM Benchmark
We adopt the standard Absolute Pose Error (APE) and Relative Pose Error (RPE) as metrics for evaluating pose estimation. We align the predicted trajectory to the ground truth using a rigid transformation to evaluate the APE . The RPE is evaluated between two frames with a distance of 1 meter.
I-A2 Baselines
We evaluate ORB-SLAM2 https://github.com/raulmur/ORB˙SLAM2 and SUMA++ https://github.com/PRBonn/semantic˙suma using their official implementations as baselines.
I-A3 Additional Results
Fig. 27 shows qualitative comparison of predicted trajectories. As can be seen, both methods achieve reasonable performance while SUMA++ has a larger maximum error than ORB-SLAM2.
I-B Geometric & Semantic Mapping
We adopt the same evaluation metrics considered in the semantic scene completion benchmark, as introduced in Appendix G-F. When evaluating the quality of reconstruction, we compare ground truth and estimated reconstruction in local windows to minimize the impact of pose drifts. Specifically, we divide the test sequences into a set of local windows, each consisting of consecutive frames. We first crop the ground truth and the reconstructed point cloud wrt. the region of interest of each window. These two local point clouds are then aligned using the similarity transformation between the corresponding poses and compared afterwards. Finally, we average the completeness, accuracy, and mIoU metrics over the entire test sequence. Following , we measure completeness and accuracy over discretized voxels such that these metrics are insensitive to the density of the point clouds.
I-B2 Baselines
To obtain dense semantic reconstruction given the localization results of ORB-SLAM2, we unproject 2D semantic segmentation maps obtained from PSPNet using depth maps estimated by semi-global matching (SGM) . We merge the unprojected 3D points using the poses predicted by ORB-SLAM2. As for SUMA++, we use the semantic estimation model pre-trained on KITTI.
I-B3 Additional Results
We show additional qualitative results of the geometric mapping in Fig. 28 in terms of completeness and accuracy at the threshold of 10cm. Consistent with the quantitative results, SUMA++ is more accurate while ORB-SLAM2+SGM is more complete.