OASIS: A Large-Scale Dataset for Single Image 3D in the Wild

Introduction

Single-view 3D is the task of recovering 3D properties such as depth and surface normals from a single RGB image. It is a core computer vision problem of critical importance. 3D scene interpretation is a foundation for understanding events and planning actions. 3D shape representation is crucial for making object recognition robust against changes in viewpoint, pose, and illumination. 3D from a single image is especially important due to the ubiquity of monocular images and videos. Even with a stereo camera with which 3D can be reconstructed by triangulating matching pixels from different views, monocular 3D cues are still necessary in textureless or specular regions where it is difficult to reliably match pixel values.

Single-image 3D is challenging. Unlike multiview 3D, it is ill-posed and resists tractable analytical formulation except in the most simplistic settings. As a result, data-driven approaches have shown greater promise, as evidenced by a plethora of works that train deep networks to map an RGB image to depth, surface normals, or 3D models . However, despite substantial progress, the best systems today still struggle with handling scenes “in the wild”— arbitrary scenes that a camera may encounter in the real world. As prior work has shown , state-of-art systems often give erroneous results when presented with unfamiliar scenes with novel shapes or layouts.

We hypothesize that a major obstacle of single-image 3D is data. Unlike object recognition, whose progress has been propelled by datasets like ImageNet covering diverse object categories with high-quality labels, single-image 3D has lacked an ImageNet equivalent that covers diverse scenes with high-quality 3D ground truth. Existing datasets are restricted to either a narrow range of scenes or simplistic annotations such as sparse relative depth pairs or surface normals .

In this paper we introduce Open Annotations of Single-Image Surfaces (OASIS), a large-scale dataset for single-image 3D in the wild. It consists of human annotations that enable pixel-wise reconstruction of 3D surfaces for 140,000 randomly sampled Internet images. Fig. LABEL:fig:teaser shows the human annotations of example images along with the reconstructed surfaces.

A key feature of OASIS is its rich annotations of human 3D perception. Six types of 3D properties are annotated for each image: occlusion boundary (depth discontinuity), fold boundary (normal discontinuity), surface normal, relative depth, relative normal (orthogonal, parallel, or neither), and planarity (planar or not). These annotations together enable a reconstruction of pixelwise depth.

To construct OASIS, we created a UI for interactive 3D annotation. The UI allows a crowd worker to annotate the aforementioned 3D properties. It also provides a live, rotatable rendering of the resulting 3D surface reconstruction to help the crowd worker fine-tune their annotations.

It is worth noting that 140,000 images may not seem very large compared to millions of images in datasets like ImageNet. But the number of images can be a misleading metric. For OASIS, annotating one image takes 305 seconds on average. In contrast, verifying a single image-level label takes no more than a few seconds. Thus in terms of the total amount of human time, OASIS is already comparable to millions of image-level labels.

OASIS opens up new research opportunities on a wide range of single-image 3D tasks—depth estimation, surface normal estimation, boundary detection, and instance segmentation of planes—by providing in-the-wild ground truths either for the first time, or at a much larger scale than prior work. For depth estimation and surface normals, pixelwise ground truth is available for images in the wild for the first time—prior data in the wild provide only sparse annotations . For the detection of occlusion boundaries and folds, OASIS provides annotations at a scale 700 times larger than prior work—existing datasets have annotations for only about 200 images. For instance segmentation of planes, ground truth annotation is available for images in the wild for the first time.

To facilitate future research, we provide extensive statistics of the annotations in OASIS, and train and evaluate leading deep learning models on a variety of single-image tasks. Experiments show that there is a large room for performance improvement, pointing to ample research opportunities for designing new learning algorithms for single-image 3D. We expect OASIS to serve as a useful resource for 3D vision research.

Related Work

3D Ground Truth from Depth-Sensors and Computer Graphics Major 3D datatsets are either collected by sensors or synthesized with Computer Graphics . But due to the limitations of depth sensors and the lack of varied 3D assets for rendering, the diversity of scenes is quite limited. For example, sensor-based ground truth is mostly for indoor or driving scenes .

3D Ground Truth from Multiview Reconstruction Single-image 3D training data can also be obtained by applying classical Structure-from-Motion (SfM) algorithms on Internet images or videos . However, classical SfM algorithms have many well known failure modes including scenes with moving objects and scenes with specular or textureless surfaces. In contrast, humans can annotate all types of scenes.

3D Ground Truth from Human Annotations Our work is connected to many previous works that crowdsource 3D annotations of Internet images. For example, prior work has crowdsourced annotations of relative depth and surface normals at sparse locations of an image (a single pair of relative depth and a single normal per image). Prior work has also aligned pre-existing 3D models to images . However, this approach has a drawback that not every shape can be perfectly aligned with available 3D models, whereas our approach can handle arbitrary geometry.

Our work is related to that of Karsch et al. , who reconstruct pixelwise depth from human annotations of boundaries, with the aid of a shape-from-shading algorithm . Our approach is different in that we annotate not only boundaries but also surface normals, planarity, and relative normals, and our reconstruction method does not rely on automatic shape from shading, which is still unsolved and has many failure modes.

One of our inspirations is LabelMe3D , which annotated 3D planes attached to a common ground plane. Another is OpenSurfaces , which also annotated 3D planes. We differ from LabelMe3D and OpenSurfaces in that our annotations recover not only planes but also curved surfaces. Our dataset is also much larger, being $600\times$ the size of LabelMe3D and $5\times$ of OpenSurfaces in terms of the number of images annotated. It is also more diverse, because LabelMe3D and OpenSurface include only city or indoor scenes.

Crowdsourcing Human Annotations

We use random keywords to query and download Creative Commons Flickr images with a known focal length (extracted from the EXIF data). Each image is presented to a crowd worker for annotation through a custom UI as shown in Fig. 1 (a). The worker is asked to mask out a region that she wishes to work on with a polygon of her choice, with the requirement that the polygon covers a pair of randomly pre-selected locations. She then works on the annotations and iteratively monitors the generated mesh (detailed in Sec 4) from an interactive preview window (Fig. 1 (a)).

Occlusion Boundary and Fold An occlusion boundary denotes locations of depth discontinuity, where the surface on one side is physically disconnected from the surface on the other side. When it is drawn, the worker also specifies which side of the occlusion is closer to the viewer, i.e. depth order of the surfaces on both sides of the occlusion. Workers need to distinguish between two kinds of occlusion boundaries. Smooth occlusion (green in Fig 1 (a)) is where the the closer surface smoothly curves away from the viewer, and the surface normals should be orthogonal to the occlusion line and parallel to the image plane, and pointing toward the further side. Sharp occlusion (red in Fig 1 (a)) has none of these constraints. On the other hand, fold denotes locations of surface normal discontinuity, where the surface geometry changes abruptly, but the surfaces on the two sides of the fold are still physically attached to each other (orange in Fig 1 (a)).

Occlusion boundaries segment a region into subregions, each of which is a continuous surface whose geometry can change abruptly but remains physically connected in 3D. Folds further segment a continuous surface into smooth surfaces where the geometry vary smoothly without discontinuity of surface normals.

Surface Normal The worker first specifies if a smooth surface is planar or curved. She annotates one normal at each planar surface which indicates the orientation of the plane. For each curved surface, she annotates normals at as many locations as she sees fit. A normal is visualized as a blue arrow originating from a green grid (see the appendix), rendered in perspective projection according to the known focal length. Such visualization helps workers perceive the normal in 3D . To rotate and adjust the normal, the worker only needs to drag the mouse.

Relative Normal Finally, to annotate normals with higher accuracy, the worker specifies the relative normal between each pair of planar surfaces. She chooses between Neither, Parallel and Orthogonal. Surfaces pairs that are parallel or orthogonal to each other then have their normals adjusted automatically to reflect the relation.

Interactive Previewing While annotating, the worker can click a button to see a visualization of the 3D shape constructed from the current annotations (detailed later in Sec. 4). Workers can rotate or zoom to inspect the shape from different angles in a preview window (Fig 1 (a)). She keeps working on it until she is satisfied with the shape.

Quality Control Completing our 3D annotation task requires knowledge of relevant concepts. To ensure good quality of the dataset, we require each worker to complete a training course to learn concepts such as occlusions, folds and normals, and usage of the UI. She then needs to pass a qualification quiz before being allowed to work on our annotation task. Besides explicitly selecting qualified workers, we also set up a separate quality verification task on each collected mesh. In this task, a worker inspects the mesh to judge if it reflects the image well. Only meshes deemed high quality are accepted.

To improve our annotation throughput, we collected annotations from three sources: Amazon Mechanical Turk, which accounts for 11% of all annotations, and two data annotation companies that employ full-time annotators, who supplied the rest of the annotations.

From Human Annotations to Dense Depth

Because humans do not directly annotate the depth value of each pixel, we need to convert the human annotations to pixelwise depth in order to visualize the 3D surface.

Generating Dense Surface Normals We first describe how we generate dense surface normals from annotations. We assume the normals to be smoothly varying in the spatial domain, except across folds or occlusion boundaries where the normals change abruptly. Therefore, our system propagates the known normals to the unknown ones by requiring the final normals to be smooth overall, but stops the propagation at fold and occlusion lines.

Generating Dense Depth Our depth generation pipeline consists of two stages: First, from surface normals and focal length, we recover the depth of each continuous surface through integration We snap the z component of the surface normals to be no smaller than 0.3 so that the generated depth would not stretch into huge distance.. Next, we adjust the depth order among these surfaces by performing surface-wise depth scaling (Fig. 1 (b)), i.e. each surface has its own scale factor.

Our design is motivated by this fact: in single-view depth recovery, depth within continuous surface can be recovered only up to an ambiguous scale; thus different surfaces may end up with different scales, leading to incorrect depth ordering between surfaces. But workers already decide which side of an occlusion boundary is closer to the viewer. Based on such knowledge, we correct depth order by scaling the depth of each surface.

We now describe the details. Let $\mathbf{S}$ denotes the set of all continuous surface. From integration, we obtain the depth $Z_{S}$ of each $S\in\mathbf{S}$. We then solve for a scaling factor $X_{S}$ for each $S$, which is used in scaling depth $Z_{S}$. Let $\mathbf{O}$ denote the set of occlusion boundaries. Along $\mathbf{O}$, we densely sample a set of point pairs $\mathbf{B}$. Each pair $(p,q)\in\mathbf{B}$ has $p$ lying on the closer side of one of the occlusion boundaries $O_{i}\in\mathbf{O}$ and $q$ the further side. The continuous surface a pixel $p$ lies on is $S(p)$, and its depth is $Z_{p}$. The set of optimal scaling factors $\mathbf{X^{*}}$ is solved for as follows:

where $\epsilon>0$ is a minimum separation between surfaces, and $\eta>0$ is a minimum scale factor. Eq.(4) requires the surfaces to meet the depth order constraints specified by point pairs $(p,q)\in\mathbf{B}$ after scaling. Meanwhile, Eq.(3) constrains the value of $\mathbf{X}$ so that they do not increase indefinitely. After correcting the depth order, the final depth for surface $S$ is $X_{S}^{*}Z_{S}$. We normalize and reproject the final depth to 3D as point clouds, and generate 3D meshes for visualization.

Dataset Statistics

Statistics of Surfaces Fig. 2 plots various statistics of the 3D surfaces. Fig. 2 (a) plots the distribution of focal length. We see that focal lengths in OASIS vary greatly: they range from wide angle to telezoom, and are mostly 1$\times$ to 10$\times$ of the width of the image. Fig. 2 (b) visualizes the distribution of surface normals. We see that a substantial proportion of normals point directly towards the camera, suggesting that parallel-frontal surfaces frequently occur in natural scenes. Fig. 2 (c) presents region-wise statistics. We see that most regions (90%+) contain occlusion boundaries and close to half have both occlusion boundaries and folds (top). We also see that most regions (70%+) contain at least one curve surface (bottom). Fig. 2 (d) shows the histogram of the number of different kinds of surfaces in an annotated region. We see that most regions consist of multiple disconnected pieces and have non-trivial geometry in terms of continuity and smoothness.

Annotation Quality We study how accurate and consistent the annotations are. To this end, we randomly sample 50 images from NYU Depth and 70 images from Tanks and Temples , and have 20 workers annotate each image. Tab. I reports the depth and normal difference between human annotations, between human annotations and sensor ground truth, and between predictions from state-of-the-art ConvNets and sensor ground truth. Depth difference is measured by the mean Euclidean distance (EDist) between corresponding points in two point clouds, after aligning one to the other through a global translation and scaling (surface-wise scaling for human annotations and CNN predictions). Normal difference is measured in Mean Angular Error (MAE). We see in Tab. I that human annotations are highly consistent with each other and with sensor ground truth, and are better than ConvNet predictions, especially when the ConvNet is not trained and tested on the same dataset.

We observe that humans often estimate the shape correctly, but the overall orientation can be slightly off, causing a large depth error against sensor ground truth (Fig. 3). This error can be particularly pronounced for planes close to orthogonal to the image plane. Thus we also compute the error after a rotational alignment with the sensor ground truth—we globally rotate the human annotated normals (up to 30 degrees) before generating the shape. After accounting for this global rotation of normals, human-sensor depth difference is further reduced by 47.96% (relative) for NYU and 62.44% (relative) for Tanks and Temples; a significant drop of normal error is also observed in human-human difference.

We also measure the qualitative aspect of human annotations by evaluating the WKDR metric , i.e. the percentage of point pairs with inconsistent depth ordering between query and reference depth. Depth pairs are sampled in the same way as . Tab. I again shows that human annotations are qualitatively accurate and highly consistent with each other.

It is worth noting that metric 3D accuracy is not required for many tasks such as navigation, object manipulation, and semantic scene understanding—humans do well without perfect metric accuracy. Therefore human perception of depth alone can be the gold standard for training and evaluating vision systems, regardless of its metric accuracy. As a result, our dataset would still be valuable even if it were less metrically accurate than it is currently.

Experiments

To facilitate future research, we use OASIS to train and evaluate leading deep learning models on a suite of single-image 3D tasks including depth estimation, normal estimation, boundary detection, plane segmentation. Qualitative results are shown in Fig. 4. A train-val-test split of 110K, 10K, 20K is used for all tasks.

For each task we estimate human performance to provide an upperbound accounting for the variance of human annotations. We randomly sample 100 images from the test set, and have each image re-annotated by 8 crowd workers. That is, each image now has “predictions” from 8 different humans. We evaluate each prediction and report the mean as the performance expected of an average human.

We first study single-view depth estimation. OASIS provides pixelwise metric depth in the wild. But as discussed in Sec 4, due to inherent single-image ambiguity, depth in OASIS is independently recovered within each continuous surface, after which the depth undergoes a surface-wise scaling to correct the depth order. The recovered depth is only accurate up to scaling within each continuous surface and ordering between continuous surfaces.

Given this, in OASIS we provide metric depth ground truths that is surface-wise accurate up to a scaling factor. This new form of depth necessitates new evaluation metrics and training losses.

Depth Metric The images in OASIS have varied focal lengths. This means that to evaluate depth estimation, we cannot simply use pixelwise difference between a predicted depth map and the ground truth map. This is because the predicted 3D shape depends greatly on the focal length—given the same depth values, decreasing the focal length will flatten the shape along the depth dimension. In practice, the focal length is often unknown for a test image. Thus, we require a depth estimator to predict a focal length along with depth. Because the predicted focal length may differ from the ground truth focal length, pixelwise depth difference is a poor indicator of how close the predicted 3D shape is to the ground truth.

A more reasonable metric is the Euclidean distance between the predicted and ground-truth 3D point cloud. Concretely, we backproject the predicted depth $Z$ to a 3D point cloud $\mathbf{P}=\{(X_{p},Y_{p},Z_{p})\}$ using $f$ (the predicted focal length), and ground truth depth $Z^{*}$ to $\mathbf{P^{*}}=\{(X^{*}_{p},Y^{*}_{p},Z^{*}_{p})\}$ using $f^{*}$ (the ground truth focal length). We then calculate the distance between $\mathbf{P}$ and $\mathbf{P^{*}}$.

The metric also needs to be invariant to surface-wise depth scaling and translation. Therefore we introduce a surface-wise scaling factor $\lambda_{S_{i}}\in\mathbf{\Lambda}$, and a surface-wise translation $\delta_{S_{i}}\in\mathbf{\Delta}$, to align each predicted surface $S_{i}\in\mathbf{S}$ in $\mathbf{P}$ to the ground truth point cloud $\mathbf{P^{*}}$ in a least square manner. The final metric, which we call Locally Scale-Invariant RMSE (LSIV_RMSE), is defined as:

where $S(p)$ denotes the surface a pixel $p$ is on. The ground truth point cloud $\mathbf{P^{*}}$ is normalized to a canonical scale by the standard deviation of its X coordinates $\sigma(X^{*})$. Under this metric, as long as $\mathbf{P}$ is accurate up to scaling and translation, it will align perfectly with $\mathbf{P^{*}}$, and get 0 error.

Note that LSIV_RMSE ignore the ordering between two separate surfaces; it allows objects floating in the air to be arbitrarily scaled. This is typically not an issue because in most scenes there are not many objects floating in the air. But we nonetheless also measure the correctness of depth ordering. We report WKDR , which is the percentage of point pairs that have incorrect depth order in the predicted depth. We evaluate on depth pairs sampled in the same way as , i.e. half are random pairs, half are from the same random horizontal lines.

Models We train and evaluate two leading depth estimation networks on OASIS: the Hourglass network , and ResNetD , a dense prediction network based on ResNet50. Each network predicts a metric depth map and a focal length, which are together used to backproject pixels to 3D points, which are compared against the ground truth to compute the LSIV_RMSE metric, which we optimize as the loss function during training. Note that we do not supervise on the predicted focal length.

We also evaluate leading pre-trained models that estimate single-image depth on OASIS, including FCRN trained on ILSVRC and NYU Depth , Hourglass trained on MegaDepth , ResNetD trained on a combination of datasets including ILSVRC , Depth in the Wild , ReDWeb and YouTube3D . For networks that do not produce a focal length, we use the validation set to find the best focal length that leads to the smallest LSIV_RMSE, and use this focal length for each test image. In addition, we also evaluate plane, a naive baseline that predicts a uniform depth map.

Tab. 2 reports the results. In terms of metric depth, we see that networks trained on OASIS perform the best. This is expected because they are trained to predict a focal length and to directly optimize the LSIV_RMSE metric. It is noteworthy that ImageNet pretraining provides a significant benefit even for this purely geometrical task. Off-the-shelf models do not perform better than the naive baseline, probably because they were not trained on diverse enough scenes or were not trained to optimize metric depth error. In terms of relative depth, it is interesting to see that ResNetD trained on ImageNet and OASIS performs the best, even though the training loss does not enforce depth ordering. We also see that there is still a significant gap between human performance and machine performance. At the same time, the gap is not hopelessly large, indicating the effectiveness of a large training set.

2 Surface Normal Estimation

We now turn to single-view surface normal estimation. We evaluate on absolute normal, i.e. the pixel-wise predicted normal values, and relative normal, i.e. the parallel and orthogonal relation predicted between planar surfaces.

Absolute Normal Evaluation We use standard metrics proposed in prior work : the mean and median of angular error measured in degrees, and the percentage of pixels whose angular error is within $\gamma$ degrees.

We evaluate on OASIS four state-of-the-art networks that are trained to directly predict normals: (1) Hourglass trained on OASIS, (2) Hourglass trained on the Surface Normal in the Wild (SNOW) dataset , (3) Hourglass trained on NYU Depth , and (4) PBRS, a normal estimation network by Zhang et al. trained on NYU Depth . We also include Front_Facing, a naive baseline predicting all normals to be orthogonal to the image plane.

Tab. 3 reports the results. As expected, the Hourglass network trained on OASIS performs the best. Although SNOW is also an in-the-wild dataset, the same network trained on it does not perform as well, but is still better than training on NYU. Notably, the human-machine gap appears fairly small numerically (17.27 versus 23.91 in mean angle error). However, we observe that the naive baseline can achieve 31.79; thus the dynamic range of this metric is small to start with, due to the natural distribution of normals in the wild. In addition, a close examination of the results suggests that these standard metrics of surface normals do not align well with perceptual quality. In natural images there can be large areas that dominate the metric but have uninteresting geometry, such as a blank wall in the background. For example, in Fig. 5, a neural network gets the background correct, but largely misses the important details in the foreground. This opens up an interesting research question about developing new evaluation metrics.

Relative Normal Evaluation We also evaluate the predicted normals in terms of relative relations, specifically orthogonality and parallelism. Getting these relations correct is important because it can help find vanishing lines and perform self-calibration.

We first define a metric to evaluate relative normal. From the human annotations, we first sample an equal number of point pairs from surface pairs that are parallel, orthogonal, and neither. Given a predicted normal map, we look at the two normals at each point pair and measure the angle $\theta$ between them. We consider them orthogonal if $|cos(\theta-90^{\circ})|<cos(\Theta_{o})$, and parallel if $|cos(\theta)|>cos(\Theta_{p})$, where $\Theta_{o}$, $\Theta_{p}$ are thresholds. We then plot the Precision-and-Recall curve for orthogonal by varying $\Theta_{o}$, and measure its Area Under Curve $\mathbf{AUC_{o}}$, using neither and parallel pairs as negative examples. Varying $\Theta_{p}$ and using neither and orthogonal as negative examples, we obtain $\mathbf{AUC_{p}}$ for parallel.

Tab. 3 reports results of relative normal evaluation. Notably, all methods perform similarly, and all perform very poorly compared to humans. This suggests that existing approaches to normal estimation have limitations in capturing orthogonality and parallelism, indicating the need for further research.

Cross-Dataset Generalization Next we study how networks trained on OASIS generalize to other datasets. Surface normal estimation is ideal for such evaluation because unlike depth, which is tricky to evaluate on a new dataset due to scale ambiguity and varying focal length, a normal estimation network can be directly evaluated on a new dataset without modification.

We train the same Hourglass network on OASIS, and NYU, and report their performance on two benchmarks not seen in training: DIODE and ETH3D . From Tab. 4 we see that training on NYU underperforms on all benchmarks, showing that networks trained on scene-specific datasets have difficulties generalizing to diverse scenes. Training on OASIS outperforms on all benchmarks, demonstrating the effectiveness of diverse annotations.

3 Fold and Occlusion Boundary Detection

Occlusion and fold are both important 3D cues, as they tell us about physical connectivity and curvature: Occlusion delineates the boundary at which surfaces are physically disconnected to each other, while Fold is where geometry changes abruptly but the surfaces remain connected.

Task We investigate joint boundary detection and occlusion-versus-fold classification: deciding whether a pixel is a boundary (fold or occlusion) and if so, which kind it is. Prior work has explored similar topics: Hoiem et al. and Stein et al. handcraft edge or motion features to perform occlusion detection, but our task involves folds, not just occlusion lines.

Evaluation Metric We adopt metrics similar to standard ones used in edge detection : F-score by optimal threshold per image (OIS), by fixed threshold (ODS) and average precision (AP). For a boundary to be considered correct, it has to be labeled correctly as either occlusion or fold. More details on the metrics can be found in the appendix.

To perform joint detection of fold and occlusion, we adapt and train two networks on OASIS: Hourglass , and a state-of-the-art edge detection network HED . The networks take in an image, and output two probabilities per pixel: $p_{e}$ is the probability of being an boundary pixel (occlusion or fold), and $p_{f}$ is the probability of being a fold pixel. Given a threshold $\tau$, pixels whose $p_{e}<\tau$ are neither fold nor occlusion. Pixels whose $p_{e}>\tau$ are fold if $p_{F}>0.5$ and otherwise occlusion.

As baselines, we also investigate how a generic edge detector would perform on this task. We use HED network trained on BSDS dataset to detect image edges, and classify the resulting edges to be either all occlusion (Edge: All Occ) or all fold (Edge: All Fold).

All results are reported on Tab 5. Hourglass outperforms HED when trained on OASIS, and significantly outperforms both the All-Fold and All-Occlusion baselines, but still underperforms humans by a large margin, suggesting that fold and occlusion boundary detection remains challenging in the wild.

4 Instance Segmentation of Planes

Our last task focuses on instance segmentation of planes in the wild. This task is important because planes often have special functional roles in a scene (e.g. supporting surfaces, walls). Prior work has explored instance segmentation of planes, but is limited to indoor or driving environments . Thanks to OASIS, we are able to present the first-ever evaluation of this task in the wild.

We follow the way prior work performs this task: a network takes in an image, and produces instance masks of planes, along with an estimate of planar parameters that define each 3D plane. To measure performance, we report metrics used in instance segmentation literature : the average precision (AP) computed and averaged across a range of overlap thresholds (ranges from 50% to 95% as in ). A ground truth plane is considered correctly detected if it overlaps with one of the detected planes by more than the overlap threshold, and we penalize multiple detection as in . We also report the AP at 50% overlap ($\text{AP}^{50\%}$) and 75% overlap ($\text{AP}^{75\%}$).

PlanarReconstruction by Yu et al. is a state-of-the-art method for planar instance segmentation. We train PlanarReconstruction on three combinations of data: (1) ScanNet only as done in , (2) OASIS only, and (3) ScanNet + OASIS. Tab. 6 compares their performance.

As expected, training on ScanNet alone performs the worse, because ScanNet only has indoor images. Training on OASIS leads to better performance. Leveraging both ScanNet and OASIS is the best overall. But even the best network significantly underperforms humans, suggesting ample space for improvement.

Conclusion

We have presented OASIS, a dataset of rich human 3D annotations. We trained and evaluated leading models on a variety of single-image tasks. We expect OASIS to be a useful resource for 3D vision research.

Acknowledgement This work was partially supported by a National Science Foundation grant (No. 1617767), a Google gift, and a Princeton SEAS innovation grant.

References

A1: Surface Normal Annotation UI

The surface normal annotation UI is shown in Fig. I.

A2: Planar versus Curved Regions

Tab. I measures the annotation quality separately for planar regions and curved regions.

A3: Comparison with Other Datasets

Tab. II compares OASIS and other datasets.

A4: Additional Examples from OASIS

Additional human annotations are shown in Fig. II.

A5: Additional Qualitative Outputs

Qualitative predictions presented in both Fig. III and Fig. 4 are produced as follows: Depth predictions are produced by a ResNetD network trained on OASIS + ImageNet . Surface normal predictions are produced by an Hourglass network trained on OASIS alone. Occlusion boundary and fold predictions are produced by an Hourglass network trained on OASIS alone. Planar instance segmentations are produced by a PlanarReconstruction network trained on Scannet + OASIS.

A6: Evaluating Fold and Occlusion Boundary Detection

This section provides details on evaluating fold and occlusion boundary detection. As discussed in Sec 6.3 of the main paper, our metric is based on the ones used in evaluating edge detection .

The input to our evaluation pipeline consists of (1) the probability of each pixel being on edge (fold or occlusion) $p_{e}$, and (2) a label of each pixel being occlusion or fold. By thresholding on $p_{e}$, we first obtain an edge map $E_{\tau}$ at threshold $\tau$. We denote the occlusion pixels as $O$ and the fold pixels as $F$. We find the intersection $O\cap E_{\tau}$ and use the same protocol as to compare it against the ground-truth occlusion $O^{*}$ and obtain true positive count TFo, false positive count FPo and false negative count FNo. We follow the same protocol to compare $F\cap E_{\tau}$ against ground-truth fold $F^{*}$ and obtain TFf, FPf and FNf.

We then calculate the joint counts TF, FP and FN: TP=TFo+TFf, FP=FPo+FPf and FN=FNo+FNf.

We iterate through different $\tau$ to obtain the joint counts TF, FP and FN at each threshold to obtain the final ODS/OIS F-score and AP.

A7: Crossed-out Numbers in Tables

We made minor quality improvements to the dataset after the camera ready deadline of CVPR 2020, affecting less than 10% of the images. The crossed-out numbers are those presented in the CVPR camera ready version of this paper https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_OASIS_A_Large-Scale_Dataset_for_Single_Image_3D_in_the_CVPR_2020_paper.pdf and are from the older, obsolete version of the dataset. The publicly released version of OASIS is the new version with the quality improvements.