3D-FUTURE: 3D Furniture shape with TextURE

Introduction

The rapid progress of modern machine learning methods, such as deep neural models, has led to various impressive breakthroughs towards 2D computer vision and natural language processing (NLP). One key to facilitating the advancement of these approaches is the availability of large-scale labeled benchmarks. Mirroring this pattern, the computer graphics and 3D vision communities have put tremendous efforts in establishing 3D datasets over the past years, expecting to enable and innovate the avenues of future research (Chang et al., 2015; Wu et al., 2015; Xiao et al., 2013; Song et al., 2015; Xiao et al., 2016; Sun et al., 2018a; Xiang et al., 2014, 2016; Silberman et al., 2012; Dai et al., 2017; Hua et al., 2016). For example, the largest 3D repositories, like ShapeNet (Chang et al., 2015) and ModelNet (Wu et al., 2015), collected massive 3D shapes from online repositories and organized them under the WordNet taxonomy. Relying on the repositories, several works, such as Pascal 3D+ (Xiang et al., 2014), ObjectNet3D (Xiang et al., 2016), Pix3D (Sun et al., 2018a), and Stanford Cars (Krause et al., 2013), further provided images and shapes associations or alignments with fine-grained pose annotations. Other works like NYU Depth Dataset (Silberman et al., 2012), SUN RGB-D (Song et al., 2015), ScanNet (Dai et al., 2017), SceneNN (Hua et al., 2016), and Matterport3D (Chang et al., 2017) introduced RGB-D scans of real-world indoor environments with many estimated and manually verified annotations. Considering that there are rich 3D benchmarks, why do we need one more?

In contrast to the 2D counterparts (Krizhevsky et al., 2012; Lin et al., 2014; Geiger et al., 2012), we realize that there is still a big gap between 3D academic research and industrial productions. For instance, the 3D CAD models in existing datasets mainly come from public online repositories like Trimble 3D Warehousehttps://3dwarehouse.sketchup.com and Yobi3Dhttps://yobi3d.com. These 3D shapes typically have fewer geometry details and uninformative textures or even no textures. Specific to shapes in the household scenario, most of them are outdated and dull furniture deprecated by modern professional designers. Therefore, the current 3D shapes are inadequate for comprehensive and subtle research in areas such as industry closely related fine-grained 3D shape understanding and texture recovery. Besides, existing benchmarks only provide pseudo image or shape alignments, and the estimated camera pose annotations. Namely, the benchmark designers manually choose a roughly matched 3D CAD model from available 3D shape benchmarks according to the object in the image. Thus, annotators may largely ignore some local shape details, which prevents the progress of fundamental data-driven studies such as high-quality 3D reconstruction from real-world images and high-accuracy image-based 3D shape retrieval. Last but not least, there is no well-organized benchmark that offers realistic synthetic indoor images with both instance-level semantic annotations and the involved 3D shapes with textures.

Motivated by the observations, we present 3D Furniture shape with TextURE (3D-FUTURE): a richly-annotated, large-scale repository of 3D furniture shapes specific to the household scenario as shown in Figure 1. At this time, 3D-FUTURE provides 20,240 realistic indoor images and the associated 9,992 unique 3D furniture models with rich geometry details and informative textures. We render these images via one of the most advanced industrial 3D rendering engines based on 5,000 exquisite room scenes developed by experienced designers. The 3D furniture shapes are used for modern industrial productions and have fine-grained geometry and texture related attributes such as category, style, theme, and material. Further, 3D-FUTURE offers instance segmentation annotation and the rendering information, including six degrees of freedom (6DoF) pose and camera field of view (FoV). Apart from these highlight features, another compelling part of 3D-FUTURE is that it enables many fundamental studies and new research opportunities such as furnishing composition, texture recovery, and other interior understanding subjects.

It is, however, nontrivial to collect thousands of aesthetic interior designs. To the best of our knowledge, it takes a designer several days to complete a house’s interior design. Thus, we considered two main research questions when establishing 3D-FUTURE: 1) can we develop a framework that allows creators to design delicate rooms efficiently? 2) can we automatically create some aesthetic designs based on the professional layout information? To investigate the former question, we build a furnishing suit composition (FSC) platformhttps://3d.shejijia.com/. The system recurrently recommends visually matched furniture by considering instance aesthetics and compatibility during the design progress. For the latter one, we reuse the expert layouts, generate multiple furnishing suit candidates with some rules and the FSC approach, render the scene, and manually select visually appealing ones. These AI-created designs will also be reviewed by designers to ensure good quality.

The remainder of this paper is organized as follows. First, we briefly review the public 3D benchmarks and discuss their imperfections. Second, we present the data acquisition process and the FSC pipeline. Third, we introduce the properties and statistics of 3D-FUTURE. Finally, we conduct various experiments leveraging on the properties. These experiments can serve as baselines for subsequent research on 3D-FUTURE.

Related Work

Lots of 3D benchmarks have been established and made publicly available over the past decades (Chang et al., 2015; Wu et al., 2015; Xiao et al., 2013; Song et al., 2015; Xiao et al., 2016; Sun et al., 2018a; Xiang et al., 2014, 2016; Silberman et al., 2012; Dai et al., 2017; Hua et al., 2016; Choi et al., 2016; Shilane et al., 2004). These datasets can be mainly divided into two groups, including 3D models and RGB-D scenes. We will briefly review some representative 3D benchmarks in the following.

One of the large and exhaustively studied 3D shape repositories is ShapeNet (Chang et al., 2015). It collected millions of raw 3D CAD models from public online repositories such as Warehouse3D and Yobi3D. By re-organizing the datasets, the subsets ShapeNetCore and ShapeNetSem have been made available, including 51,300 and 12,000 models. ShapeNet assigned rich semantic annotations to part of the shapes, such as synsets in the WordNet taxonomy, functional patterns, parts, keypoints, and categories. 3D shape repositories like ModelNet (Wu et al., 2015) and Princeton Shape Benchmark (Shilane et al., 2004) also share similar content as ShapeNet. Several other works like (Choi et al., 2016) and ScanObjectNN (Uy et al., 2019) create the datasets of 3D scans of real objects based on state of the art (SOTA) RGB-D reconstruction approaches. These benchmarks have largely driven the fundamental 3D studies, including 3D representation, 3D shape recognition, 3D object reconstruction, and part segmentation. However, since the 3D shapes are collected online, many may lack geometry details and have dreamlike or no textures.

Relying on these large-scale 3D shape databases, the community also builds benchmarks with image and shape associations to facilitate the research of 3D object understanding from images. For example, PASCAL3D+ (Xiang et al., 2014) and ObjectNet3D (Xiang et al., 2016) aligned objects in the 2D images with the 3D shapes and provided raw 3D pose annotation. Further, Pix3D (Sun et al., 2018a) contributed more accurate 2D-3D alignment for 395 3D shapes of nine object categories. Unluckily, these pseudo alignments may largely ignore some local shape details. Moreover, the expensive labor efforts make it difficult to build a large-scale benchmark with precise pixel-level 2D-3D alignment.

2 RGB-D Scenes

In recent years, the community has put significant efforts into building RGB-D datasets to expand researches on 3D scene understanding. For example, NYU Depth V2 (Silberman et al., 2012) captured 464 short Kinect RGB-D sequences from 464 different indoor scenes, where 1,449 images are with dense per-pixel labeling, including depth, surface normal, and semantic labels. SUN RGB-D (Song et al., 2015) followed the pattern by annotating 10,335 RGB-D frames, and offered 3D bounding boxes. To capture the full 3D extent of indoor environments, SUN3D (Xiao et al., 2013) obtained 415 long sequences in 254 unique spaces with comprehensive views. Further, Dai et al. established ScanNet (Dai et al., 2017), an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with estimated 3D camera poses, surface reconstructions, semantic segmentation, and a broad set of CAD model alignments. Later, a more extensive dataset Matterport3D (Chang et al., 2017) was made publicly available, contributing to panoramic HDR color images with 3D scene annotations. Different from these RGB-D real-world databases, we focus on experienced exquisite interior designs used in industrial productions.

The works most closely related to ours are InteriorNet (Li et al., 2018) and Structured3D (Zheng et al., 2019), which also offer photorealistic images by rendering professional house designs. However, there are two significant differences. First of all, we provide furniture shapes with textures in the scenes. The 6DoF pose and camera FoV are shared in 3D-FUTURE. Second, 3D-FUTURE additionally expects to foster studies of exquisite interior design understanding. Thus, for each room, the camera viewpoints are suggested by designers, so that the captured images contain the whole design idea.

Data Acquisition Process

In this section, we introduce the pipeline of our dataset construction procedure. We mainly address the two issues, i.e., designing efficiency and aesthetic design creation, as stated in Sec. 1.

We construct a 3D pool containing a large amount of industrial 3D computer-aided design (CAD) furnishing and interior finish models. We associate each shape with multiple textures and materials, resulting in enlarged shape repositories. The models are richly annotated with diverse attributes, including theme color, style, material, brand, real-world size, and category in the WordNet taxonomy. There are 500 fine-grained categories in five levels of the taxonomy. High-resolution 2D rendering for each textured model is also available in the database. Based on these objects, hundreds of experienced designers have created $\sim$60,000 decorative houses for different scenarios in several years, where $\sim$30,000 homes have been evaluated as excellent or brilliant designs. A design sample is shown in Figure 1. The large-scale interior data is offered by Alibaba Topping Homestylerhttps://www.tangping.com/. We set up a project based on the large-scale interior data to build 3D-FUTURE.

2 Furnishing Suit Composition (FSC)

One of the main challenges in establishing 3D-FUTURE is how to collect many exquisite interior designs in an acceptable project cycle. To address this issue, we develop a high-performing furnishing suit composition framework. We mainly borrow the concepts of attribute-based interpretable compatibility methods (Yang et al., 2019; Wang et al., 2018b; Chen et al., 2019a) in fashion outfit compatibility learning. An overview of this framework is presented in Figure 2.

Our training set for FSC has the form as $\{\mathcal{X}^{1},\mathcal{X}^{2},...,\mathcal{X}^{N}\}$, and $\mathcal{X}^{i}=\{x^{i}_{1},x^{i}_{2},...,x^{i}_{m_{i}}\}$. Here, $N$ is the total number of experienced house designs. $m_{i}$ is the number of items in house $\mathcal{X}_{i}$, and $x^{i}_{j}$ is a specific furnishing item contained in house $\mathcal{X}^{i}$. Note that the elements in $\mathcal{X}^{i}$ are in order, which means $x^{i}_{j}$ is a former furnishing item selected by designers followed by $x^{i}_{j+1}$.

As aforementioned, we have rich attribute annotations for each furnishing item. These interpretable attributes show significance in understanding the item’s content and are thus beneficial to FSC learning. However, we can not expect a limited number of attributes to represent an item comprehensively. Therefore, we propose a Deep Furnishing Suit Model (DFSM), which consists of a visual embedding network (VEN) and two transformer encoders (TransEnc1 and TransEnc2), to learn representative deep visual embedding leveraging on the large-scale excellent house designs. DFSM is driven by a margin ranking loss with hard sample mining and a variant of classification loss.

In specific, given a furnishing suit $\mathcal{X}^{i}$, we randomly capture a subset $X^{i}_{j\sim k}=\{x^{i}_{j},x^{i}_{j+1},...,x^{i}_{k}\}$, where $1\leq j\leq k<m_{i}$. Our goal is to predict $x^{i}_{k+1}$ given $X^{i}_{j\sim k}$. According to the category label of $X^{i}_{j\sim k}$, we randomly choose three negative examples from the furnishing pool to construct a candidate set $C=\{x^{i}_{k+1},z^{0}_{x^{i}_{k+1}},z^{1}_{x^{i}_{k+1}},z^{2}_{x^{i}_{k+1}}\}$ in an online manner. We also ensure that the negative examples $z^{0}$ / $z^{1}$ / $z^{2}$ have the same style / color / material as $x^{i}_{k+1}$. We feed both $X^{i}_{j\sim k}$ and the candidate images into VEN to extract visual features. In our paper, we take the CNN part of MobileNetV2 followed by a projection layer as VEN, and pre-train it via the unsupervised learning strategy stated in (Wu et al., 2018).

After obtaining the image features, we construct two tasks, i.e., mask prediction and compatibility scoring, based on the impressive transformer architecture in NLP (Devlin et al., 2018; Vaswani et al., 2017). For the former one, we have a sequence of feature vectors $\mathcal{F}^{i}=${VEN($X^{i}_{j\sim k}$), [Mask]} with dimension $d$, where [Mask] denotes a particular mask embedding. The task is to predict the masked item given the previous ones. We thus feed $\mathcal{F}$ into TransEnc1 to capture the enhanced feature $\mathcal{\widetilde{F}}^{i}$, and optimize the model via the following loss:

For the second task, we take the candidate suits as inputs and directly learn their compatibility scores. Let $F_{(X^{i}_{j\sim k},c)}=$ {[Start], VEN($X^{i}_{j\sim k}$), $f_{c}$} be the visual feature vectors of a candidate suit $O_{(X^{i}_{j\sim k},c)}$, where [Start] is a particular start token embedding. To estimate the compatibility score of a suit, we need to first capture an embedding that can represent it. We thus employ TransEnc2 to acquire $\widetilde{F}_{(X^{i}_{j\sim k},c)}$, and use the feature vector of the [Start] token as the representation of suit $O_{(X^{i}_{j\sim k},c)}$ (denoted as $r_{(X^{i}_{j\sim k},c)}$). Further, we utilize two fully connected layers and a sigmoid function to secure a score ($s_{(X^{i}_{j\sim k},c)}$), which is the measure of the quality of the suit. For conventional presentation, $s_{(X^{i}_{j\sim k},c)}$ is abbreviated as $s(x^{i}_{k+1})$ hereafter. Since the ground truth compatibility scores are not available, we minimize a margin ranking loss with a simple hard sample mining policy. The objective is expressed as:

where $\alpha$ is set to 0.1 in our experiments. The trained visual embedding network (VEN) is used to extract the visual feature for each furniture item.

2.2 Decision Tree Based FSC

The main goal here is to infer attribute-based matching patterns, i.e., attribute crosses, for FSC. Considering both interpretability and scalability, we utilize GBDT (Friedman, 2001) to automatically construct attribute crosses as shown in Figure 6. We will not introduce the details of GBDT here, but only present some facts in training the decision trees.

We employ six attributes to represent a specific item, including theme color, style, material, real-world size, the second-level category, and visual information (the learned visual embedding). Here, we denote the learned visual embedding as an attribute. For the discrete attributes (style, material, and the second-level category), we directly convert them to one-hot vectors. For theme color and real-world size, we first adopt k-Means Clustering (Kanungo et al., 2002) to discretize real values and then transform them into one-hot vectors. By further considering the visual embedding, we can represent each item as a feature vector. We assign a label (positive or negative) to each specific furnishing suit to train the decision trees. Both the positive and negative suits are constructed similarly in Sec. 3.2.1.

2.3 Re-training via Hard Sample Mining

The negative furnishing suits construction strategy may return some naive negative samples, due to the large-scale furnishing pool, causing some inaccuracies in both the deep embedding networks and the decision trees. We fine-tune the visual embedding network and re-train the decision tree model via a straightforward hard sample mining strategy to address the issue. Specifically, given $X^{i}_{j\sim k}$, we can have the TopK recommendations using the trained DFSM. We then randomly select negative samples from the TopK pool. After the re-training stage, we fix VEN’s parameters and establish an automatically re-training system to update the decision tree model daily using continuously enlarged online designs.

3 Topping Homestyler Design Platform

Our DFSM is integrated into the online Topping Homestyler Design Platformhttp://3d.shejijia.com to improve the house design efficiency. There are also other highlight features that can facilitate the design procedure, such as the large-scale shape pool, image-based furniture retrieval, 2D display, 3D Roaming, various professional design templates, texture and item replacement, and online rendering.

4 Create Aesthetic Interior Design

We have collected 5,000 exquisite interior room designs in the 3D-FUTURE project. We do not plan to provide several synthetic images in different viewpoints for each room. Instead, we expect to deliver more superlative designs to bring more research possibilities. Thus, we take these experienced designs as templates and create several aesthetic interior designs for each template.

For example, given a template room with professional design information, we first replace the interior finishing according to the materials, room style, and other descriptions. Second, we choose a furniture seed (e.g., bed) based on the interior finishing information. Third, we iteratively perform recommendations based on our DFSM and other rules to generate a furnishing list. Finally, we put the items contained in the furnishing list into their corresponding positions. In the third step, we also learn one-to-one visual compatibility models (e.g, bed-nightstand and sofa-coffee table) as additional rules to improve the robustness.

With the pipeline, we can automatically create many interior designs as shown in Figure 3. To ensure the quality, we render an image for each design and manually select 15,240 visually appealing designs. Our experienced designers further review these designs to assure good quality.

The 3D designs are rendered by one of the most advanced computer-generated imagery rendering software applications, V-Rayhttps://www.chaosgroup.com/. To ensure reality, we enable as many functions as possible supported by V-Ray.

Properties of 3D-FUTURE

In this section, we summarize the properties of our 3D-FUTURE database. Compared to previous 3D benchmarks, 3D-FUTURE has some prominent properties that can bring more possibilities for future 3D research.

3D-FUTURE offers 20,240 photo-realistic synthetic images corresponding to 20,240 interior designs. As aforementioned, we have 5,000 experienced designs and 15,240 automatically created aesthetic designs. We render one image for each design. Previous datasets, such as Structured3D (Zheng et al., 2019) and InteriorNet (Li et al., 2018), also provide realistic indoor images and scene parsing annotations. However, they put cameras in random positions and capture redundant images for each house. These images were not manually verified, thus suffer from unexpected viewpoints.

In contrast, 3D-FUTURE focuses more on inspiring the understanding of exquisite interior designs. Thus, the camera positions are suggested by professional designers to obtain the best viewpoint for each room. Besides, 3D-FUTURE provides instance semantic labels of 34 categories and ten supper-categories. Moreover, the images contained in 3D-FUTURE are visually more appealing and realistic compared to previous ones.

2 2D-3D Alignments

Previous benchmarks only provide pseudo 2D-3D alignment annotations (Xiang et al., 2016; Sun et al., 2018a; Dai et al., 2017; Krause et al., 2013; Xiang et al., 2014). Namely, they manually choose a roughly matched 3D CAD model from public 3D shape benchmarks according to the object contained in an image. Annotators thus may largely ignore some local shape details. As a result, these benchmarks offer a small number of matched 3D shape and 2D image pairs. Besides, previous benchmarks with alignment annotations do not come with scene images. In contrast, 3D-FUTURE provides precious 2D-3D alignments and 3D pose annotations. It contains 9,992 unique 3D shapes and 20,240 scene images. By cropping instances from the scene images, we can further secure 37,441 image and shape pairs with slight occlusions, as reported in Table 3. Some samples are presented in Figure 4.

3 High-quality Shapes with Informative Textures

The 3D shapes contained in previous large-scale shape repositories (Chang et al., 2015; Shilane et al., 2004; Wu et al., 2015) are mainly collected from online repositories. These 3D CAD models usually contain few geometry details and low informative textures. Luckily, 3D-FUTURE provides high-quality 3D furniture shapes with rich details in various styles, including European furniture, which often contains intricate carvings. All the shapes come with informative textures and have been used for modern industrial productions. We show some samples in Figure 5. We believe these features can potentially facilitate innovative research on high-quality 3D shape understanding and generation. In Figure 10, we compare the proportion of different number of vertices and faces over ShapeNetCore (Chang et al., 2015), ModelNet40 (Wu et al., 2015) and our dataset. While other datasets have some extremely low-resolution shapes, 3D shapes in 3D-FUTURE show uniform distributions in both vertices and faces.

4 Fine-Grained Attributes

Previous 3D benchmark provides functional attribute annotations in WordNet taxonomy for 3D shapes (Chang et al., 2015). However, these attributes are not well organized and do not have corresponding textures. In contrast, for each textured shape in 3D-FUTURE, we provide four types of attributes verified by professional designers. We have 34 shape categories, 8 super-categories, 19 styles, 15 materials, and 16 themes. These attributes have been demonstrated valuable for interior designs and content understanding by industrial productions. We present the statistics of these attributes in Figure 9 and Figure 8. These figures imply the preferences of experienced modern designers when designing the rooms.

Baseline Experiments

In the section, we conduct several baseline experiments by leveraging the properties of 3D-FUTURE, including shape recognition, joint 2D instance segmentation and 3D pose estimation, image-based shape retrieval, 3D object reconstruction, and texture synthesis. We split our 3D shapes into a training set with 6,699 models, and a test set with 3,293 models. The scene images are divided according to the training and test splits of 3D shapes. There are 14,761 images for training and 5,479 images for test. We will briefly present the experimental details for each task and report the scores.

Over the past years, most 3D object recognition methods extend deep convolutional neural networks (DCNNs) to modeling 3D data. Because 3D CNNs are too memory intensive (Ji et al., 2012), some researchers prefer to either develop special deep learning operations on point clouds and mesh surfaces (Qi et al., 2017a, b; Hanocka et al., 2019; Feng et al., 2019), or project 3D shapes to several 2D images and then apply 2D convolutional networks (Su et al., 2015). However, it is nontrivial to extend the projection-based methods to high-resolution 3D scene understanding. Moreover, point and mesh-based approaches suffer from computation bottlenecks and are thus limited to sparse point clouds and a small number of surfaces.

In contrast to ShapeNet (Chang et al., 2015) and ModelNet (Wu et al., 2015), 3D-FUTURE enables the study of fine-grained 3D furniture recognition, which requires the networks to capture more local and global geometric details. Here we consider the well-known MVCNN (Su et al., 2015) and PointNet++ (Qi et al., 2017b) as the baselines. In specific, we train a 12-view MVCNN with ResNet50 as the backbone. For PointNet++, we sample 1024 points for each shape instance and adopt the multi-scale grouping (MSG) strategy (Qi et al., 2017b) and normal vectors to secure the best performance. We train the networks using 6,699 shapes and evaluate the trained models via the remaining 3293 shapes. The classification accuracy for each category is presented in Table 2 and Figure 11. While these methods can reach 90% accuracy on ModelNet40 and ShapenetCore, they do not perform well (69.2% $\sim$ 69.9%) on 3D-FUTURE, due to the presence of fine-grained furniture categories. This observation would motivate researchers to exploit more efficient 3D representation learning approaches for deeper 3D shape analysis.

2 Image-based 3D Shape Retrieval

Cross-domain image-based 3D shape retrieval (IBSR) is to identify the CAD models of the objects contained in query images. The primary issue in IBSR is the large appearance gaps between 3D shapes and 2D images. To tackle this challenge, early works made efforts to map cross-domain representations into a unified constrained embedding space via adaptation techniques such as weight-sharing constraints, metric learning, and distance matching (Li et al., 2015; Aubry et al., 2014; Lee et al., 2018; Massa et al., 2016; Tasse and Dodgson, 2016; Girdhar et al., 2016). Recent works (Sun et al., 2018a; Huang et al., 2018; Wu et al., 2017; Bansal et al., 2016; Bachman, 1978; Grabner et al., 2018, 2019) predict 2.5D sketches from images, such as surface normal, depth, and location field, to bridge the gaps between 3D and 2D domains. However, the performance of state-of-the-art IBSR methods show a large gap than its 2D counterpart, i.e., content-based image retrieval. This is because there are no large-scale benchmarks that offer large amounts of precious 2D-3D alignment annotations.

In this experiment, we train the baseline using 31,444 image-shape pairs and evaluate the retrieval algorithm via the other 5,994 image-shape pairs. Then we crop the furniture instances with occlusion levels of “NO”, “Slight” and “Standard” from the scene images to produce the image-shape pairs. The statistics of the train and test sets are presented in Table. 3. We develop a DCNN based metric learning network to study the cross-domain shape similarities, as shown in 14. Specifically, we first project the selected 3D shapes into 2D planes using the toolboxhttps://github.com/3D-FRONT-FUTURE to bridge the 3D and 2D gaps. Given a query image and its corresponding 3D shape, we randomly sample a negative 3D shape from the 3D pool to construct a triplet. We then feed the triples (2D images) into a ResNet-34 feature extractor and adopt a margin ranking loss to push the query image close to its corresponding 3D shape.We utilize a category classification loss and an instance classification loss (Wu et al., 2018) such that the network can capture shape similarity among furniture instances.

We take TopK Recall (TopK@R) and Top5 average F-score (mean F-score) as our metrics. The latter is used to measure the retrieval sequences. The retrieval results for each category are reports in Table. 4. We also show some qualitative retrieval sequences in Figure 15. We can see that while the captured Top1@R for a large portion of categories is less than 30.0%, the retrieval sequences seem to be visually acceptable. Besides, there is a remarkable gap between Top1@R (23.4%) and Top3@R (40.6%). The observations demonstrate that our large 3D pool contains many furniture with similar shape characteristics, which would provide potential opportunities for fine-grained shape retrieval studies.

3 Jointly 2D Instance Segmentation and 3D Pose Estimation

Image-based 6DoF pose estimation is a fundamental 3D vision task that can benefit many intelligent applications such as autonomous driving, augmented reality, and robotic manipulation. Typical methods 6DoF pose estimation first build point-wise correspondences between 3D models and 2D images, followed by the Perspective-n-Point (PnP) algorithm to compute pose parameters (Collet et al., 2011; Rothganger et al., 2006). These approaches perform well for objects with rich textures but are not robust to featureless or occluded cases. Recent works thus employ RGB-D sensors and deep learning to improve keypoints detection or directly predict 6DoF pose from images (Kehl et al., 2016; Brachmann et al., 2014; Bo et al., 2014; Hinterstoisser et al., 2012; Xiang et al., 2017; Peng et al., 2019; Song et al., 2020; Tekin et al., 2018; Rad and Lepetit, 2017; Park et al., 2020). Nevertheless, the main issues such as occlusion and clutter, scalability to multiple objects, and symmetries have not been well addressed.

Instance segmentation is the task of detecting and delineating each distinct object of interest appearing in an image. Current instance segmentation methods can be roughly categorized into two paradigms: segmentation-based methods and detection-based methods. The former category of approaches group the predicted category labels via techniques such as clustering (Dhanachandra et al., 2015), metric learning (Fathi et al., 2017), and watershed algorithms (Najman and Schmitt, 1994), to form instance segmentation results. The latter predicts the mask for region instances detected by SOTA object detectors. Methods such as Mask R-CNN series (He et al., 2017; Huang et al., 2019; Cai and Vasconcelos, 2019) have achieved impressive performance for daily objects.

In this experiment, we learn to predict instance segmentation in 2D images and estimate their 6DoF poses in a unified framework. In contrast to the well-studied benchmarks such as ObjectNet3D (Xiang et al., 2016), PASCAL3D+ (Xiang et al., 2014), and Pix3D (Sun et al., 2018a), 3D-FUTURE encourages estimating pose parameters for multiple objects with occlusions in diverse indoor scenes. We provide 3D pose annotations for 100K+ objects in the scene images. The objects are further divided into five occlusion levels, including “NO”, “Slight”, “Standard”, “Heavy”, and “N/A”. Here, an object labeled as “N/A” means that its corresponding 3D shape is not available, or a part of the object is out of the camera view. We train our model on the 14,761 training images and test it on the remaining 5,479 test images.

We modify Cascade Mask-RCNN (Cai and Vasconcelos, 2019; He et al., 2017) as our baseline. The network architecture is shown in Figure. 13. Specifically, we take ResNeXt-101 (Xie et al., 2017) with the setting of 64-4d (group number: 64, width of group: 4) as the backbone, and adopt FPN (Lin et al., 2017) to extract the dense features. Then, we utilize a three-stage cascade architecture to perform bounding box regression and object classification. Finally, we add two branches that consist of several fully connected layers to predict the instance masks and their 6DoF poses simultaneously. We cast rotation estimation as a viewpoint classification problem. In detail, we convert the rotation matrices to Euler angles and divide the 360-degree azimuth, 180-degree elevation, and 360-degree in-plane rotation into 18 bins, 9 bins, and 18 bins, respectively. For translation estimation, we use L1 smooth loss to regress the translation parameters directly.

For 2D instance segmentation, we report Average Precision (AP) and Average Recall (AR) over different IoU thresholds following (He et al., 2017). For 3D pose estimation, we take both Average Viewpoint Precision (AVP) in PASCAL3D+ (Xiang et al., 2014) and Average Orientation Similarity (AOS) in KITTI (Geiger et al., 2012) to measure the rotation predictions as (Xiang et al., 2016), and employ Root Mean Square Error (RMSE) to evaluate the translation predictions. In specific, we define the difference between an estimated rotation matrix $R$ and its ground truth $R_{gt}$ as $\nabla(R,R_{gt})=\frac{1}{\sqrt{2}}\parallel\log(R^{T}R_{gt})\parallel_{F}$. In AVP, a correct estimation should satisfy $\nabla(R,R_{gt})<\frac{\pi}{6}$. The cosine similarity between rotations in AOS is computed as $cos(\nabla(R,R_{gt}))$.

We present the instance segmentation and pose estimation results in Table 5. Here, the metrics for camera poses are with respect to AP and AR, where the IoU thresholds range from 0.5 to 0.95. For instance segmentation, our baseline captures a mean AP of 0.55 on 3D-FUTURE. The score is at a similar level to those reported on the MSCOCO leaderboard achieved by recent SOTA methods. For 3D pose estimation, our baseline yields a mean AVP of 43%. Besides, as analyzed in (Xiang et al., 2016), AP is an upper bound of AOS. This means the closer AOS is to AP, the more accurate the rotation estimation is. By showing the gaps between AOS and AP in Figure 12, we can see that the estimated rotation (0.43) can be further improved. From the observations, we conclude that most objects’ 3D poses are not well modeled in our challenging setting. This suggests that researchers may need to carefully study 3D pose estimation with different levels of occlusions based on 3D-FUTURE. Some qualitative results are shown in Figure 16 and Figure 17 to further justify our conclusions.

4 Single-View 3D Object Reconstruction

Inferring 3D structure from a single image has been an active research area for a long time. In the supervised setting, traditional methods investigated shape from shading (Durou et al., 2008; Zhang et al., 1999) and defoce (Favaro and Soatto, 2005) to reason the visible parts of objects. Leveraging on large-scale shape repositories, various works examined deep architectures to produce shapes in 3D volume (Choy et al., 2016), point cloud (Fan et al., 2017), and mesh surface (Groueix et al., 2018) directly. Recently, several SOTA methods recovered 3D meshes from initializations using shape deformation based on deep networks (Wang et al., 2018a). In the unsupervised setting, 3D recovery has been recast as a 2D image reconstruction progress of unobserved views with differentiable rendering (Liu et al., 2019; Chen et al., 2019b).

In this paper, we examine several SOTA reconstruction algorithms as the baselines, including ONet (Mescheder et al., 2019), Pixel2Mesh (Wang et al., 2018a), and DISN (Xu et al., 2019). We report the widely studied Intersection over Union (IoU), Chamfer Distance (CD), and F-score to evaluate these approaches on 3D-FUTURE. We refer (Xu et al., 2019) for the definitions of these metrics. We randomly render 24 different view images each model for training and a random view image for testing. The resolution of each image is $256\times 256$. As shown in Table 6 and Figure 18, Pixel2Mesh is more robust in general 3D object reconstruction. However, all the SOTA methods cannot recover good-quality shapes when the 3D shapes contain many geometric details.

5 Texture Synthesis For 3D Shapes

Unlike geometry reconstruction, texture reconstruction of 3D objects has received less attention from the community. Previous works studied the subject by learning colored 3D reconstruction on voxels or point clouds (Sun et al., 2018b; Tulsiani et al., 2017) based on view synthesis and multi-view geometry. While voxel representations are limited to the low resolutions, point representations are sparse and thus ignore geometric details. Recent approaches alternatively learned a 2D texture atlas (UV mapping) for 3D meshes to map a point on the shape manifold to a pixel in the texture atlas. These methods mainly take advantage of differentiable rendering to recast the problem as an unobserved view synthesis problem (Raj et al., 2019; Oechsle et al., 2019).

Existing 3D repositories contain less dreamlike or uninformative textures and cannot support high-quality texture recovery studies. In contrast, 3D-FUTURE provides furniture shapes with informative textures, which are widely used in industrial productions. We examine two baselines for texture synthesis, i.e., Texture Fileds (Oechsle et al., 2019) and a novel BicycleGAN++ method. Here, BicycleGAN++ extends BicycleGAN (Zhu et al., 2017) for texture synthesis. An illustration of the network is shown in Figure 19. In specific, we incorporate a texture encoder such that the learned model can perform controllable texture synthesis. Importantly, by enlarging the weights of the reconstruction losses and introducing a texture consistency loss, we find that the produced multi-view textured images will show preferable consistency in overlap regions.

We conduct experiments on four super-categories, including Sofa, Bed, Chair, and Table. The details of our train and test splits are reported in Table 7. We randomly render 32 views of images for each shape to enlarge the training set. For each baseline, we first train them on the whole train set and then perform category-specific fine-tuning. Following (Oechsle et al., 2019), we use structure similarity image metric (SSIM) (Wang et al., 2004), L1, Frechet Inception Distance (FID) (Heusel et al., 2017), and Feat1 as our metrics to evaluate the quality of the synthetic texture. Here, L1 is the L1 distance between the ground-truth view rendering and the produced textured image under the same viewpoint. Feat1 is a global perceptual measure operated on the Inception-net (Szegedy et al., 2015) feature space using the L1 distance. As shown in Table 8, while BicycleGAN++ earns higher scores on FID and Feat1, Texture Fields performs better in terms of SSIM and L1, indicating that BicycleGAN++ produces more realistic images with higher quality and Texture Fields focuses more on structured texture details. We also give some qualitative results in Figure 20 and Figure 21. We can see that BicylcGAN++ can only learn the main color information while largely ignores the semantic parts of objects. Texture Fields can partially preserve the structured texture details but produces dreamlike textures. These observations demonstrate that achieving visually appealing texture recovery for 3D meshes is still very challenging, especially for the industrial 3D shapes with informative texture details.

Conclusion

In this paper, we have built the large-scale 3D-FUTURE benchmark specific to the household scenario with rich 3D and 2D annotations. 3D-FUTURE contains 20,240 realistic synthetic images and 9,992 high-quality 3D CAD furniture shapes. The exciting features include but are not limited to the exhausting interior designs by experienced designers, photo-realistic renderings, 2D-3D alignments, and most significantly the industrial 3D furniture shapes with informative textures. We conduct several experiments to show the remarkable properties of 3D-FUTURE. The experiments can serve as baselines for future research using our database. We hope that 3D-FUTURE can facilitate innovative research on high-quality 3D shape understanding and generation, bring new research opportunities for 3D vision, and build a bridge between academic study and 3D industrial applications.

References