Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence

Introduction

3D objects in a same class share some common shape features and semantic correspondences, which can be used to construct a deformable shape model beneficial for a diverse array of downstream tasks in 3D and 2D domains such as shape understanding , reconstruction , manipulation , and image synthesis .

Learning a 3D shape model with dense correspondences is a longstanding task in computer vision and graphics. However, existing works mostly focus on object classes with consistent geometric topologies such as human face and body . Shapes in these object categories can be pre-aligned for 3D model construction. Recent deep learning based approaches directly learn a latent space of 3D objects . Although these methods can model complex objects, they do not deal with dense correspondences between 3D shapes.

In this paper, we investigate learning model of 3D shapes and their dense correspondences for more generic objects such as cars and chairs. Compared to human face and body, these object classes exhibit much larger shape variations and structure changes, rendering correspondence construction extremely challenging. For these object categories, even human cannot reliably label the dense correspondences between two arbitrary shapes.

To achieve this goal, we act on recent advances in deep implicit fields, which have shown extraordinary power of representing complicated 3D geometry , and propose a novel Deformed Implicit Field (DIF) representation for joint shape latent space and dense correspondence learning. With DIF, a 3D shape is represented by a template implicit field, shared across the category, together with a 3D deformation field and a scalar correction field, dedicated for each shape instance. The output implicit field of a shape can be constructed by deforming the template implicit field and applying correction. The deformation field serves as a shape alignment function, with which dense correspondence between two shapes can be established by deforming their surfaces to the aligned 3D space. The correction field is introduced to handle structure variations among shapes.

We apply a neural network called DIF-Net to learn these fields together given a collection of shapes. To achieve unsupervised correspondence learning without any label, our key observation is that the normal direction of a shape point is highly correlated to its semantic information and very useful for correspondence reasoning. In light of this, we simply enforce the normals of two corresponding points connected by deformation to be close. In addition, we impose a spatial smoothness constrain on the deformation fields and enforce the correction fields to be minimal to make it only focus on dealing with structure variations. Thorough qualitative and quantitative evaluations show that our DIF-Net trained in this way can produce high quality correspondences. Moreover, correspondence uncertainty reflecting structure discrepancy between two shapes can be reliably measured by our method.

The contribution of this paper is as follows:

We propose DIF, a novel implicit field based 3D shape representation for shapes of an object category.

We propose DIF-Net, the first method devoted to 3D shape modeling with dense correspondences learned in an unsupervised fashion for objects with structure variation.

We show that our method can achieve high-quality dense correspondences and compelling texture transfer and shape editing results that cannot be achieved by previous methods. We believe our method can be applied in a wide range of 3D shape analysis and manipulation tasks.

Related Work

Building 3D shape model for a class of shapes has been actively studied in the past. Perhaps the most famous 3D shape model is the 3D morphable model (3DMM) introduced by Blanz et al. for human faces. To build a 3DMM model, face scans are aligned by shape registration methods to derive correspondences, based on which shape deformation bases can be obtained via PCA. The 3DMM model has brought a profound impact to human face related research . Apart from face, Loper et al. build a 3D morphable model for skinned human body which can control body shapes and poses. This model has been applied in various tasks such as pose estimation and image manipulation . Similarly, Zuffi et al. propose a morphable model for animals. The object categories handled by these methods typically have consistent topologies where shapes can be aligned to build correspondences. They do not address more complex object classes containing structure variations.

Learning Shape Latent Space.

A large volume of methods have been proposed in recent years to model 3D shapes and learn a latent shape space using deep neural networks, especially generative adversarial networks (GANs) and variational auto-encoders . However these methods do not explicitly model the dense correspondence among different shapes. Our method not only learns a shape latent space but also generates dense shape correspondence.

Implicit Shape Representation.

Recent studies show that learning implicit functions for 3D shapes excels at representing complicated geometry . For example, Park et al. use a neural network to approximate the signed distance field (SDF) of 3D shapes and show superior results compare to voxel and mesh based representations . Sitzmann et al. show that surfaces of complex scene can be represented by a simple 5-layer MLP with periodic functions as activation. However, these methods mainly target at high-fidelity surface reconstruction and cannot reveal shape correspondence. Genova et al. introduce an implicit template constructed with multiple RBF kernels. They can obtain a coarse dense correspondence between shapes by deforming and relocating RBF kernels to fit different shapes. However, their learning process is designed only for shape reconstruction thus the obtained correspondences are not reliable. Our representation in this paper is also based on implicit fields, but enables correspondence reasoning. Our new loss functions leads to high-quality correspondences learned without any label. A concurrent work from Zheng et al. also uses deformed implicit field for shape modeling. Nevertheless, they do not consider structure differences between shapes.

Structured Shape Representation.

Structured representations are also widely used to model complex shapes with varying structures . By decomposing 3D shapes into small parts, a complicated shape can be represented by primitive elements such as cuboids , superquadrics , convexes , and RBF kernels . Many of these methods can provide part-level correspondences among shapes, but do not model dense correspondences. Recently, a concurrent work also predicts dense correspondence for 3D shapes with structure variations based on the part-based implicit representation of . In contrast, our method does not divide a shape into parts. We obtain dense correspondence via shape deformation, whereas achieves this by using a part-aware embedding space.

Shape Deformation.

Shape deformation aims at deforming a shape to best fit a target while preserving local geometric details. Although our method also models shape deformation, our goal is to design a novel shape representation for surface reconstruction as well as correspondence reasoning. Besides, our implicit field based method can handle structure mismatch, which is problematic for previous mesh deformation methods.

Approach

with a neural network. We adopt the auto-decoder framework presented in to jointly learn the shape codes $\{\alpha_{j}\}$ for the given objects and the weights of model $f$. This auto-decoder framework can give rise to a decent latent space as shown in .

After training, new shapes can be generated by latent space sampling, and a shape can be embedded into the latent shape via inverse optimization.

where the magnitude of $s$ represents the distance from $p$ to its closest shape surface and the sign indicates whether $p$ is inside (negative) the shape or outside (positive). With an SDF, shape surface can be implicitly represented by the iso-surface of $SDF(\cdot)=0$. A 3D mesh can be extracted from this implicit surface using off-the-shelf algorithms such as Marching Cubes . Using SDF to represent shapes, our neural shape model can be rewritten as

Network Components.

The task in Eq. (3) intertwines shape information decoding from latent codes and SDF prediction for 3D points. Inspired by , we employ two networks to decompose this task: a DIF-Net for SDF prediction and a Hyper-Net for shape information decoding, as illustrated in Fig. 2. Hyper-Net $\Psi$ predicts the weights $\omega$ for the neurons in DIF-Net $\Phi$, and the two networks can be written as

Our DIF-Net $\Phi$ consists of two sub-networks: a template SDF generation network $T$ and a Deform-Net $D$. We will introduce these two sub-networks and our deformed implicit field representation in the next section.

2 Deformed Implicit Field Representation

For a given object class, we assume that the object instances are mostly composed by a few common patterns or semantic structures. This is a mild assumption valid for many real-world object classes. For example, all cars consist of bodies and tires, and most chairs have back, seat, and legs. We seek to find a template implicit field which depicts common structures of the class and can derive SDFs for different objects through 3D deformation and correction.

To capture common structures of an object category, we learn a template SDF generation network $T$:

which maps a 3D point $p$ to a scalar value $\widetilde{s}$. The latter is used to construct the SDF for a specific object via deformation and correction, which will be described later. The network weights of $T$ is shared across the whole class therefore it is enforced to learn common patterns within the class.

Note that although the SDF of an object can be derived from $T$, $T$ itself need not to be a valid SDF of any certain object. In fact, Figure 3 shows that the values of $T$ in the free space are “fused” SDF values of different shapes in the object category but not distances to its iso-surface $s=0$ (\eg, compare the lower part of chair’s template field slice and the extracted iso-surface at $s=0$). This makes our template field fundamentally different from previous mesh-based templates . Our template field records all structure variations in a category via its different iso-surfaces in the volume while a mesh-based template is only meaningful at its surface. This difference enables our method to learn better correspondence for objects with structure discrepancies, as we will show in the experiments.

Deformation Field and Correction Field.

To obtain the SDF for a certain object, we learn a Deform-Net $D$ to predict a deformation field as well as a correction field on top of the template field $T$:

The per-point offset $v$ deforms a point $p$ to the template space to obtain its SDF value via the template field $T$, which naturally induces dense correspondence between an object instance and the template field. Dense correspondences between two shapes can be further established by deforming their surface to the aligned template space and then searching nearest neighbors. The per-point correction $\Delta s$ modifies the assigned SDF value of $p$ if it still differs from the ground truth value. The correction field therefore helps to add or delete structures to enhance the shape representation ability, as illustrated in Fig. 4.

Note that the correction field is also crucial to learn reasonable correspondences under structure variations. Consider the chair example in Fig. 4, to generate the side stretcher which does not exist in the template surface using only deformation, some surface points must be deformed to the desired positions, necessitating a complex deformation field that is difficult to learn by network $D_{\omega}$. Even if such a deformation field can be learned, the correspondences derived from it for the stretcher part are wrong. In contrast, with a correction field, the stretcher structure can be added by simply altering the SDF values in this region rather than deforming existing surface points. This way, not only wrong correspondences can be avoided, but also a simple, smooth deformation field can be easily learned by the network. The ablation study in Sec. 4.3 also shows the effectiveness of our correction field.

In summary, our neural shape model can be written as

which is parameterized by the weights of network $\Psi$ and $T$.

3 Learning Shape and Correspondence with DIF

We use the auto-decoder framework presented in to jointly train weights of networks $\Psi$ and $T$ and learn latent codes $\{\alpha_{j}\}$. We design new loss functions for DIF to learn desirable dense correspondences.

Given a collection of shapes, we first apply an SDF regression loss similar to to learn the SDFs of these shapes. Let $\Phi_{i}(p)$ be the short-hand notation for $\Phi_{\Psi(\alpha_{i})}(p)$ which is the predicted SDF value, we have

where $\bar{s}$ and $\bar{n}$ denote the ground-truth SDF value and surface normal respectively, $\nabla$ denotes the spacial gradient of a 3D field, $\Omega$ is the 3D space and $\mathcal{S}_{i}$ denotes shape surfaces. In practice, points will be sampled in the free space and on shape surface to calculate the loss. The second term in Eq. (10) is used to learn correct normals on shape surfaces – the gradient function of an SDF equals the surface normal given surface points as input and can be easily computed using network backpropagration. The third term is derived from the Eikonal equation which enforces the norm of spatial gradients $\nabla\Phi_{i}$ to be $1$. The last term penalizes SDF values close to for non-surface points through $\rho({s})={\rm exp}(-\delta\cdot|s|),\delta\gg 1$. We refer the readers to for mode details about this loss. As in , we also apply a regularization loss to constrain the learned latent codes:

Alternatively, we can apply stronger regularization on the latent space akin to the VAE training scheme . More details and results can be found in the suppl. material.

To learn desired correspondences, our key observation is that the normal of a surface point is highly correlated with its semantic information. For example, normals on car hoods always point to the sky, and normals on the left doors always point to the left. In light of this, we encourage the normal directions of points in the template space to be consistent with their correspondences on all given shape instances:

where $\nabla T$ is the spatial gradient of template field $T$, and $\bar{n}$ denotes the ground-truth normal of point $p$ on object surface $\mathcal{S}_{i}$. Since the template field is used to derive all final shapes through deformation, this loss essentially enforces the normal consistency for correspondences across all the shapes generated by our network. Note that this loss is different with the normal term in Eq. (10): generating correct normals for each shape, as enforced by the latter, does not necessitate consistent correspondence normals between shapes, which is yet imposed by the former.

Deformation Smoothness Prior.

To encourage smooth deformation and avoid large shape distortion, we add a simple smoothness loss on the deformation field:

which penalizes the spatial gradient of the deformation field along $X$, $Y$ and $Z$ directions.

Minimal Correction Prior.

To encourage shape representation through implicit field deformation rather than correction, we minimize the correction field via

In summary, the whole training process can be formulated as the following optimization problem:

where $w$’s are the balancing weights for different loss terms.

4 Correspondence Uncertainty Measurement

In practice, it is desirable to have a quality or uncertainty metric for the obtained shape correspondences, which can be used for structure difference analysis, bad correspondence removal \etc. As mentioned previously, the correspondence between two objects $\mathcal{O}_{i}$ and $\mathcal{O}_{j}$ can be built by nearest neighbor search in the template space. Let $p_{i}$ be a point on $\mathcal{O}_{i}$ and $p_{j}$ its corresponding points found on $\mathcal{O}_{j}$, we propose a simple yet surprisingly-effective uncertainty metric based on their distance in the template space:

where $v_{i}=D^{v}_{\omega_{i}}(p_{i})$ is the deformation vector (similarly for $v_{j}$) and $\gamma$ is a scaling factor. The examples in Fig. 5 show that the regions with high uncertainty computed by Eq. (16) conform well to structure discrepancy between shapes.

Experiments

The Hyper-Net $\Psi$, Deform-Net $D$ and template field network $T$ are all implemented as MLPs. We train them on four categories in ShapeNet-V2 , including car, airplane, chair, and table. All parameters are trained end-to-end using the Adam optimizer. Training takes about 4 hours on 8 NVIDIA V100 GPUs with batchsize 256 for one category. See suppl. material for more details.

1 Ability of Shape Representation

To evaluate the representation power of DIF and the latent shape space learned by DIF-Net, we embed new shapes unseen in the training stage and measure the reconstruction accuracy. With our trained networks $\Psi$ and $T$, we embed a test shape to the latent space by solving the following simplified optimization problem of Eq. (15):

We compare with two state-of-the-art shape modeling methods based on deep implicit field: OccNet and DeepSDF . For OccNet, we train an individual model for each category using our training data for a fair comparison. For DeepSDF, we use a per-category model trained by to evaluate its performance.

Table 1 shows the shape reconstruction accuracy on 100 test shapes for each category, measured with chamfer distance (CD) and earth mover distance (EMD), and Fig. 6 visually compares some results. It can be seen that all three methods perform well in representing unseen shapes, and our method is slightly better in terms of numerical error.

We also compare with a variant of our method which does not model dense correspondence. Specifically, we replace DIF-Net $\Phi$ with a MLP of three hidden layers that directly predicts the SDF of a shape. The numerical results are presented in Table 1, which are slightly better than our DIF-Net. It indicates that the deformation-based implicit design only leads to moderate decease of representation capability. However, high-quality dense correspondences can be achieved with this design, as we will show later.

Due to space limitation, more evaluations of our trained DIF-Net including latent space interpolation, sampling and retrieval are deferred to the suppl. materials.

2 Learned Dense Correspondence

Figure 7 visualizes the correspondences generated by our method, where we manually paint salient color patterns on the shapes to better check the correspondence quality. Visually inspected, our method produces convincing correspondences across various shapes despite their structure differences. It not only correctly matches shared semantic components between two shapes but also preserves the original color patterns. Moreover, the uncertain regions revealed by our method well reflect structure differences between two shapes.

Quantitative Evaluation via Label Transfer.

To our knowledge, there is no dataset offering ground truth dense correspondence for objects with structure variation. Therefore, we resort to a semantic label transfer experiment for quantitative evaluation. We use the ShapeNet-Part dataset which contains part labels for ShapeNet objects. For each of the four object categories, we selected 5 labeled shapes as source shapes, and transfer their labels to other shapes leveraging dense correspondences. This task can be viewed as few-shot 3D shape segmentation learning using 5 samples as training data.

For this task, we first deform all labeled points on the 5 source shapes into our template 3D space, as shown in Fig. 8. For an unlabeled shape, we deform its surface points into the template 3D space, find 10 nearest labeled points for each, and then conduct simple label voting.

We compare our method with AtlasNet , AtlasNet-v2 , SIF , ShapeFlow , and DualSDF , which can be used to build correspondences, as well as a naive closest point based method. For fair comparison, we train AtlasNet, AtlasNet-v2, and SIF for each category individually with our training data. For Shapeflow and DualSDF, we use a per-category model trained by the original papers and use all shapes in the intersection of our training set and theirs as target shapes. More details and the visual results can be found in the suppl. material.

Table 2 compares the accuracy measured by IoU between ground-truth and transferred labels. Our method outperforms all others by a wide margin. AtlasNet, AtlasNet-v2, DualSDF, and SIF may generate inconsistent correspondences (\eg, a point labeled as chair “arm” corresponds to “arm” regions for some shapes but to “seat” for others) due to the limitation of their shape representations. Their results are worse than the naive closest point method for categories with large shape variation. ShapeFlow alleviates this problem by learning a volumetric deformation flow, but still suffers from inaccurate correspondence reasoning due to a mesh-based shape representation.

3 Ablation Study

In this section, we conduct ablation study to validate the efficacy of our training loss terms and the correction field. The main results of different settings are shown in Fig. 9. As can be seen, without the normal loss $L_{normal}$, the learned correspondences are inferior as indicated by the significant IoU drop in label transfer. Without the deformation smoothness loss $L_{smooth}$, the learned correspondences are highly distorted, as shown in the transferred textures. Without the correction field, shape representation ability of the model decreases significantly. Figure 9 shows that the correction field is also crucial to obtain high quality correspondences. Without the correction field, more complex deformations are needed to represent the final shape with various structures, which significantly increases the learning difficulty. Under this situation, learning may get stuck into local minima, leading to an inferior template field lacking rich structural information (as shown by the iso-surfaces in Fig. 9), which further results in correspondence accuracy drop.

Applications

Using dense correspondence generated by DIF-Net, we are able to transfer textures from one object to another. Successfully transferring rich textures among various shapes necessitates high quality dense correspondence. Figure 10 shows texture transfer results between ground truth shapes in ShapeNet. Visually inspected, the rich texture patterns are well preserved and transferred to correct semantic areas in new shapes. Figure ​​​Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence​​​ contains two more texture transfer results of our method for embedded 3D shapes.

2 Shape Editing

With the learned latent shape and dense correspondence, our method can be used to manipulate 3D shapes by moving one or a sparse set of points. Specifically, give a shape with embedded latent code $\alpha$, we can freely select one 3D point $p_{1}$ on the shape and specify its desired new position $p_{2}$. Let $p_{1}^{\prime}=p_{1}+D_{\Psi({\alpha})}^{v}(p_{1})$ be the deformed point of $p_{1}$ in the template 3D space, we achieve shape editing via solving for a new shape code $\hat{\alpha}$ minimizing the following equation:

where $v=D_{\Psi(\hat{\alpha})}^{v}(p_{2})$ is the deformation vector for $p_{2}$ with the new shape code $\hat{\alpha}$. In this equation, the first term enforces the original and new points on the shapes before and after editing to be a correspondence pair thus having same semantic meaning. The second term ensures the new point lies on the new shape surface. The third term requires the code change to be small. Figure 11 shows the editing results for two shapes and another two examples can be found in Fig. ​​​Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence​​​. We can even add new structures to a given shape via Eq. (18), where in this case we directly select $p_{1}^{\prime}$ in the template space and a free point $p_{2}$ in the shape space. An example is shown in Fig. 11. More details and results regarding shape editing can be found in the suppl. material.

Conclusion

We have presented Deformed Implicit Field, a novel implicit-based representation modeling a class of 3D shapes and providing dense correspondences. We also presented DIF-Net, a neural shape model that learns high-quality dense correspondences in an unsupervised manner through our proposed loss functions. Various experiments and applications collectively demonstrated the high quality shapes and correspondences generated by our method. In future, we plan to extend the DIF representation to handling more generic 3D objects and scenes.

References

Appendix A More Implementation Details

To train DIF-Net via the SDF regression loss $L_{sdf}$ defined in Eq. (10) in the main paper, we randomly sample points on shape surface and in the free space. Specifically, we follow a similar step as in to normalize each ground truth mesh into a sphere with radius of 1/1.03. Then, for surface points, we render 100 virtual images for each normalized mesh using 100 virtual cameras regularly sampled on the unit sphere. Surface points are obtained via back-projecting the depth pixels from these virtual images. Normals of these surface points are obtained in a similar way from virtual normal images. This sampling strategy helps us get rid of inner structures of each mesh that are invisible from outside (e.g car seats).

For free-space points, we uniformly sample them within a cube of $^{3}$, and calculate their distance to the nearest surface points sampled using the above strategy. To decide the sign of distance for a free-space point, we render it with the same virtual cameras used to obtain surface points, and check if its depth is smaller than the depth of a surface point falling into the same pixel. As long as it has a smaller depth value in any virtual image, it will be classified as an external point and get a positive sign. Otherwise, it will be classified as an internal point and get a negative sign.

In the end, we randomly sample 500K surface points along with normals and 500K free space points with their SDF values for each mesh.

Network Architecture.

Figure I shows the structures of Hyper-Net $\Psi$ and DIF-Net $\Phi$ used in our method. All networks are MLPs. The Hyper-Net consists of multiple MLPs each responsible for the weights $\omega_{i}$ of a single fully-connected layer $i$ in the Deform-Net $D_{\omega}$. The weights of Template Field $T$ are shared across the class. For Hyper-Net $\Psi$, we use ReLU activations. For Deform-Net $D_{\omega}$ and Template Field $T$, we use sine activations advocated by .

Training Details.

We train our model on four categories in ShapeNet-V2 , including car, airplane, chair, and table using 3K, 3.5K, 4K, and 4K instances as training set, respectively.

We jointly learn all latent codes $\{\alpha_{i}\}$, the Hyper-Net $\Psi$, and the Template Field $T$ using the losses in Eq. (15) in the main paper. Similar to , we initialize all latent codes using $\mathcal{N}(0,0.01^{2})$. The weights of Hyper-Net $\Psi$ are initialized using . The weights of Template Field $T$ are initialized as in .

We train an individual model on each category for 60 epochs. At each iteration, we randomly select 4K surface points and 4K free-space points for each shape in a batch to calculate the losses in Eq. (15) and update learnable parameters. During each epoch, there are in total 200K surface points and 200K free-space points used for each shape.

We set the balancing weights for four terms in the SDF regression loss $L_{sdf}$ in Eq. (10) to $3e3$, $1e2$, $5e1$, and $5e2$ respectively following . $w_{1}$ and $w_{4}$ are set to $1e2$ and $1e6$ respectively for all categories. For $w_{2}$, we set it to $5$, $2$, $5$, and $1$ for car, airplane, chair, and table respectively. For $w_{3}$, we set it to $1e2$, $1e2$, $5e1$, and $1e2$ for each of the above categories, respectively. Models for all categories are trained using an Adam optimizer with a learning rate of $1e-4$ and a batchsize of 256.

Inference Details.

At inference time, given a test shape, we obtain its latent code via optimizing Eq. (17). Weights for different loss terms and point sampling strategy are identical to the training phase. We use Adam optimizer with a learning rate of $1e-4$, and update the latent code for 30 epochs in total.

Appendix B Different Latent Code Regularization

In the main paper, we constrain the norm of learned latent code $\alpha$ to be small using $L_{reg}$ defined in Eq. (11). Alternatively, we can also constrain the posterior distribution of latent code to be close to a Gaussian distribution to learn a latent space for better sampling. Specifically, we can replace the $L_{reg}$ in Eq. (11) with the following loss:

where $KL$ denotes the Kullback–Leibler divergence and $q(\alpha_{i}|\mathcal{O}_{i})$ is the posterior distribution of $\alpha_{i}$ represented by a Gaussian distribution with mean equals to $\alpha_{i}$. To calculate $L_{reg^{\prime}}$ in Eq. (I), we have to obtain the standard deviation for $q(\alpha_{i}|\mathcal{O}_{i})$ as well. Therefore, we introduce an extra learnable latent code $\sigma$ with the same dimension of $\alpha$ to represent its standard deviation.

Specifically, at training stage, the input latent code $\alpha$ to the Hyper-Net $\Psi$ is replaced by a random variable $\widetilde{\alpha}$ sampled from $\mathcal{N}(\alpha,\sigma^{2}I)$. Then, we train all learnable parameters using the following losses:

where $L_{reg^{\prime}}$ is defined in Eq. (I) and all other losses are the same as in Eq. (15). In practice, we set the Gaussian distribution in Eq. (I) to $\mathcal{N}(0,0.01^{2})$, and set the weight $w_{5}$ to $1e2$. Other balancing weights are the same as described in the previous section.

Appendix C Learned Template Field

Figure II shows the learned template fields for four categories. Different structures of a category are recorded in different iso-surfaces of the learned template field.

Appendix D Learned Shape Latent Space

In this part, we show latent space interpolation results using the model trained with Eq. (15) in Fig. VI. Shapes in even positions are interpolated from their two neighbors. We also show the color-coded correspondence of them. As depicted, interpolated shapes are reasonable and correspondence between different shapes are consistent.

Latent Space Retrieval.

Given a source shape, we can search for its nearest neighbors using the Euclidean distance between shape latent codes as metrics. Retrieval results are shown in Fig. VII. Shapes on the right-hand side are nearest neighbors of the left-most shapes, and are sorted in an ascending distance order. Our learned latent space can capture the shape similarities in the original shape space—similar shapes are embeded close to each other.

Latent Space Sampling.

We show shape sampling results using the model trained with Eq. (II) in Fig. VIII. Our model can well capture the distribution of 3D shapes and generate new shapes.

Appendix E More Details of Label Transfer

Figure III shows the source shapes used to conduct the label transfer experiment. We manually select 5 sources for each category characterizing various structures in the shape distribution.

Closest Point.

In the naive closest point method, we label each point by direct label voting using its 10 nearest labeled points in the original shape space.

AtlasNet.

For AtlasNet, we train individual model for each category using two different settings: one using a sphere mesh as the template, dubbed Atlas-sph., and the other using 25 square meshes as the template, dubbed Atlas-25.

For each trained model, we first fit the template to all source shapes, and label each grid vertex on the template using label voting of its 5 nearest labeled points on 5 source shapes respectively. Then, given unlabeled points on a target shape, we fit the template to that shape, and label all points using label voting of 10 nearest grid vertices on the template.

AtlasNet-v2.

For AtlasNet-v2, we train individual model for each category using the Patch Deformation module with MLP adjustment. For trained models, we follow the same step as for AtlasNet to transfer labels to target shapes.

SIF.

We train SIF for each category using a template of 100 implicit kernels as in . To conduct label transfer, we follow to first calculate the template coordinates for all point on different shapes. The template coordinates has a dimension of 300 (100 kernels each with a 3-dimensional coordinates). Then, given an unlabeled point on the target shape, we find its 10 nearest labeled points in the template coordinate system and conduct label voting.

ShapeFlow.

We use the model trained on chair category provided by the authorshttps://github.com/maxjiang93/ShapeFlow.. To conduct label transfer, we deform surface points of source shapes and target shapes into the template space, find 10 nearest labeled source points for each target point, and then conduct label voting.

DualSDF.

We use models trained on plane and chair categories provided by the authorshttps://github.com/zekunhao1995/DualSDF.. For the label transfer task, we label each primitive using label voting of its 5 nearest labeled points on 5 source shapes respectively. Then, given an unlabeled point on a target shape, we assign it with the label of its closest primitive.

Qualitative Results.

Figure IV shows the visual results of label transfer task.

Appendix F More Details of Shape Editing

We achieve shape editing via Eq. (18) as described in the main paper. Given a shape with latent code $\alpha$, we initialize the variable $\hat{\alpha}$ to $\alpha$ at the beginning of the optimization. Balancing weight for three terms in Eq. (18) are set to $1$, $1$, and $5$ respectively. We use Adam optimizer with a learning rate of $1e-4$, and update $\hat{\alpha}$ with 1,000 iterations. The optimization process takes about 10 seconds.

Adding New Structures.

To add new structures to a given shape, we select $p_{1}^{\prime}$ defined in Eq. (18) in the template space, as is shown in Fig. V. Then, we follow the same optimization process to learn $\hat{\alpha}$ for the new shape as described in the above paragraph.

More Editing Results.

We show more shape editing results in Fig. IX. Our method is able to move selected points on the shape to desired position or add new structures. Features of the original shapes can be preserved after editing.