SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator

Introduction

Geometric deep learning has led to a series of breakthroughs in a broad spectrum of problems ranging from biochemistry , physics to recommender systems . This method allows computational models that are composed of multiple layers to learn representations of irregular data structures, such as graphs and meshes. The majority of current works focus on the study of generic graphs , whereas it is still challenging to extract non-linear low-dimensional features from manifolds.

A path to ‘solving’ issues related to 3D computer vision then appears to be paved by defining intrinsic convolution operators. Attempts along this path started from formulating local intrinsic patches on meshes , and some other efforts exploit the similar idea of learning the filter weights between the nodes in a local graph neighborhood with utilizing pre-defined local pseudo-coordinate systems over the graphs.

Driven by the significance of the design of kernel weight function, a few questions arise: Is designing better weight function the vital part of learning representations of manifolds? Can we find more efficient convolution operators without introducing elusive kernel functions and pseudo-coordinates? It is somewhat intricate to answer if considering the problems defined on generic graphs with varied topologies. These problems, however, are possible to be addressed in terms of meshes, where data are generally aligned.

In this paper, we address these problems by introducing a simple operator, called SpiralNet++, which captures local geometric structure from serializing the local neighborhood of vertices. Instead of randomly generating sequences per epoch , SpiralNet++ generates spiral sequences only once in order to employ the prior knowledge of fixed meshes, which improves robustness. Since our approach explicitly encodes local information, the model is capable of efficiently learning discriminative features on 3D shapes. We further propose a dilated SpiralNet++ which allows to leverage neighborhoods at multiple scales to achieve detailed captures.

SpiralNet++ is fast, efficient, and easy to apply to various tasks in the domain of 3D computer vision. In our experiments, we bring this operator into three types of challenging problems, i.e., dense shape correspondence, 3D facial expression classification, and 3D shape reconstruction. Without relying on pre-processed shape descriptors or pseudo-coordinate systems, our approach outperforms the competitive baselines by a large margin in all the tasks.

Related Work

Geometric deep learning. Geometric deep learning began with attempts to generalize convolutional neural networks for data with an underlying structure that is non-Euclidean. It has been widely adopted to the tasks of graphs and 3D geometry, such as node classification , community detection , molecule prediction , mesh deformation prediction , protein interaction prediction .

Dense Shape Correspondence. We refer to related surveys on shape correspondence. Ovsjanikov et al. formulated a function correspondence problem to find a compact representation that could be used for point-to-point maps. Litany et al. took dense descriptor fields defined on two shapes as inputs and established a soft map between the two given objects, allowing end-to-end training. Masci et al. proposed to apply filters to local patches represented in geodesic polar coordinates. Boscaini et al. proposed the ACNN by using an anisotropic patch extraction method, exploiting the maximum curvature directions to orient patches. Monti et al. established a unified framework generalizing CNN architectures to non-euclidean domains. Verma et al. proposed a graph-convolution operator of dynamic correspondence between filter weights and neighboring nodes with arbitrary connectivity, which is computed from features learned by the network. Lim et al. firstly proposed SpiralNet and applied it on this task, which achieved highly competitive results. However, we observe that because spiral sequences are randomly generated at each epoch, the model is hard to converge and normally requires a larger sequence length as well as high dimensional shape descriptors as input. In order to solve these issues, we present SpiralNet++ that overcomes all of these drawbacks.

3D Facial Expression Classification. Facial expression recognition is a long-established computer vision problem with numerous datasets and methods having been proposed to address it. Cheng et al. proposed a high-resolution 4D facial expression dataset, 4DFAB, building a statistical learning model for static and dynamic expression recognition. In this paper, we are the first to introduce SpiralNet++ and other geometric deep learning methods into this task.

Shape Reconstruction. Shape reconstruction is a task that recreates the surface or creates another cross-section . Ranjan et al. proposed a convolutional mesh autoencoder (CoMA) based on ChebyNet and spatial pooling to generate 3D facial meshes. Bouritsas et al. then integrated the idea of spiral convolution into mesh autoencoder based on the architecture of CoMA, called Neural3DMM. In contrast to SpiralNet , they manually selected a reference vertex on the template mesh and defined the spiral sequence based on the shortest geodesic distance from the reference vertex. We argue that it is actually unnecessary to calculate specific spirals but only introducing redundant procedures, since under the assumption of meshes having the same topology, the spirals are already fixed and the same across all the meshes once defined. Additionally, to allow fixed-size spirals for explicit $k$-disk, they do zero-padding for the vertices that have a smaller spiral length than the average length of $k$-disk. Intuitively, vertices with a shorter spiral sequence than the average would decrease training efficiency of the weights applied on the concatenated feature vectors, since non-negligible zero paddings always have them not updated. In this paper, our approach addresses these deficiencies and shows the state-of-the-art performance on this task.

Our Approach

We assume the input domain is represented as a manifold triangle mesh $\mathcal{M}=(\mathcal{V},\mathcal{E},\mathcal{F})$, where $\mathcal{V},\mathcal{E},\mathcal{F}$ correspond to sets of vertices, edges and faces.

In contrast to previous approaches which aggregate neighboring node features based on trainable weight functions, our method encodes node features under a explicitly defined spiral sequence, and a fully connected layer follows to encode input features combined with ordering information. It is a simple yet efficient approach. In the following sections, we will elaborate on the definition of spiral sequence and the convolution operation in detail.

2 Spiral Sequence

We begin with the definition of spiral sequences, which is the core step of our proposed operator. Given a center vertex, the sequence can be quite naturally enumerated by intuitively following a spiral, as illustrated in Figure 2. The degrees of freedom are merely the orientation within each ring (clockwise or counter-clockwise) and the choice of the starting direction. We fix the orientation to counter-clockwise here and choose an arbitrary starting direction. The spirals are pre-computed only once.

We first define a $k$-ring and a $k$-disk around a center vertex $v$ as follows:

where $\mathcal{N}(V)$ is the set of all vertices adjacent to any vertex in set $V$.

It shows remarkable advantages to allow the model to learn a high-level feature representation in terms of each vertex in a consistent and robust way when we freeze spirals during training. Compared with SpiralNet , we credit the major improvement of our approach in terms of speed and efficiency to employing the nature of aligned meshes. Note that since we do not restrict spirals to the scope of a predefined number of rings, we are not involved in performance decays caused by introducing zero-padding . Furthermore, under the assumption of meshes having the same topology, the same vertex across meshes will always have the same spiral sequence regardless of the choice of starting direction, which eases the pain of manually defining the reference point and calculating the start point. By serializing the local neighborhood of vertices we are able to encode relevant information in a straightforward way with very little preprocessing.

3 Spiral Convolution

An euclidean CNN designs a two-dimensional kernel sliding on 2D images and maps $D$ input feature maps to $E$ output feature maps.

Thanks to the nature of the spiral serialization of neighboring nodes, we can define our spiral convolution in an equivalent manner to the euclidean CNNs, easing the pain of calculating the assignment value of $\mathbf{x}_{j}$ to the weight matrix. We define our spiral convolution operator for a node $i$ as

With the motivation of exponentially expanding the receptive field without losing resolution or coverage, we define dilated spiral convolution operators. Obviously, spiral convolution operators could immediately gain the power of capturing multi-scale contexts without increasing complexity from uniformly sampling the spiral sequence while keeping the same spiral length, as illustrated in Figure 2.

Experiments

In this section, we evaluate our method on three tasks, i.e., dense shape correspondence, 3D facial expression classification, and 3D shape reconstruction. We compare our method against FeaStNet , MoNet , ChebyNet and SpiralNet . To enable a fair comparison, the model architectures and the kernel size of different convolutions are the same and fixed, which yields the same level of parameterization. Furthermore, we use raw 3D coordinates as input node features instead of 3D shape descriptors as traditionally used for shape analysis. All the compared methods are with our implementation in order to enforce the same experimental setting except for Neural3DMM that we utilize their code directly. We train and evaluate each method on a single NVIDIA RTX 2080 Ti.

We validate our method on a collection of three-dimensional meshes solving the task of shape correspondence similar to . Shape correspondence refers to the task of labeling each node of a given shape to the corresponding node of a reference shape . We use the FAUST dataset , containing 10 scanned human shapes in 10 different poses, resulting in a total of 100 non-watertight meshes with 6,890 nodes each. The first 80 subjects in FAUST were used for training with the remaining 20 for testing.

As for all the experiments, we follow the network architecture of . It consists of the following sequence of linear layers (1x1 convolutions) and graph convolutions: Lin(16)$\rightarrow$Conv(32)$\rightarrow$Conv(64)$\rightarrow$Conv(128)$\rightarrow$Lin(256) $\rightarrow$Lin(6890), where the numbers indicate the amount of output channels of each layer. A non-linear activation function, ELU (exponential linear unit), is used after each Conv and the first linear layer. The kernel size or spiral length of all the Convs is 10.

The models are trained with the standard cross-entropy classification loss. We take Adam as the optimizer with the learning rate of 3e-3 (SpiralNet++, MoNet, ChebyNet), 1e-3 (SpiralNet), 1e-2 (FeaStNet), and dropout probability 0.5. As for input features we use the raw 3D XYZ vertice coordinates instead of 544 dimensional SHOT descriptors which was previously used in MoNet , SpiralNet .

Discussion.

In Table 1, we present the accuracy of the exact correspondence (with $0\%$ geodesic error) obtained by SpiralNet++ and other approaches. It shows that our method significantly outperforms all the baselines with 99.88% accuracy and it’s counterpart SpiralNet. It should be noted that our method enjoys an extremely fast speed with the training time of 0.98s per epoch in average, which owes to our method exploiting the essence of the fixed mesh topologies. From experiments, We also observed that SpiralNet generally requires around 2500 epochs to converge while it is sufficient for SpiralNet++ to converge within 100 epochs. In Figure 4, we plot the percentage of correspondences that are within a certain geodesic error. In Figure 3, it can be seen that most nodes are classified correctly with our method, which is much better than SpiralNet. Figure 1 visualizes the obtained correspondence using texture transfer.

2 3D Facial Expression Classification

As the second experiment, we address the problem of 3D facial expression classification using the 4DFAB dataset , which is a large scale dataset of high-resolution 3D faces. Previous efforts against this task focused on extracting low-dimensional features with PCA and LDA based on manually defined facial landmarks and a multi-class SVM was then employed to classify expressions . Similar to the deep convolutional neural networks used to classify the high-resolution images in the ImageNet , we develop an end-to-end hierarchical architecture with our method and other geometric deep learning approaches (e.g., ChebyConv , FeaStConv , MoNet ) to solve this 3D mesh classification problem. Following the experimental setup introduced in , we partition the data into 10 folds, and 17 distinct participants in testset are not shown in trainset (with 153 distinct participants). The number of each class is balanced in both training set and test set.

The models use a mesh pooling operation based on edge contraction . The pooling operation iteratively contracts vertex pairs to simplify meshes, while maintaining surface error approximation using quadric metrics. The output feature is then directly obtained by the multiplication of input feature with a downsampling transform matrix. We denote a pooling layer using this algorithm with Pool($c$), with $c$ being the downsampling factor.

Architectures and parameters.

We design the following end-to-end architecture to classify 3D facial expressions: Conv(16) $\rightarrow$ Pool(4) $\rightarrow$ Conv(16) $\rightarrow$ Pool(4) $\rightarrow$ FC(32) $\rightarrow$ FC(6). Dropout with a probability of 0.5 is used before each FC layer. We take a standard cross entropy loss function and ELU activation function. Training is done for 300 epochs with the learning rate of 1e-3, learning rate decay of 0.99 per epoch, L2 regularization of 5e-4, batch size of 32.

Discussion

All results of the 3D facial expression classification are shown in Table 2. It shows that with our proposed architecture, all of the graph convolution operations outperform the baseline . We credit these improvements to the capacity of learning intrinsic shape features compared to the baseline method. Specifically, our method achieves the highest recognition rate of 78.59% on average. This indicates that SpiralNet++ can be successfully applied to multi-scale mesh data improving previous results in this domain. Furthermore, it can be seen that our method is much more faster than all the other approaches.

3 3D Shape Reconstruction

As our largest experiment, we evaluate the effectiveness of SpiralNet++ on an extreme facial expression dataset. We demonstrate that a standard autoencoder architecture with SpiralNet++ allows the synthesis of high-fidelity 3D face with rich expression details. We use the dataset introduced in , which consists of 12 classes of extreme expressions, containing over 20,465 3D meshes, each with about 5,023 vertices and 14,995 edges. Following the interpolation experimental setup , we divide the dataset into training and test sets with a split ratio of 9:1. We compare our SpiralNet++ against a number of baselines including CoMA and Neural3DMM , and furthermore, for the first time, we bring MoNet and FeaStNet into this problem to explore the performance of other intrinsic convolution operations on generative models. It is worth highlighting that in the original work of CoMA , they used ChebyNet with $K=6$. However, in order to have a fair comparison with other experiments, we show both results obtained with $K=6$ (i.e., CoMA) and $K=9$. In the end, we evaluate our proposed dilated spiral convolution on this problem.

The performance of each generative model is closely related to the pooling and unpooling procedures. The same pooling strategy introduced in Section 4.2 is used here. In the unpooling stage, contracted vertices are recovered using the barycentric coordinates of the closet triangle in the decimated mesh .

Architectures and parameters.

We build a standard autoencoder architecture, consisting of an encoder and a decoder. The encoder includes several convolutional layers interleaved between pooling layers, and one fully connected layer is applied in the end of the encoder to encode non-linear mesh representations. Specifically, the structure is: 3 $\times$ {Conv(32)$\rightarrow$ Pool(4)} $\rightarrow$ {Conv(64) $\rightarrow$ Pool(4)} $\rightarrow$ FC(16), with ELU activation function after each Conv layer. The structure of the decoder is the reversed order of the encoder with the replacement of pooling layers to unpooling layers. Note that one more convolutional layer with the output dimensional of 3 should be added to the end of the decoder to reconstruct 3D shape coordinates. Training is done using Adam for 300 epochs with learning rate of 0.001, learning rate decay of 0.99 per epoch and a batch size of 32.

We evaluate all the methods with the same architecture and hyperparameters. The kernel size of each methods is set as 9 in order to keep aligned with Neural3DMM , where they chose 1-hop deriving the spiral length of 9.

Discussion.

Table 3 shows mean euclidean errors with standard deviations, median errors and the training time per epoch. Our SpiralNet++ and its dilated version outperform all the other approaches. The result of our proposed dilated spiral convolution validates our assumption, which shows the higher capacity of capturing non-linear low-dimensional representations of 3D shape meshes without increasing parameters. We credit this improvement to its larger receptive field brought by sampling larger input feature space. Moreover, we should stress the remarkable speed of our method. With the same autoencoder architecture, SpiralNet++ is a few times faster than all the other methods. It should be noted that the performance of Neural3DMM is even worse than CoMA when bring weight matrices to the same number, which can be attributed to the fact that model learning is disrupted from introducing non-negligible information (i.e., zero-padding). The performance of Neural3DMM would decrease with the variance of vertex degrees increase. Figure 5 shows the visualization of reconstructed faces in the test set. Larger errors can be seen from the faces generated by CoMA and Neural3DMM, and in particular, it become worse on the faces with extreme expressions. However, SpiralNet++ shows better reconstruction quality in these cases.

Conclusions

We explicitly introduce SpiralNet++ to the domain of 3D shape meshes, where data are generally aligned instead of varied topologies, which allows SpiralNet++ to efficiently fuse neighboring node features with local geometric structure information. We further apply this method to the tasks of dense shape correspondence, 3D facial expression classification and 3D shape reconstruction. Extensive experimental results show that our approach are faster and outperform competitive baselines in all the tasks.

Acknowledgements

SG and MB were supported in part by the ERC Consolidator Grant No.724228 (LEMAN), Google Faculty Research Awards, Amazon AWS Machine Learning Research grant, and the Royal Society Wolfson Research Merit award. SZ was partially supported by the EPSRC Fellowship DEFORM (EP/S010203/1) and a Google Faculty Award.

References