Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance

Introduction

Disentangling factors of variation is important for the broader goal of controlling and understanding deep networks, but also for applications such as image manipulation through interpretable operations. Progress in the direction of disentangling the latent space of deep generative models has facilitated the separation of latent image representations into dimensions that account for independent factors of variation, such as identity, illumination, normals, and spatial support , low-dimensional transformations, such as rotations, translation, or scaling, or finer-levels of variation, including age, gender, wearing glasses, or other attributes e.g. for particular classes, such as faces.

Shape variation is more challenging as it amounts to a transformation of a function’s domain, rather than its values. Even simple, supervised additive models of shape result in complex nonlinear optimization problems . Despite this challenge several works in the previous decade aimed at learning shape/appearance factorizations in an unsupervised manner, exploring groupwise image alignment, . In the context of deep learning several works have aimed at incorporating deformations and alignment in a supervised setting, including Spatial Transformers , Deep Epitomic Networks , Deformable CNNs , Mass Displacement Networks , Mnemonic Descent , or Densereg . These works have shown that one can improve the accuracy of both classification and localization tasks by injecting deformations and alignment within traditional CNN architectures.

Turning to unsupervised deep learning, even though most works focus on rigid, or low-dimensional parametric deformations, e.g. , several works have attempted to incorporate richer non-rigid deformations within learning. A thread of works has been aimed at dynamically rerouting the processing of information within the network’s graph based on the input, starting from neural computation arguments and eventually translating into concrete algorithms, such as the ‘capsule’ works of that bind neurons on-the-fly. Still, these works lack a transparent, parametric handling of non-rigid deformations. Working on a more geometric direction, several works have recently aimed at recovering dense correspondences between pairs or sets of RGB images, as e.g. in the recent works of . These works however do not have the notion of a reference coordinate system (‘template’) to which images can get mapped - this makes the image generation and manipulation harder. More recently, use the equivariance principle in order to align sets of images to a common coordinate system, but do not develop this into a full-blown generative model of images.

Our work pushes the envelope of this line of research by following the deformable template paradigm . In particular, we consider that object instances are obtained by deforming a prototypical object, or ‘template’, through dense, diffeomorphic deformation fields. This makes it possible to factor object variability within a category into variations that are associated to spatial transformations, generally linked to the object’s 2D/3D shape, and variations that are associated to appearance (or, ‘texture’ in graphics), e.g. due to facial hair, skin color, or illumination. In particular we consider that both sources of variation can be modelled in terms of a low-dimensional latent code that is learnable in an unsupervised manner from images. We achieve disentangling by breaking this latent code into separate parts that are fed into separate decoder networks that deliver appearance and deformation estimates. Even though one could hope that a generic convolutional architecture will learn to represent such effects, we argue that explicitly injecting this inductive bias in a network can help with the training, while also yielding control over the generative process.

Our main contributions in this work can be summarized as follows:

First, we introduce the Deforming Autoencoder architecture, bringing together the deformable modeling paradigm with unsupervised deep learning. We treat the template-to-image correspondence task as that of predicting a smooth and invertible transformation. As shown in Fig. 1, our network predicts this transformation field alongside with the template-aligned appearance and subsequently deforms the synthesized appearance to generate an image similar to its input. This allows for a disentanglement of the shape and appearance parts of image generation by explicitly modelling the effects of image deformation during the decoding stage.

Second, we explore different ways in which deformations can be represented and predicted by the decoder. Instead of building a generic deformation model, we compose a global, affine deformation field, with a non-rigid field that is synthesized as a convolutional decoder network. We develop a method that allows us to constrain the synthesized field to be a diffeomorphism, namely an invertible and smooth transformation, and show that it simplifies training and improves accuracy. We also show that class-related information can be exploited, when available, to learn better deformation models: this yields sharper images and can be used to learn models that jointly account for multiple classes - e.g. all MNIST digits.

Third, we show that disentangling appearance from deformation comes with several advantages when it comes to modeling and manipulating images. By using disentangling we obtain clearly better synthesis results when manipulating images for tasks such as expression, pose or identity interpolation when compared to standard autoencoder architectures. Along the same lines, we show that accounting for deformations facilitates a further disentangling of the appearance components into an intrinsic, shading-albedo decomposition which completely fails when naively performed in the original image coordinates. This allows us to perform re-shading through simple operations on the latent shading coordinate space.

We complement these qualitative results with a quantitative analysis of the learned model in terms of landmark localization accuracy. We show that our method is not too far below supervised methods and outperforms with a margin the latest state-of-the-art works on self-supervised correspondence estimation , even though we never explicitly trained our network for correspondence estimation, but rather only aimed at reconstructing pixel intensities.

Deforming Autoencoders

Our architecture embodies the deformable template paradigm in an autoencoder architecture. The premise of our work is that image generation can be interpreted as the combination of two processes: a synthesis of appearance on a deformation-free coordinate system (‘template’), followed by a subsequent deformation that introduces shape variability. Denoting by $T(\mathbf{p})$ the value of the synthesized appearance (or, texture) at coordinate $\mathbf{p}=(x,y)$ and by $W(\mathbf{p})$ the estimated deformation field, we consider that the observed image, $I(\mathbf{p})$, can be reconstructed as follows:

namely the image appearance at position $\mathbf{p}$ is obtained by looking up the synthesized appearance at position $W(\mathbf{p})$. This is implemented in terms of a spatial transformer layer that allows us to pass gradients through the warping process.

The appearance and deformation functions are synthesized by independent decoder networks. The inputs to the decoders are delivered by a joint encoder network that takes as input the observed image and delivers a low-dimensional latent representation, $Z$, of shape and appearance. This is split into two parts, $Z=[Z_{T},Z_{S}]$ which feed into the appearance and shape networks respectively, providing us with a clear separation of shape and appearance.

Rather than leave deformation modeling entirely to back-propagation, we use some domain knowledge to simplify and accelerate learning. The first observation is that global aspects can be expressed using low-dimensional linear models. We account for global deformations by an affine Spatial Transformer layer, that uses a six-dimensional input to synthesize a deformation field as an expansion on a fixed basis . This means that the shape representation, $Z_{S}$ described above is decomposed into two parts, $Z_{W},Z_{A}$, where $Z_{A}$ accounts for the affine, and $Z_{W}$ for the non-rigid, learned part of the deformation field. These deformation fields are generated by separate decoders, and are composed, so that the affine transformation warps the detailed non-rigid warps to the image positions where they should apply. This is also a common decomposition in deformable models for faces .

Turning to local deformation effects, we quickly realized that not every deformation field is plausible. Without appropriate regularization we would often obtain deformation fields that could expand small areas to occupy whole regions, and/or would be non-diffeomorphic, meaning that the deformation could spread a connected texture pattern to a disconnected image area (Figure 2-(f)).

To prevent this problem, instead of making the shape decoder CNN directly predict the local warping field $W(\mathbf{p})=(W_{x}(x,y),W_{y}(x,y))$, we consider a ‘differential decoder’ that generates the spatial gradient of the warping field: $\nabla_{x}W_{x}$ and $\nabla_{y}W_{y}$, where $\nabla_{c}$ denotes the $c-th$ component of the spatial gradient vector. These two quantities measure the displacement of consecutive pixels - for instance $\nabla_{x}W_{x}=1$ amounts to translation in the horizontal axis, $\nabla_{x}W_{x}=2$ amounts to horizontal shifting by a size of 2, while $\nabla_{x}W_{x}=-1$ amounts to left-right flipping; a similar behavior is associated with $\nabla_{y}W_{y}$ in the vertical axis. We note that global rotations are handled by the affine warping field, and the $\nabla_{x}W_{y},\nabla_{y}W_{x}$ are associated with small local rotations of minor importance - we therefore focus on $\nabla_{x}W_{x},\nabla_{y}W_{y}$.

Having access to these two values gives us a handle on the deformation field, since we can prevent folding/excessive stretching by controlling $\nabla_{x}W_{x},\nabla_{y}W_{y}$.

2 Class-aware Deforming Autoencoder

We can require our network’s latent representation to be predictive of not only shape and appearance, but also of instance class, if that is available during training. We note that this information, being discrete may be easier to acquire than the actual deformation field, which would require manual landmark annotation. For instance, for faces such discrete information could represent the expression or a person’s identity.

In particular we consider that the latent representation can be decomposed as follows: $Z=[Z_{T},Z_{C},Z_{S}]$, where $Z_{T},Z_{S}$ are as previously the appearance- and shape- related parts of the representation, respectively, while $Z_{C}$ is fed as input to a sub-network trained to predict the class associated with the input image. Apart from assisting the classification task, the latent vector $Z_{C}$ is fed into both the appearance and shape decoders. Intuitively this allows our decoder network to learn a mixture model that is conditioned on class information, rather than treating the joint, multi-modal distribution through a monolithic model. Even though the class label is only used during training, and not for reconstruction, our experimental results show that a network trained with class supervision can deliver more accurate synthesis results.

3 Intrinsic Deforming Autoencoder: Deformation, Albedo and Shading Decomposition

Having outlined Deforming Autoencoders, we now use a Deforming Autoencoder to model complex physical image signals, such as illumination effects, without a supervision signal. For this we design the Intrinsic Deforming-Autoencoder, named Intrinsic-DAE to model shading and albedo for in-the-wild face images. As shown in Fig. 4-(a), we introduce two separate decoders for shading $S$ and albedo $A$, each of with has the same structure as the original texture decoder. The texture is computed by $T=S\circ A$ where $\circ$ denotes the Hadamard product.

In order to model the physical properties of shading and albedo, we follow the intrinsic decomposition regularization loss used in : we apply the L2 smoothness loss on $\nabla S$, meaning that shading is expected to be smooth, while leaving albedo unconstrained. As shown in Fig. 4 and more extensively in the experimental results section, when used in tandem with an Deforming Autoencoder this allows us to successfully decompose of face image into shape, albedo, and shading components, while a standard Autoencoder completely fails at decomposing unaligned images into shading and albedo.

4 Training

Our objective function is formed as the sum of three losses, combining the reconstruction error with the regularization terms required for the modules described above. Concretely, the loss of the deforming autoencoder can be written as

and the warping loss is decomposed as follows:

In particular the smoothness cost, $E_{\text{smooth}}$, penalizes quickly-changing deformations encoded by the local warping field. It is measured in terms of the total variation norm of the horizontal and vertical differential warping fields, and is given by

where $\lambda_{1}=1e-6$. Finally, $E_{\text{BiasReduce}}$ aims at removing any systematic bias introduced by the fitting process, e.g. the average template becoming small, or a distorted version of the data. It consists of regularization on (1) the affine parameters defined as the L2-distance between $S_{A}$ and $S_{0}$, with $S_{0}$ being the identity affine transform and (2) on free-form deformations defined as the L2-distance between the average deformation grid within a minibatch, $\bar{W}$ and the identity grid $W_{0}$:

where $\lambda_{2}=\lambda_{2}^{\prime}=0.01$.

In the class-aware variant described in Sec. 2.2 we augment the loss above with the cross-entropy loss evaluated on the classification network’s outputs, while for Intrinsic-DAE, we add the following objective function in training: $E_{\text{Shade}}=\lambda_{3}\|\nabla S\|^{2}$ where $\lambda_{3}=$1e-6.

We experiment with two types of architectures; the majority of our results are obtained with a standard auto-encoder architecture, where both encoder and decoders are CNNs with standard convolution-BatchNorm-ReLU blocks. The number of filters and the texture bottleneck capacity can vary per experiment, image resolution, and dataset, as detailed in the Appendix 0.A.

Follow the recent work on densely connected convolutional networks , we have also experimented with incorporating dense connections into our encoder and decoders architectures respectively (no skip connections over the bottleneck layer for latent representations). In particular, we follow the architecture of DenseNet-121, but without the $1\times 1$ convolutional layers inside each dense block. These have been shown to better exploit larger datasets, as indicated in the quantitative analysis of unsupervised face alignment. We call this version of the deforming autoencoder Dense-DAE.

Experiments

To demonstrate the properties of our deformation disentangling network, we conduct experiments on the following three datasets:

Deformed MNIST. A synthetic dataset designed specifically to explore the deformation modelling power of our network. Deformed MNIST consists of handwritten MNIST images randomly distorted using a mixture of sinusoidal waveforms.

MUG facial expression dataset . This dataset consists of videos of individuals performing facial expressions, with simple blue background and minor translation. The dataset also offers frames from the videos, classified according to the facial expression, as well as the subject.

Faces-in-the-wild dataset: MAFL and CelebA . These datasets consist of uncontrolled “in-the-wild” faces with variability in pose, illumination, expression, age, etc.

Using these datasets we experimentally explored the ability of the unsupervised appearance-shape (or texture-deformation) disentangling network on 1) unsupervised image alignment/appearance inference; 2) learning semantically meaningful manifolds for shape and appearance; 3) decomposition into illumination intrinsics (shading, albedo); 4) unsupervised landmark detection, as detailed below. We intend to make all of the code of our system publicly available in order to facilitate the reproduction of our results.

We first use our network to model canonical appearance and deformation for single category objects. For this purpose, we demonstrate the results in the MNIST and MUG facial expression datasets (Fig. 5, 6, 7).

We observe that by heavily limiting the size of $Z_{T}$ (1 in Fig. 5 and 0 in Fig. 7), we can successfully infer a canonical appearance for such a class. In Fig. 5, all different types of handwritten digits ’3’ are aligned to a simple canonical shape. In Fig. 7, by limiting the dimension of $Z_{T}$ to , the network learns to encode a single texture image for all expressions, and successfully distills expression-related information exclusively in the shape space. In Fig. 7-(b) we show that by interpolating the learned latent representations, we can generate meaningful shape interpolations that mimic facial expressions.

In cases where data has a multi-modal distribution exhibiting multiple different canonical appearances, e.g., multi-class MNIST digit images, learning a single appearance is less meaningful and often challenging (Fig. 6-(b)). In such cases, utilizing class information (Sec. 2.2) significantly improves the quality of multi-modal appearance learning (Fig. 6-(d)). As the network learns to classify the images implicitly in its latent space, it learns to generate a single canonical appearance for each class. Misclassified data will be decoded into an incorrect class: the image at position (2,4) in Fig. 6-(c,d) is interpreted as a 6.

We now demonstrate the effectiveness of texture inference using our network on in-the-wild human faces. Using the MAFL face dataset, we show that our network is able to align the faces to a common texture space under various poses, illumination conditions, or facial expressions (Fig. 10)-(d). The aligned textures retain the information of the input image such as lighting, gender, and facial hair, without a relevant supervision training signal. We further demonstrate the alignment on the 11k Hands dataset , where we align palmar images of the left hand of several subjects 8. This property of our network is especially useful for applications such as computer graphics, where establishing correspondences (UV map) between a class of objects is important but usually difficult.

2 Autoencoders vs. Deforming Autoencoders

We show the ability of our network to learn meaningful deformation representations without supervision. We compare our disentangling network with a plain auto-encoder (Fig. 9). Contrary to our network which disentangles an image into a template texture and a deformation field, the auto-encoder is trained to encode all of the image in a single latent representation, i.e., the bottleneck.

We train both networks in the MAFL faces-in-the-wild dataset. To evaluate the learned representation, we conduct manifold traversal (i.e., latent representation interpolation) between two randomly sampled face images: given a source face image $I^{s}$ and a target image $I^{t}$, we first compute their latent representations $Z$s. We use $Z_{T}(I^{s})$ and $Z_{S}(I^{s})$ to denote the latent representations in our network for $I^{s}$, and $Z_{ae}(I^{s})$ for the latent representation learned by a plain autoencoder. We then conduct linear interpolation on $Z$, between $Z^{s}$ and $Z^{t}$: $Z^{\lambda}=\lambda Z^{s}+(1-\lambda)Z^{t}$. We subsequently reconstruct the image $I^{\lambda}$ from $Z^{\lambda}$ using the corresponding decoder(s), as shown in Fig. 9.

By traversing the learned deformation representation only, we can change the shape and pose of a face while maintaining its texture (Fig. 9-(1)); interpolating the texture representation results in pose-aligned texture transfer (Fig. 9-(2)); traversing on both representations will generate a smooth deformation from one image to another (Fig. 9-(3,5,7)). Compared to the interpolation using the autoencoder (Fig. 9-(4,6,8)), which often exhibits artifacts, our traversal stays on the semantic manifold of faces and generates sharp facial features.

3 Intrinsic Deforming Autoencoders

Having demonstrated the disentanglement abilities of Deforming Autoencoders, we now explore the disentanglement capabilities of Intrinsic-DAE described in Sec. 2.3. Using only the $E_{\text{DA}}$ and regularization losses, the Intrinsic-DAE is able to generate convincing shading and albedo estimates without direct supervision (Fig. 10-(b) to (g)). Without the “learning-to-align” property, a baseline autoencoder structure with an intrinsic decomposition design (Fig. 4-(b)) cannot decompose the image into plausible shading and albedo components (Fig. 10-(h),(i),(j)).

In addition, we show that by manipulating the learned latent representation of $S$, Intrinsic-DAE allows us to simulate illumination effects for face images, such as interpolating lighting directions (Fig. 11).

Training with $L2$ reconstruction losses, autoencoder-like architectures are prone to generating smooth images which lack visual realism (Fig. 10). Inspired by the success of generative adversarial networks (GANs) , we follow previous work where an adversarial loss is adopted to generate visually realistic images: we train the Intrinsic-DAE with an extra adversarial loss term $E_{\text{Adversarial}}$ applied on the final output. The loss function becomes:

In practice, we apply a PatchGAN as the discriminator and set $\lambda_{4}=0.1$. We found that the adversarial loss improves the visual sharpness of the reconstruction while the deformation, shading are still successfully disentangled (Fig. 12).

4 Unsupervised alignment evaluation

Having qualitatively analyzed the disentanglement capabilities of our networks, we now turn to quantifying their performance on the task of unsupervised image alignment. We report the performance of our face DAE’s alignment on landmark detection on face images, specifically, the eyes, the nose, and corners of the mouth. We report performance on the MAFL dataset, which contains manually annotated landmark locations for 19,000 training and 1,000 test images. In our experiments, we use a model trained on the CelebA dataset without any form of supervision to estimate deformation fields on the MAFL training set. Following the evaluation protocol of the work that we directly compare to , we train a landmark regressor post-hoc on these deformation fields using the provided annotations. We use landmark locations from the MAFL training set as training data for this regressor, but do not pass gradients to the Deforming Autoencoder, which thereby remains fixed to the model learned without supervision. The regressor is a 2-layer fully-connected neural network. Its inputs are flattened deformation fields (vectors of size $64\times 64\times 2$), which are provided as input to a 100-dimensional hidden layer, followed by a ReLU and a 10-D output layer to predict the spatial coordinates ($(x,y)$) for five landmarks corresponding to the eyes, nose, and mouth corner landmarks. We use L1 loss as the objective function for this regression task.

In testing, we predict landmark locations using the trained regressor and the deformation fields on the MAFL test set. In Table 1 we report the mean error in landmark localization as a percentage of the inter-ocular distance. As the deformation field determines the alignment in the texture space, it serves as an effective mapping between landmark locations on the aligned texture and those on the original, unaligned faces. Hence, the mean error we report directly quantifies the quality of the (unsupervised) face alignment.

In Table 2 we compare with the results of the best current method for semi-supervised image registration . We observe that by better modeling of the deformation space we quickly bridge the gap in performance, even though we never explicitly trained to learn correspondences.

Conclusion and Future Work

In this paper we have developed deep autoencoders that can disentangle shape and appearance in latent representation space. We have shown that this method can be used for unsupervised groupwise image alignment. Our experiments with expression morphing in humans, image manipulation, such as shape and appearance interpolation, as well as unsupervised landmark localization, show the generality of our approach. We have shown that bringing images in a canonical coordinate system allows for a more extensive form of image disentangling, facilitating the estimation of decompositions into shape, albedo and shading without any form of supervision. We expect that this will lead in the future to a full-fledged disentanglement into normals, illumination, and 3D geometry.

Acknowledgment

This work was supported by a gift from Adobe, NSF grants CNS-1718014 and DMS 1737876, the Partner University Fund, and the SUNY2020 Infrastructure Transportation Security Center.

References

Appendix 0.A Architectural Details

In our experiments, where input images are of size $64\times 64\times\text{Nc}$ (Nc is 1 for MNIST and 3 for faces), we use identical architectures for convolutional encoders and decoders.

Conv(n): convolution layer with $n$ output feature map;

ConvT(n): transposed convolution (deconvolution) layer with $n$ output feature map;

A.2 DenseNet-style Encoders and Decoders

For DenseNet-style architectures, we employ dense convolutional connections. The architecture for the encoder is

whereas the architecture for the decoder is

DBE(n,k): A dense encoder block with $k$ $3\times 3$ convolutions with $n$ channels.

TBE(m,n,p): An encoder transition block of $1\times 1$ convolutions with $m$ input channels and $n$ output channels. Also includes a max-pooling operation of size $p$.

DBD(n,k): A dense decoder block with $k$ $3\times 3$ transposed convolution operations with $n$ channels.

TBD(m,n): A decoder transition block of $4\times 4$ convolutions, stride of $2$ and padding of $1$. It has $m$ input channels, and $n$ output channels.

We describe the tensor sizes for intermediate convolution operations in Tables 3 and 4.

Appendix 0.B Ablation Study

In this section, we show experimental results on single deformed MNIST images of the digit 3 (Figure 14) as well as in-the-wild faces (without masking) from the MAFL dataset (Figure 15) to demonstrate the effect of varying the dimension of $Z_{T}$.

B.2 Methods for deformation modeling

In this section, we demonstrate the effect of using different warping modules.

We first show additional comparisons between using our proposed affine + integral warping and a non-rigid warping field directly output from a convolutional decoder for non-rigid deformation modeling (Figure 16).

We visualize the utility of affine and integral warping modules in our network with face images (Figure 17). We can see that the affine transformation handles global pose variance (Figure. 17-(b)) but not local non-rigid deformation. Our proposed integral warping module aligns the faces in a non-rigid manner (Figure 17-(c)). Incorporating both deformation modules improves the non-rigid alignment (Figure 17-(d)).

Appendix 0.C Latent Manifold Traversal

We provide additional results and comparisons with a plain autoencoder on traversing the learned manifolds. In addition to Figure 13 in our manuscript, we provide two more sets of results in Figure 18 and Figure 19. Compared to a plain autoencoder, our deforming autoencoder not only generates better reconstructions, but also learns a better face manifold - interpolating between learned latent representations generates sharper and more realistic face images. For this experiment, we use the convolutional encoder and decoder architecture as described in Sec. 0.A.1.

Appendix 0.D Intrinsic Decomposition with DAE

In Fig.21 we provide additional results of unsupervised intrinsic disentangling for faces-in-the-wild using Intrinsic-DAE. Using the architecture and objective functions described in Sec. 2.3 of the main paper the network learns to bring faces under different poses and illumination conditions, shown in Fig. 21-(a), to a canonical view, as shown in Fig. 21-(d), while separating the shading, shown in Fig. 21-(b) and albedo, shown in Fig. 21-(c) components in the canonical view using two independent decoders. With the learned deformation from the deformation decoder, we can warp the aligned shading and aligned albedo to its original view as in the input image, as shown in Fig. 21-(e,f).

In Fig. 22, we provide additional results for “changing lighting direction” of a face image using Intinsic-DAE. We show that even without explicitly modeling of geometry, we can simulate smooth and reasonable lighting direction changes in the image by interpolating the learned latent representation for shading, as shown in Fig. 22-a-(4),b-(4).

For Intrinsic-DAE, we use the DenseNet architecture as the encoders and decoders (0.A.2). The network is trained with a subset of $200,000$ images in the CelebA dataset. The dimensions of latent representations are: 16 for albedo, 16 for shading, and 128 for deformation field.