Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression

Introduction

Algorithms for automatic recovery of physical scene properties from an input image are of interest for many applications across computer vision and graphics; examples include material recognition and re-rendering. The intrinsic image model assumes that color image $I$ is the point-wise product of albedo $A$ and shading $S$:

Here, albedo is the physical reflectivity of surfaces in the scene. Considerable research focuses on automated recovery of $A$ and $S$ given as input only color image $I$ , or given $I$ and a depth map $D$ for the scene . Our work falls into the former category as we predict the decomposition using only color input. Yet, we outperform modern approaches that rely on color and depth input .

We achieve such results through a drastic departure from most traditional approaches to the intrinsic image problem. Many works attack this problem by incorporating strong physics-inspired priors. One expects albedo and material changes to be correlated, motivating priors such as piecewise constancy of albedo or sparseness of the set of unique albedo values in a scene . One also expects shading to vary smoothly over the image . Tang et al. explore generative learning of priors using deep belief networks. Though learning aligns with our philosophy, we take a discriminative approach.

Systems motivated by physical priors are usually formulated as optimization routines solving for a point-wise decomposition that satisfies Equation 1 and also fits with priors imposed over an extended spatial domain. Hence, graph-based inference algorithms and conditional random fields (CRFs) in particular are often used.

We forgo both physical modeling constraints and graph-based inference methods. Our direct intrinsics approach is purely data-driven and learns a convolutional regression which maps a color image input to its corresponding albedo and shading outputs. It is instantiated in the form of a multiscale fully convolutional neural network (Figure 1).

Key to enabling our direct intrinsics approach is availability of a large-scale dataset with example ground-truth albedo-shading decompositions. Unfortunately, collecting such ground-truth for real images is a challenging task as it requires full control over the lighting environment in which images are acquired. This is possible in a laboratory setting , but difficult for more realistic scenes.

The Intrinsic Images in the Wild (IIW) dataset attempts to circumvent the lack of training data through large-scale human labeling effort. However, its ground-truth is not in the form of actual decompositions, but only relative reflectance judgements over a sparse set of point pairs. These are human judgements rather than physical properties. They may be sufficient for training models with strong priors , or most recently, CNNs for replicating human judgements . But they are insufficient for data-driven learning of intrinsic image decompositions from scratch.

We circumvent the data availability roadblock by training on purely synthetic images and testing on both real and synthetic images. The MPI Sintel dataset provides photo-realistic rendered images and corresponding albedo-shading ground-truth derived from the underlying 3D models and art assets. These were first used as training data by Chen and Koltun for deriving a more accurate physics-based intrinsics model. Figure 2 shows examples.

Section 2 describes the details of our CNN architecture and learning objectives for direct intrinsics. Our design is motivated by recent work on using CNNs to recover depth and surface normal estimates from a single image . Section 3 provides experimental results and benchmarks on the Sintel dataset, and examines the portability of our model to real images. Section 4 concludes.

Direct Intrinsics

We break the full account of our system into specification of the CNN architecture, description of the training data, and details of the loss function during learning.

Intrinsic decomposition requires access to all the precise details of an image patch as well as overall gist of the entire scene. The multiscale model of Eigen and Fergus for predicting scene depth has these ingredients and we build upon their network architecture. In their two-scale network, they first extract global contextual information in a coarse subnetwork (scale $1$), and use that subnetwork’s output as an additional input to a finer-scale network (scale $2$). As Figure 3 shows, we adopt a Multiscale CNN Regression (MSCR) architecture with important differences from :

Instead of fully connected layers in scale $1$, we use a $1\times 1$ convolution layer following the upsampling layer. This choice enables our model to run on arbitrary-sized images in a fully convolutional fashion.

For nonlinear activations, we use Parametric Rectified Linear Units (PReLUs) . With PReLUs, a negative slope $a$ for each activation map channel appears as a learnable parameter:

where $x_{i}$ is pre-activation value at $i$-th dimension of a feature map. During experiments, we observe better convergence with PReLUs compared to ReLUs.

Our network has two outputs, albedo and shading (-a and -s in Figure 3), which it predicts simultaneously.

We optionally use deconvolution to learn to upsample the scale $2$ output to the resolution of the original images . Without deconvolution, we upsample an RGB output ($C^{\prime}=C=3$ in Figure 3 and the layer between uses fixed bilinear interpolation). With deconvolution, we set $C^{\prime}=64$ channels, $C=3$, and learn to upsample from a richer representation.

In addition to these basic changes, we explore a variant of our model, shown on the right side of Figure 3 that connects multiple layers of the scale $1$ subnetwork directly to that subnetwork’s output. The reasoning follows that of Maire et al. and Hariharan et al. , with the objective of directly capturing a representation of the input at multiple levels of abstraction. We adopt the ”hypercolumn“ (HC) terminology to designate this modification to MSCR.

The remaining architectural details are as follows. For convolutional layers $1$ through $5$ in the scale $1$ net, we take the common AlexNet design. Following those, we upsample the feature map to a quarter of the original image size, and feed it to a $1\times 1$ convolutional layer with $64$-dimensional output (conv6). Scale $2$ consists of $4$ convolutional layers for feature extraction followed by albedo and shading prediction. The first of these layers has $9\times 9$ filters and $96$ output maps. Subsequently, we concatenate output of the scale $1$ subnetwork and feed the result into the remaining convolutional and prediction layers, all of which use $5\times 5$ filters. The optional learned deconvolutional layer uses $8\times 8$ filters with stride $4$. Whether using deconvolution or simple upsampling, we evaluate our output on a grid of the same spatial resolution as the original image.

2 MPI Sintel Dataset

For training data, we follow Chen and Koltun and use the “clean pass” images of MPI Sintel dataset instead of their “final” images, which are the result of additional computer graphics tricks which distract from our application. This eliminates effects such as depth of field, motion blur, and fog. Ground-truth shading images are generated by rendering the scene with all elements assigned a constant grey albedo.

Some images contain defect pixels due to software rendering issues. We follow and do not use images with defects in evaluation. However, limited variation within the Sintel dataset is a concern for data-driven learning. Hence, we use defective images in training by masking out defective pixels (ignoring their contribution to training error).

3 MIT Intrinsic Image Dataset

To demonstrate adaptability of our model to the real world images, we use the MIT intrinsic image dataset . Images in this dataset are acquired with special apparatus, yielding ground-truth reflectance and shading components for real world objects. Here, reflectance is synonymous with our terminology of albedo.

Due to the limited scalability of the collection method, the MIT dataset contains only $20$ different objects, with each object having $11$ images from different light sources. Only $1$ image of $11$ has shading ground-truth. We generate each of $10$ shading images $S$ from a corresponding original image $I$ and reflectance image $A$ (identical for all the images because they are taken from the same object and the same camera settings) by element-wise division: $S=I^{\prime}/(\alpha A^{\prime})$, where $I^{\prime}$ and $A^{\prime}$ denote mean values of RGB channels of $I$ and $A$ respectively, and $\alpha$ is the value that minimizes the sum of squared error of $I-\alpha A\cdot S$.

For our models trained on MIT, we denote inclusion of these additional generated examples in training by appending the designation GenMIT to the model name. We find that some images in the MIT dataset do not exactly follow $I=\alpha A\cdot S$, but including these generated shadings still improves overall performance.

4 Data Synthesis: Matching Sintel to MIT

Even after generating shading images, the size of the MIT dataset is still small enough to be problematic for data-driven approaches. While we can simply train on Sintel and test on MIT, we observed some differences in dataset characteristics. Specifically, the rendering procedure generating Sintel ground-truth produces output that does not satisfy $I=\alpha A\cdot S$. In order to shift the Sintel training data into a domain more representative of real images, we resynthesized ground-truth $I$ from the ground-truth $A$ and $S$. In experiments, we denote this variant by ResynthSintel and find benefit from training with it when testing on MIT.

5 Data Augmentation

Throughout all experiments, we crop and mirror training images to generate additional training examples. We optionally utilize further data augmentation, denoted DA in experiments, consisting of scaling and rotating images.

6 Learning

Given an image $I$, we denote our dense prediction of albedo $A$ and shading $S$ maps as:

where $\Theta$ consists of all CNN parameters to be learned.

Since the intensity of our ground-truth albedo and shading is not absolute, imposing standard regression loss (L2 error) does not work. Hence, to learn $\Theta$, we use the scale invariant L2 loss described in . Let $Y^{\ast}$ be a ground-truth image in $\log$ space of either albedo or shading and $Y$ be a prediction map. By denoting $y=Y^{\ast}-Y$ as their difference, the scale invariant L2 loss is:

where $i,j$ are image coordinates, $c$ is the channel index (RGB) and $n$ is the number of evaluated pixels. $\lambda$ is a coefficient for balancing the scale invariant term: it is simply least square loss when $\lambda=0$, scale invariant loss when $\lambda=1$, and an average of the two when $\lambda=0.5$. We select $\lambda=0.5$ for training on MIT or Sintel separately, as it has been found to produce good absolute-scale predictions while slightly improving qualitative output . We select $\lambda=1$ for training on MIT and Sintel jointly, as the intensity scales from the two datasets differ and the generated images no longer preserve the original intensity scale. Note that $n$ is not necessarily equal to the number of image pixels because we ignore defective pixels in the training set.

6.2 Gradient L2 Loss

We also consider training with a loss that favors recovery of piecewise constant output. To do so, we use the gradient loss, which is an L2 error loss between the gradient of prediction and that of the ground-truth. By letting $\nabla_{i}$ and $\nabla_{j}$ be derivative operators in the $i$- and $j$-dimensions, respectively, of an image, the gradient L2 loss is:

Shading cannot be assumed piecewise constant; we do not use gradient loss for it. Our objective with gradient loss is:

We denote as MSCR+GL the version of our model using it.

6.3 Dropout

Though large compared to other datasets for intrinsic image decomposition, MPI Sintel, with $890$ examples, is still small compared to the large-scale datasets for image classification on which deep networks have seen success. We find it necessary to add additional regularization during training and employ dropout with probability $0.5$ for all convolutional layers except conv1 though conv5 in scale $1$.

7 Implementation Details

We implement our algorithms in the Caffe framework . We use stochastic gradient descent with random initialization and momentum of $0.9$ to optimize our networks. Learning rates for each layer are tuned by hand to get reasonable convergences. We train networks with batch size $32$ for $8000$ to $50000$ mini-batch iterations (depending on convergence speed and dataset). We randomly crop images at a size of $416\times 416$ pixels and mirror them horizontally. For additional data augmentation (DA), we also randomly rotate images in the range of [$-15$, $15$] degrees and zoom by a random factor in the range [$0.8$, $1.2$].

Due to the architecture of our scale $1$ subnetwork, our CNN may take as input any image whose width and height are each a multiple of $32$ pixels. For testing, we pad the images to fit this requirement and then crop the output map to the original input size.

Empirical Evaluation

MPI Sintel dataset: We use a total of $890$ images in the Sintel albedo/shading dataset, from $18$ scenes with $50$ frames each (one of the scenes has only $40$ frames). We use two-fold cross validation, that is, training on half of the images and testing on the remaining images, to obtain our test results on all $890$ images. Our training/testing split is a scene split, placing an entire scene (all images it contains) either completely in training or completely in testing. For comparison to prior work, we retrain with the less stringent historically-used image split of Chen and Koltun , which randomly assigns each image to the train/test set.

MIT-intrinsic image dataset: MIT has $20$ objects with $11$ different light source images, for $220$ images total. For MIT-intrinsic evaluation, we also split into two and use two-fold cross validation. Following best practices, we split the validation set by objects rather than images.

We adopt the same three error measures as :

MSE is the mean-squared error between albedo/shading results and their ground-truth. Following , we use scale-invariant measures when benchmarking intrinsics results; the absolute brightness of each image is adjusted to minimize the error.

LMSE is the local mean-squared error, which is the average of the scale-invariant MSE errors computed on overlapping square windows of size 10% of the image along its larger dimension.

On Sintel, we compare our model with two trivial decomposition baselines where either shading or albedo is assumed uniform grey, the classical Retinex algorithm ( version) which obtains intrinsics by thresholding gradients, and three state-of-the-art intrinsics approaches which use not only RGB image input but also depth input. Barron et al. estimate the most likely intrinsics using a shading rendering engine and learned priors on shapes and illuminations. Lee et al. estimate intrinsic image sequences from RGB+D video subject to additional shading and temporal constraints. Chen and Koltun use a refined shading model by decomposing it into direct irradiance, indirect irradiance, and a color component. On MIT, we compare with Barron et al. as well as the trivial baseline.

The top panel of Table 1 (image split case) shows that evaluated on Chen and Koltun’s test set, our MSCR+dropout+GL model significantly outperforms all competing methods according to MSE and LMSE. It is also overall better according to DSSIM than the current state-of-art method of Chen and Koltun: while our albedo DSSIM is $0.0054$ larger, our shading DSSIM is $0.0145$ smaller. Note that Chen and Koltun’s method utilizes depth information and is also trained on the DSSIM measure directly, whereas ours is based on the color image alone and is not trained to optimize the DSSIM score.

The bottom panel of Table 1 (scene-split case) is more indicative of an algorithm’s out-of-sample generalization performance; the test scenes have not been seen during training. These results show that: 1) The out-of-sample errors in the scene-split case are generally larger than the in-sample errors in the image-split case; 2) While HC has negligible effect, each tweak with dropout, gradient loss, learned deconvolutional layers, and data augmentation improves performance; 3) Training on Sintel and MIT together provides a small improvement when testing on Sintel.

Figure 2 shows sample results from our best model, while Figure 4 displays a side-by-side comparison with three other approaches. An important distinction is that our results are based on RGB alone, while the other approaches require both RGB and depth input. Across a diversity of scenes, any of the three RGB+D approaches could break down in one of the scenes on either albedo or shading: e.g. Lee et al.’s method on the bamboo scene, Barron et al.’s method on the dragon scene, Chen and Koltun’s method on the old man scene. The quality of our results is even across scenes and remains overall consistent with both albedo and shading ground-truth.

Table 2 shows that our model graciously adapts to real images. Trained on MIT alone, it produces reasonable results. Naively adding Sintel data to training hurts performance, but mixing our resynthesized version of Sintel into training results in noticeable improvements to albedo estimation when testing on MIT. The behavior of ablated system variants on MIT mirrors our findings on Sintel. On MIT, the learned deconvolutional layer is especially important. Output in Figure 5 exhibits clear visual degradation upon its removal. Figure 5 illustrates a tradeoff when using resynthesized Sintel training data: there is an overall benefit, but a Sintel-specific shading prior (bluish tint) leaks in.

In addition to Sintel and MIT, we briefly experiment with testing, but not training, our models on the IIW dataset . Here, performance is less than satisfactory (WHDR=$27.2$), compared to both our own prior work and the current state-of-the-art , which are trained specifically for IIW. We speculate that there could be some discrepancy between the tasks of predicting human reflectance judgements (WHDR metric) and physically-correct albedo-shading decompositions. As we observed when moving from Sintel to MIT, there could be a domain shift between Sintel/MIT and IIW for which we are not compensating. We leave these interesting issues for future work.

Conclusion

We propose direct intrinsics, a new intrinsic image decomposition approach that is not based on the physics of image formation or the statistics of shading and albedo priors, but learns the dual associations between the image and the albedo+shading components directly from training data.

We develop a two-level feed-forward CNN architecture based on a successful previous model for RGB to depth prediction, where the coarse level architecture predicts the global context and the finer network uses the output of the coarse network to predict the finer resolution result. Combined with well-designed loss functions, data augmentation, dropout, and deconvolution, we demonstrate that direct intrinsics outperforms state-of-the-art methods that rely not only on more complex priors and graph-based inference, but also on the additional input of scene depth.

Our data-driven learning approach is more flexible, generalizable, and easier to model. It only needs training data, requires no hand-designed features or representations, and can adapt to unrealistic illuminations and complex albedo, shape, and lighting patterns. Our model works with both synthetic and real images and can further improve on real images when augmenting training with synthetic examples.

Acknowledgments. We thank Ayan Chakrabarti for valuable discussion. We thank both Qifeng Chen and Jon Barron for providing their code and accompanying support.

References