Preserving Color in Neural Artistic Style Transfer

Introduction

The recent neural artistic style algorithm takes as input two images — a style image and a content image — and outputs a new image depicting the objects of the content image in the style of the other image. For example, Figure 1 shows a photograph of a farmhouse garden as the content image, and a painting by Vincent van Gogh as the style image. Applying the algorithm produces a new painting of the farmhouse in the style of the van Gogh painting. This method works by matching statistics on the feature responses in a Convolutional Neural Network trained on object recognition; for a detailed description of the algorithm see . As illustrated in Figures 1 and 2, the output reproduces the style of brushstrokes, geometric shapes, and painterly structures exhibited in the style image. However, it also copies the color distribution of the style image which might be undesirable in many cases (Fig. 1(c), 2(c)). For example, the painting of the farmhouse has the colors of the original van Gogh painting (Fig. 1(c)), whereas one might prefer the output painting to preserve the colors of the farmhouse photograph. In particular, one might also imagine that the artist would have used the colors of the scene if they were to paint the farmhouse.

This note presents two simple methods to preserve the colors of the source image during neural style transfer—in other words, to transfer the style without transferring the colors. We compare two different approaches to color preservation: color histogram matching and luminance-only transfer (Fig. 1(d,e), 2(d,e)). We compare these methods and discuss their advantages and disadvantages.

Approach #1: Color histogram matching

The first method we present works as follows. Given the style image ($S$), and the content image ($C$), the style image’s colors are tranformed to match the colors of the content image. This produces a new style image $S^{\prime}$ that replaces $S$ as input to the neural style transfer algorithm. The algorithm is otherwise unchanged.

The one choice to be made is the color transfer procedure. There are many color transformation algorithms to choose from; see for a survey. Here we use linear methods, which are simple and very effective for color style transfer.

In particular, let $\textbf{x}_{i}=(R,G,B)^{T}$ be a pixel of an image. Each pixel is transformed as:

where A is a $3\times 3$ matrix and b is a 3-vector.

We choose this transformation so that the mean and covariance of the RGB values in the new style image $S^{\prime}$ match those of $C^{\prime}$. In other words, let \mbox{\boldmath\mu}_{S} and \mbox{\boldmath\mu}_{C} be the mean colors of the style and content images, and \mbox{\boldmath\Sigma}_{S} and \mbox{\boldmath\Sigma}_{C} be the pixel covariances. The mean and covariance of the color pixels is given by \mbox{\boldmath\mu}=\sum_{i}\textbf{x}_{i}/N and \mbox{\boldmath\Sigma}=\sum_{i}(\textbf{x}_{i}-\mbox{\boldmath\mu})(\textbf{x}_{i}-\mbox{\boldmath\mu})^{T}/N. We want to choose A and b to be satisfy \mbox{\boldmath\mu}_{S^{\prime}}=\mbox{\boldmath\mu}_{C} and \mbox{\boldmath\Sigma}_{S^{\prime}}=\mbox{\boldmath\Sigma}_{C}. These are satisfied by the constraints:

There remains a family of solutions for A that satisfies these constraints.

We consider two variants. The first variant uses the Cholesky decomposition:

where \mbox{\boldmath\Sigma}=\textbf{L}\textbf{L}^{T} is the Cholesky decomposition of $\Sigma$.

The second variant is a 3D color matching formulations explored in Image Analogies (Appendix B). First, let the eigenvalue decomposition of a covariance matrix be \mbox{\boldmath\Sigma}=\textbf{U}\mbox{\boldmath\Lambda}\textbf{U}^{T}. Then, we define a matrix square-root as: \mbox{\boldmath\Sigma}^{1/2}=\textbf{U}\mbox{\boldmath\Lambda}^{1/2}\textbf{U}^{T}. Then, the transformation is given by

In general, we find that results using the Image Analogies color transfer (Fig. 3(b)) look better than those using the Cholesky transfer (Fig. 3(a)). Furthermore, the Cholesky transform has the conceptually-undesirable property that it depends on the channel ordering, i.e., using RGB images will give different results from BGR. We also experimented with the Monge-Kantorovitch linear transform that minimizes the pixel-wise $L_{2}$-distance to the image before color transfer, and found the results to be essentially indistinguishable from the Image Analogies approach.

In general, we find that the color matching method works reasonably well with neural style transfer. This is in contrast to Image Analogies, where it gave poor synthesis results .

We also test transferring the color distribution to the output of neural style transfer instead of to the style image. In other words, neural style transfer is computed from the original inputs $S$ and $C$, and then the output $T$ is color-matched to $C$, producing a new output $T^{\prime}$. We find that transferring the color histogram before style transfer leads to better results (Fig. 4). This is particularly apparent for the second example in Figure 4, where the cubist texture is not transferred as completely to the sky. It appears to be easier for the algorithm to transfer the style between two images with similar color distribution. This can be explained by the reduced competition between reconstructing the content image and simultaneously matching the texture information from the style image.

Approach #2: Luminance-only transfer

The second method we consider is to perform style transfer only in the luminance channel, as used in Image Analogies . This is motivated by the observation that visual perception is far more sensitive to changes in luminance than in color .

The modification is simple. The luminance channels $L_{S}$ and $L_{C}$ are first extracted from the style and content images. Then the neural style transfer algorithm is applied to these images to produce an output luminance image $L_{T}$. Using the YIQ color space, the color information of the content image are represented by the $I$ and $Q$ channels; these are combined with $L_{T}$ to produce the final color output image (Fig. 5(c,d)).

If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be helpful to match the histogram of the style luminance channel $L_{S}$ to that of the content image $L_{C}$ before transferring the style. As in the previous section, we use a linear map that matches the second-order statistics of the content image (Fig. 5(e,f)). Let $\mu_{S}$ and $\mu_{C}$ be the mean luminances of the two images, and $\sigma_{S}$ and $\sigma_{C}$ be their standard deviations. Then each luminance pixel in the style image is updated as:

Comparison and discussion

Here we present two simple methods to preserve the color of the content image in the neural style transfer algorithm :

Linear color transfer onto the style image, before style transfer.

Style transfer only in the luminance channel.

Both methods give perceptually-interesting results but have different advantages and disadvantages.

The first method is naturally limited by how well the color transfer from the content image onto the style image works. The color distribution often cannot be matched perfectly, leading to a mismatch between the colors of the output image and that of the content image (Fig. 6(e)). The synthesis also replicates “content” structures from the van Gogh style scene, i.e., the pattern of reflections on the river appear as vertical yellow stripes of brushstrokes in the output.

In contrast, the second method preserves the colors of the content image perfectly. However, dependencies between the luminance and the color channels are lost in the output image (Fig. 6 (d)). This is particularly apparent for styles with prominent brushstrokes. In Figure 6(d), colors are no longer aligned to strokes. That means a single brushstroke can have multiple colors, which does not happen in real paintings. In comparison, when using full style transfer and color matching, the output image really consists of strokes which are blotches of paint, not just variations of light and dark.

One potential advantage of the luminance-based method is that it reduces the dimensionality of the optimization problem for the neural synthesis. The neural synthesis algorithm performs numerical optimization of the output image, and luminance-only synthesis has one-third fewer parameters. However, it is unclear that there is any practical advantage in typical GPU implementations.

In future work, it would be interesting to explore how the two statistical models in here (color statistics vs. CNN activations) might be unified, and to explore more sophisticated color transfer and adjustment procedures .

References