EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

Introduction

Image inpainting, or image completion, involves filling in missing regions of an image. It is an important step in many image editing tasks. It can, for example, be used to fill in the holes left after removing unwanted objects from an image. Humans have an uncanny ability to zero in on visual inconsistencies. Consequently, the filled regions must be perceptually plausible. Among other things, the lack of fine structure in the filled region is a giveaway that something is amiss, especially when the rest of the image contain sharp details. The work presented in this paper is motivated by our observation that many existing image inpainting techniques generate over-smoothed and/or blurry regions, failing to reproduce fine details.

We divide image inpainting into a two-stage process (Figure 1): edge generation and image completion. Edge generation is solely focused on hallucinating edges in the missing regions. The image completion network uses the hallucinated edges and estimates RGB pixel intensities of the missing regions. Both stages follow an adversarial framework to ensure that the hallucinated edges and the RGB pixel intensities are visually consistent. Both networks incorporate losses based on deep features to enforce perceptually realistic results.

Like most computer vision problems, image inpainting predates the wide-spread use of deep learning techniques. Broadly speaking, traditional approaches for image inpainting can be divided into two groups: diffusion-based and patch-based. Diffusion-based methods propagate background data into the missing region by following a diffusive process typically modeled using differential operators . Patch-based methods, on the other hand, fill in missing regions with patches from a collection of source images that maximize patch similarity . These methods, however, do a poor job of reconstructing complex details that may be local to the missing region.

More recently deep learning approaches have found remarkable success at the task of image inpainting. These schemes fill the missing pixels using learned data distribution. They are able to generate coherent structures in the missing regions, a feat that was nearly impossible for traditional techniques. While these approaches are able to generate missing regions with meaningful structures, the generated regions are often blurry or suffer from artifacts, suggesting that these methods struggle to reconstruct high frequency information accurately.

Then, how does one force an image inpainting network to generate fine details? Since image structure is well-represented in its edge mask, we show that it is possible to generate superior results by conditioning an image inpainting network on edges in the missing regions. Clearly, we do not have access to edges in the missing regions. Rather, we train an edge generator that hallucinates edges in these areas. Our approach of “lines first, color next” is partly inspired by our understanding of how artists work . “In line drawing, the lines not only delineate and define spaces and shapes; they also play a vital role in the composition”, says Betty Edwards, highlights the importance of sketches from an artistic viewpoint . Edge recovery, we suppose, is an easier task than image completion. Our proposed model essentially decouples the recovery of high and low-frequency information of the inpainted region.

We evaluate our proposed model on standard datasets CelebA , Places2 , and Paris StreetView . We compare the performance of our model against current state-of-the-art schemes. Furthermore, we provide results of experiments carried out to study the effects of edge information on the image inpainting task. Our paper makes the following contributions:

An edge generator capable of hallucinating edges in missing regions given edges and grayscale pixel intensities of the rest of the image.

An image completion network that combines edges in the missing regions with color and texture information of the rest of the image to fill the missing regions.

An end-to-end trainable network that combines edge generation and image completion to fill in missing regions exhibiting fine details.

We show that our model can be used in some common image editing applications, such as object removal and scene generation. Our source code is available at: https://github.com/knazeri/edge-connect

Related Work

Diffusion-based methods propagate neighboring information into the missing regions . adapted the Mumford-Shah segmentation model for image inpainting by introducing Euler’s Elastica. However, reconstruction is restricted to locally available information for these diffusion-based methods, and these methods fail to recover meaningful structures in the missing regions. These methods also cannot adequately deal with large missing regions.

Patch-based methods fill in missing regions (i.e., targets) by copying information from similar regions (i.e., sources) of the same image (or a collection of images). Source regions are often blended into the target regions to minimize discontinuities . These methods are computationally expensive since similarity scores must be computed for every target-source pair. PatchMatch addressed this issue by using a fast nearest neighbor field algorithm. These methods, however, assume that the texture of the inpainted region can be found elsewhere in the image. This assumption does not always hold. Consequently, these methods excel at recovering highly patterned regions such as background completion but struggle at reconstructing patterns that are locally unique.

One of the first deep learning methods designed for image inpainting is context encoder , which uses an encoder-decoder architecture. The encoder maps an image with missing regions to a low-dimensional feature space, which the decoder uses to construct the output image. However, the recovered regions of the output image often contain visual artifacts and exhibit blurriness due to the information bottleneck in the channel-wise fully connected layer. This was addressed by Iizuka et al. by reducing the number of downsampling layers, and replacing the channel-wise fully connected layer with a series of dilated convolution layers . The reduction of downsampling layers are compensated by using varying dilation factors. However, training time was increased significantlyModel by required two months of training over four GPUs. due to extremely sparse filters created using large dilation factors. Yang et al. uses a pre-trained VGG network to improve the output of the context-encoder, by minimizing the feature difference of image background. This approach requires solving a multi-scale optimization problem iteratively, which noticeably increases computational cost during inference time. Liu et al. introduced “partial convolution” for image inpainting, where convolution weights are normalized by the mask area of the window that the convolution filter currently resides over. This effectively prevents the convolution filters from capturing too many zeros when they traverse over the incomplete region.

Recently, several methods were introduced by providing additional information prior to inpainting. Yeh et al. trains a GAN for image inpainting with uncorrupted data. During inference, back-propagation is employed for $1,500$ iterations to find the representation of the corrupted image on a uniform noise distribution. However, the model is slow during inference since back-propagation must be performed for every image it attempts to recover. Dolhansky and Ferrer demonstrate the importance of exemplar information for inpainting. Their method is able to achieve both sharp and realistic inpainting results. Their method, however, is geared towards filling in missing eye regions in frontal human face images. It is highly specialized and does not generalize well. Contextual Attention takes a two-step approach to the problem of image inpainting. First, it produces a coarse estimate of the missing region. Next, a refinement network sharpens the result using an attention mechanism by searching for a collection of background patches with the highest similarity to the coarse estimate. takes a similar approach and introduces a “patch-swap” layer which replaces each patch inside the missing region with the most similar patch on the boundary. These schemes suffer from two limitations: 1) the refinement network assumes that the coarse estimate is reasonably accurate, and 2) these methods cannot handle missing regions with arbitrary shapes. Free-form inpainting method proposed in is perhaps closest in spirit to our scheme. It uses hand-drawn sketches to guide the inpainting process. Our method does away with hand-drawn sketches and instead learns to hallucinate edges in the missing regions.

The inpainting technique proposed in this paper subsumes two disparate computer vision problems: Image-to-Edges and Edges-to-Image. There is a large body of literature that addresses “Image-to-Edges” problems . Canny edge detector, an early scheme for constructing edge maps, for example, is roughly 30 years old . Dollár and Zitnikc use structured learning on random decision forests to predict local edge masks. Holistically-nested Edge Detection (HED) is a fully convolutional network that learns edge information based on its importance as a feature of the overall image. In our work, we train on edge maps computed using Canny edge detector. We explain this in detail in Section 4.1 and Section 5.3.

Traditional “Edges-to-Image” methods typically follow a bag-of-words approach, where image content is constructed through a pre-defined set of keywords. These methods, however, are unable to accurately construct fine-grained details especially near object boundaries. Scribbler is a learning-based model where images are generated using line sketches as the input. The results of their work possess an art-like quality, where color distribution of the generated result is guided by the use of color in the input sketch. Isola et al. proposed a conditional GAN framework , called pix2pix, for image-to-image translation problems. This scheme can use available edge information as a priori. CycleGAN extends this framework and finds a reverse mapping back to the original data distribution. This approach yields superior results since the aim is to learn the inverse of the forward mapping.

EdgeConnect

We propose an image inpainting network that consists of two stages: 1) edge generator, and 2) image completion network (Figure 2). Both stages follow an adversarial model , i.e. each stage consists of a generator/discriminator pair. Let $G_{1}$ and $D_{1}$ be the generator and discriminator for the edge generator, and $G_{2}$ and $D_{2}$ be the generator and discriminator for the image completion network, respectively. To simplify notation, we will use these symbols also to represent the function mappings of their respective networks.

Our generators follow an architecture similar to the method proposed by Johnson et al. , which has achieved impressive results for style transfer, super-resolution , and image-to-image translation . Specifically, the generators consist of encoders that down-sample twice, followed by eight residual blocks and decoders that up-sample images back to the original size. Dilated convolutions with a dilation factor of two are used instead of regular convolutions in the residual layers, resulting in a receptive field of $205$ at the final residual block. For discriminators, we use a $70\times 70$ PatchGAN architecture, which determines whether or not overlapping image patches of size $70\times 70$ are real. We use instance normalization across all layers of the networkThe details of our architecture are in appendix A.

We use $\mathbf{C}_{gt}$ and $\mathbf{C}_{pred}$ conditioned on $\mathbf{I}_{gray}$ as inputs of the discriminator that predicts whether or not an edge map is real. The network is trained with an objective comprised of an adversarial loss and feature-matching loss

where $\lambda_{adv,1}$ and $\lambda_{FM}$ are regularization parameters. The adversarial loss is defined as

The feature-matching loss $\mathcal{L}_{FM}$ compares the activation maps in the intermediate layers of the discriminator. This stabilizes the training process by forcing the generator to produce results with representations that are similar to real images. This is similar to perceptual loss , where activation maps are compared with those from the pre-trained VGG network. However, since the VGG network is not trained to produce edge information, it fails to capture the result that we seek in the initial stage. The feature matching loss $\mathcal{L}_{FM}$ is defined as

where $L$ is the final convolution layer of the discriminator, $N_{i}$ is the number of elements in the $i$’th activation layer, and $D^{(i)}_{1}$ is the activation in the $i$’th layer of the discriminator. Spectral normalization (SN) further stabilizes training by scaling down weight matrices by their respective largest singular values, effectively restricting the Lipschitz constant of the network to one. Although this was originally proposed to be used only on the discriminator, recent works suggest that generator can also benefit from SN by suppressing sudden changes of parameter and gradient values. Therefore, we apply SN to both generator and discriminator. Spectral normalization was chosen over Wasserstein GAN (WGAN), as we found that WGAN was several times slower in our early tests. Note that only $\mathcal{L}_{adv,1}$ is maximized over $D_{1}$ since $D_{1}$ is used to retrieve activation maps for $\mathcal{L}_{FM}$. For our experiments, we choose $\lambda_{adv,1}=1$ and $\lambda_{FM}=10$.

2 Image Completion Network

We include the two losses proposed in commonly known as perceptual loss $\mathcal{L}_{perc}$ and style loss $\mathcal{L}_{style}$. As the name suggests, $\mathcal{L}_{perc}$ penalizes results that are not perceptually similar to labels by defining a distance measure between activation maps of a pre-trained network. Perceptual loss is defined as

where $\phi_{i}$ is the activation map of the $i$’th layer of a pre-trained network. For our work, $\phi_{i}$ corresponds to activation maps from layers $\tt{relu1\_1,~{}relu2\_1,~{}relu3\_1,~{}relu4\_1}$ and $\tt{relu5\_1}$ of the VGG-19 network pre-trained on the ImageNet dataset . These activation maps are also used to compute style loss which measures the differences between covariances of the activation maps. Given feature maps of sizes $C_{j}\times H_{j}\times W_{j}$, style loss is computed by

where $G_{j}^{\phi}$ is a $C_{j}\times C_{j}$ Gram matrix constructed from activation maps $\phi_{j}$. We choose to use style loss as it was shown by Sajjadi et al. to be an effective tool to combat “checkerboard” artifacts caused by transpose convolution layers . Our overall loss function is

Experiments

To train $G_{1}$, we generate training labels (i.e. edge maps) using Canny edge detector. The sensitivity of Canny edge detector is controlled by the standard deviation of the Gaussian smoothing filter $\sigma$. For our tests, we empirically found that $\sigma\approx 2$ yields the best results (Figure 6). In Section 5.3, we investigate the effect of the quality of edge maps on the overall image completion.

For our experiments, we use two types of image masks: regular and irregular. Regular masks are square masks of fixed size (25% of total image pixels) centered at a random location within the image. We obtain irregular masks from the work of Liu et al. . Irregular masks are augmented by introducing four rotations ($0^{\circ},90^{\circ},180^{\circ},270^{\circ}$) and a horizontal reflection for each mask. They are classified based on their sizes relative to the entire image in increments of $10\%$ (e.g., 0-10%, 10-20%, etc.).

2 Training Setup and Strategy

Our proposed model is implemented in PyTorch. The network is trained using $256\times 256$ images with a batch size of eight. The model is optimized using Adam optimizer with $\beta_{1}=0$ and $\beta_{2}=0.9$. Generators $G_{1},G_{2}$ are trained separately using Canny edges with learning rate $10^{-4}$ until the losses plateau. We lower the learning rate to $10^{-5}$ and continue to train $G_{1}$ and $G_{2}$ until convergence. Finally, we fine-tune the networks by removing $D_{1}$, then train $G_{1}$ and $G_{2}$ end-to-end with learning rate $10^{-6}$ until convergence. Discriminators are trained with a learning rate one tenth of the generators’.

Results

Our proposed model is evaluated on the datasets CelebA , Places2 , and Paris StreetView . Results are compared against the current state-of-the-art methods both qualitatively and quantitatively.

Figure 4 shows a sample of images generated by our model. For visualization purposes, we reverse the colors of $\mathbf{C}_{comp}$. Our model is able to generate photo-realistic results with a large fraction of image structures remaining intact. Furthermore, by including style loss, the inpainted images lack any “checkerboard” artifacts in the generated results. As importantly, the inpainted images exhibit minimal blurriness.

Figure 3 compares images generated by our method with those generated by other state-of-the-art techniques. The images generated by our proposed model are closer to ground truth than images from other methods. We conjecture that when edge information is present, the network only needs to learn the color distribution, without having to worry about preserving image structure.

2 Quantitative Comparison

Visual Turing Tests

We evaluate our results using the human perceptual metrics 2 alternative forced choice (2AFC) and just noticeable differences (JND). For 2AFC, we ask participants whether or not an image is real from a pool of randomized images. For JND, we ask participants to select the more realistic image from pairs of real and generated images. Participants are given two seconds for each test. The tests are performed over 300 images for each model and mask size. Each image is shown 10 times in total. The results are summarized in Table 2.

3 Ablation Study

We now turn our attention to the key assumption of this work: edge information helps with image inpainting. Table 3 shows inpainting results with and without edge information. Our model achieved better scores for every metric when edge information was incorporated into the inpainting model, even when a significant portion of the image is missing.

Next, we turn to a more interesting question: How much edge information is needed to see improvements in the generated images? We again use Canny edge detector to construct edge information. We use the parameter $\sigma$ to control the amount of edge information available to the image completion network. Specifically, we train our image completion network using edge maps generated for $\sigma=0,0.5,\dots,5.5$, and we found that the best image inpainting results are obtained with edges corresponding to $\sigma\in[1.5,2.5]$, across all datasets shown in Figure 6. For large values of $\sigma$, too few edges are available to make a difference in the quality of generated images. On the other hand, when $\sigma$ is too small, too many edges are produced, which adversely affect the quality of the generated images. We used this study to set $\sigma=2$ when creating ground truth edge maps for the training of the edge generator network.

Figure 7 shows how different values of $\sigma$ affects the inpainting task. Note that in a region where edge data is sparse, the quality of the inpainted region degrades. For instance, in the generated image for $\sigma=5$, the left eye was reconstructed much sharper than the right eye.

Alternative Edge Detection Systems

We use Canny edge detector to produce training labels for the edge generator network due to its speed, robustness, and ease of use. Canny edges are one-pixel wide, and are represented as binary masks (1 for edge, 0 for background). Edges produced with HED , however, are of varying thickness, and pixels can have intensities ranging between 0 and 1. We noticed that it is possible to create edge maps that look eerily similar to human sketches by performing element-wise multiplication on Canny and HED edge maps (Figure 8). We trained our image completion network using the combined edge map. However, we did not notice any improvements in the inpainting results.Further analysis with HED are available in appendix C.

Discussions and Future Work

We proposed EdgeConnect, a new deep learning model for image inpainting tasks. EdgeConnect comprises of an edge generator and an image completion network, both following an adversarial model. We demonstrate that edge information plays an important role in the task of image inpainting. Our method achieves state-of-the-art results on standard benchmarks, and is able to deal with images with multiple, irregularly shaped missing regions.

The trained model can be used as an interactive image editing tool. We can, for example, manipulate objects in the edge domain and transform the edge maps back to generate a new image. This is demonstrated in Figure 10. Here we have removed the right-half of a given image to be used as input. The edge maps, however, are provided by a different image. The generated image seems to share characteristics of the two images. Figure 11 shows examples where we attempt to remove unwanted objects from existing images.

We plan to investigate better edge detectors. While effectively delineating the edges is more useful than hundreds of detailed lines, our edge generating model sometimes fails to accurately depict the edges in highly textured areas, or when a large portion of the image is missing (as seen in Figure 9). We believe our fully convolutional generative model can be extended to very high-resolution inpainting applications with an improved edge generating system.

This research was supported in part by NSERC Discovery Grant. The authors would like to thank Konstantinos G. Derpanis for helpful discussions and feedbacks. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research.

References

Appendix A Network Architectures

We follow a similar naming convention as those presented in . Let ck denote a $7\times 7$ Convolution-SpectralNorm-InstanceNorm-ReLU layer with $k$ filters and stride $1$ with reflection padding. Let dk denote a $4\times 4$ Convolution-SpectralNorm-InstanceNorm-ReLU layer with $k$ filters and stride $2$ for down-sampling. Let uk be defined in the same manner as dk with transpose convolution for up-sampling. Let Rk denote a residual block of channel size $k$ across both layers. We use dilated convolution in the first layer of Rk with dilation factor of $2$, followed by spectral normalization and instance normalization.

The architecture of our generators is adopted from the model proposed by Johnson et al. : c64, d128, d256, R256, R256, R256, R256, R256, R256, R256, R256, u128, u64, c*.

The final layer c* varies depending on the generator. In the edge generator $G_{1}$, c* has channel size of $1$ with sigmoid activation for edge prediction. In the image completion network $G_{2}$, c* has channel size of $3$ with $\tanh$ (scaled) activation for the prediction of RGB pixel intensities. In addition, we remove spectral normalization from all layers of $G_{2}$.

A.2 Discriminators

The discriminators $D_{1}$ and $D_{2}$ follow the same architecture based on the $70\times 70$ PatchGAN . Let Ck-s denote a $4\times 4$ Convolution-SpectralNorm-LeakyReLU layer with $k$ filters of stride $s$. The discriminators have the architecture C64-2, C128-2, C256-2, C512-1, C1-1. The final convolution layer produces scores predicting whether $70\times 70$ overlapping image patches are real or fake. LeakyReLU is employed with slope $0.2$.

Appendix B Experimental Results

We provide additional results produced by our model over the following datasets:

With CelebA, we cropped the center $178\times 178$ of the images, then resized them to $256\times 256$ using bilinear interpolation. For Paris StreetView, since the images in the dataset are elongated ($936\times 537$), we separate each image into three: 1) Left $537\times 537$, 2) middle $537\times 537$, 3) right $537\times 537$, of the image. These images are scaled down to $256\times 256$ for our model, totaling $44,700$ images.

Table 4 shows the accuracy of our edge generator $G_{1}$ across all three datasets. We measure precision and recall for various mask sizes. We emphasize that the goal of this experiment is not to achieve best precision and recall results, but instead to showcase how close the generated edges are to the oracle.

B.2 Comprehensive Results

Tables 7 and 7 shows the performance of our model compared to existing methods over the datasets CelebA and Paris StreetView respectively. Our method produces noticeably better results. Note that we did not include values for PConv over the CelebA and Paris StreetView datasets as the source code was not available at the time of writing. Figures 13, and 14 display these results graphically. Additional inpainting results of our proposed model are shown in figures 15, 16, and 17. Our source code, pre-trained models, and more results are available at: https://github.com/knazeri/edge-connect

Appendix C Alternative Edge Generating Systems

In drawing, an edge is a boundary that separates two areas. A thick line brings the shape forward thin line indicates a plane receding into the background. In other words edges create a sense of distance and are not just about lines. Here we use HED as an alternative edge detection system. Edges produced with HED, are of varying thickness, and pixels can have intensities ranging between 0 and 1. We noticed that it is possible to create edge maps that look eerily similar to human sketches by performing element-wise multiplication on Canny and HED edge maps. We compare the quantitative results between Canny and a combination of HED and Canny edges (i.e. HED$\odot$Canny). Generated images based on the combined edges gave the best performance. However, our generator $G_{1}$ is unable to generate these type of edges accurately during training. Table 5 shows $G_{1}$ trained on HED$\odot$Canny had the poorest performance out of all methods despite its ground truth counterpart achieving the best performance. These results suggest that better edge detectors result in better inpainting, however, effectively drawing those edges remains an open question in our research. Figure 12 shows the results of $G_{1}$ trained using hybrid edges.