3D Photography using Context-aware Layered Depth Inpainting

Introduction

3D photography—capturing views of the world with a camera and using image-based rendering techniques for novel view synthesis—is a fascinating way to record and reproduce visual perception. It provides a dramatically more immersive experience than old 2D photography: almost lifelike in Virtual Reality, and even to some degree on normal flat displays when displayed with parallax.

Classic image-based reconstruction and rendering techniques, however, require elaborate capture setups involving many images with large baselines hedman2017casual; whelan2018reconstructing; kopf2013image; penner2017soft; hedman2018deep; flynn2016deepstereo, and/or special hardware (e.g., Lytro Immerge, Facebook Manifold camera https://facebook360.fb.com/2018/05/01/red-facebook-6dof-camera/).

Recently, we have seen work to make capture for 3D photography more effortless by using cell phone cameras and lowering baseline requirements hedman2017casual; hedman2018instant. In the most extreme cases, novel techniques such as Facebook 3D Photos https://facebook360.fb.com/2018/10/11/3d-photos-now-rolling-out-on-facebook-and-in-vr/ now just require capturing a single snapshot with a dual lens camera phone, which essentially provides an RGB-D (color and depth) input image.

In this work we are interested in rendering novel views from such an RGB-D input. The most salient features in rendered novel views are the disocclusions due to parallax: naïve depth-based warping techniques either produce gaps here (Figure 1a) or stretched content (1b). Recent methods try to provide better extrapolations.

Stereo magnification zhou2018stereo and recent variants srinivasan2019pushing; mildenhall2019llff use a fronto-parallel multi-plane representation (MPI), which is synthesized from the small-baseline dual camera stereo input. However, MPI produces artifacts on sloped surfaces. Besides, the excessive redundancy in the multi-plane representation makes it memory and storage inefficient and costly to render.

Facebook 3D Photos use a layered depth image (LDI) representation shade1998layered, which is more compact due to its sparsity, and can be converted into a light-weight mesh representation for rendering. The color and depth in occluded regions are synthesized using heuristics that are optimized for fast runtime on mobile devices. In particular it uses a isotropic diffusion algorithm for inpainting colors, which produces overly smooth results and is unable to extrapolate texture and structures (Figure 1c).

Several recent learning-based methods also use similar multi-layer image representations dhamo2018peeking; tulsiani2018layer. However, these methods use “rigid” layer structures, in the sense that every pixel in the image has the same (fixed and predetermined) number of layers. At every pixel, they store the nearest surface in the first layer, the second-nearest in the next layer, etc. This is problematic, because across depth discontinuities the content within a layer changes abruptly, which destroys locality in receptive fields of convolution kernels.

In this work we present a new learning-based method that generates a 3D photo from an RGB-D input. The depth can either come from dual camera cell phone stereo, or be estimated from a single RGB image li2018megadepth; lasinger2019towards; godard2019digging. We use the LDI representation (similar to Facebook 3D Photos) because it is compact and allows us to handle situations of arbitrary depth-complexity. Unlike the “rigid” layer structures described above, we explicitly store connectivity across pixels in our representation. However, as a result it is more difficult to apply a global CNN to the problem, because our topology is more complex than a standard tensor. Instead, we break the problem into many local inpainting sub-problems, which we solve iteratively. Each problem is locally like an image, so we can apply standard CNN.We use an inpainting model that is conditioned on spatially-adaptive context regions, which are extracted from the local connectivity of the LDI. After synthesis we fuse the inpainted regions back into the LDI, leading to a recursive algorithm that proceeds until all depth edges are treated.

The result of our algorithm are 3D photos with synthesized texture and structures in occluded regions (Figure 1d). Unlike most previous approaches we do not require predetermining a fixed number of layers. Instead our algorithm adapts by design to the local depth-complexity of the input and generates a varying number of layers across the image. We have validated our approach on a wide variety of photos captured in different situations.

Related Work

Representation for novel view synthesis. Different types of representations have been explored for novel view synthesis, including light fields gortler1996lumigraph; levoy1996light; buehler2001unstructured, multi-plane images zhou2018stereo; srinivasan2019pushing; mildenhall2019llff, and layered depth images shade1998layered; swirski2011layered; dhamo2018peeking; tulsiani2018layer; hedman2017casual; hedman2018instant; dhamo2019object; niklaus2019KenBurns. Light fields enable photorealistic rendering of novel views, but generally require many input images to achieve good results. The multi-plane image representation zhou2018stereo; srinivasan2019pushing; mildenhall2019llff stores multiple layers of RGB-$\alpha$ images at fixed depths. The main advantage of this representation is its ability to capture semi-reflective or semi-transparent surfaces. However, due to the fixed depth discretization, sloped surfaces often do not reproduce well, unless an excessive number of planes is used. Many variants of layered depth image representations have been used over time. Representations with a fixed number of layers everywhere have recently been used dhamo2018peeking; tulsiani2018layer, but they do not preserve locality well, as described in the previous section. Other recent work hedman2017casual; hedman2018instant extends the original work of Shade et al. shade1998layered to explicitly store connectivity information. This representation can locally adapt to any depth-complexity and can be easily converted into a textured mesh for efficient rendering. Our work uses this representation as well.

Image-based rendering. Image-based rendering techniques enable photorealistic synthesis of novel views from a collection of posed images. These methods work best when the images have sufficiently large baselines (so that multi-view stereo algorithms can work well) or are captured with depth sensors. Recent advances include learning-based blending hedman2018deep, soft 3D reconstruction penner2017soft, handling reflection sinha2012image; kopf2013image, relighting xu2019deep, and reconstructing mirror and glass surfaces whelan2018reconstructing. Our focus in this work lies in novel view synthesis from one single image.

Learning-based view synthesis. CNN-based methods have been applied to synthesizing novel views from sparse light field data kalantari2016learning or two or more posed images flynn2016deepstereo; hedman2018deep; choi2019extreme. Several recent methods explore view synthesis from a single image. These methods, however, often focus on a specific domain srinivasan2017learning; Wiles20SynSin, synthetic 3D scenes/objects zhou2016view; park2017transformation; sun2018multi; dhamo2019object; dhamo2018peeking; eslami2018neural, hallucinating only one specific view xie2016deep3d; zeng2015hallucinating, or assuming piecewise planar scenes liu2018planenet; liu2018geometry.

Many of these learning-based view synthesis methods require running a forward pass of the pre-trained network to synthesize the image of a given viewpoint. This makes these approaches less applicable to display on resource-constrained devices. Our representation, on the other hand, can be easily converted into a textured mesh and efficiently rendered with standard graphics engines.

Image inpainting. The task of image inpainting aims to fill missing regions in images with plausible content. Inspired by the success of texture synthesis efros1999texture; efros2001image, example-based methods complete the missing regions by transferring the contents from the known regions of the image, either through non-parametric patch-based synthesis wexler2007space; barnes2009patchmatch; darabi2012image; huang2014image or solving a Markov Random Field model using belief propagation komodakis2007image or graph cut pritch2009shift; kwatra2003graphcut; he2014image. Driven by the progress of convolutional neural networks, CNN-based methods have received considerable attention due to their ability to predict semantically meaningful contents that are not available in the known regions pathak2016context; song2017contextual; iizuka2017globally; yang2017high; yu2018generative. Recent efforts include designing CNN architectures to better handle holes with irregular shapes liu2018image; yu2019free; Yan_2018_Shift and two-stage methods with structure-content disentanglement, e.g., predicting structure (e.g., contour/edges in the missing regions) and followed by content completion conditioned on the predicted structures nazeri2019edgeconnect; xiong2019foreground; ren2019structureflow.

Our inpainting model builds upon the recent two-stage approaches nazeri2019edgeconnect; xiong2019foreground; ren2019structureflow but with two key differences. First, unlike existing image inpainting algorithms where the hole and the available contexts are static (e.g., the known regions in the entire input image), we apply the inpainting locally around each depth discontinuity with adaptive hole and context regions. Second, in addition to inpaint the color image, we also inpaint the depth values as well as the depth discontinuity in the missing regions.

Depth inpainting. Depth inpainting has applications in filling missing depth values where commodity-grade depth cameras fail (e.g., transparent/reflective/distant surfaces) liu2017robust; zhang2018deep; lu2014depth or performing image editing tasks such as object removal on stereo images wang2008stereoscopic; mu2014stereoscopic. The goal of these algorithms, however, is to inpaint the depth of the visible surfaces. In contrast, our focus is on recovering the depth of the hidden surface.

CNN-based single depth estimation. CNN-based methods have recently demonstrated promising results on estimating depth from a single image. Due to the difficulty of collecting labeled datasets, earlier approaches often focus on specific visual domains such as indoor scenes eigen2015predicting or street view godard2017unsupervised; zhou2017unsupervised. While the accuracy of these approaches is not yet competitive with multi-view stereo algorithms, this line of research is particularly promising due to the availability of larger and more diverse training datasets from relative depth annotations chen2016single, multi-view stereo li2018megadepth, 3D movies lasinger2019towards and synthetic data niklaus2019KenBurns.

For cases where only one single color image is available, we obtain the depth estimate through a pre-trained depth estimation model li2018megadepth; lasinger2019towards. Removing the dependency on stereo or multiple images as input makes our method more widely applicable to all the existing photos.

Method

Layered depth image. Our method takes as input an RGB-D image (i.e., an aligned color-and-depth image pair) and generates a Layered Depth Image (LDI, shade1998layered) with inpainted color and depth in parts that were occluded in the input.

An LDI is similar to a regular 4-connected image, except at every position in the pixel lattice it can hold any number of pixels, from zero to many. Each LDI pixel stores a color and a depth value. Unlike the original LDI work shade1998layered, we explicitly represent the local connectivity of pixels: each pixel stores pointers to either zero or at most one direct neighbor in each of the four cardinal directions (left, right, top, bottom). LDI pixels are 4-connected like normal image pixels within smooth regions, but do not have neighbors across depth discontinuities.

LDIs are a useful representation for 3D photography, because (1) they naturally handle an arbitrary number of layers, i.e., can adapt to depth-complex situations as necessary, and (2) they are sparse, i.e., memory and storage efficient and can be converted into a light-weight textured mesh representation that renders fast.

The quality of the depth input to our method does not need to be perfect, as long as discontinuities are reasonably well aligned in the color and depth channels. In practice, we have successfully used our method with inputs from dual camera cell phones as well as with estimated depth maps from learning-based methods li2018megadepth; lasinger2019towards.

Method overview. Given an input RGB-D image, our method proceeds as follows. We first initialize a trivial LDI, which uses a single layer everywhere and is fully 4-connected. In a pre-process we detect major depth discontinuities and group them into simple connected depth edges (Section 3.1). These form the basic units for our main algorithm below. In the core part of our algorithm, we iteratively select a depth edge for inpainting. We then disconnect the LDI pixels across the edge and only consider the background pixels of the edge for inpainting. We extract a local context region from the “known” side of the edge, and generate a synthesis region on the “unknown” side (Section 3.2). The synthesis region is a contiguous 2D region of new pixels, whose color and depth values we generate from the given context using a learning-based method (Section 3.3). Once inpainted, we merge the synthesized pixels back into the LDI (Section 3.4). Our method iteratively proceeds in this manner until all depth edges have been treated.

The only input to our method is a single RGB-D image. Every step of the algorithm below proceeds fully automatically. We normalize the depth channel, by mapping the min and max disparity values (i.e., 1 / depth) to 0 and 1, respectively. All parameters related to spatial dimensions below are tuned for images with 1024 pixels along the longer dimension, and should be adjusted proportionally for images of different sizes.

We start by lifting the image onto an LDI, i.e., creating a single layer everywhere and connecting every LDI pixel to its four cardinal neighbors. Since our goal is to inpaint the occluded parts of the scene, we need to find depth discontinuities since these are the places where we need to extend the existing content. In most depth maps produced by stereo methods (dual camera cell phones) or depth estimation networks, discontinuities are blurred across multiple pixels (Figure 2c), making it difficult to precisely localize them. We, therefore, sharpen the depth maps using a bilateral median filter ma2013median (Figure 2d), using a $7\!\times\!7$ window size, and $\sigma_{\textit{spatial}}=4.0$, $\sigma_{\textit{intensity}}=0.5$.

After sharpening the depth map, we find discontinuities by thresholding the disparity difference between neighboring pixels. This results in many spurious responses, such as isolated speckles and short segments dangling off longer edges (Figure 2e). We clean this up as follows: First, we create a binary map by labeling depth discontinuities as 1 (and others as 0). Next, we use connected component analysis to merge adjacent discontinuities into a collection of “linked depth edges”. To avoid merging edges at junctions, we separate them based on the local connectivity of the LDI. Finally, we remove short segments ($<10$ pixels), including both isolated and dangling ones. We determine the threshold 10 by conducting five-fold cross-validation with LPIPS zhang2018unreasonable metric on 50 samples randomly selected from RealEstate10K training set. The final edges (Figures 2f) form the basic unit of our iterative inpainting procedure, which is described in the following sections.

2 Context and synthesis regions

Our inpainting algorithm operates on one of the previously computed depth edges at a time. Given one of these edges (Figure 3a), the goal is to synthesize new color and depth content in the adjacent occluded region. We start by disconnecting the LDI pixels across the discontinuity (Figure 3b). We call the pixels that became disconnected (i.e., are now missing a neighbor) silhouette pixels. We see in Figure 3b that a foreground silhouette (marked green) and a background silhouette (marked red) forms. Only the background silhouette requires inpainting. We are interested in extending its surrounding content into the occluded region.

We start by generating a synthesis region, a contiguous region of new pixels (Figure 3c, red pixels). These are essentially just 2D pixel coordinates at this point. We initialize the color and depth values in the synthesis region using a simple iterative flood-fill like algorithm. It starts by stepping from all silhouette pixels one step in the direction where they are disconnected. These pixels form the initial synthesis region. We then iteratively expand (for 40 iterations) all pixels of the region by stepping left/right/up/down and adding any pixels that have not been visited before. For each iteration, we expand the context and synthesis regions alternately and thus a pixel only belong to either one of the two regions Additionally, we do not step back across the silhouette, so the synthesis region remains strictly in the occluded part of the image. Figure 4 shows a few examples.

We describe our learning-based technique for inpainting the synthesis region in the next section. Similar techniques liu2018image; nazeri2019edgeconnect were previously used for filling holes in images. One important difference to our work is that these image holes were always fully surrounded by known content, which constrained the synthesis. In our case, however, the inpainting is performed on a connected layer of an LDI pixels, and it should only be constrained by surrounding pixels that are directly connected to it. Any other region in the LDI, for example on other foreground or background layer, is entirely irrelevant for this synthesis unit, and should not constrain or influence it in any way.

We achieve this behavior by explicitly defining a context region (Figure 3c, blue region) for the synthesis. Our inpainting networks only considers the content in the context region and does not see any other parts of the LDI. The context region is generated using a similar flood-fill like algorithm. One difference, however, is that this algorithm selects actual LDI pixels and follows their connection links, so the context region expansion halts at silhouettes. We run this algorithm for 100 iterations, as we found that synthesis performs better with slightly larger context regions. In practice, the silhouette pixels may not align well with the actual occluding boundaries due to imperfect depth estimation. To tackle this issue, we dilate the synthesis region near the depth edge by 5 pixels (the context region erodes correspondingly). Figure 5 shows the effect of this heuristic.

3 Context-aware color and depth inpainting

Model. Given the context and synthesis regions, our next goal is to synthesize color and depth values. Even though we perform the synthesis on an LDI, the extracted context and synthesis regions are locally like images, so we can use standard network architectures designed for images. Specifically, we build our color and depth inpainting models upon image inpainting methods in nazeri2019edgeconnect; liu2018image; xiong2019foreground.

One straightforward approach is to inpaint the color image and depth map independently. The inpainted depth map, however, may not be well-aligned with respect to the inpainted color.

To address this issue, we design our color and depth inpainting network similar to nazeri2019edgeconnect; xiong2019foreground: we break down the inpainting tasks into three sub-networks: (1) edge inpainting network, (2) color inpainting network, and (3) depth inpainting network (Figure 6). First, given the context edges as input, we use the edge inpainting network to predict the depth edges in the synthesis regions, producing the inpainted edges. Performing this step first helps infer the structure (in terms of depth edges) that can be used for constraining the content prediction (the color and depth values). We take the concatenated inpainted edges and context color as input and use the color inpainting network to produce inpainted color. We perform the depth inpainting similarly.

Figure 7 shows an example of how the edge-guided inpainting is able to extend the depth structures accurately and alleviate the color/depth misalignment issue.

Multi-layer inpainting. In depth-complex scenarios, applying our inpainting model once is not sufficient as we can still see the hole through the discontinuity created by the inpainted depth edges. We thus apply our inpainting model until no further inpainted depth edges are generated. Figure 8 shows an example of the effects. Here, applying our inpainting model once fills in missing layers. However, several holes are still visible when viewed at a certain viewpoint (Figure 8b). Applying the inpainting model one more time fixes the artifacts.

Training data generation. For training, our proposed model can be simply trained on any image dataset without the need of annotated data. Here, we choose to use MSCOCO dataset lin2014microsoft for its wide diversity in object types and scenes. To generate the training data for the inpainting model, we create a synthetic dataset as follows. First, we apply the pre-trained MegaDepth li2018megadepth on the COCO dataset to obtain pseudo ground truth depth maps. We extract context/synthesis regions (as described in Section 3.2) to form a pool of these regions. We then randomly sample and place these context-synthesis regions on different images in the COCO dataset. We thus can obtain the ground truth content (RGB-D) from the simulated occluded region.

4 Converting to 3D textured mesh

We form the 3D textured mesh by integrating all the inpainted depth and color values back into the original LDI. Using mesh representations for rendering allows us to quickly render novel views, without the need to perform per-view inference step. Consequently, the 3D representation produced by our algorithm can easily be rendered using standard graphics engines on edge devices.

Experimental Results

In this section, we start with describing implementation details (Section 4.1). We then show visual comparisons with the state-of-the-art novel view synthesis methods (Section 4.2). We refer to the readers to supplementary material for extensive results and comparisons. Next, we follow the evaluation protocol in zhou2018stereo and report the quantitative comparisons on the RealEstate10K dataset (Section 4.3). We present an ablation study to justify our model design (Section 4.4). Finally, we show that our method works well with depth maps from different sources (Section 4.5).Additional details and visual comparisons can be found in our supplementary material.

Training the inpainting model. For the edge-generator, we follow the hyper-parameters in nazeri2019edgeconnect. Specifically, we train the edge-generator model using the ADAM optimizer kingma2014adam with $\beta=0.9$ and an initial learning rate of $0.0001$. We train both the edge and depth generator model using the context-synthesis regions dataset on the MS-COCO dataset for 5 epochs. We train the depth generator and color image generator for 5 and 10 epochs, respectively.

Inpainting model architecture. For the edge inpainting network, we adopt the architecture provided by nazeri2019edgeconnect. For the depth and color inpainting networks, we use a standard U-Net architecture with partial covolution liu2018image. Due to the space limitation, we leave additional implementation details (specific network architecture, the training loss and the weights for each network) to the supplementary material. We will make the source code and pre-trained model publicly available to foster future work.

Training data. We use the 118k images from COCO 2017 set for training. We select at most 3 pairs of regions from each image to form the context-synthesis pool. During training, we sample one pair of regions for each image, and resize it by a factor between $\left[1.0,1.3\right]$.

2 Visual comparisons

Comparisons with methods with MPI representations. We compare our proposed model against MPI-based approaches on RealEstate10K dataset. We use DPSNet im2019dpsnet to obtain the input depth maps for our method. We render the novel views of MPI-based methods using the pre-trained weights provided by the authors. Figure 9 shows two challenging examples with complex depth structures. Our method synthesizes plausible structures around depth boundaries; on the other hand, stereo magnification and PB-MPI produce artifacts around depth discontinuities. LLFF mcmillan1995plenoptic suffers from ghosting effects when extrapolating new views.

Comparisons with Facebook 3D photo. Here, we aim to evaluate the capability of our method on photos taken in the wild. We extract the color images and the corresponding depth maps estimated from an iPhone X (with dual camera lens). We use the same set of RGB-D inputs for both Facebook 3D photo and our algorithm. Figure 10 shows the view synthesis result in comparison with Facebook 3D photo. The diffused color and depth values by the facebook 3D photo algorithm work well when small or thin occluded regions are revealed at novel views. These artifacts, however, become clearly visible with larger occluded regions. On the other hand, our results in general fills in the synthesis regions with visually plausible contents and structures.

3 Quantitative comparisons

We evaluate how well our model can extrapolate views compared to MPI-based methods srinivasan2019pushing; zhou2018stereo; choi2019extreme; mildenhall2019llff. We randomly sample 1500 video sequences from RealEstate10K to generate testing triplets. For each triplet, we set $t=10$ for target view, so that all the methods need to extrapolate beyond the source ($t=0$) and reference ($t=4$) frame. We use DPSNet im2019dpsnet to generate the input depth maps required for our model. We quantify the performance of each model using SSIM and PSNR metrics between the synthesized target views and the ground truth. As these metrics do not capture the perceptual quality of the synthesized view, we include LPIPS zhang2018unreasonable metric to quantify how well does the generated view align with human perception. For PB-MPI, we set the number of depth layers to 64 as it yields the best result. We report the evaluation results in Table 1. Our proposed method performs competitively on SSIM and PSNR. In addition, our synthesis views exhibit better perceptual quality, as reflected in the superior LPIPS score.

4 Ablation study

We conduct ablation studies to see how each of our proposed components contribute to the final performance. We first verify the effectiveness of edge-guided depth inpainting. We sample 130 triplets from our testing sequences, evaluate the inpainted color on both the entire image and disoccluded regions, and report the numbers in Table 2. The results show that our proposed edge-guided inpainting leads to minor improvement in numerical metrics. Next, we examine the efficacy of our color inpainting model following the same procedure described above. We present the performance in both entire image and occluded regions in Table 3. We observe that our proposed model yields better perceptual quality. Figure 11 shows an example.

5 Handling different depth maps

We test our method using depth maps generated using different approaches (Figure 12). We select images from SUNRGBD song2015sun dataset, and obtain the corresponding depth maps from three different sources: 1) depth estimated with MegaDepth li2018megadepth, 2) MiDas lasinger2019towards and 3) Kinect depth sensor. We present the resulting 3D photos in Figure 12. The results show that our method can handle depth maps from different sources reasonably well.

Conclusions

In this paper, we present an algorithm for creating compelling 3D photography from a single RGB-D image. Our core technical novelty lies in creating a completed layered depth image representation through context-aware color and depth inpainting. We validate our method on a wide variety of everyday scenes. Our experimental results show that our algorithm produces considerably fewer visual artifacts when compared with the state-of-the-art novel view synthesis techniques. We believe that such technology can bring 3D photography to a broader community, allowing people to easily capture scenes for immersive viewing.

Acknowledgement. This project is supported in part by NSF (#1755785) and MOST-108-2634-F-007-006 and MOST-109-2634-F-007-016.

References

Additional Quantitative Results

We further evaluate the PB-MPI method srinivasan2019pushing with various number of depth layers. We report the results in Table 4.

Visual Results

Comparisons with the state-of-the-arts. We provide a collection of rendered 3D photos with comparisons with the state-of-the-art novel view synthesis algorithms. In addition, we show that our method can synthesize novel view for legacy photos. Please refer to the website https://shihmengli.github.io/3D-Photo-Inpainting/ for viewing the results.

Ablation studies. To showcase how each of our proposed component contribute to the quality of the synthesized view, we include a set of rendered 3D photos using the same ablation settings in Section 4.4 of the main paper. Please refer to the website3 for viewing the photos.

Implementation Details

In this section, we provide additional implementation details of our model, including model architectures, training objectives, and training dataset collection. We will release the source code to facilitate future research in this area.

We adopt the same U-Net ronneberger2015u architecture as in liu2018image for our depth inpainting and color inpainting models (see Table 5), and change the input channels for each model accordingly. For the edge inpainting model, we use a design similar to nazeri2019edgeconnect (see Table 6). We set the input depth and RGB values in the synthesis region to zeros for all three models. The input edge values in the synthesis region are similarly set to zeros for depth and color inpainting models, but remain intact for the edge inpainting network. We show the input details of each model in Table 7

Training objective.

To train our color inpainting model, we adopt similar objective functions as in liu2018image. First, we define the reconstruction loss for context and synthesis regions:

where $S$ and $C$ are the binary mask indicating synthesis and context regions, respectively, $\odot$ denotes the Hadamard product, $N$ is the total number of pixels, $I$ is the inpainted result, and $I_{gt}$ is the ground truth image.

Next, we define the perceptual loss johnson2016perceptual:

Here, $\psi_{p}(\cdot)$ is the output of the $p$th layer from VGG-16 simonyan2014very, and $N_{\psi_{p}}$ is the total number of elements in $\psi_{p}(\cdot)$.

where $C_{p}$, $H_{p}$, $W_{p}$ is the number of channels, height, and width of the output $\psi_{p}(\cdot)$.

Finally, we adopt the Total Variation (TV) loss:

Here, We overload the notation $S$ to denote the synthesis region. This term can be interpreted as a smoothing penalty on the synthesis area. Combine all these loss terms, we obtain the training objective for our color inpainting model:

For our depth inpainting model, we use only $L_{context}+L_{synthesis}$ as the objective functions. For edge inpainting model, we follow the identical training protocol as in nazeri2019edgeconnect.

Training details.

We illustrate the data generation process in Figure 13. We use the depth map predicted by MegaDepth li2018megadepth as our pseudo ground truth. We train our method using 1 Nvidia V100 GPU with batch size of 8, and the total training time take about 5 days.

Failure cases

As estimating depth/disparity map from a single image remain a challenging problem (particularly for scenes with complex, thin structures), our method fails to produce satisfactory results with plausible motion parallax for scenes with complex structures. Due to the use of explicit depth map, our method is unable to handle reflective/transparent surfaces well. We show in Figure 14 two examples of such cases. Here, we show the input RGB image as well as the estimated depth map from the pre-trained MegaDepth model. The rendered 3D photos can be found in the supplementary webpage.