NeRF++: Analyzing and Improving Neural Radiance Fields

Introduction

Recall your last vacation where you captured a few photos of your favorite place. Now at home you wish to walk around in this special place again, if only virtually. This requires you to render the same scene from different, freely placed viewpoints in a possibly unbounded scene. This novel view synthesis task is a long-standing problem in computer vision and graphics (Chen & Williams 1993; Debevec et al. 1996; Levoy & Hanrahan 1996; Gortler et al. 1996; Shum & Kang 2000).

Recently, learning-based methods have led to significant progress towards photo-realistic novel view synthesis. The method of Neural Radiance Fields (NeRF), in particular, has attracted significant attention (Mildenhall et al. 2020). NeRF is an implicit MLP-based model that maps 5D vectors—3D coordinates plus 2D viewing directions—to opacity and color values, computed by fitting the model to a set of training views. The resulting 5D function can then be used to generate novel views with conventional volume rendering techniques.

In this technical report, we first present an analysis of potential failure modes in NeRF, and an analysis of why NeRF avoids these failure modes in practice. Second, we present a novel spatial parameterization scheme that we call inverted sphere parameterization that allows NeRF to work on a new class of captures of unbounded scenes.

In particular, we find that in theory, optimizing the 5D function from a set of training images can encounter critical degenerate solutions that fail to generalize to novel test views, in the absence of any regularization. Such phenomena are encapsulated in the shape-radiance ambiguity (Figure 1, left), wherein one can fit a set of training images perfectly for an arbitrary incorrect geometry by a suitable choice of outgoing 2D radiance at each surface point. We empirically show that the specific MLP structure used in NeRF plays an important role in avoiding such ambiguities, yielding an impressive ability to synthesize novel views. Our analysis offers a new view into NeRF’s impressive success.

We also address a spatial parameterization issue that arises in challenging scenarios involving 360∘ captures around objects within unbounded environments (Figure 1, right). For 360∘ captures, NeRF assumes that the entire scene can be packed into a bounded volume, which is problematic for large-scale scenes: either we fit a small part of the scene into the volume and sample it in detail, but completely fail to capture background elements; or, we fit the full scene into the volume and lack detail everywhere due to limited sampling resolution. We propose a simple yet effective solution that separately models foreground and background, addressing the challenge of modeling unbounded 3D background content with an inverted sphere scene parameterization. We show improved quantitative and qualitative results on real-world captures from the Tanks and Temples dataset (Knapitsch et al. 2017) and from the light field dataset of Yücer et al. 2016.

In summary, we present an analysis on how NeRF manages to resolve the shape-radiance ambiguity, as well as a remedy for the parameterization of unbounded scenes in the case of 360∘ captures.

Preliminaries

Ideally, $\sigma$ should peak at the ground-truth surface location for opaque materials, in which case $\mathbf{c}$ reduces to the surface light field (Wood et al. 2000). Given $n$ training images, NeRF uses stochastic gradient descent to optimize $\sigma$ and $\mathbf{c}$ by minimizing the discrepancy between the ground truth observed images $I_{i}$, and the predicted images $\hat{I}_{i}(\sigma,\mathbf{c})$ rendered from $\sigma$ and $\mathbf{c}$ at the same viewpoints:

To compensate for the network’s spectral bias and to synthesize sharper images, NeRF uses a positional encoding $\gamma$ that maps $\mathbf{x}$ and $\mathbf{d}$ to their Fourier features (Tancik et al. 2020):

where $k$ is a hyper-parameter specifying the dimensionality of the Fourier feature vector.

Shape-radiance ambiguity

The capacity of NeRF to model view-dependent appearance leads to an inherent ambiguity between 3D shape and radiance that can admit degenerate solutions, in the absence of regularization. For an arbitrary, incorrect shape, one can show that there exists a family of radiance fields that perfectly explains the training images, but that generalizes poorly to novel test views.

To illustrate this ambiguity, imagine that for a given scene we represent the geometry as a unit sphere. In other words, let us fix NeRF’s opacity field to be 1 at the surface of the unit sphere, and 0 elsewhere. Then, for each pixel in each training image, we intersect a ray through that pixel with the sphere, and define the radiance value at the intersection point (and along the ray direction) to be the color of that pixel. This artificially constructed solution is a valid NeRF reconstruction that perfectly fits the input images. However, this solution’s ability to synthesize novel views is very limited: accurately generating such a view would involve reconstructing an arbitrarily complex view-dependent function at each surface point. The model is very unlikely to accurately interpolate such a complex function, unless the training views are extremely dense, as in conventional light field rendering works (Buehler et al. 2001; Levoy & Hanrahan 1996; Gortler et al. 1996). This shape-radiance ambiguity is illustrated in Figure 2.

Why does NeRF avoid such degenerate solutions? We hypothesize that two related factors come to NeRF’s rescue: 1) incorrect geometry forces the radiance field to have higher intrinsic complexity (i.e., much higher frequencies) while in contrast 2) NeRF’s specific MLP structure implicitly encodes a smooth BRDF prior on surface reflectance.

Factor 1: As $\sigma$ deviates from the correct shape, $\mathbf{c}$ must in general become a high-frequency function with respect to $\mathbf{d}$ to reconstruct the input images. For the correct shape, the surface light field will generally be much smoother (in fact, constant for Lambertian materials). The higher complexity required for incorrect shapes is more difficult to represent with a limited capacity MLP.

Factor 2: In particular, NeRF’s specific MLP structure encodes an implicit prior favoring smooth surface reflectance functions where $\mathbf{c}$ is smooth with respect to $\mathbf{d}$ at any given surface point $\mathbf{x}$. This MLP structure, shown in Figure 3, treats the scene position $\mathbf{x}$ and the viewing direction $\mathbf{d}$ asymmetrically: $\mathbf{d}$ is injected into the network close to the end of the MLP, meaning that there are fewer MLP parameters, as well as fewer non-linear activations, involved in the creation of view-dependent effects. In addition, the Fourier features used to encode the viewing direction consist only of low-frequency components, i.e., $\gamma^{4}(\cdot)$ vs. $\gamma^{10}(\cdot)$ for encoding $\mathbf{d}$ vs. $\mathbf{x}$ (see Eq. 3). In other words, for a fixed $\mathbf{x}$, the radiance $\mathbf{c}(\mathbf{x},\mathbf{d})$ has limited expressivity with respect to $\mathbf{d}$.

To validate this hypothesis, we perform an experiment where we instead represent $\mathbf{c}$ with a vanilla MLP that treats $\mathbf{x}$ and $\mathbf{d}$ symmetrically—i.e., accepting both as inputs to the first layer and encoding both with $\gamma^{10}(\cdot)$—to eliminate any implicit priors involving viewing direction that arise from the network structure. If we train NeRF from scratch with this alternate model for $\mathbf{c}$, we observe reduced test image quality compared with NeRF’s special MLP, as shown in Figure 4 and Table 1. This result is consistent with our hypothesis that implicit regularization of reflectance in NeRF’s MLP model of radiance $\mathbf{c}$ helps recover correct solutions.

Inverted Sphere Parametrization

The volumetric rendering formula in Eq. 2 integrates over Euclidean depth. When the dynamic range of the true scene depth is small, the integral can be numerically well-approximated with a finite number of samples. However, for outdoor, 360∘ captures centered on nearby objects that also observe the surrounding environment, the dynamic depth range can be extremely large, as the background (buildings, mountains, clouds, etc.) can be arbitrarily far away. Such a high dynamic depth range leads to severe resolution issues in NeRF’s volumetric scene representation, because to synthesize photo-realistic images, the integral in Eq. 2 needs sufficient resolution in both foreground and background areas, which is challenging to achieve by simply sampling points according to a Euclidean parameterization of 3D space. Fig. 5 demonstrates this tradeoff between scene coverage and capturing detail. In a more restricted scenario where all cameras are forward-facing towards a plane separating the cameras from the scene content, NeRF addresses this resolution issue by projectively mapping a subset of the Euclidean space, i.e., a reference camera’s view frustum, to Normalized Device Coordinates (NDC) (McReynolds & Blythe 2005), and integrating in this NDC space. However, this NDC parameterization also fundamentally limits the possible viewpoints, due to its failure to cover the space outside the reference view frustum.

We address this restriction with an inverted sphere parameterization that facilitates free view synthesis. In our representation, we first partition the scene space into two volumes, an inner unit sphere and an outer volume represented by an inverted sphere covering the complement of the inner volume (see Figure 6 for an illustration and Figure 7 for a real-world example of a scene modeled in this way). The inner volume contains the foreground and all the cameras, while the outer volume contains the remainder of the environment.

These two volumes are modelled with two separate NeRFs. To render the color for a ray, they are raycast individually, followed by a final compositition. No re-parameterization is needed for the inner NeRF, as that part of the scene is nicely bounded. For the outer NeRF, we apply an inverted sphere parametrization.

Specifically, a 3D point $(x,y,z),\ \ r=\sqrt{x^{2}+y^{2}+z^{2}}>1$ in the outer volume can be re-parameterized by the quadruple $(x^{\prime},y^{\prime},z^{\prime},1/r),\ \ x^{\prime 2}+y^{\prime 2}+z^{\prime 2}=1$, where $(x^{\prime},y^{\prime},z^{\prime})$ is a unit vector along the same direction as $(x,y,z)$ representing a direction on the sphere, and $0<1/r<1$ is the inverse radius along this direction specifying the point $r\cdot(x^{\prime},y^{\prime},z^{\prime})$ outside the sphere. Unlike Euclidean space where objects can be at unlimited distance from the origin, all the numbers in the re-parametrized quadruple are bounded, i.e., $x^{\prime},y^{\prime},z^{\prime}\in,1/r\in$. This not only improves numeric stability, but also respects the fact that farther objects should get less resolution. We can directly raycast this 4D bounded volume (only 3 degrees of freedom) to render a camera ray’s color. Note that the composite of foreground and background is equivalent to breaking the integral in Eq. 2 into two parts, integration inside the inner and outer volumes. In particular, consider that the ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$ is partitioned into two segments by the unit sphere: in the first, $t\in(0,t^{\prime})$ is inside the sphere; in the second, $t\in(t^{\prime},\infty)$ is outside the sphere. We can rewrite the volumetric rendering integral in Eq. 2 as

Terms (i) and (ii) are computed in Euclidean space, while term (iii) is computed in inverted sphere space with $\frac{1}{r}$ as the integration variable. In other words, we use $\sigma_{\mathit{in}}(\mathbf{o}+t\mathbf{d}),\mathbf{c}_{\mathit{in}}(\mathbf{o}+t\mathbf{d},\mathbf{d})$ in terms (i) and (ii), and $\sigma_{\mathit{out}}(x^{\prime},y^{\prime},z^{\prime},1/r),\mathbf{c}_{\mathit{out}}(x^{\prime},y^{\prime},z^{\prime},1/r,\mathbf{d})$ in term (iii).

In order to compute term (iii) for the ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$, we first need to be able to evaluate $\sigma_{\mathit{out}},\mathbf{c}_{\mathit{out}}$ at any $1/r$; in other words, we need a way to compute $(x^{\prime},y^{\prime},z^{\prime})$ corresponding to a given $1/r$, so that $\sigma_{\mathit{out}},\mathbf{c}_{\mathit{out}}$ can take $(x^{\prime},y^{\prime},z^{\prime},1/r)$ as input. This can be achieved as follows.

As shown in Fig. 8, let the ray intersect the unit sphere at point $\mathbf{a}$, and the midpoint of the chord aligning with the ray be the point $\mathbf{b}$. Point ${\mathbf{a}=\mathbf{o}+t_{a}\mathbf{d}}$ is computed by solving $|\mathbf{o}+t_{a}\mathbf{d}|=1$, while $\mathbf{b}=\mathbf{o}+t_{b}\mathbf{d}$ is acquired by solving ${\mathbf{d}^{T}(\mathbf{o}+t_{b}\mathbf{d})=0}$. Then to get $(x^{\prime},y^{\prime},z^{\prime})$ given $1/r$, we can rotate the vector $\mathbf{a}$ along the vector $\mathbf{b}\times\mathbf{d}$ by the angle $\omega=\arcsin|\mathbf{b}|-\arcsin(|\mathbf{b}|\cdot\frac{1}{r})$. Once we can evaluate $\sigma_{\mathit{out}},\mathbf{c}_{\mathit{out}}$ at any $1/r$, we simply sample a finite number of points from the interval $$ to compute term (iii).

The inverted sphere parameterization for the outer volume has an intuitive physical explanation. It can be viewed in terms of a virtual camera whose image plane is the unit sphere at the scene origin. Hence, the 3D point $(x,y,z)$ is projected to the pixel $(x^{\prime},y^{\prime},z^{\prime})$ on the image plane, while the term $1/r\in(0,1)$ serves as the (inverse) depth, or disparity, of the point. From this perspective, the NDC parameterization suitable only for forward-facing capture is related to our representation, as it uses a virtual pinhole camera rather than a spherical projection surface. In this sense, our inverted sphere parameterization is related to the concept of multi-sphere images (a scene representation consisting of nested concentric spheres sampled according to inverse depth from the sphere centers) proposed in recent work on view synthesis (Attal et al. 2020; Broxton et al. 2020).

Experiments

We validate NeRF++ and compare it with NeRF on two real-world datasets captured with hand-held cameras: Tanks and Temples (T&T) (Knapitsch et al. 2017) and the Light Field (LF) dataset of Yücer et al. 2016. We report PSNR, SSIM, and LPIPS (Zhang et al. 2018) as our quantitative metrics for measuring the quality of synthesized test images.

T&T dataset. We use the training/testing images and SfM poses provided by Riegler & Koltun 2020. This dataset consists of hand-held 360∘ captures of four large-scale scenes; the camera poses are estimated by COLMAP SfM (Schönberger & Frahm 2016). For NeRF, we normalize the scene such that all cameras are inside the sphere of radius $\frac{1}{8}$. This normalization ensures that the unit sphere covers the majority of the background scene content (although some background geometry still lies outside the bounding unit sphere). To numerically compute the volumetric rendering integral for each camera ray, we uniformly sample points from the ray origin to its intersection with the unit sphere. Since NeRF++ ray-casts both inner and outer volumes and hence uses twice the number of samples per camera ray compared to a single volume, we also double the number of samples used by NeRF for the sake of fairness. Specifically, NeRF’s coarse-level MLP uses 128 uniform samples, while the fine-level MLP uses 256 additional importance samples. Under this hyper-parameter setting, NeRF has roughly the same computation cost and GPU memory footprint during training and testing as NeRF++. We randomly sample 2048 camera rays at each training iteration, and train NeRF and NeRF++ for 250k iterations with a learning rate of 5e-4.

LF dataset. We use four scenes from the LF dataset: Africa, Basket, Ship, and Torch. Each scene is densely covered by 2k–4k hand-held captured images, and camera parameters are recovered using SfM. We construct a sparse surrounding capture by temporally subsampling the images by a factor of 40. In particular, the training images are frames 0, 40, 80, …, and the testing images are frames 20, 60, 100, … . The scene normalization method, number of samples per camera ray, batch size, training iterations, and learning rate are the same as in the T&T dataset.

Results. As shown in Table 2, NeRF++ significantly outperforms NeRF in challenging scenarios involving 360∘ captures of objects within large-scale unbounded scenes. In Figure 9, we can see that images synthesized by NeRF++ have significantly higher fidelity.

Open challenges

NeRF++ improves the parameterization of unbounded scenes in which both the foreground and the background need to be faithfully represented for photorealism. However, there remain a number of open challenges. First, the training and testing of NeRF and NeRF++ on a single large-scale scene is quite time-consuming and memory-intensive. Training NeRF++ on a node with 4 RTX 2080 Ti GPUs takes $\sim$24 hours. Rendering a single 1280x720 image on one such GPU takes $\sim$30 seconds at test time. Liu et al. 2020 have sped up the inference, but rendering is still far from real-time. Second, small camera calibration errors may impede photorealistic synthesis. Robust loss functions, such as the contextual loss (Mechrez et al. 2018), could be applied. Third, photometric effects such as auto-exposure and vignetting can also be taken into account to increase image fidelity. This line of investigation is related to the lighting changes addressed in the orthogonal work of Martin-Brualla et al. 2020.

References