PlenOctrees for Real-time Rendering of Neural Radiance Fields

Introduction

Despite the progress of real-time graphics, interactive 3D content with truly photorealistic scenes and objects are still time consuming and costly to produce due to the necessity of optimized 3D assets and dedicated shaders. Instead, many graphics applications opt for image-based solutions. E-commerce websites often use a fixed set of views to showcase their products; VR experiences often rely on 360 video recordings to avoid the costly production of real 3D scenes, and mapping services such as Google Street View stitch images into panoramic views limited to 3-DOF.

Recent advances in neural rendering, such as neural volumes and neural radiance fields (NeRFs) , open a promising new avenue to model arbitrary objects and scenes in 3D from a set of calibrated images. NeRFs in particular can faithfully render detailed scenes and appearances with non-Lambertian effects from any view, while simultaneously offering a high degree of compression in terms of storage. Partly due to these exciting properties, of late, there has been an explosion of research based on NeRF.

Nevertheless, for practical applications, runtime performance remains a critical limitation of NeRFs: due to the extreme sampling requirements and costly neural network queries, rendering a NeRF is agonizingly slow. For illustration, it takes roughly 30 seconds to render an 800x800 image from a NeRF using a high performance GPU, making it impractical for real-time interactive applications.

In this work, we propose a method for rendering a NeRF in real time, achieved by distilling the NeRF into a hierarchical 3D volumetric representation. Our approach preserves NeRF’s ability to synthesize arbitrarily complex geometry and view-dependent effects from any viewpoint and requires no additional supervision. In fact, our method achieves and in many cases surpasses the quality of the original NeRF formulation, while providing significant acceleration. Our model allows us to render an 800x800 image at 167.68 FPS on a NVIDIA V100 GPU and does not rely on a deep neural network during test time. Moreover, our representation is amenable to modern web technologies, allowing interactive rendering in a browser on consumer laptops.

Naive NeRF rendering is slow because it requires dense sampling of the scene, where every sample requires a neural network inference. Because these queries depend on the viewing direction as well as the spatial position, one cannot naively cache these color values for all viewing directions.

We overcome these challenges and enable real-time rendering by pre-sampling the NeRF into a tabulated view-dependent volume which we refer to as a PlenOctree, named after the plenoptic functions of Adelsen and Bergen . Specifically, we use a sparse voxel-based octree where every leaf of the tree stores the appearance and density values required to model the radiance at a point in the volume. In order to account for non-Lambertian materials that exhibit view-dependent effects, we propose to represent the RGB values at a location with spherical harmonics (SH), a standard basis for functions defined on the surface of the sphere. The spherical harmonics can be evaluated at arbitrary query viewing directions to recover the view dependent color.

Although one could convert an existing NeRF into such a representations via projection onto the SH basis functions, we show that we can in fact modify a NeRF network to predict appearances explicitly in terms of spherical harmonics. Specifically, we train a network that produces coefficients for the SH functions instead of raw RGB values, so that the predicted values can later be directly stored within the leaves of the PlenOctree. We also introduce a sparsity prior during NeRF training to improve the memory efficiency of our octrees, consequently allowing us to render higher quality images. Furthermore, once the structure is created, the values stored in PlenOctree can be optimized because the rendering procedure remains differentiable. This enables the PlenOctree to obtain similar or better image quality compared to NeRF. Our pipeline is illustrated in Fig. 2.

Additionally, we demonstrate how our proposed pipeline can be used to accelerate NeRF model training, making our solution more practical to train than the original NeRF approach. Specifically, we can stop training the NeRF model early to convert it into a PlenOctree, which can then be trained significantly faster as it no longer involves any neural networks.

Our experiments demonstrate that our approach can accelerate NeRF-based rendering by 5 orders of magnitude without loss in image quality. We compare our approach on standard benchmarks with scenes and objects captured from $360^{\circ}$ views, and demonstrate state-of-the-art level performance for image quality and rendering speed.

Our interactive viewer can enable operations such as object insertion, visualizing radiance distributions, decomposing the SH components, and slicing the scene. We hope that these real-time operations can be useful to the community for visualizing and debugging NeRF-based representations.

To summarize, we make the following contributions:

The first method that achieves real-time rendering of NeRFs with similar or improved quality.

NeRF-SH: a modified NeRF that is trained to output appearance in terms of spherical basis functions.

PlenOctree, a data structure derived from NeRFs which enables highly efficient view-dependent rendering of complex scenes.

Accelerated NeRF training method using an early training termination, followed by a direct fine-tuning process on PlenOctree values.

Related Work

Novel View Synthesis. The task of synthesizing novel views of a scene given a set of photographs is a well-studied problem with various approaches. All methods predict an underlying geometric or image-based 3D representation that allows rendering from novel viewpoints. Mesh based methods represent the scene with surfaces, and have been used to model Lambertian (diffuse) and non-Lambertian scenes .

Mesh based representations are compact and easy to render; however, optimizing a mesh to fit a complex scene of arbitrary topology is challenging. Image-based rendering methods , on the other hand, enable easy capture as well as photo-realistic and fast rendering, however are often bounded in the viewing angle and do not allow easy editing of the underlying scene.

Volume rendering is a classical technique with a long history of research in the graphics community . Volume-based representations such as voxel grids and multi-plane images (MPIs) are a popular alternative to mesh representations due to their topology-free nature: gradient-based optimization is therefore straightforward, while rendering can still be real-time. However, such naive volumetric representations are often memory bound, limiting the maximum resolution that can be captured. Volumetric octrees are a popular approach for reducing memory and compute in such cases. We refer the reader to this survey for a historical perspective on octree volume rendering. Octrees have been used in recent work to decrease the memory requirements during training for other 3D tasks . Concurrent with this work, NeX extends MPIs to encode spherical basis functions that enable view-dependent rendering effects in real-time. Also concurrently, Lombardi et al. propose to model data using geometric primitives, and also distill NeRFs to enable real-time rendering.

Coordinate-Based Neural Networks. Recently, coordinate-based neural networks have emerged as a popular alternative to explicit volumetric representations, as they are not limited to a fixed voxel representation. These methods train a multilayer perceptron (MLP) whose input is a coordinate and output is some property of space corresponding to that location. These networks have been used to predict occupancy , signed distance fields , and radiance . Coordinate-based neural networks have been used for view synthesis in Scene Representation Networks , NeRFs , and many NeRF extensions . These networks represent a continuous function that can be sampled at arbitrarily fine resolutions without increasing the memory footprint. Unfortunately, this compactness is achieved at the expense of computational efficiency as each sample must be processed by a neural network. As a result, these representations are often slow and impractical for real-time rendering.

NeRF Accelerations. While NeRFs are able to produce high quality results, their computationally expensive rendering leads to slow training and inference. One way to speed up the process of fitting a NeRF to a new scene is to incorporate priors learned from a dataset of similar scenes. This can be accomplished by conditioning on predicted images features or meta-learning . To improve inference speed, Neural Sparse Voxel Fields (NSVF) learns a sparse voxel grid of features that are input into a NeRF like model. The sparse voxel grid allows the renderer to skip over empty regions when tracing a ray which improves the render time $\sim$10x. Decomposed Radiance Fields spatially decomposes a scene into multiple smaller networks. AutoInt modifies the architecture of the NeRF so that inference requires fewer samples but produces lower quality results. None of these approaches achieve real-time. The concurrent work DoNeRF adds a depth classifier to NeRF in order to drastically improve the efficiency of sampling, but requires ground-truth depth for training. Although not based on NeRF, recently Takikawa et al. propose a method to accelerate neural SDF rendering with an octree. Note that this work does not model appearance properties. In contrast, we employ a volumetric representation that can capture photorealistic view-dependent appearances while achieving even higher framerates.

Preliminaries

Neural radiance fields (NeRF) are 3D representations that can be rendered from arbitrary novel viewpoints while capturing continuous geometry and view-dependent appearance. The radiance field is encoded into the weights of a multilayer perceptron (MLP) that can be queried at a position $\mathbf{x}=(x,y,z)$ from a viewing direction $\mathbf{d}=(\theta,\phi)$ to recover the corresponding density $\sigma$ and color $c=(r,g,b)$. A pixel’s predicted color $C(\mathbf{r})$ is computed by casting a ray, $\mathbf{r}$, into the volume and accumulating the color based on density along the ray. NeRF estimates the accumulated color by taking $N$ point samples along the ray to perform volume rendering:

Where $\delta_{i}$ are the distances between point samples. To train the NeRF network, the predicted colors $\hat{C}$ for a batch of rays $\mathcal{R}$ corresponding to pixels in the training images are optimized using Adam to match the target pixel colors:

To better represent high frequency details in the scene the inputs are positionally encoded and two stages of sampling are performed, one coarse and one fine. We refer the interested reader to the NeRF paper paper for details.

One notable consequence of this architecture is that each sample along the ray must be fed to the MLP to obtain the corresponding $\sigma_{i}$ and $c_{i}$. A total of 192 samples were taken for each ray in the examples presented in NeRF. This is inefficient as most samples are sampling free space which do not contribute to the integrated color. To render a single target image at $800\times 800$ resolution, the network must be run on over 100 million inputs. Therefore it takes about 30 seconds to render a single frame using a NVIDIA V100 GPU, making it impractical for real-time applications. Our use of a sparse voxel octree avoids excess compute in regions without content. Additionally we precompute the values for each voxel so that network queries are not performed during inference.

Method

We propose a pipeline that enables real-time rendering of NeRFs. Given a trained NeRF, we can convert it into a PlenOctree, an efficient data structure that is able to represent non-Lambertian effects in a scene. Specifically, it is an octree which stores spherical harmonics (SH) coefficients at the leaves, encoding view-dependent radiance.

To make the conversion to PlenOctree more straightforward, we also propose NeRF-SH, a variant of the NeRF network which directly outputs the SH coefficients, thus eliminating the need for a view-direction input to the network. With this change, the conversion can then be performed by evaluating on a uniform grid followed by thresholding. We fine-tune the octree on the training images to further improve image quality, Please see Fig. 2 for a graphical illustration of our pipeline.

The conversion process leverages the continuous nature of NeRF to dynamically obtain the spatial structure of the octree. We show that even with a partially trained NeRF, our PlenOctree is capable of producing results competitive with the fully trained NeRF.

SHs have been a popular low-dimensional representation for spherical functions and have been used to model Lambertian surfaces or even glossy surfaces . Here we explore its use in a volumetric context. Specifically, we adapt the NeRF network $f$ to output spherical harmonics coefficients $\mathbf{k}$ , rather than RGB values.

Where $S:x\mapsto(1+\exp(-x))^{-1}$ is the sigmoid function for normalizing the colors. In other words, we factorize the view-dependent appearance with the SH basis, eliminating the view-direction input to the network and removing the need to sample view directions at conversion time. Please see the appendix for more technical discussion of SHs. With a single evaluation of the network, we can now efficiently query colors from arbitrary viewing angles at inference time. In Fig. 7, one can see that NeRF-SH training speed is similar to, but slightly faster than, NeRF (by about 10%).

Note that we can also project a trained NeRF to SHs directly at each point by sampling NeRF at random directions and multiplying by the SH component values to form Monte Carlo estimates of the inner products. However, this sampling process takes several hours to achieve reasonable quality and imposes a quality loss of about 2 dB.With 10000 view-direction samples per point, taking about 2 hours, the PSNR is 29.21 vs. 31.02 for our main method prior to optimization. Nevertheless, this alternative approach offers a pathway to convert existing NeRFs into PlenOctrees.

Other than SHs, we also experiment with Spherical Gaussians (SG) , a learnable spherical basis which have been used to represent all-frequency lighting . We find that SHs perform better in our use case and provide an ablation in the appendix.

Sparsity prior. Without any regularization, the model is free to generate arbitrary geometry in unobserved regions. While this does not directly worsen image quality, it would adversely impact our conversion process as the extra geometry occupies significant voxel space.

To solve this problem, we introduce an additional sparsity prior during NeRF training. Intuitively, this prior encourages NeRF to choose empty space when both space and solid colors are possible solutions. Formally,

Here, $\{\sigma_{k}\}_{k=1}^{K}$ are the evaluated density values at $K$ uniformly random points within the bounding box, and $\lambda$ is a hyperparameter. The final training loss is then $\beta_{\text{sparsity}}\mathcal{L}_{\text{sparsity}}+\mathcal{L}_{\text{RGB}}$, where $\beta_{\text{sparsity}}$ is a hyperparameter. Fig. 3 illustrates the effect of the prior.

2 PlenOctree: Octree-based Radiance Fields

Once we have trained a NeRF-SH model, we can convert it into a sparse octree representation for real time rendering. A PlenOctree stores density and SH coefficients modelling view-dependent appearance at each leaf. We describe the conversion and rendering processes below.

Rendering. To render the PlenOctree, for each ray, we first determine ray-voxel intersections in the octree structure. This produces a sequence of segments between voxel boundaries with constant density and color, of lengths $\{\delta_{i}\}_{i=1}^{N}$. NeRF’s volume rendering model (1) is then applied to assign a color to the ray. This approach allows for skipping large voxels in one step while also not missing small voxels.

At test-time, we further accelerate this rendering process by applying early-stopping when the ray has accumulated transmittance $T_{i}$ less than $\gamma=0.01$.

Conversion from NeRF-SH. The conversion process can be divided into three steps. At a high level, we evaluate the network on a grid, retaining only density values, then filter the voxels via thresholding. Finally we sample random points within each remaining voxel and average them to obtain SH coefficients to store in the octree leaves. More details are given below:

Evaluation. We first evaluate the NeRF-SH model to obtain $\sigma$ values on a uniformly spaced 3D grid. The grid is automatically scaled to tightly fit the scene content. By pre-evaluating $\sigma$ on a larger grid and finding the bounding box of all points with $\sigma\geq\tau_{a}$.

Filtering. Next, we filter this grid to obtain a sparse set of voxels centered at the grid points sufficient for representing the scene. Specifically, we render alpha maps for all the training views using this voxel grid, keeping track of the maximum ray weight $1-\exp(-\sigma_{i}\delta_{i})$ at each voxel. We then eliminate the voxels whose weights are lower than a threshold $\tau_{w}$. The octree is constructed to contain the remaining voxels as leaves at the deepest level while being empty elsewhere. Compared to naively thresholding by $\sigma$ at each point, this method eliminates non-visible voxels.

Sampling. Finally, we sample a set of $256$ random points in each remaining voxel and set the associated leaf of the octree to the mean of these values to reduce aliasing. Each leaf now contains the density $\sigma$ and a vector of spherical harmonics coefficients for each of the RGB color channels.

This full extraction process takes about 15 minutes. Note that sampling $8$ points instead of $256$ allows for extraction in about $1.5$ minutes, with minimal loss in quality.

3 PlenOctree Optimization

Since this volume rendering process is fully differentiable with respect to the tree values, we can directly fine-tune the resulting octree on the original training images using the NeRF loss (3) with SGD in order to improve the image quality. Note that the tree structure is fixed to that obtained from NeRF in this process. PlenOctree optimization operates at about $3$ million rays per second, compared to about $9000$ for NeRF training, allowing us to optimize for many epochs in a relatively short time. The analytic derivatives for this process are implemented in custom CUDA kernels. We defer technical details to the appendix.

The fast octree optimization indirectly allows us to accelerate NeRF training, as seen in Fig. 7, since we can stop the NeRF-SH training at an earlier time for constructing the PlenOctree, with only a slight degradation in quality.

Results

Datasets. For our experiments, we use the NeRF-synthetic dataset and a subset of the Tanks and Temples dataset . The NeRF-synthetic dataset consists of 8 scenes where each scene has a central object with 100 inward facing cameras distributed randomly on the upper hemisphere. The images are $800\times 800$ with provided ground truth camera poses. The Tanks and Temples subset is from NSVF and contains 5 scenes of real objects captured by an inward facing camera circling the scene. We use foreground masks provided by the NSVF authors. Each scene contains between 152-384 images of size $1920\times 1080$.

Baselines. The principal baseline for our experiments is NeRF ; we report results for both the original NeRF implementation, denoted NeRF (original) as well as a reimplementation in Jax , denoted simply NeRF, which our NeRF-SH code is based off of. Unless otherwise stated, all NeRF results and timings are from the latter implementation. We compare also to two recent papers introducing NeRF accelerations, neural sparse voxel fields (NSVF) and AutoInt , as well as two older methods, scene representation networks (SRN) and Neural Volumes .

2 Quality Evaluation

We evaluate our approach against prior works on the synthetic and real datasets mentioned above. The results are in Tables 1 and Table 2 respectively. Note that none of the baselines achieve real-time performance; nevertheless, our quality results are competitive in all cases and better in terms of some metrics.

In Figures 4 and 6, we show qualitative examples that demonstrate that our PlenOctree conversion does not perceptually worsen the rendered images compared to NeRF; rather, we observe that the PlenOctree optimization process enhances fine details such as text. Additionally, we note that our modifications of NeRF to predict spherical function coefficients (NeRF-SH) does not significantly change the performance.

3 Speed Trade-off Analysis

A number of parameters for PlenOctree conversion and rendering can be tuned to trade-off between speed and image quality. In Figure 5 and Table 3 we compared image accuracy and inference time for four variants of PlenOctree that sweep this trade-off.

4 Indirect Acceleration of NeRF Training

Since we can efficiently fine-tune the octree on the original training data, as briefly discussed in §4.3, we can choose to stop the NeRF-SH training at an earlier time before converting it to a PlenOctree. Indeed, we have found that the image quality improvements gained during fine-tuning can often be greater than continuing to train the NeRF-SH an equivalent amount of time. Therefore it can be more time efficient to stop the NeRF-SH training before it has converged and transition to PlenOctree conversion and fine-tuning.

In Figure 7 we compare NeRF and NeRF-SH models trained for 2 million iterations each to a sequence of PlenOctree models extracted from NeRF-SH checkpoints. We find that given a time constraint, it is almost always preferable to stop the NeRF training and transition to PlenOctree optimization.

5 Real-time and In-browser Applications

Interactive demos. Within our desktop viewer, we are able to perform a variety of real-time scene operations on the PlenOctree representation. For example, it is possible to insert meshes while maintaining proper occlusion, slice the PlenOctree to visualize a cross-section, or render the depth map to verify the geometry. Other features include probing the radiance distribution at any point in space, and inspecting subsets of SH components. These examples are demonstrated in Figure 9. The ability to perform these actions in real-time is beneficial both for interactive entertainment and debugging NeRF-related applications.

Web renderer. We have implemented a web-based renderer enabling interactive viewing of PlenOctrees in the browser. This is achieved by rewriting our CUDA-based PlenOctree renderer as a WebGL-compatible fragment shader and applying compressions to reduce file sizes. Please see the appendix for more information.

Discussion

We have introduced a new data representation for NeRFs using PlenOctrees, which enables real-time rendering capabilities for arbitrary objects and scenes. Not only can we accelerate the rendering performance of the original NeRF method by more than 3000 times, but we can produce images that are either equal or better quality than NeRF thanks to our hierarchical data structure. As training time poses another hurdle for adopting NeRFs in practice (taking 1-2 days to fully converge), we also showed that our PlenOctrees can accelerate effective training time for our NeRF-SH. Finally, we have implemented an in-browser viewer based on WebGL to demonstrate real-time and 6-DOF rendering capabilities of NeRFs on consumer laptops. In the future, our approach may enable virtual online stores in VR, where any products with arbitrary complexity and materials can be visualized in real-time while enabling 6-DOF viewing.

While we achieve state-of-the-art rendering performance and frame rates, the octree representation is much larger than the compact representation of the original NeRF model and has a larger memory footprint. The average uncompressed octree size for the full model is 1.93 GB on the synthetic dataset and 3.53 GB on the Tanks and Temples dataset. For online delivery, we use lower-resolution compressed models which are about 30-120 MB; please see the appendix for details. Although already possible in some form (Fig. 8), applying our method to unbounded and forward-facing scenes optimally requires further work as the data distribution is different for unbounded scenes. The forward-facing scenes inherently do not support 6-DOF viewing, and we suggest MPIs may be more appropriate in this case .

In the future, we plan to explore extensions of our method to enable real-time 6-DOF immersive viewing of large-scale scenes, as well as of dynamic scenes. We believe that real-time rendering of NeRFs has the potential to become a new standard for next-generation AR/VR technologies, as photorealistic 3D content can be digitized as easily as recording 2D videos.

Acknowledgements

We thank Vickie Ye and Ben Recht for help discussions as well as reviewing the manuscript, Zejian Wang of Pinscreen for helping with video capture, and BAIR commons for an allocation of GCP credits.

References

Appendix

Appendix A Additional Results

Here we provide further qualitative comparisons with baselines: SRN , Neural Volumes , NSVF in Figure 10. We show more qualitative results of our method in Figure 11 and Figure 12. We also report a per-scene breakdown of the quantitative metrics against all approaches in Table 5, 6, 7, 8.

A.2 Spherical Basis Function Ablation

We also provide ablation studies on the choice of spherical basis functions. We first ablate the effect on the number of spherical harmonics basis, then we explore the use of a learnable spherical basis functions. All experiments are conducted on NeRF-synthetic dataset and we report the average metric directly after training NeRF with spherical basis functions and after converting it to PlenOctrees with fine-tuning.

Furthermore, we also experimented with spherical Gaussians (SGs) , which is another form of spherical basis functions similar to spherical harmonics, but with learnable Gaussian kernels. Please see §B.1 for a brief introduction of SHs and SGs. SG-25 denotes our model using 25 SG components instead of SH, all with learnable lobe axis $\mathbf{p}$ and bandwidth $\lambda$. However, while this model has marginally better PSNR, the advantage disappears following PlenOctree conversion and fine-tuning.

Appendix B Technical Details

In the main paper, we used the SH functions without defining their exact form. Here, we provide a brief technical discussion of both spherical harmonics (SH) and spherical Gaussians (SG) for completeness.

Spherical Gaussians (SGs), also known as the von Mises-Fisher distribution , is another form of spherical basis functions that have been widely adopted to approximate spherical functions. Unlike SHs, SGs are a learnable basis. A normalized SG is defined as:

B.2 PlenOctree Compression

The uncompressed PlenOctree file would be unpleasantly time-consuming for users to download for in-browser rendering. Thus, to minimize the size of PlenOctrees for viewing in the browser, we use SH-9 instead of SH-16 or SH-25 and apply a looser bounding box, which reduces the number of occupied voxels. On top of this, we compress the PlenOctrees directly in the following ways:

We compress the entire tree, including pointers, using the standard DEFLATE algorithm from ZLIB .

This process reduces the file size by as much as $20$-$30$ times. The tree is fully decompressed before it is displayed in the web renderer. We will also release this code.

B.3 Analytic Derivatives of PlenOctree Rendering

In this section, we derive the analytic derivatives of the NeRF piecewise constant volume rendering model for optimizing PlenOctrees directly. Throughout this section we will consider a fixed ray with a given origin and direction.

For preciseness, we provide definitions of quantities used in NeRF volume rendering. The NeRF rendering model considers a ray divided into $N$ consecutive segments with endpoints $\{t_{i}\}_{i=0}^{N}$, where $t_{0}$ and $t_{N}$ are the near and far bounds. The segments have constant densities $\bm{\sigma}=(\sigma_{0},\ldots,\sigma_{N-1})$ where each $\sigma_{i}\geq 0$. If we shine a light of intensity $1$ at $t_{i}$, then at the camera position $t_{0}$, the light intensity is given by

Where $\delta_{i}:=t_{i+1}-t_{i}$ are segment lengths as in $\S$3 of the main paper. Note that $T_{i}$ is also known as the accumulated transmittance from $t_{0}$ to $t_{i}$, and is the same as the definition in (1). It can be shown that this precisely models the absorption within each segment in the piecewise-constant setting.

Let $\mathbf{c}=(c_{0},\ldots c_{N-1})$ be the color associated with segments $0,\ldots,N-1$, and $c_{N}$ be the background light intensity; each $c_{0},\ldots,c_{N}\in^{3}$ is an RGB color. We are interested in the derivative of the rendered color $\hat{C}(\bm{\sigma},\mathbf{c})$ with respect to $\bm{\sigma}$ and $\mathbf{c}$. Note $c_{N}$ (background) is typically considered a hyperparameter.

B.3.2 Derivation of the Derivatives

From the original NeRF rendering equation (1), we can express the rendered ray color $\hat{C}(\bm{\sigma},\mathbf{c})$ as:

Where $w_{i}(\bm{\sigma})=T_{i}(\bm{\sigma})\left(1-\exp(-\sigma_{i}\delta_{i})\right)=T_{i}(\bm{\sigma})-T_{i+1}(\bm{\sigma})$ are segment weights, and $w_{N}(\bm{\sigma})=T_{N}(\bm{\sigma})$. Note that the background color $c_{N}$ was omitted in equation (1) of the main paper for simplicity.

Since the rendered color are a convex combination of the segment colors, it’s immediately clear that

Handling spherical harmonics colors is straightforward by applying the chain rule, noting that the SH basis function values are constant across the ray.

This is slightly more tricky. We can write the derivative wrt. $\sigma_{i}$,

Where the derivative of the intensity $T_{k}$, is

$1_{k>i}$ denotes an indicator function whose value is $1$ if $k>i$ or else. Basically, we can delete any $T_{k}$ for $k\leq i$ from the original expression, then multiply by $-\delta_{i}$. Therefore we can simplify (16) as follows

Within the PlenOctree renderer, this gradient can be computed in two rendering passes; the second pass is needed due to dependency on “future” weights and colors not seen by the ray marching process. The first pass store $\sum_{k=0}^{N}c_{k}\;w_{k}(\bm{\sigma})$, then subtracting a prefix from it. The overhead is still relatively small, and auxiliary memory use is constant.

B.4 NeRF-SH Training Details

Our NeRF-SH model is built upon a Jax reimplementation of NeRF . In our experiments, we use a batch size of 1024 rays, each with 64 sampled points in the coarse volume and 128 additional sampled points in the fine volume. The model is optimized with the Adam optimizer using a learning rate that starts at $5\times 10^{-4}$ and decays exponentially to $5\times 10^{-6}$ over the training process. All of our models are trained for 2M iterations under the same protocol. Training takes around 50 hours to converge for each model on a single NVIDIA V100 GPU.

B.5 PlenOctree Optimization Details

After converting the NeRF-SH model into a PlenOctree, we further optimize the PlenOctree on the training set with SGD using the NeRF loss; note we no longer apply the sparsity loss here since the octree is already sparse. For NeRF-synthetic dataset, we use a constant $1\times 10^{7}$ learning rate and optimize for maximum 80 epochs. For Tanks&Temples dataset, we set the learning rate to $1.5\times 10^{6}$ and the maximum epochs to 40. We applied early stopping for the optimization process by monitoring the PSNR on the validation setFor Tanks&Temples dataset, we hold out 10% of the training set as validation set only for PlenOctree optimization.. On average it takes around 10 minutes to finish the PlenOctree optimization for each scene on a single NVIDIA V100 GPU. The entire optimization process is done in float32 for stability, but after it we storage the PlenOctree with float16 to reduce the model size.