Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

Introduction

Computer graphics primitives are fundamentally represented by mathematical functions that parameterize appearance. The quality and performance characteristics of the mathematical representation are crucial for visual fidelity: we desire representations that remain fast and compact while capturing high-frequency, local detail. Functions represented by multi-layer perceptrons (MLPs), used as neural graphics primitives, have been shown to match these criteria (to varying degree), for example as representations of shape [Park et al., 2019; Martel et al., 2021] and radiance fields [Mildenhall et al., 2020; Liu et al., 2020; Müller et al., 2020, 2021].

The important commonality of the these approaches is an encoding that maps neural network inputs to a higher-dimensional space, which is key for extracting high approximation quality from compact models. Most successful among these encodings are trainable, task-specific data structures [Takikawa et al., 2021; Liu et al., 2020] that take on a large portion of the learning task. This enables the use of smaller, more efficient MLPs. However, such data structures rely on heuristics and structural modifications (such as pruning, splitting, or merging) that may complicate the training process, limit the method to a specific task, or limit performance on GPUs where control flow and pointer chasing is expensive.

Key to both the task-independent adaptivity and efficiency is a multiresolution hierarchy of hash tables:

Adaptivity: we map a cascade of grids to corresponding fixed-size arrays of feature vectors. At coarse resolutions, there is a 1:1 mapping from grid points to array entries. At fine resolutions, the array is treated as a hash table and indexed using a spatial hash function, where multiple grid points alias each array entry. Such hash collisions cause the colliding training gradients to average, meaning that the largest gradients—those most relevant to the loss function—will dominate. The hash tables thus automatically prioritize the sparse areas with the most important fine scale detail. Unlike prior work, no structural updates to the data structure are needed at any point during training.

Efficiency: our hash table lookups are $\mathcal{O}(1)$ and do not require control flow. This maps well to modern GPUs, avoiding execution divergence and serial pointer-chasing inherent in tree traversals. The hash tables for all resolutions may be queried in parallel.

We validate our multiresolution hash encoding in four representative tasks (see Figure 1):

Gigapixel image: the MLP learns the mapping from 2D coordinates to RGB colors of a high-resolution image.

Neural signed distance functions (SDF): the MLP learns the mapping from 3D coordinates to the distance to a surface.

Neural radiance caching (NRC): the MLP learns the 5D light field of a given scene from a Monte Carlo path tracer.

Neural radiance and density fields (NeRF): the MLP learns the 3D density and 5D light field of a given scene from image observations and corresponding perspective transforms.

In the following, we first review prior neural network encodings (Section 2), then we describe our encoding (Section 3) and its implementation (Section 4), followed lastly by our experiments (Section 5) and discussion thereof (Section 6).

Background and Related Work

Early examples of encoding the inputs of a machine learning model into a higher-dimensional space include the one-hot encoding [Harris and Harris, 2013] and the kernel trick [Theodoridis, 2008] by which complex arrangements of data can be made linearly separable.

This has been adopted in computer graphics to encode the spatio-directionally varying light field and volume density in the NeRF algorithm [Mildenhall et al., 2020]. The five dimensions of this light field are independently encoded using the above formula; this was later extended to randomly oriented parallel wavefronts [Tancik et al., 2020] and level-of-detail filtering [Barron et al., 2021a]. We will refer to this family of encodings as frequency encodings. Notably, frequency encodings followed by a linear transformation have been used in other computer graphics tasks, such as approximating the visibility function [Jansen and Bavoil, 2010; Annen et al., 2007].

Müller et al. [2019; 2020] suggested a continuous variant of the one-hot encoding based on rasterizing a kernel, the one-blob encoding, which can achieve more accurate results than frequency encodings in bounded domains at the cost of being single-scale.

Sparse parametric encodings.

While existing parametric encodings tend to yield much greater accuracy than their non-parametric predecessors, they also come with downsides in efficiency and versatility. Dense grids of trainable features consume much more memory than the neural network weights. To illustrate the trade-offs and to motivate our method, Figure 2 shows the effect on reconstruction quality of a neural radiance field for several different encodings. Without any input encoding at all (a), the network is only able to learn a fairly smooth function of position, resulting in a poor approximation of the light field. The frequency encoding (b) allows the same moderately sized network (8 hidden layers, each 256 wide) to represent the scene much more accurately. The middle image (c) pairs a smaller network with a dense grid of $128^{3}$ trilinearly interpolated, 16-dimensional feature vectors, for a total of 33.6 million trainable parameters. The large number of trainable parameters can be efficiently updated, as each sample only affects 8 grid points.

However, the dense grid is wasteful in two ways. First, it allocates as many features to areas of empty space as it does to those areas near the surface. The number of parameters grows as $\mathcal{O}(N^{3})$, while the visible surface of interest has surface area that grows only as $\mathcal{O}(N^{2})$. In this example, the grid has resolution $128^{3}$, but only $53\,807$ $(2.57\%)$ of its cells touch the visible surface.

Second, natural scenes exhibit smoothness, motivating the use of a multi-resolution decomposition [Hadadan et al., 2021; Chibane et al., 2020]. Figure 2 (d) shows the result of using an encoding in which interpolated features are stored in eight co-located grids with resolutions from $16^{3}$ to $173^{3}$, each containing 2-dimensional feature vectors. These are concatenated to form a 16-dimensional (same as (c)) input to the network. Despite having less than half the number of parameters as (c), the reconstruction quality is similar.

If the surface of interest is known a priori, a data structure such as an octree [Takikawa et al., 2021] or sparse grid [Liu et al., 2020; Chabra et al., 2020; Jiang et al., 2020; Peng et al., 2020a; Hadadan et al., 2021; Chibane et al., 2020] can be used to cull away the unused features in the dense grid. However, in the NeRF setting, surfaces only emerge during training. NSVF [Liu et al., 2020] and several concurrent works [Yu et al., 2021a; Sun et al., 2021] adopt a multi-stage, coarse to fine strategy in which regions of the feature grid are progressively refined and culled away as necessary. While effective, this leads to a more complex training process in which the sparse data structure must be periodically updated.

Our method—Figure 2 (e,f)—combines both ideas to reduce waste. We store the trainable feature vectors in a compact spatial hash table, whose size is a hyper-parameter $T$ which can be tuned to trade the number of parameters for reconstruction quality. It neither relies on progressive pruning during training nor on a priori knowledge of the geometry of the scene. Analogous to the multi-resolution grid in (d), we use multiple separate hash tables indexed at different resolutions, whose interpolated outputs are concatenated before being passed through the MLP. The reconstruction quality is comparable to the dense grid encoding, despite having ${20\times}$ fewer parameters.

Unlike prior work that used spatial hashing [Teschner et al., 2003] for 3D reconstruction [Nießner et al., 2013], we do not explicitly handle collisions of the hash functions by typical means like probing, bucketing, or chaining. Instead, we rely on the neural network to learn to disambiguate hash collisions itself, avoiding control flow divergence, reducing implementation complexity and improving performance. Another performance benefit is the predictable memory layout of the hash tables that is independent of the data that is represented. While good caching behavior is often hard to achieve with tree-like data structures, our hash tables can be fine-tuned for low-level architectural details such as cache size.

Multiresolution Hash Encoding

We use a spatial hash function [Teschner et al., 2003] of the form

where $\oplus$ denotes the bit-wise XOR operation and $\pi_{i}$ are unique, large prime numbers. Effectively, this formula XORs the results of a per-dimension linear congruential (pseudo-random) permutation [Lehmer, 1951], decorrelating the effect of the dimensions on the hashed value. Notably, to achieve (pseudo-)independence, only ${d-1}$ of the $d$ dimensions must be permuted, so we choose ${\pi_{1}:=1}$ for better cache coherence, {\pi_{2}=2\,654\,435\,761}, and {\pi_{3}=805\,459\,861}.

Lastly, the feature vectors at each corner are $d$-linearly interpolated according to the relative position of $\mathbf{x}$ within its hypercube, i.e. the interpolation weight is $\mathbf{w}_{l}:=\mathbf{x}_{l}-\lfloor\mathbf{x}_{l}\rfloor$.

Choosing the hash table size $T$ provides a trade-off between performance, memory and quality. Higher values of $T$ result in higher quality and lower performance. The memory footprint is linear in $T$, whereas quality and performance tend to scale sub-linearly. We analyze the impact of $T$ in Figure 4, where we report test error vs. training time for a wide range of $T$-values for three neural graphics primitives. We recommend practitioners to use $T$ to tweak the encoding to their desired performance characteristics.

The hyperparameters $L$ (number of levels) and $F$ (number of feature dimensions) also trade off quality and performance, which we analyze for an approximately constant number of trainable encoding parameters $\theta$ in Figure 5. In this analysis, we found ${(F=2,L=16)}$ to be a favorable Pareto optimum in all our applications, so we use these values in all other results and recommend them as the default.

Implicit hash collision resolution.

It may appear counter-intuitive that this encoding is able to reconstruct scenes faithfully in the presence of hash collisions. Key to its success is that the different resolution levels have different strengths that complement each other. The coarser levels, and thus the encoding as a whole, are injective—that is, they suffer from no collisions at all. However, they can only represent a low-resolution version of the scene, since they offer features which are linearly interpolated from a widely spaced grid of points. Conversely, fine levels can capture small features due to their fine grid resolution, but suffer from many collisions—that is, disparate points which hash to the same table entry. Nearby inputs with equal integer coordinates ${\lfloor\mathbf{x}_{l}\rfloor}$ are not considered a collision; a collision occurs when different integer coordinates hash to the same index. Luckily, such collisions are pseudo-randomly scattered across space, and statistically unlikely to occur simultaneously at every level for a given pair of points.

When training samples collide in this way, their gradients average. Consider that the importance to the final reconstruction of such samples is rarely equal. For example, a point on a visible surface of a radiance field will contribute strongly to the reconstructed image (having high visibility and high density, both multiplicatively affecting the magnitude of gradients) causing large changes to its table entries, while a point in empty space that happens to refer to the same entry will have a much smaller weight. As a result, the gradients of the more important samples dominate the collision average and the aliased table entry will naturally be optimized in such a way that it reflects the needs of the higher-weighted point.

Online adaptivity.

Note that if the distribution of inputs $\mathbf{x}$ changes over time during training, for example if they become concentrated in a small region, then finer grid levels will experience fewer collisions and a more accurate function can be learned. In other words, the multiresolution hash encoding automatically adapts to the training data distribution, inheriting the benefits of tree-based encodings [Takikawa et al., 2021] without task-specific data structure maintenance that might cause discrete jumps during training. One of our applications, neural radiance caching in Section 5.3, continually adapts to animated viewpoints and 3D content, greatly benefitting from this feature.

d𝑑d-linear interpolation.

Implementation

To demonstrate the speed of the multiresolution hash encoding, we implemented it in CUDA and integrated it with the fast fully-fused MLPs of the tiny-cuda-nn framework [Müller, 2021].We observe speed-ups on the order of ${10\times}$ compared to a naïve Python implementation. We therefore also release PyTorch bindings around our hash encoding and fully fused MLPs to permit their use in existing projects with little overhead. We release the source code of the multiresolution hash encoding as an update to Müller and the source code pertaining to the neural graphics primitives at https://github.com/nvlabs/instant-ngp.

In order to optimize inference and backpropagation performance, we store hash table entries at half precision (2 bytes per entry). We additionally maintain a master copy of the parameters in full precision for stable mixed-precision parameter updates, following Micikevicius et al. .

To optimally use the GPU’s caches, we evaluate the hash tables level by level: when processing a batch of input positions, we schedule the computation to look up the first level of the multiresolution hash encoding for all inputs, followed by the second level for all inputs, and so on. Thus, only a small number of consecutive hash tables have to reside in caches at any given time, depending on how much parallelism is available on the GPU. Importantly, this structure of computation automatically makes good use of the available caches and parallelism for a wide range of hash table sizes $T$.

The optimal number of feature dimensions $F$ per lookup depends on the GPU architecture. On one hand, a small number favors cache locality in the previously mentioned streaming approach, but on the other hand, a large $F$ favors memory coherence by allowing for $F$-wide vector load instructions. ${F=2}$ gave us the best cost-quality trade-off (see Figure 5) and we use it in all experiments.

Architecture.

Initialization.

We initialize neural network weights according to Glorot and Bengio to provide a reasonable scaling of activations and their gradients throughout the layers of the neural network. We initialize the hash table entries using the uniform distribution ${\mathcal{U}}(-10^{-4},10^{-4})$ to provide a small amount of randomness while encouraging initial predictions close to zero. We also tried a variety of different distributions, including zero-initialization, which all resulted in a very slightly worse initial convergence speed. The hash table appears to be robust to the initialization scheme.

Training.

We jointly train the neural network weights and the hash table entries by applying Adam [Kingma and Ba, 2014], where we set ${\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=10^{-15}}$, The choice of $\beta_{1}$ and $\beta_{2}$ makes only a small difference, but the small value of $\epsilon=10^{-15}$ can significantly accelerate the convergence of the hash table entries when their gradients are sparse and weak. To prevent divergence after long training periods, we apply a weak L2 regularization (factor $10^{-6}$) to the neural network weights, but not to the hash table entries.

We observed fastest convergence with a learning rate of $10^{-4}$ for signed distance functions and $10^{-2}$ otherwise, as well a a batch size of $2^{14}$ for neural radiance caching and $2^{18}$ otherwise.

Experiments

To highlight the versatility and high quality of the encoding, we compare it with previous encodings in four distinct computer graphics primitives that benefit from encoding spatial coordinates.

Learning the 2D to RGB mapping of image coordinates to colors has become a popular benchmark for testing a model’s ability to represent high-frequency detail [Sitzmann et al., 2020; Müller et al., 2019; Martel et al., 2021; Tancik et al., 2020]. Recent breakthroughs in adaptive coordinate networks (ACORN) [Martel et al., 2021] have shown impressive results when fitting very large images—up to a billion pixels—with high fidelity at even the smallest scales. We target our multiresolution hash encoding at the same task and converge to high-fidelity images in seconds to minutes (Figure 4).

It is difficult to directly compare the performance of our encoding to ACORN; a factor of ${\sim\!10}$ stems from our use of fully fused CUDA kernels, provided by the tiny-cuda-nn framework [Müller, 2021]. The input encoding allows for the use of a much smaller MLP than with ACORN, which accounts for much of the remaining ${10\times}$–${100\times}$ speedup. That said, we believe that the biggest value-add of the multiresolution hash encoding is its simplicity. ACORN relies on an adaptive subdivision of the scene as part of a learning curriculum, none of which is necessary with our encoding.

2. Signed Distance Functions

Signed distance functions (SDFs), in which a 3D shape is represented as the zero level-set of a function of position $\mathbf{x}$, are used in many applications including simulation, path planning, 3D modeling, and video games. DeepSDF [Park et al., 2019] uses a large MLP to represent one or more SDFs at a time. In contrast, when just a single SDF needs to be fit, a spatially learned encoding, such as ours can be employed and the MLP shrunk significantly. This is the application we investigate in this section. As baseline, we compare with NGLOD [Takikawa et al., 2021], which achieves state-of-the-art results in both quality and speed by prefixing its small MLP with a lookup from an octree of trainable feature vectors. Lookups along the hierarchy of this octree act similarly to our multiresolution cascade of grids: they are a collision-free analog to our technique, with a fixed growth factor ${b=2}$. To allow meaningful comparisons in terms of both performance and quality, we implemented an optimized version of NGLOD in our framework, details of which we describe in Appendix B. Details pertaining to real-time training of SDFs are described in Appendix C.

In Figure 7, we compare NGLOD with our multiresolution hash encoding at roughly equal parameter count. We also show a straightforward application of the frequency encoding [Mildenhall et al., 2020] to provide a baseline, details of which are found in Appendix D. By using a data structure tailored to the reference shape, NGLOD achieves the highest visual reconstruction quality. However, even without such a dedicated data structure, our encoding approaches a similar fidelity to NGLOD in terms of the intersection-over-union metric (IoUIoU is the ratio of volumes of the interiors of the intersection and union of the pair of shapes being compared. IoU is always ${\leq 1}$ with a perfect fit corresponding to ${=1}$. We measure IoU by comparing the signs of the SDFs at 128 million points uniformly distributed within the bounding box of the scene.) with similar performance and memory cost.

Furthermore, the SDF is defined everywhere within the training volume, as opposed to NGLOD, which is only defined within the octree (i.e. close to the surface). This permits the use of certain SDF rendering techniques such as approximate soft shadows from a small number of off-surface distance samples [Evans, 2006], as shown in the adjacent figure.

To emphasize differences between the compared methods, we visualize the SDF using a shading model. The resulting colors are sensitive to even slight changes in the surface normal, which emphasizes small fluctuations in the prediction more strongly than in other graphics primitives where color is predicted directly. This sensitivity reveals undesired microstructure in our hash encoding on the scale of the finest grid resolution, which is absent in NGLOD and does not disappear with longer training times. Since NGLOD is essentially a collision-free analog to our hash encoding, we attribute this artifact to hash collisions. Upon close inspection, similar microstructure can be seen in other neural graphics primitives, although with significantly lower magnitude.

3. Neural Radiance Caching

In neural radiance caching [Müller et al., 2021], the task of the MLP is to predict photorealistic pixel colors from feature buffers; see Figure 8. The MLP is run independently for each pixel (i.e. the model is not convolutional), so the feature buffers can be treated as per-pixel feature vectors that contain the 3D coordinate $\mathbf{x}$ as well as additional features. We can therefore directly apply our multiresolution hash encoding to $\mathbf{x}$ while treating all additional features as auxiliary encoded dimensions $\mathbf{\xi}$ to be concatenated with the encoded position, using the same encoding as Müller et al. . We integrated our work into Müller et al.’s implementation of neural radiance caching and therefore refer to their paper for implementation details.

For photorealistic rendering, the neural radiance cache is typically queried only for indirect path contributions, which masks its reconstruction error behind the first reflection. In contrast, we would like to emphasize the neural radiance cache’s error, and thus the improvement that can be obtained by using our multiresolution hash encoding, so we directly visualize the neural radiance cache at the first path vertex.

4. Neural Radiance and Density Fields (NeRF)

In the NeRF setting, a volumetric shape is represented in terms of a spatial (3D) density function and a spatiodirectional (5D) emission function, which we represent by a similar neural network architecture as Mildenhall et al. . We train the model in the same ways as Mildenhall et al.: by backpropagating through a differentiable ray marcher driven by 2D RGB images from known camera poses.

the $16$ output values of the density MLP, and

the view direction projected onto the first $16$ coefficients of the spherical harmonics basis (i.e. up to degree $4$). This is a natural frequency encoding over unit vectors.

Its output is an RGB color triplet, for which we use either a sigmoid activation when the training data has low dynamic-range (sRGB) or an exponential activation when it has high dynamic range (linear HDR). We prefer HDR training data due to the closer resemblance to physical light transport. This brings numerous advantages as has also been noted in concurrent work [Mildenhall et al., 2021].

Informed by the analysis in Figure 10, our results were generated with a 1-hidden-layer density MLP and a 2-hidden-layer color MLP, both $64$ neurons wide.

Accelerated ray marching.

When marching along rays for both training and rendering, we would like to place samples such that they contribute somewhat uniformly to the image, minimizing wasted computation. Thus, we concentrate samples near surfaces by maintaining an occupancy grid that coarsely marks empty vs. non-empty space. In large scenes, we additionally cascade the occupancy grid and distribute samples exponentially rather than uniformly along the ray. Appendix E describes these procedures in detail.

At HD resolutions, synthetic and even real-world scenes can be trained in seconds and rendered at $60$ FPS, without the need of caching of the MLP outputs [Garbin et al., 2021; Yu et al., 2021b; Wizadwongsa et al., 2021]. This high performance makes it tractable to add effects such as anti-aliasing, motion blur and depth of field by brute-force tracing of multiple rays per pixel, as shown in Figure 12.

Comparison with direct voxel lookups.

Figure 11 shows an ablation where we replace the entire neural network with a single linear matrix multiplication, in the spirit of (although not identical to) concurrent direct voxel-based NeRF [Yu et al., 2021a; Sun et al., 2021]. While the linear layer is capable of reproducing view-dependent effects, the quality is significantly compromised as compared to the MLP, which is better able to capture specular effects and to resolve hash collisions across the interpolated multiresolution hash tables (which manifest as high-frequency artifacts). Fortunately, the MLP is only 15% more expensive than the linear layer, thanks to its small size and efficient implementation.

Comparison with high-quality offline NeRF

On one hand, our method performs best on scenes with high geometric detail, such as Ficus, Drums, Ship and Lego, achieving the best PSNR of all methods. On the other hand, mip-NeRF and NSVF outperform our method on scenes with complex, view-dependent reflections, such as Materials; we attribute this to the much smaller MLP that we necessarily employ to obtain our speedup of several orders of magnitude over these competing implementations.

While we isolated the performance and convergence impact of our hash encoding and its small MLP, we believe an additional study is required to quantify the impact of advanced ray marching schemes (such as ours, coarse-fine [Mildenhall et al., 2020], or DONeRF [Neff et al., 2021]) independently from the encoding and network architecture. We report additional information in Section E.3 to aid in such an analysis.

Discussion and Future Work

At the end of the encoding, we concatenate rather than reduce (for example, by summing) the $F$-dimensional feature vectors obtained from each resolution. We prefer concatenation for two reasons. First, it allows for independent, fully parallel processing of each resolution. Second, a reduction of the dimensionality of the encoded result $\mathbf{y}$ from $LF$ to $F$ may be too small to encode useful information. While $F$ could be increased proportionally, it would make the encoding much more expensive.

However, we recognize that there may be applications in which reduction is favorable, such as when the neural network is significantly more expensive than the encoding, in which case the added computational cost of increasing $F$ could be insignificant. We thus argue for concatenation by default and not as a hard-and-fast rule. In our applications, concatenation, coupled with ${F=2}$ always yielded by far the best results.

Choice of hash function.

A good hash function is efficient to compute, leads to coherent look-ups, and uniformly covers the feature vector array regardless of the structure of query points. We chose our hash function for its good mixture of these properties and also experimented with three others:

The PCG32 [O’Neill, 2014] RNG, which has superior statistical properties. Unfortunately, it did not yield a higher-quality reconstruction, making its higher cost not worthwhile.

Even better coherence can be achieved by treating the hash function as a tiling of space into dense grids. Like (2), the speed-up is small in practice with significant detriment to quality.

Alternatively to hand-crafted hash functions, it is conceivable to optimize the hash function in future work, turning the method into a dictionary-learning approach. Two possible avenues are (1) developing a continuous formulation of indexing that is amenable to analytic differentiation or (2) applying an evolutionary optimization algorithm that can efficiently explore the discrete function space.

Microstructure due to hash collisions.

The salient artifact of our encoding is a small amount of “grainy” microstructure, most visible on the learned signed distance functions (Figure 1 and Figure 7). The graininess is a result of hash collisions that the MLP is unable to fully compensate for. We believe that the key to achieving state-of-the-art quality on SDFs with our encoding will be to find a way to overcome this microstructure, for example by filtering hash table lookups or by imposing an additional smoothness prior on the loss.

Generative setting

Parametric input encodings, when used in a generative setting, typically arrange their features in a dense grid which can then be populated by a separate generator network, typically a CNN such as StyleGAN [Chan et al., 2021; DeVries et al., 2021; Peng et al., 2020b]. Our hash encoding adds an additional layer of complexity, as the features are not arranged in a regular pattern through the input domain; that is, the features are not bijective with a regular grid of points. We leave it to future work to determine how best to overcome this difficulty.

Other applications.

We are interested in applying the multiresolution hash encoding to other low-dimensional tasks that require accurate, high-frequency fits. The frequency encoding originated from the attention mechanism of transformer networks [Vaswani et al., 2017]. We hope that parametric encodings such as ours can lead to a meaningful improvement in general, attention-based tasks.

Heterogenous volumetric density fields, such as cloud and smoke stored in a VDB [Museth, 2013, 2021] data structure, often include empty space on the outside, a solid core on the inside, and sparse detail on the volumetric surface. This makes them a good fit for our encoding. In the code released alongside this paper, we have included a preliminary implementation that fits a radiance and density field directly from the noisy output of a volumetric path tracer. The initial results are promising, as shown in Figure 13, and we intend to pursue this direction further in future work.

Conclusion

Many graphics problems rely on task specific data structures to exploit the sparsity or smoothness of the problem at hand. Our multi-resolution hash encoding provides a practical learning-based alternative that automatically focuses on relevant detail, independent of the task. Its low overhead allows it to be used even in time-constrained settings like online training and inference. In the context of neural network input encodings, it is a drop-in replacement, for example speeding up NeRF by several orders of magnitude and matching the performance of concurrent non-neural 3D reconstruction techniques.

Slow computational processes in any setting, from lightmap baking to the training of neural networks, can lead to frustrating workflows due to long iteration times [Enderton and Wexler, 2011]. We have demonstrated that single-GPU training times measured in seconds are within reach for many graphics applications, allowing neural approaches to be applied where previously they may have been discounted.

References

Appendix A Smooth Interpolation

One may desire smoother interpolation than the $d$-linear interpolation that our multiresolution hash encoding uses by default.

In this case, the obvious solution would be using a $d$-quadratic or $d$-cubic interpolation, both of which are however very expensive due to requiring the lookup of $3^{d}$ and $4^{d}$ instead of $2^{d}$ vertices, respectively. As a low-cost alternative, we recommend applying the smoothstep function,

to the $d$-linear interpolation weights. Crucially, the derivative of the smoothstep,

vanishes at and at $1$, causing the discontinuity in the derivatives of the encoding to vanish by the chain rule. The encoding thus becomes $C^{1}$-smooth.

However, by this trick, we have merely traded discontinuities for zero-points in the individual levels which are not necessarily more desirable. So, we offset each level by half of its voxel size $1/(2N_{l})$, which prevents the zero derivatives from aligning across all levels. The encoding is thus able to learn smooth, non-zero derivatives for all spatial locations $\mathbf{x}$.

For higher-order smoothness, higher-order smoothstep functions $S_{n}$ can be used at small additional cost. In practice, the computational cost of the $1$st order smoothstep function $S_{1}$ is hidden by memory bottlenecks, making it essentially free. However, the reconstruction quality tends to decrease as higher-order interpolation is used. This is why we do not use it by default. Future research is needed to explain the loss of quality.

Appendix B Implementation Details of NGLOD

We designed our implementation of NGLOD [Takikawa et al., 2021] such that it closely resembles that of our hash encoding, only differing in the underlying data structure; i.e. using the vertices of an octree around ground-truth triangle mesh to store collision-free feature vectors, rather than relying on hash tables. This results in a notable difference to the original NGLOD: the looked-up feature vectors are concatenated rather than summed, which in our implementation serendipitously resulted in higher reconstruction quality compared to the summation of an equal number of trainable parameters.

The octree implies a fixed growth factor ${b=2}$, which leads to a smaller number of levels than our hash encoding. We obtained the most favorable performance vs. quality trade-off at a roughly equal number of trainable parameters as our method, through the following configuration:

the number of feature dimensions per entry is ${F=8}$,

Appendix C Real-time SDF Training Data Generation

In order to not bottleneck our SDF training, we must be able to generate a large number of ground truth signed distances to high-resolution meshes very quickly ($\sim$millions per second).

Similar to prior work [Takikawa et al., 2021], we distribute some (${1/8}$th) of our training positions uniformly in the unit cube, some (${4/8}$ths) uniformly on the surface of the mesh, and the remainder (${3/8}$ths) perturbed from the surface of the mesh.

The uniform samples in the unit cube are trivial to generate using any pseudorandom number generator; we use a GPU implementation of PCG32 [O’Neill, 2014].

To generate the uniform samples on the surface of the mesh, we compute the area of each triangle in a preprocessing step, normalize the areas to represent a probability distribution, and store the corresponding cumulative distribution function (CDF) in an array. Then, for each sample, we select a triangle proportional to its area by the inversion method—a binary search of a uniform random number over the CDF array—and sample a uniformly random position on that triangle by standard sample warping [Pharr et al., 2016].

Lastly, for those surface samples that must be perturbed, we add a random 3D vector, each dimension independently drawn from a logistic distribution (similar shape to a Gaussian, but cheaper to compute) with standard deviation $r/1024$, where $r$ is the bounding radius of the mesh.

When training our implementation of Takikawa et al. , we must be careful to rarely generate training positions outside of octree leaf nodes. To this end, we replace the uniform unit cube sampling routine with one that creates uniform 3D positions in the leaf nodes of the octree by first rejection sampling a uniformly random leaf node from the array of all nodes and then generating a uniform random position within the node’s voxel. Fortunately, the standard deviation $r/1024$ of our logistic perturbation is small enough to almost never leave the octree, so we do not need to modify the surface sampling routine.

C.2. Efficient Signed Distances to the Triangle Mesh

Next, we sign these distances by tracing $32$ “stab rays” [Nooruddin and Turk, 2003], which we distribute uniformly over the sphere using a Fibonacci lattice that is pseudorandomly and independently offset for every training position. If any of these rays reaches infinity, the corresponding position $\mathbf{x}$ is deemed “outside” of the object and the distance is marked positive. Otherwise, it is marked negative.If the mesh is watertight, it is cheaper to sign the distance based on the normal(s) of the closest triangle(s) from the previous step. We also implemented this procedure, but disable it by default due to its incompatibility with typical meshes in the wild.

For maximum efficiency, we use NVIDIA ray tracing hardware through the OptiX $7$ framework, which is over an order of magnitude faster than using the previously mentioned triangle BVH for ray-shape intersections on our RTX 3090 GPU.

Appendix D Baseline MLPs with Frequency Encoding

In our signed distance function (SDF), neural radiance caching (NRC), and neural radiance and density fields (NeRF) experiments, we use an MLP prefixed by a frequency encoding as baseline. The respective architectures are equal to those in the main text, except that the MLPs are larger and that the hash encoding is replaced by sine and cosine waves (SDF and NeRF) or triangle waves (NRC). The following table lists the number of hidden layers, neurons per hidden layer, frequency cascades (each scaled by a factor of $2$ as per Vaswani et al. ), and adjusted learning rates.

For NeRF, the first listed number corresponds to the density MLP and the second number to the color MLP. For SDFs, we make two additional changes: (1) we optimize against the relative $\mathcal{L}^{2}$ loss [Lehtinen et al., 2018] instead of the MAPE described in the main text, and (2) we perturb training samples with a standard deviation of ${r/128}$ as opposed to the value of ${r/1024}$ from Appendix C.1. Both changes smooth the loss landscape, resulting in a better reconstruction with the above configuration.

Notably, even though the above configurations have fewer parameters and are slower than our configurations with hash encoding, they represent favorable performance vs. quality trade-offs. An equal parameter count comparison would make pure MLPs too expensive due to their scaling with $\mathcal{O}(n^{2})$ as opposed to the sub-linear scaling of trainable encodings. On the other hand, an equal throughput comparison would require prohibitively small MLPs, thus underselling the reconstruction quality that pure MLPs are capable of.

We also experimented with Fourier features [Tancik et al., 2020] but did not obtain better results compared to the axis-aligned frequency encodings mentioned previously.

Appendix E Accelerated NeRF Ray Marching

The performance of ray marching algorithms such as NeRF strongly depends on the marching scheme. We utilize three techniques with imperceivable error to optimize our implementation:

skipping of empty space and occluded regions, and

compaction of samples into dense buffers for efficient execution.

In synthetic NeRF scenes, which we bound to the unit cube $^{3}$, we use a fixed ray marching step size equal to ${\Delta t:=\sqrt{3}/1024}$; $\sqrt{3}$ represents the diagonal of the unit cube.

In all other scenes, based on the intercept theoremThe appearance of objects stays the same as long as their size and distance from the observer remain proportional., we set the step size proportional to the distance $t$ along the ray ${\Delta t:=t/256}$, clamped to the interval \big{[}\sqrt{3}/1024,s\cdot\sqrt{3}/1024\big{]}, where $s$ is size of the largest axis of the scene’s bounding box. This choice of step size exhibits exponential growth in $t$, which means that the computation cost grows only logarithmically in scene diameter, with no perceivable loss of quality.

Lastly, we stop ray marching and set the remaining contribution to zero as soon as the transmittance of the ray drops below a threshold; in our case ${\epsilon=10^{-4}}$.

Mildenhall et al. already identified a non-linear step size as benefitial: they recommend sampling uniformly in the disparity-space of the average camera frame, which is more aggressive than our exponential stepping, requiring on one hand only a constant number of steps, but on the other hand can lead to a loss of fidelity compared to exponential stepping [Neff et al., 2021].

E.2. Occupancy Grids

To skip ray marching steps in empty space, we maintain a cascade of $K$ multiscale occupancy grids, where ${K=1}$ for all synthetic NeRF scenes (single grid) and ${K\in}$ for larger real-world scenes (up to $5$ grids, depending on scene size). Each grid has a resolution of $128^{3}$, spanning a geometrically growing domain $[-2^{k-1}+0.5,2^{k-1}+0.5]^{3}$ that is centered around ${(0.5,0.5,0.5)}$.

Each grid cell stores occupancy as a single bit. The cells are laid out in Morton (z-curve) order to facilitate memory-coherent traversal by a digital differential analyzer (DDA). During ray marching, whenever a sample is to be placed according to the step size from the previous section, the sample is skipped if its grid cell’s bit is low.

Which one of the $K$ grids is queried is determined by both the sample position $\mathbf{x}$ and the step size $\Delta t$: among the grids covering $\mathbf{x}$, the finest one with cell side-length larger than $\Delta t$ is queried.

To continually update the occupancy grids while training, we maintain a second set of grids that have the same layout, except that they store full-precision floating point density values rather than single bits.

We update the grids after every $16$ training iterations by performing the following steps. We

decay the density value in each grid cell by a factor of $0.95$,

randomly sample $M$ candidate cells, and set their value to the maximum of their current value and the density component of the NeRF model at a random location within the cell, and

update the occupancy bits by thresholding each cell’s density with ${t=0.01\cdot 1024/\sqrt{3}}$, which corresponds to thresholding the opacity of a minimal ray marching step by ${1-\exp(-0.01)\approx 0.01}$.

The sampling strategy of the $M$ candidate cells depends on the training progress since the occupancy grid does not store reliable information in early iterations. During the first $256$ training steps, we sample ${M=K\cdot 128^{3}}$ cells uniformly without repetition. For subsequent training steps we set ${M=K\cdot 128^{3}/2}$ which we partition into two sets. The first ${M/2}$ cells are sampled uniformly among all cells. Rejection sampling is used for the remaining samples to restrict selection to cells that are currently occupied.

Related work.

The idea of constraining the MLP evaluation to occupied cells has already been exploited in prior work on trainable, cell-based encodings [Liu et al., 2020; Yu et al., 2021b; Yu et al., 2021a; Sun et al., 2021]. In contrast to these papers, our occupancy grid is independent from the learned encoding, allowing us to represent it more compactly as a bitfield (and thereby at a resolution that is decoupled from that of the encoding) and to utilize it when comparing against other methods that do not have a trained spatial encoding, e.g. “Ours: Frequency” in Table 2.

Empty space can also be skipped by importance sampling the depth distribution, such as by resampling the result of a coarse prediction [Mildenhall et al., 2020] or through neural importance sampling [Müller et al., 2019] as done in DONeRF [Neff et al., 2021].

E.3. Number of Rays Versus Batch Size

The batch size has a significant effect on the quality and speed of NeRF convergence. We found that training from a larger number of rays, i.e. incorporating more viewpoint variation into the batch, converged to lower error in fewer steps. In our implementation where the number of samples per ray is variable due to occupancy, we therefore include as many rays as possible in batches of fixed size rather than building variable-size batches from a fixed ray count.

In Table 3, we list ranges of the resulting number of rays per batch and corresponding samples per ray.

Lastly, we note that the occupancy grid in our frequency-encoding baseline (“Ours: Freq.”; Appendix D) produces even fewer samples than when used alongside our hash encoding. This can be explained by the slightly more detailed reconstruction of the hash encoding: when the extra detail is finer than the occupancy grid resolution, its surrounding empty space can not be effectively culled away and must be traversed by extra steps.