Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Introduction

We leverage recent progress in modeling the behavior of deep networks using kernel regression with a neural tangent kernel (NTK) to theoretically and experimentally show that standard MLPs are poorly suited for these low-dimensional coordinate-based vision and graphics tasks. In particular, MLPs have difficulty learning high frequency functions, a phenomenon referred to in the literature as “spectral bias” . NTK theory suggests that this is because standard coordinate-based MLPs correspond to kernels with a rapid frequency falloff, which effectively prevents them from being able to represent the high-frequency content present in natural images and scenes.

We leverage NTK theory and simple experiments to show that a Fourier feature mapping can be used to overcome the spectral bias of coordinate-based MLPs towards low frequencies by allowing them to learn much higher frequencies (Section 4).

We demonstrate that a random Fourier feature mapping with an appropriately chosen scale can dramatically improve the performance of coordinate-based MLPs across many low-dimensional tasks in computer vision and graphics (Section 5).

Related Work

Our work is motivated by the widespread use of coordinate-based MLPs to represent a variety of visual signals, including images and 3D scenes . In particular, our analysis is intended to clarify experimental results demonstrating that an input mapping of coordinates (which they called a “positional encoding”) using sinusoids with logarithmically-spaced axis-aligned frequencies improves the performance of coordinate-based MLPs on the tasks of novel view synthesis from 2D images and protein structure modeling from cryo-electron microscopy . We analyze this technique to show that it corresponds to a modification of the MLP’s NTK, and we show that other non-axis-aligned frequency distributions can outperform this positional encoding.

Prior works in natural language processing and time series analysis have used a similar positional encoding to represent time or 1D position. In particular, Xu et al. use random Fourier features (RFF) to approximate stationary kernels with a sinusoidal input mapping and propose techniques to tune the mapping parameters. Our work extends this by directly explaining such mappings as a modification of the resulting network’s NTK. Additionally, we address the embedding of multidimensional coordinates, which is necessary for vision and graphics tasks.

To analyze the effects of applying a Fourier feature mapping to input coordinates before passing them through an MLP, we rely on recent theoretical work that models neural networks in the limits of infinite width and infinitesimal learning rate as kernel regression using the NTK . In particular, we use the analyses from Lee et al. and Arora et al. , which show that the outputs of a network throughout gradient descent remain close to those of a linear dynamical system whose convergence rate is governed by the eigenvalues of the NTK matrix . Analysis of the NTK’s eigendecomposition shows that its eigenvalue spectrum decays rapidly as a function of frequency, which explains the widely-observed “spectral bias” of deep networks towards learning low-frequency functions .

We leverage this analysis to consider the implications of adding a Fourier feature mapping before the network, and we show that this mapping has a significant effect on the NTK’s eigenvalue spectrum and on the corresponding network’s convergence properties in practice.

Background and Notation

To lay the foundation for our theoretical analysis, we first review classic kernel regression and its connection to recent results that analyze the training dynamics and generalization behavior of deep fully-connected networks. In later sections, we use these tools to analyze the effects of training coordinate-based MLPs with Fourier feature mappings.

Kernel regression. Kernel regression is a classic nonlinear regression algorithm . Given a training dataset $(\mathbf{X},\mathbf{y})=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n}$, where $\mathbf{x}_{i}$ are input points and $y_{i}=f(\mathbf{x}_{i})$ are the corresponding scalar output labels, kernel regression constructs an estimate $\hat{f}$ of the underlying function at any point $\mathbf{x}$ as:

where $\mathbf{K}$ is an $n\times n$ kernel (Gram) matrix with entries $\mathbf{K}_{ij}=k(\mathbf{x}_{i},\mathbf{x}_{j})$ and $k$ is a symmetric positive semidefinite (PSD) kernel function which represents the “similarity” between two input vectors. Intuitively, the kernel regression estimate at any point $\mathbf{x}$ can be thought of as a weighted sum of training labels $y_{i}$ using the similarity between the corresponding $\mathbf{x}_{i}$ and $\mathbf{x}$.

Approximating deep networks with kernel regression. Let $f$ be a fully-connected deep network with weights $\theta$ initialized from a Gaussian distribution $\mathcal{N}$. Theory proposed by Jacot et al. and extended by others shows that when the width of the layers in $f$ tends to infinity and the learning rate for SGD tends to zero, the function $f(\mathbf{x};\theta)$ converges over the course of training to the kernel regression solution using the neural tangent kernel (NTK), defined as:

Fourier Features for a Tunable Stationary Neural Tangent Kernel

We would like the composed NTK to be shift-invariant over the input domain, since the training points are distributed with uniform density. In problems where the inputs are normalized to the surface of a hypersphere (common in machine learning), a dot product kernel (such as the regular NTK) corresponds to spherical convolution. However, inputs in our setting are dense in Euclidean space. A Fourier feature mapping of input coordinates makes the composed NTK stationary (shift-invariant), acting as a convolution kernel over the input domain (see Appendix C for additional discussion on stationary kernels).

We would like to control the bandwidth of the NTK to improve training speed and generalization. As we see from Eqn. 4, a “wider” kernel with a slower spectral falloff achieves faster training convergence for high frequency components. However, we know from signal processing that reconstructing a signal using a kernel whose spectrum is too wide causes high frequency aliasing artifacts. We show in Section 5 that a Fourier feature input mapping can be tuned to lie between these “underfitting’ and “overfitting” extremes, enabling both fast convergence and low test error.

Fourier features and the composed neural tangent kernel. Fourier feature mappings have been used in many applications since their introduction in the seminal work of Rahimi and Recht , which used random Fourier features to approximate an arbitrary stationary kernel function by applying Bochner’s theorem. Extending this technique, we use a Fourier feature mapping $\gamma$ to featurize input coordinates before passing them through a coordinate-based MLP, and investigate the theoretical and practical effect this has on convergence speed and generalization. The function $\gamma$ maps input points $\mathbf{v}\in[0,1)^{d}$ to the surface of a higher dimensional hypersphere with a set of sinusoids:

Because $\cos(\alpha-\beta)=\cos\alpha\cos\beta+\sin\alpha\sin\beta$, the kernel function induced by this mapping is:

Note that this kernel is stationary (a function of only the difference between points). We can think of the mapping as a Fourier approximation of a kernel function: $\mathbf{b}_{j}$ are the Fourier basis frequencies used to approximate the kernel, and $a_{j}^{2}$ are the corresponding Fourier series coefficients.

where $\mathbf{w}=\mathbf{K}^{-1}\mathbf{y}$ (from Eqn. 1). This allows us to draw analogies to signal processing, where the composed NTK acts similarly to a reconstruction filter. In the next section, we show that the frequency decay of the composed NTK determines the behavior of the reconstructed signal.

Manipulating the Fourier Feature Mapping

Preprocessing the inputs to a coordinate-based MLP with a Fourier feature mapping creates a composed NTK that is not only stationary but also tunable. By manipulating the settings of the $a_{j}$ and $\mathbf{b}_{j}$ parameters in Eqn. 5, it is possible to dramatically change both the rate of convergence and the generalization behavior of the resulting network. In this section, we investigate the effects of the Fourier feature mapping in the setting of 1D function regression.

We train MLPs to learn signals $f$ defined on the interval $[0,1)$. We sample $cn$ linearly spaced points on the interval, using every $c^{\textrm{th}}$ point as the training set and the remaining points as the test set. Since our composed kernel function is stationary, evaluating it at linearly spaced points on a periodic domain makes the resulting kernel matrix circulant: it represents a convolution and is diagonalizable by the Fourier transform. Thus, we can compute the eigenvalues of the composed NTK matrix by simply taking the Fourier transform of a single row. All experiments are implemented in JAX and the NTK functions are calculated automatically using the Neural Tangents library .

Effects of Fourier features on network convergence. We generate ground truth 1D functions by sampling $cn$ values from a family with parameter $\alpha$ as follows: we sample a standard i.i.d. Gaussian vector of length $cn$, scale its $i^{\textrm{th}}$ entry by $1/i^{\alpha}$, then return the real component of its inverse Fourier transform. We will refer to this as a “$1/f^{\alpha}$ noise” signal.

In Figure 3, we train MLPs (4 layers, 1024 channels, ReLU activations) to fit a bandlimited $1/f^{1}$ noise signal ($c=8,n=32$) using Fourier feature mappings with different $p$ values. Figures 3b and 3d show that the NTK linear dynamics model accurately predict the effects of modifying the Fourier feature mapping parameters. Separating different frequency components of the training error in Figure 3c reveals that networks with narrower NTK spectra converge faster for low frequency components but essentially never converge for high frequency components, whereas networks with wider NTK spectra successfully converge across all components. The Fourier feature mapping $p=1$ has adequate power across frequencies present in the target signal (so the network converges rapidly during training) but limited power in higher frequencies (preventing overfitting or aliasing).

Tuning Fourier features in practice. Eqn. 3 allows us to estimate a trained network’s theoretical loss on a validation set using the composed kernel. For small 1D problems, we can minimize this loss with gradient-based optimization to choose mapping parameters $a_{j}$ (given a dense sampling of $b_{j}$). In this carefully controlled setting (1D signals, small training dataset, gradient descent with small learning rate, very wide networks), we find that this optimized mapping also achieves the best performance when training networks. Please refer to Appendix A.1 for details and experiments.

In real-world problems, especially in multiple dimensions, it is not feasible to use a feature mapping that densely samples Fourier basis functions; the number of Fourier basis functions scales with the number of training data points, which grows exponentially with dimension. Instead, we sample a set of random Fourier features from a parametric distribution. We find that the exact sampling distribution family is much less important than the distribution’s scale (standard deviation).

Figure 4 demonstrates this point using hyperparameter sweeps for a variety of sampling distributions. In each subfigure, we draw 1D target signals ($c=2,n=1024$) from a fixed $1/f^{\alpha}$ distribution and train networks to learn them. We use random Fourier feature mappings (of length 16) sampled from different distribution families (Gaussian, uniform, uniform in log space, and Laplacian) and sweep over each distribution’s scale. Perhaps surprisingly, the standard deviation of the sampled frequencies alone is enough to predict test set performance, regardless of the underlying distribution’s shape. We show that this holds for higher-dimensional tasks in Appendix A.4. We also observe that passing this sparse sampling of Fourier features through an MLP matches the performance of using a dense set of Fourier features with the same MLP, suggesting a strategy for scaling to higher dimensions. We proceed with a Gaussian distribution for our higher-dimensional experiments in Section 6 and treat the scale as a hyperparameter to tune on a validation dataset.

Experiments

We validate the benefits of using Fourier feature mappings for coordinate-based MLPs with experiments on a variety of regression tasks relevant to the computer vision and graphics communities.

In Table 1, we compare the performance of coordinate-based MLPs with no input mapping and with the following Fourier feature mappings ($\cos,\sin$ are applied elementwise):

Our experiments show that all of the Fourier feature mappings improve the performance of coordinate-based MLPs over using no mapping and that the Gaussian RFF mapping performs best.

2 Tasks

We conduct experiments with direct regression, where supervision labels are in the same space as the network outputs, as well as indirect regression, where the network outputs are passed through a forward model to produce observations in the same space as the supervision labels (Appendix D contains a theoretical analysis of indirect regression through a linear forward model). For each task and dataset, we tune Fourier feature scales on a held-out set of signals. For each target signal, we train an MLP on a training subset of the signal and compute error over the remaining test subset. All tasks (except 3D shape regression) use L2 loss and a ReLU MLP with 4 layers and 256 channels. The 3D shape regression task uses cross-entropy loss and a ReLU MLP with 8 layers and 256 channels. We apply a sigmoid activation to the output for each task (except the view synthesis density prediction). We use 256 frequencies for the feature mapping in all experiments (see Appendix A.2 for experiments that investigate the effects of network depth and feature mapping sparsity). Appendix E provides additional details on each task and our implementations, and Appendix F shows more result figures.

2D image regression. In this task, we train an MLP to regress from a 2D input pixel coordinate to the corresponding RGB value of an image. For each test image, we train an MLP on a regularly-spaced grid containing $\nicefrac{{1}}{{4}}$ of the pixels and report test error on the remaining pixels. We compare input mappings over a dataset of natural images and a dataset of text images.

3D shape regression. Occupancy Networks implicitly represent a 3D shape as the “decision boundary” of an MLP, which is trained to output 0 for points outside the shape and 1 for points inside the shape. Each batch of training data is generated by sampling points uniformly at random from the bounding box of the shape and calculating their labels using the ground truth mesh. Test error is calculated using intersection-over-union versus ground truth on a set of points randomly sampled near the mesh surface to better highlight the different mappings’ abilities to resolve fine details.

2D computed tomography (CT). In CT, we observe integral projections of a density field instead of direct measurements. In our 2D CT experiments, we train an MLP that takes in a 2D pixel coordinate and predicts the corresponding volume density at that location. The network is indirectly supervised by the loss between a sparse set of ground-truth integral projections and integral projections computed from the network’s output. We conduct experiments using two datasets: procedurally-generated Shepp-Logan phantoms and 2D brain images from the ATLAS dataset .

3D magnetic resonance imaging (MRI). In MRI, we observe Fourier transform coefficients of atomic response to radio waves under a magnetic field. In our 3D MRI experiments, we train an MLP that takes in a 3D voxel coordinate and predicts the corresponding response at that location. The network is indirectly supervised by the loss between a sparse set of ground-truth Fourier transform coefficients and Fourier transform coefficients computed from discretely querying the MLP on a voxel grid. We conduct experiments using the ATLAS dataset .

3D inverse rendering for view synthesis. In view synthesis, we observe 2D photographs of a 3D scene, reconstruct a representation of that scene, then render images from new viewpoints. To perform this task, we train a coordinate-based MLP that takes in a 3D location and outputs a color and volume density. This MLP is indirectly supervised by the loss between the set of 2D image observations and the same viewpoints re-rendered from the predicted scene representation. We use a simplified version of the method described in NeRF , where we remove hierarchical sampling and view dependence and replace the original positional encoding with our compared input mappings.

Conclusion

We leverage NTK theory to show that a Fourier feature mapping can make coordinate-based MLPs better suited for modeling functions in low dimensions, thereby overcoming the spectral bias inherent in coordinate-based MLPs. We experimentally show that tuning the Fourier feature parameters offers control over the frequency falloff of the combined NTK and significantly improves performance across a range of graphics and imaging tasks. These findings shed light on the burgeoning technique of using coordinate-based MLPs to represent 3D shapes in computer vision and graphics pipelines, and provide a simple strategy for practitioners to improve results in these domains.

Acknowledgements

We thank Ben Recht for advice, and Cecilia Zhang and Tim Brooks for their comments on the text. BM is funded by a Hertz Foundation Fellowship and acknowledges support from the Google BAIR Commons program. MT, PS and SFK are funded by NSF Graduate Fellowships. RR was supported in part by ONR grants N000141712687 and N000142012529 and the Ronald L. Graham Chair. RN was supported in part by an FHL Vive Center Seed Grant. Google University Relations provided a generous donation of compute credits.

References

Appendix A Further experiments

Using Eqn. 3 in the main paper, we can predict what error a trained network will achieve on a set of testing points. Since this equation depends on the composed NTK, we can directly relate predicted test set loss to the Fourier feature mapping parameters $a$ and $b$ for a validation set of signals $\mathbf{y}_{val}$:

In Figure 5, we show the results of minimizing Eqn. 10 by gradient descent on $a_{j}$ values (with fixed corresponding “densely sampled” $b_{j}=j$) for validation sets sampled from three different $1/f^{\alpha}$ noise families. Note that gradient descent on this theoretical loss approximation produces $a_{j}$ values which are able to perform as well as the best “power law” $a_{j}$ values for each respective signal class (compared dashed lines versus $\times$ markers in Figure 5b). As mentioned in the main text, we find that this optimization strategy is only viable for small 1D regression problems. In our multidimensional tasks, using densely sampled $\mathbf{b}_{j}$ values is not tractable due to memory constraints. In addition, the theoretical approximation only holds when training the network using SGD, and in practice we train using the Adam optimizer .

A.2 Feature sparsity and network depth

In our experiments, we observe that deeper networks need fewer Fourier features than shallow networks. As the depth of the MLP increases, we observe that a sparser set of frequencies can achieve similar performance; Figure 6 illustrates this effect in the context of 2D image regression.

Again drawing on NTK theory, we understand this tradeoff as an effect of frequency “spreading,” as illustrated in Figure 7. A Fourier featurization consists of only discrete frequencies, but when composed with the NTK, the influence of each discrete frequency “spreads” over its local neighborhood in the final spectrum. We find that the “spread” around each frequency feature increases for deeper networks. For an MLP to learn all of the frequency components in the target signal, its corresponding composed NTK must contain adequate power across the frequency support of the target signal. This is accomplished either by including more frequencies in the Fourier features or by spreading those frequencies through sufficient NTK depth.

A.3 Gradient descent does not optimize Fourier features

One may wonder if the Fourier feature mapping parameters $a_{j}$ and $\mathbf{b}_{j}$ can be optimized alongside network weights using gradient descent, which may circumvent the need for careful initialization. We performed an experiment in which the $a_{j},\mathbf{b}_{j}$ values are treated as trainable variables (along with the weights of the network) and optimize all variables with Adam to minimize training loss. Figure 8 shows that jointly optimizing these parameters does not improve performance compared to leaving them fixed.

A.4 Visualizing underfitting and overfitting in 2D

Figure 4 in the main text shows (in a 1D setting) that as the scale of the Fourier feature sampling distribution increases, the trained network’s error traces out a curve that starts in an underfitting regime (only low frequencies are learned) and ends in an overfitting regime (the learned function includes high-frequency detail not present in the training data). In Figure 9, we show analogous behavior for 2D image regression, demonstrating that the same phenomenon holds in a multidimensional problem. In Figure 10, we show how changing the scale for Gaussian Fourier features qualitatively affects the final result in the 2D image regression task.

A.5 Failures of positional encoding (axis-aligned bias)

Here we present a simple experiment to directly showcase the benefits of using an isotropic frequency distribution, such as Gaussian RFF, compared to the axis-aligned “positional encoding” used in prior work . As discussed in the main paper, the positional encoding mapping only uses on-axis frequencies. This approach is well-suited to data that has more frequency content along the coordinate axes, but is not as effective for more natural signals.

In Figure 11, we conduct a simple 2D image experiment where we train a coordinate-based MLP (2 layers, 256 channels) to fit target 2D sinusoid images ($512\times 512$ resolution). We sample 64 such 2D sinusoid images (regularly-sampled in polar coordinates, with 16 angles and 4 radii) and train a 2D coordinate-based MLP to fit each, using the same setup as the 2D image experiments described in Section E.1. The isotropic Gaussian RFF mapping performs well across all angles, while the positional encoding mapping performs worse for frequencies that are not axis-aligned.

Appendix B Additional details for main text figures

In Figure 12, we present an alternate version of Figure 3 from the main text showing a denser sampling of $p$ values to better visualize the effect of changing Fourier feature falloff on the resulting trained network. Again, the feature mapping used here is $a_{j}=1/j^{p},b_{j}=j$ for $j=1,\ldots,n/2$.

B.2 Main text Figure 4 (different random feature distributions in 1D)

Exact details for the sampling distributions used to generate $b_{j}$ values for Figure 4 in the main text are shown in Table 2. In Figure 13, we present an alternate version showing both train and test performance, emphasizing the underfitting/overfitting regimes created by manipulating the scale of the Fourier features.

We include the Uniform log distribution because it is the random equivalent of the “positional encoding” sometimes used in prior work. One observation is that the sampling for uniform-log variables ($X^{\prime}=\sigma_{ul}^{X}$ where $X\sim\mathcal{U}[0,1)$) corresponds to the following CDF:

This shows that the randomized equivalent of positional encoding is sampling from a distribution proportional to a $1/f$ falloff power law.

Appendix C Stationary kernels

One of the primary benefits of our Fourier feature mapping is that it results in a stationary composed NTK function. In this section, we offer some intuition for why stationarity is desirable for our low-dimensional graphics and imaging problems.

First, let us consider the implications of using an MLP applied directly to a low-dimensional input (without any Fourier feature mapping). In this setting, the NTK is a function of the dot product between its inputs and of their norms . This makes the NTK rotation-invariant, but not translation-invariant. For our graphics and imaging applications, we want to be able to model an object or scene equally well regardless of its location, so translation-invariance or stationarity is a crucial property. We can then add approximate rotation invariance back by using an isotropic frequency sampling distribution.

This effect is illustrated in a simple 1D example in Figure 14, which shows that the benefits of a stationary composed NTK indeed appear in the MLP setting with a basic Fourier featurization (using a single frequency). We train MLPs with and without this basic Fourier embedding to learn a set of shifted 1D Gaussian probability density functions. The plain MLP successfully fits a zero-centered function but struggles to fit shifted functions, while the MLP with basic Fourier embedding exhibits stationary behavior, with good performance regardless of shifts.

Appendix D Indirect supervision through a linear map

In some of the tasks we explore in this work, such as image regression or 3D shape regression, optimization is performed by minimizing a loss between the output of a network and a directly observed quantity, such as the color of a pixel or the occupancy of a voxel. But in many graphics and imaging applications of interest, measurements are indirect, and the loss must be computed on the output of a network after it has been processed by some physical forward model. In NeRF , measurements are taken by sampling and compositing along rays in each viewing direction. In MRI, measurements are taken along various curves through the frequency domain. In CT, measurements are integral projections of the subject at various angles, which correspond to measuring lines through the origin in the frequency domain. Although the measurement transformation for NeRF is nonlinear (in density, although it is linear in color), those for both CT and MRI are linear. In this section, we extend the linearized training dynamics of Lee et al. to the setting of training through a linear operator denoted by a matrix $\mathbf{A}$. This allows us to modify Eqn. 3 to incorporate $\mathbf{A}$, thereby demonstrating that the conclusions drawn in this work for the “direct” regression case also apply to the “indirect” case.

Our derivation closely follows Lee et al. , and begins by replacing the neural network $f$ with its linearization around the initial parameters $\theta_{0}$:

where $\omega_{t}\triangleq\theta_{t}-\theta_{0}$ denotes the change in network parameters since initialization and $t$ denotes time in continuous-time gradient flow dynamics. Then describes the dynamics of gradient flow:

Substituting this into the gradient flow dynamics of Eqn. 14 gives us:

Assuming our initialization is small, i.e., $f_{0}(\mathbf{x})\approx 0~{}\forall\mathbf{x}$, we can write our approximate linearized network output as:

In our previous analysis, we work instead with the expected or infinite-width NTK matrix $\mathbf{K}$, which is fixed throughout training. Using this notation, we have

Appendix E Task details

We present additional details for each task from Section 6 in the main text, including training parameters, forward models, datasets, etc. All experiments are implemented using JAX and trained on a single K80 or RTX2080Ti GPU. Training a single MLP took between 10 seconds (for the 2D image task) and 30 minutes (for the inverse rendering task).

The 2D image regression tasks presented in the main text all use $512\times 512$ resolution images. A subsampled grid of $256\times 256$ pixels is used as training data, and an offset grid of $256\times 256$ pixels is used for testing. We use two image datasets: Natural and Text, each consisting of 32 images. The Natural images are generated by taking center crops of randomly sampled images from the Div2K dataset . The Text images are generated by placing random strings of text with random sizes and colors on a white background (examples can be seen in Figure 15). For each dataset we perform a hyperparameter sweep over feature mapping scales on 16 images. We find that scales $\sigma_{g}=10$ and $\sigma_{p}=6$ work best for the Natural dataset and $\sigma_{g}=14$ and $\sigma_{p}=5$ work best for the Text dataset (see Table 2 for mapping definitions). In Table 3, we report model performance using the optimal mapping scale on the remaining 16 images.

Each model (MLP with 4 layers, 256 channels, ReLU activation, sigmoid output) is trained for 2000 iterations using the Adam optimizer with default settings ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$). Learning rates are manually tuned for each dataset and method. For Natural images a learning rate of $10^{-3}$ is used for the Gaussian RFF and the positional encoding, and a learning rate of $10^{-2}$ is used for the basic mapping and “no mapping” methods. For the Text images a learning rate of $10^{-3}$ is used for all methods.

E.2 3D shape

We evaluate the 3D shape regression task (similar to Occupancy Networks ) on four complex triangle meshes commonly used in computer graphics applications (Dragon, Armadillo, Buddha, and Lucy, shown in Figure 16), each containing hundreds of thousands of vertices. We train one coordinate-based MLP network to represent a single mesh rather than trying to generalize one network to encode multiple objects, since our goal is to demonstrate that a network with no mapping or the low frequency “basic” mapping cannot accurately represent even a single shape, let alone a whole class of objects.

We use a network with 8 layers of 256 channels each and a ReLU nonlinearity between each layer. We apply a sigmoid activation to the output. Our batch size is $32^{3}$ points, and we use the Adam optimizer with a learning rate starting at $5\times 10^{-4}$ and exponentially decaying by a factor of $0.01$ over the course of 10000 total training iterations. At each training iteration, we sample a batch of 3D points uniformly at random from the bounding box of the mesh, and then calculate ground truth labels (using the point-in-mesh method implemented in the Trimesh library , which relies on the Embree kernel for acceleration ). We use cross-entropy loss to train the network to match these classification labels (0 for points outside the mesh, 1 for points inside).

The meshes are scaled to fit inside the unit cube $^{3}$ such that the centroid of the mesh is $(0.5,0.5,0.5)$. We use the Lucy statue mesh as a validation object to find optimal scale values for the positional encoding and Gaussian feature mapping. As described in the caption for Table 4, we calculate error on both a uniformly random test set and a test set that is close to the mesh surface (randomly chosen mesh vertices that have been perturbed by a random Gaussian vector with standard deviation $0.01$) in order to illustrate that Fourier feature mappings provide a large benefit in resolving fine surface details. Both test sets have $64^{3}$ points.

In Figure 16, we visualize additional results on all four meshes mentioned above (including the validation mesh Lucy). We render normal maps, which are computed by taking the cross product of the numerical horizontal and vertical derivatives of the depth map. The original depth map is generated by intersecting camera rays with the first $0.5$ isosurface of the network. We select the Fourier feature scales for (d) and (e) by doing a hyperparameter search based on validation loss for the Lucy mesh in the last row and report test loss over the other three meshes (Table 4). Note that the weights for each trained MLP are only 2MB, while the triangle mesh files for the objects shown are 61MB, 7MB, 79MB, and 32MB respectively.

E.3 2D CT

In computed tomography (CT), we observe measurements that are integral projections (integrals along parallel lines) of a density field. We construct a 2D CT task by using ground truth $512\times 512$ resolution images, and computing 20 synthetic integral projections at evenly-spaced angles. For each of these images, the supervision data is the set of integral projections, and the test PSNR is evaluated over the original image.

We use two datasets for our 2D CT task: randomized Shepp-Logan phantoms , and the ATLAS brain dataset . For each dataset, we perform a hyperparameter sweep over mapping scales on 8 examples. We found that scales $\sigma_{g}=4$ and $\sigma_{p}=3$ work best for the Shepp dataset and $\sigma_{g}=5$ and $\sigma_{p}=5$ work best for the ATLAS dataset. In Table 5, we report model performance using the optimal mapping scale on a distinct set of 8 images.

Each model (MLP with 4 layers, 256 channels, ReLU activation, sigmoid output) is trained for 1000 iterations using the Adam optimizer with default settings ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$). The learning rate is manually tuned for each method. Gaussian RFF and positional encoding use a learning rate of $10^{-3}$, and the basic and “no mapping” method use a learning rate of $10^{-2}$.

E.4 3D MRI

In magnetic resonance imaging (MRI), we observe measurements that are Fourier coefficients of the atomic response to radio waves under a magnetic field. We construct a toy 3D MRI task by using ground truth $96\times 96\times 96$ resolution volumes and randomly sampling $\sim\!13\%$ of the Fourier coefficients for each volume from an isotropic Gaussian. For each of these volumes, the supervision data is the set of sampled Fourier coefficients, and the test PSNR is evaluated over the original volume.

We use the ATLAS brain dataset for our 3D MRI experiments. We perform a hyperparameter sweep over mapping scales on 6 examples. We find that scales $\sigma_{g}=5$ and $\sigma_{p}=4$ perform best. In Table 6, we report model performance using the optimal mapping scale on a distinct set of 6 images. Each model (MLP with 4 layers, 256 channels, ReLU activation, sigmoid output) is trained for 1000 iterations using the Adam optimizer with default settings ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$). We use a manually-tuned learning rate of $2\times 10^{-3}$ for each method. Results are visualized in Figure 18.

E.5 3D inverse rendering for view synthesis

In this task we use the “tiny NeRF” simplified version of the view synthesis method NeRF where hierarchical sampling and view dependence have been removed. The model is trained to predict the color and volume density at an input 3D point. Volumetric rendering is used to render novel viewpoints of the object. The loss is calculated between the rendered views and ground truth renders. In our experiments we use the NeRF Lego dataset of 120 images downsampled to $400\times 400$ pixel resolution. The dataset is split into 100 training images, 7 validation images, and 13 test images. The reconstruction quality on the validation images is used to determine the best mapping scale; for this scene we find $\sigma_{g}=6.05$ and $\sigma_{p}=1.27$ perform best.

The model (MLP with 4 layers, 256 channels, ReLU activation, sigmoid on RGB output) is trained for $5\times 10^{5}$ iterations using the Adam optimizer with default settings ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$). The learning rate is manually tuned for each mapping: $10^{-2}$ for no mapping, $5\times 10^{-3}$ for basic, $5\times 10^{-4}$ for positional encoding, and $5\times 10^{-4}$ for Gaussian. During training we use batches of 1024 rays.

The original NeRF method uses an input mapping similar to the Positional encoding we compare against. The original NeRF mapping is smaller than our mappings (8 vs. 256 frequencies). We include metrics for this mapping in Table 7 under Original pos. enc. The positional encoding mappings only contain frequencies on the axes, and are therefore biased towards signals with on-axis frequency content (as demonstrated in Section A.5). In our experiments we rotate the Lego scene, which was manually axis-aligned in the original dataset, for a more equitable comparison. Table 7 also reports metrics for positional encodings on the original axis-aligned scene. Results are visualized in Figure 19.

Appendix F Additional results figures