Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

Introduction

The wide success of deep learning in many fields (LeCun et al., 2015) remains a mystery. For example, a puzzle recently attracts a lot of attention, that is, why Deep Neural Networks (DNNs), with more parameters than samples, often generalize well (Zhang et al., 2016). A major difficulty of resolving this puzzle may be attributed to the lack of an effective model which can accurately predict the final output function of DNNs and yet is simple enough for analysis. Devising such an effective model could propel deep learning into a new era in which quantitative understandings of deep learning replace the qualitative or empirical ones.

Towards this end, we begin with a widely observed phenomenon of DNNs, that is, Frequency Principle (F-Principle) (Xu et al., 2018; Rahaman et al., 2018; Xu et al., 2019):

DNNs initialized with small parameters often fit target functions from low to high frequencies during the training.

Without an explicit mathematical description, it is unclear how this implicit bias of the F-Principle functions quantitatively during the training. Inspired by the F-Principle, we construct a model of linear F-Principle (LFP) dynamics, which explicitly imposes different priorities on different frequencies in the gradient flow dynamics. Experimentally, we show that the LFP model can accurately predict the output of two-layer ReLU neural networks (NNs) of large widths. We then rationalize the LFP model using a linearized mean field residual dynamics of DNNs, which is widely considered in recent theoretical studies of DNNs (Mei et al., 2018; Rotskoff & Vanden-Eijnden, 2018; Mei et al., 2019). We prove that the long-time limit solution of this LFP dynamics is equivalent to the solution of a constrained optimization problem minimizing an F-Principle norm (FP-norm), in which higher frequencies of feasible solutions are more heavily penalized. Therefore, by analyzing the explicit regularity underlying the FP-norm, we can obtain a quantitative understanding of the behavior of two-layer NNs.

With a reasonable construction process and an ability of making accurate predictions for two-layer ReLU NNs of large widths, the LFP model qualifies as a primitive candidate of an effective model of DNNs, which is capable of providing quantitative understandings of deep learning. To analyze the generalization error of the LFP model, we first use the FP-norm, the explicit penalty, to induce an FP function space, and estimate its Rademacher complexity. We then provide an a priori estimate, i.e., an estimate without the knowledge of the model solution, of the generalization error for the LFP model, which is bounded by the FP-norm of the target function, scales as $1/\sqrt{M}$ as the number of training samples $M$ increases, and is independent of the number of parameters in NNs.

Related works

Various approaches have been proposed in an attempt to resolve the generalization puzzle of DNNs. For example, the generalization error has been related to various complexity measures (Bartlett et al., 1999; Bartlett & Mendelson, 2002; Bartlett, Foster & Telgarsky, 2017; Bartlett, Harvey, Liaw & Mehrabian, 2017; Neyshabur et al., 2017; Golowich et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2018; E et al., 2018), local properties (sharpness/flatness) of loss functions at minima (Hochreiter & Schmidhuber, 1995; Keskar et al., 2016; Dinh et al., 2017; Wu et al., 2017), stability of optimization algorithms (Bousquet & Elisseeff, 2002; Xu & Mannor, 2012; Hardt et al., 2015), and implicit bias of the training process (Arpit et al. (2017); Rahaman et al. (2018); Xu (2018a, b); Pérez et al. (2018); Xu et al. (2019, 2018); Neyshabur et al. (2014); Poggio et al. (2018); Soudry et al. (2018)). Recently, analyzing DNNs in an extremely over-parameterized regime renders a promising approach. For example, the training process of two-layer neural networks at the mean-field limit can be described by a partial differential equation (Rotskoff & Vanden-Eijnden, 2018; Mei et al., 2018; Sirignano & Spiliopoulos, 2018). In addition, the training dynamics of a DNN in an extremely over-parameterized regime is found to be well approximated by the gradient flow of a linearized model of the DNN resembling kernel methods (Jacot et al., 2018; Lee et al., 2019). This result initiates a series of works. For example, Arora et al. (2019); Cao & Gu (2019); E, Ma, Wang & Wu (2019); E, Ma & Wu (2019) utilize the linearized model to study the generalization error bounds of DNN. Note that an a priori generalization error bound for two-layer NNs is provided in E et al. (2018), in which an explicit penalty is imposed to the loss function of NNs. In contrast, our a prior generalization bound works for NNs without any extra penalty.

Notation and experimental setup

For a two-layer neural network, its output (also known as the hypothesis function) reads as

2 Experimental setup

In our experiments, we use two-layer ReLU NNs of form $h(x,\theta)=\sum_{i=1}^{N}w_{i}\sigma(r_{i}\cdot x-|r_{i}|l_{i})$ for input dimension $d>1$ and $h(x,\theta)=\sum_{i=1}^{N}w_{i}\sigma(r_{i}(x-l_{i}))$ for $d=1$. The NNs are trained with MSE loss and full batch size. The learning rate for Fig. 1 and 2 is $3\times 10^{-5}$, for Fig. 3 is $10^{-4}$. The training algorithm for Fig. 1 and 2 is gradient descent, for Fig. 3 is Adam (Kingma & Ba, 2014). $l_{i}$’s are initialized by a uniform distribution on $$for Fig. 1 and on$$ for Fig. 2. For Fig. 2, $w_{i}$’s and $r_{i}$’s are initialized by ${\cal N}(0,1)$ and ${\cal N}(0,0.49)$. For Fig. 3, We use an NN of $5000$ hidden neurons initialized by Xavier normal initialization.

A random non-zero initial output of DNN leads to a specific type of generalization error. To eliminate this error, we use DNNs with an antisymmetrical initialization (ASI) trick(Zhang et al., 2019).

An effective model of Linear F-Principle (LFP) dynamics

It is difficult to analyze DNNs theoretically due to its huge number of parameters and highly non-linear dynamics. In this section, inspired by the F-Principle, we propose a Linear F-Principle (LFP) dynamics to effectively model a two-layer ReLU NN of a large width. Specifically, with the loss of MSE, up to a multiplicative constant in time scale, we model the gradient descent dynamics of the two-layer NN of a sufficiently large width $N$ as

where, different from $\hat{u}(\xi,t)$ in the left hand side (LHS), $\widehat{u_{p}}(\xi,t)=\mathcal{F}(u(\cdot,t)p(\cdot))=\mathcal{F}\left[\sum_{i=1}^{M}\left(h(x_{i},\theta(t))-y_{i}\right)\delta(\cdot-x_{i})\right]$ in the right hand side (RHS) incorportates the information from the training dataset. In our numerical experiments, we only consider NNs with ASI trick(Zhang et al., 2019), which guarantees $h(\cdot,\theta_{0})=0$. Note that, for $d=1$, an NN of the form $h(x,\theta)=\sum_{i=1}^{N}w_{i}\sigma(r_{i}(x-l_{i}))$ is modeled by the same LFP dynamics (3). For convenience, we refer to the long-time limit solution of the LFP dynamics as the solution of the LFP model. Intuitively, the coefficient as a function of $\xi$ in the RHS characterizes a decaying priority of convergence for $\hat{u}(\xi,t)$ from low to high frequencies, conforming with the phenomenon of the F-Principle (Xu et al., 2018, 2019).

Intuitively, a higher order of decay in the frequency domain, say $1/|\xi|^{d+3}$ comparing to $1/|\xi|^{d+1}$, leads to a more “smooth” solution. Therefore, adjusting the relative importance of $1/|\xi|^{d+3}$ and $1/|\xi|^{d+1}$ through their coefficients $\frac{1}{N}\sum_{i=1}^{N}\left(|r_{i}(0)|^{2}+w_{i}(0)^{2}\right)$ and $4\pi^{2}\frac{1}{N}\sum_{i=1}^{N}\left(|r_{i}(0)|^{2}w_{i}(0)^{2}\right)$, we can obtain solutions of different regularity/smoothness for a given training dataset.

Before we rationalize this model, we first demonstrate the effectiveness of this model through experiments on the synthetic training data of 1-d and 2-d input. Note that the long-time limit solution of dynamics (3) is obtained by solving an equivalent optimization problem numerically (see Section 5 and Appendix 11 for details.).

For the case of 2-d input, i.e., $d=2$, we consider the training dataset of the famous XOR problem, which cannot be solved by one-layer neural networks. This training dataset consists of four points represented by white stars in Fig. 2(a). As shown in Fig. 2(a-c), our LFP model predicts the final output of NN very accurately over the domain $^{2}$. Similar to the 1-d case, as the number of hidden neurons increases, the prediction by the LFP model becomes more accurate. We also present similar experimental results for an asymmetrical training dataset in Fig. 4 in Appendix.

Our starting point is the following linearized mean field residual dynamics (Mei et al. (2018, 2019)):

where $\theta_{0}$ denotes the initial parameters of the NN in the mean field kernel limit. The kernel is defined as

where $\sigma^{*}(\cdot,\theta_{0})=w_{0}\sigma\left(r_{0}\cdot[\cdot]-\left\lvert r_{0}\right\rvert l_{0})\right)$, $\sigma$ is the ReLU function. By applying the Fourier transform with respect to $x$ to both sides of Eq. (4), we can approximately derive the following frequency domain dynamics up to a time constant (see Appendix 8 for details), that is

Explicitizing the implicit bias of the F-Principle

In our LFP model, the solution is implicitly regularized by a decaying coefficient for different frequencies of $\hat{u}$ throughout the training. For a quantitative analysis of this solution, we explicitize such an implicit dynamical regularization by a constrained optimization problem as follows.

First, we present a general theorem that the long-time limit solution of a gradient flow dynamics is equivalent to the solution of a constrained optimization problem. All proofs are in Appendix 9.

Given $g\in H_{2}$, we consider the following two problems.

Since this equation is linear and with nonpositive eigenvalues on the right hand side, there exists a unique global-in-time solution $u(t)$ for all $t\in[0,+\infty)$ satisfying the initial condition. Moreover, the long-time limit $\lim_{t\rightarrow+\infty}u(t)$ exists and will be denoted as $u_{\infty}$.

In the following, we will show that it has a unique minimizer which is denoted as $u_{\min}$. Now we present the following theorem of the equivalence relation.

Suppose that $PP^{*}$ is surjective. The above Problems (i) and (ii) are equivalent in the sense that $u_{\infty}=u_{\min}$. More precisely, we have

The following corollary is obtained directly from Theorem 1.

Then the following two problems are equivalent in the sense that $\hat{h}_{\infty}=\hat{h}_{\min}$.

Note that in Appendix 9, we provide another version of Corollary 2 for the discretized frequency, which is considered in Section 6.

2 Explicitizing the implicit bias for two-layer NNs

By Corollary 2, we derive the following constrained optimization problem explicitly minimizing an FP-norm (see Section 6.1), whose solution is equivalent to that of the LFP model (3), that is,

subject to constraints $h(x_{i})=y_{i}$ for $i=1,\cdots,M$. Note that the solutions of the LFP models in Figs. (1, 2) are obtained by solving another form of this optimization problem (see Appendix 11). This explicit penalty indicates that the learning of DNN is biased towards functions with more power at low frequencies (more precisely, functions of smaller FP-norm), which is speculated in Xu et al. (2018); Rahaman et al. (2018); Xu et al. (2019). Next, we extend this special example to the case of a general weight function $(\gamma(\xi))^{-2}$ in the frequency domain.

FP-norm and an a priori generalization error bound

The equivalent explicit optimization problem (10) provides a way to analyze the generalization of sufficiently wide two-layer NNs. We begin with the definition of an FP-norm, which naturally induces a FP-space containing all possible solutions of a target NN, whose Rademacher complexity can be controlled by the FP-norm of the target function. Thus we obtain an a priori estimate of the generalization error of NN by the theory of Rademacher complexity. Our a priori estimates follows the Monte Carlo error rates with respect to the sample size. Importantly, Our estimate unravels how frequency components of the target function affect the generalization performance of DNNs.

we define the FP-norm for all function $h\in L^{2}(\Omega)$:

2 a priori generalization error bound

The following lemma shows that the FP-norm closely relates to the Rademacher complexity, which is defined as

Suppose that the real-valued target function $f\in F_{\gamma}(\Omega)$, the training dataset $\{x_{i};y_{i}\}_{i=1}^{M}$ satisfies $y_{i}=f(x_{i})$, $i=1,\cdots,M$, and $h_{M}$ is the solution of the regularized model

By the assumption in the theorem, the target function $f$ belongs to $F_{\gamma}(\Omega)$ which is a subspace of $L^{2}(\Omega)$. In most applications, $f$ is also a continuous function. In any case, $f$ can be well-approximated by a large neural network due to universal approximation theory Cybenko (1989).

Our a priori generalization error bound in Theorem. 4 is large if the target function possesses significant high frequency components. Thus, it explains the failure of DNNs in generalization for learning the parity function (Shalev-Shwartz et al., 2017), whose power concentrates at high frequencies. In the following, We use experiments to illustrate that, as predicted by our a priori generalization error bound, larger FP-norm of the target function indicates a larger generalization error.

3 Experiment

In this section, we train a ReLU-NN of width 1-5000-1 to fit 20 uniform samples of $f(x)=\sin(2\pi vx)$ on $$until the training MSE loss is smaller than$10^{-6}$, where$v$ is the frequency. We then use 500 uniform samples to test the NN. The FP-norm of the target function is computed by

where $\hat{f}(\xi)$ is computed by the discrete Fourier transform of $f(x)$. As shown in Fig. 3, a larger FP-norm of the target function is related to a larger test error.

Discussion

In this work, inspired by the F-Principle, we propose an effective LFP model for NNs — a model quantitatively well predicts the output of wide two-layer ReLU NNs and is theoretically rationalized by their training dynamics in an extremely over-parameterized regime. We explicitize the implicit bias of the F-Principle by a constrained optimization problem equivalent to the LFP model. This explicitization leads to an a priori estimate of the generalization error bound, which depends on the FP-norm of the target function. Note that, our LFP model based on the ReLU transfer function can be naturally extended to other transfer functions following a similar construction process.

As a candidate of an effective model of DNNs, the LFP model advances our qualitative/empirical understandings of the F-Principle to a quantitative level. i) With ASI trick (Zhang et al., 2019) offsetting the initial DNN output to zero, the LFP model indicates that the F-Principle also holds for DNNs initialized with large weights. Therefore, “initialized with small parameters” (Xu et al., 2018, 2019) is not a necessary condition for the F-Principle. ii) Based on the qualitative behavior of F-Principle, previous works (Xu et al., 2018, 2019; Rahaman et al., 2018) speculate that “DNNs prefer to learn the training data by a low frequency function”. With an equivalent optimization problem explicitizing the F-Principle, the LFP model quantifies this speculation.

Our a priori generalization error bound increases as the FP-norm of the target function increases. This explains several important phenomena. First, DNNs fail to generalize well for the parity function (Shalev-Shwartz et al., 2017). Xu et al. (2019) shows that this is due to the inconsistency between the high frequency dominant property of the parity function and the low frequency preference of DNNs. In this work, by our a priori generalization error bound, the dominant high frequency of the parity function quantitatively results in a large FP-norm, thus, a large generalization error. Second, because randomly labeled data possesses large high frequency components, which induces a large FP-norm of any function well matches the training data and test data, we expect a very large generalization error, e.g., no generalization, as observed in experiments. Intuitively, our estimate indicates good generalization of Ns forNwell-structured low-frequency dominant real dataset as well as bad generalization of NNs for randomly labeled data, thus providing insight into the well known puzzle of generalization of DNNs (Zhang et al., 2016).

The F-Principle, a widely observed implicit bias of DNNs, is also a natural bias for human. Empirically, when a human see several points of training data, without a specific prior, one tends to interpolate these points by a low frequency dominant function. Therefore, the success of DNN may partly result from its adoption of a similar interpolation bias as human’s. In general, there could be multiple types of implicit biases underlying the training dynamics of a DNN. Inspired by the LFP model, discovering and explicitizing these implicit biases could be a key step towards a thorough quantitative understanding of deep learning.

Acknowledgments

The authors want to thank Prof. Weinan E for helpful discussions. ZX, YZ are supported by the NYU Abu Dhabi Institute G1301.

Appendix

In this section, we derive the dynamics of each frequency component of the loss function when a two-layer ReLU-NN is used to fit a $d$-dimensional function. Under mild assumption, we can clearly see that a lower frequency component has a faster convergence speed.

The starting point is the following linearized mean field residual dynamics Mei et al. (2018, 2019):

here $\theta_{0}$ denotes the initial parameters of the training dynamics, considering the linear kernel regime. The kernel is defined as

with $\sigma^{*}(\cdot,\theta_{0})=w_{0}\sigma\left(r_{0}\cdot[\cdot]-\left\lvert r_{0}\right\rvert l_{0})\right)$ and $\sigma$ is the ReLU function. For simplicity, we assume the parameters are isotropic and $\rho(l_{0})=C_{l_{0}}$, $\rho(e_{r_{0}})=C_{e_{r_{0}}}$ to be specified later that is,

where $e_{r_{0}}$ is the unit vector of $r_{0}$. Then gradient of $\sigma^{*}$ with respect to the parameters is

For any given vector $r_{0}$, we can decompose the gradient with respect to $r_{0}$, that is, second row in the above gradient vector, into two directions: parallel and perpendicular to $r_{0}$, i.e.,

where $[\cdot]_{\perp r_{0}}=[\cdot]-\frac{[\cdot]\cdot r_{0}}{\left\lvert r_{0}\right\rvert^{2}}r_{0}$ and $[\cdot]\sigma^{\prime}(\cdot)=\sigma(\cdot)$ due to the property of the ReLU function. Thus, the gradients (24) can be written into two parts

Then the kernel (22) can be split into two parts,

In the following computation, we will drop the term $G$. Since this term corresponds to the direction perpendicular to $r_{0}$, it is not very easy to check by numerical experiments and we only consider the dynamics of the parallel part.

Taking Fourier transform with respect to $x$,

where $x_{\parallel r_{0}}=\frac{r_{0}\cdot x}{\left\lvert r_{0}\right\rvert^{2}}r_{0}$, $x_{\perp r_{0}}=x-x_{\parallel r_{0}}$, $\xi_{\perp r_{0}}=\xi-\frac{\xi\cdot r_{0}}{\left\lvert r_{0}\right\rvert^{2}}r_{0}$ and $y=e_{r_{0}}\cdot x$. The above delta function is defined as, for any function $f(e_{r_{0}})$

where $C_{e_{r_{0}}}=\frac{\Gamma(d/2)}{2\pi^{d/2}}$. Combining the above results, one obtains

then we first integrate the variable $e_{r_{0}}$ using (42) to get

Finally, by choosing a suitable time scale the above dynamics can be written as

Proofs of the equivalence theorems

Suppose that $H_{1}$ and $H_{2}$ are two seperable Hilbert spaces and $P:H_{1}\rightarrow H_{2}$ and $P^{*}:H_{2}\rightarrow H_{1}$ is the adjoint of $P$. Then all eigenvalues of $P^{*}P$ and $PP^{*}$ are non-negative. Moreover, they have the same positive spectrum. If in particular, we assume that the operator $PP^{*}$ is surjective, then the operator $PP^{*}$ is invertible.

We consider the eigenvalue problem $P^{*}Pu_{1}=\lambda u_{1}$. Taking inner product with $u_{1}$, we have $\langle u_{1},P^{*}Pu_{1}\rangle_{H_{1}}=\lambda\left\lVert u_{1}\right\rVert^{2}_{H_{1}}$. Note that the left hand side is $\left\lVert Pu_{1}\right\rVert^{2}_{H_{2}}$ which is non-negative. Thus $\lambda\geq 0$. Similarly, the eigenvalues of $PP^{*}$ are also non-negative.

Now if $P^{*}P$ has a positive eigenvalue $\lambda>0$, then $P^{*}Pu_{1}=\lambda u_{1}$ with non-zero vector $u_{1}\in H_{1}$. It follows that $PP^{*}(Pu_{1})=\lambda(Pu_{1})$. It is sufficient to prove that $Pu_{1}$ is non-zero. Indeed, if $Pu_{1}=0$, then $P^{*}Pu_{1}=0$ and $\lambda=0$ which contradicts with our assumption. Therefore, any positive eigenvalue of $P^{*}P$ is an eigenvalue of $PP^{*}$. Similarly, any positive eigenvalue of $PP^{*}$ is an eigenvalue of $P^{*}P$.

Next, suppose that $PP^{*}$ is surjective. We show that $PP^{*}u_{2}=0$ has only the trivial solution $u_{2}=0$. In fact, $PP^{*}u_{2}=0$ implies that $\left\lVert P^{*}u_{2}\right\rVert^{2}_{H_{1}}=\langle u_{2},PP^{*}u_{2}\rangle_{H_{2}}=0$, i.e., $P^{*}u_{2}=0$. Thanks to the surjectivity of $PP^{*}$, there exists a vector $u_{3}\in H_{2}$ such that $u_{2}=PP^{*}u_{3}$. Let $u_{1}=P^{*}u_{3}\in H_{1}$. Hence $u_{2}=Pu_{1}$ and $P^{*}Pu_{1}=0$. Taking inner product with $u_{1}$, we have $\left\lVert Pu_{1}\right\rVert^{2}_{H_{2}}=\langle u_{1},P^{*}Pu_{1}\rangle_{H_{1}}=0$, i.e., $u_{2}=Pu_{1}=0$. Therefore $PP^{*}$ is injective. This with the surjectivity assumption of $PP^{*}$ leads to that $PP^{*}$ is invertible. ∎

Given $g\in H_{2}$, we consider the following two problems.

Since this equation is linear and with nonpositive eigenvalues on the right hand side, there exists a unique global-in-time solution $u(t)$ for all $t\in[0,+\infty)$ satisfying the initial condition. Moreover, the long-time limit $\lim_{t\rightarrow+\infty}u(t)$ exists and will be denoted as $u_{\infty}$.

In the following, we will show it has a unique minimizer which is denoted as $u_{\min}$.

Now we show the following equivalent theorem.

Suppose that $PP^{*}$ is surjective. The above Problems (i) and (ii) are equivalent in the sense that $u_{\infty}=u_{\min}$. More precisely, we have

For problem (i’), from the theory of ordinary differential equations on Hilbert spaces, we have that its solution can be written as

The following corollaries are obtained directly from Theorem 8.

The next corollary is a weighted version of Theorem 8.

The operator $\Gamma:H_{1}\rightarrow H_{W}$ is bijective. Hence $\Gamma^{-1}:H_{\Gamma}\rightarrow H_{1}$ is well-defined and $H_{\Gamma}$ with norm $\left\lVert\cdot\right\rVert_{H_{\Gamma}}$ is a Hilbert space. The equivalence result holds by applying Theorem 8 with proper replacements. More precisely, we replace $u$ by $\Gamma u$ and $P$ by $P\Gamma$. ∎

Then the following two problems are equivalent in the sense that $\hat{h}_{\infty}=\hat{h}_{\min}$.

The equivalence result then follows from Corollary 10. ∎

We remark that $P^{*}P\hat{h}=\sum_{i=1}^{M}\widehat{h\delta_{x_{i}}}$, where $\delta_{x_{i}}(\cdot)=\delta(\cdot-x_{i})$, $i=1,\cdots,M$. Therefore problem (C1) can also be written as:

In the following, we study the discretized version of this dynamics-optimization problem (C1&C2).

Then the following two problems are equivalent in the sense that $\hat{h}_{\infty}=\hat{h}_{\min}$.

The equivalence result then follows from Corollary 10. ∎

Proof of the a priori generalization error bound

2 FP-space

we define the FP-norm for all function $h\in L^{2}(\Omega)$:

3 A prior generalization error bound

We first prove (ii) since it is more involved. By the definition of the Rademacher complexity

For (ii), the proof is similar to (i). We have

By the definition of the FP-norm, we have $\left\lVert h_{M}-h_{\rm ini}\right\rVert_{\gamma}=\left\lVert\hat{h}_{M}-\hat{h}_{\rm ini}\right\rVert_{H_{\Gamma}}$. According to Corollary 12, the minimizer of problem (77) exists, i.e., $h_{M}$ exists. Since the target function $f(x)$ satisfies the constraints $f(x_{i})=y_{i}$, $i=1,\cdots,M$, we have $\left\lVert h_{M}-h_{{\rm ini}}\right\rVert_{\gamma}\leq\left\lVert f-h_{{\rm ini}}\right\rVert_{\gamma}$. ∎

Let $f^{\prime}=f-h_{{\rm ini}}$. Since $h_{M}(x_{i})-f(x_{i})=0$ for $i=1,\cdots,M$, we have $h_{M}(x_{i})-f^{\prime}(x_{i})-h_{{\rm ini}}(x_{i})=0$. Therefore

We remark that the last step is due to the same reason as Lemma 14. ∎

Suppose that the real-valued target function $f\in F_{\gamma}(\Omega)$, the training dataset $\{x_{i};y_{i}\}_{i=1}^{M}$ satisfies $y_{i}=f(x_{i})$, $i=1,\cdots,M$, and $h_{M}$ is the solution of the regularized model

One of the component can be bounded as follows

By the assumption in the theorem, the target function $f$ belongs to $F_{\gamma}(\Omega)$ which is a subspace of $L^{2}(\Omega)$. In most applications, $f$ is also a continuous function. In any case, $f$ can be well-approximated by a large neural network due to universal approximation theory Cybenko (1989).

Numerical solution of the LFP model

Numerically, we solve the following ridge regression problem (Mei et al. (2019)) to approximate the solution of the optimization problem (10)