Neural Network Approximation

Introduction

Approximation using Neural Networks (NNs) is the method of choice for building numerical algorithms in Machine Learning (ML) and Artificial Intelligence (AI). It is now being looked at as a possible platform for computation in many other areas. Although NNs have been around for over 70 years, starting with the work of Hebb in the late 1940’s (?) and Rosenblatt in the 1950’s (?), it is only recently that their popularity has surged as they have achieved state-of-the-art performance in a striking variety of machine learning problems. Examples of these are computer vision (?), employed for instance in self-driving cars, natural language processing (?), used in Google Translate, or reinforcement learning, such as superhuman performance at Go (?, ?), to name a few.

Nevertheless, it is generally agreed upon that there is still a lack of solid mathematical analysis to explain the reasons behind these empirical successes. As a start, the understanding of the approximation properties of NNs is of vital importance since approximation is one of the main components of any algorithmic design. A rigorous analysis of what special properties NNs hold as a method of approximation could lead to both significant practical improvements (?, ?) and a priori performance guarantees for computational algorithms based on NNs.

At the heart of providing such a rigorous theory is understanding the benefits of using NNs as an approximation tool when compared with other more classical methods of approximation such as polynomials, wavelets, splines, and sparse approximation from bases, frames, and dictionaries. Indeed, most applications of NNs are built on some form of function approximation. This includes not only learning theory and statistical estimation, but also the new forays of NNs into other application domains such as numerical methods for solving partial differential equations (PDEs).

An often cited theoretical feature of neural networks is that they produce universal function approximants (?, ?) in the sense that, given any continuous target function $f$ and a target accuracy $\epsilon>0$, neural networks with enough judiciously chosen parameters produce an approximation to $f$ within an error of size $\epsilon$. This universal approximation capacity has been known since the $1980$’s. But surely, this cannot be the main reason why neural networks are so effective in practice. Indeed, all families of functions used in numerical approximation such as polynomials, splines, wavelets, etc., produce universal approximants. What we need to understand is in what way NNs are more effective than other methods as an approximation tool.

The purpose of the present article is to describe the approximation properties of NNs as we presently understand them, and to compare their performance with other methods of approximation. To accomplish such a comparative analysis, we introduce, starting in §5, the tools by which various methods of approximation are evaluated. These include approximation rates on model classes, $n$–widths, metric entropy, and approximation classes. Since NN approximation is a form of nonlinear manifold approximation, we make this particular form of approximation the focal point of our exposition. The ensuing sections of the paper examine the specific approximation properties of NNs. After making some remarks that apply to general activation functions $\sigma$, we turn our attention to the performance of Rectified Linear Unit (ReLU) networks. These are the most heavily used in numerical settings and fortunately also the NNs most amenable to analysis.

Since the output of a ReLU network is a continuous piecewise linear function (CPwL), it is important to understand what the class of outputs of a ReLU network depending on $n$ parameters looks like in terms of their allowable partitions and the correlation between the linear pieces. This topic is addressed in §3. This structure increases in complexity with the depth of the network. It turns out that deeper NNs give a richer set of outputs than shallow networks. Therefore, much of our analysis centers on deep ReLU networks.

The key takeaways from this paper are the following. For a fixed value of $n$, the outputs of ReLU networks depending on $n$ parameters form a rich parametric family of CPwL functions. This manifold exhibits certain space filling properties (in the Banach space where we measure performance error), which are both a boon and a bottleneck. On one hand, space filling provides the possibility to approximate with relatively few parameters larger classes of functions than the classes that are currently approximated by classical methods. On the other hand, this flexibility comes at the expense of both the stability of the algorithm by which one selects the right parameters and the a priori performance guarantees and uncertainty quantifications of performance when using NNs in numerical algorithms. This points to the need for a comprehensive study of the trade-offs between stability of numerical algorithms based on NNs and their numerical efficiency.

This exposition is far from providing a satisfactory theory for approximation by NNs, even when we restrict ourselves to ReLU networks. We highlight several fundamental questions that remain unanswered. Their solution would not only lead to a better understanding of NN approximation but would most likely guarantee better performance in numerical algorithms. These issues include:

matching upper and lower for the rate of approximation of standard model classes when using ReLU networks

how to precisely describe the types of function classes that benefit from NN approximation.

how to numerically impose stability in parameter selection;

how the imposition of stability limits the performance of the network;

What is a Neural Network?

This section begins by introducing feed-forward neural networks and their elementary properties. We begin with a general setting and then specialize to the case of fully connected networks. While the latter networks are generally not the architecture of choice in most targeted applications, their architecture provides the most convenient way to understand the trade-offs between approximation efficiency and the complexity of the network. They also allow for a clearer picture of the balance between width and depth in the assignment of parameters.

In its most general formulation, a feed-forward neural network $\mathcal{N}$ is associated with a directed acyclic graph (DAG),

called the architecture of $\mathcal{N}$, determined by a finite set ${\cal V}$ of vertices and a finite set of directed edges ${\cal E}$, in which every vertex $v\in{\cal V}$ must belong to at least one edge $e\in{\cal E}$. The set ${\cal V}$ consists of three distinguished subsets. The first is the set ${\cal I}$ of input vertices. These vertices have no incoming edges and are placeholders for independent variables (i.e. network inputs). The second is the set ${\cal O}$ of output vertices. These vertices have no outgoing edges and will store, for given inputs, the corresponding value of the dependent variables (i.e. the network output). The third is the set of hidden vertices ${\cal H}={\cal V}\backslash\left\{{\cal I},{\cal O}\right\}$. For a given input, hidden vertices store certain intermediate values used to compute the corresponding output. The vertices and edges also have the following adornments:

In going forward, we often refer to the vertices as nodes. The weights and biases are referred to as the trainable parameters of ${\cal N}$. For a fixed network architecture, varying the values of these trainable parameters produces a family of output functions. The key to describing how these functions are constructed is that to each vertex $v\in{\cal V}\setminus{\cal I}$ we associate a computational unit called a neuron. This unit takes as inputs the scalar outputs $x_{v^{\prime}}$ from vertices $v^{\prime}\in{\cal V}\setminus{\cal O}$ with an edge $e=(v^{\prime},v)\in{\cal E}$ terminating at $v$, and outputs the scalar

The word neuron comes from the fact that (1) can be viewed as a simple computational model for a single biological neuron. A neuron associated to a vertex $v\in{\cal V}\setminus{\cal I}$ observes signals $x_{v^{\prime}}$ computed by upstream neurons associated to $v^{\prime}$, takes a superposition of these signals, mediated by synaptic weights $w_{e}$, $e=(v^{\prime},v)$, and outputs $x_{v}$ which is then seen by the downstream neurons. For all neurons associated to vertices $v\in{\cal O}$, the activation function $\sigma_{v}$ is the identity. The neuron associated to the $i$-th input vertex $v\in{\cal I}$, $i=1,\ldots,d$, where $d:=|{\cal I}|$, observes a scalar incoming (i.e. externally provided) signal $x_{i}$ and outputs $x_{i}$, which is then seen by the downstream neurons.

The preceding is a very general definition of neural networks and encompasses virtually all network architectures used in practice. In this article, however, we restrict our study to rather special examples of such networks, the so-called fully connected networks. The architecture of such a network is given by a directed acyclic graph whose vertices are organized into layers.

As is customary, we specify that there is a single activation function $\sigma$ that is used at each hidden vertex $v\in{\cal H}$, i.e., $\sigma_{v}=\sigma$ for all $v\in{\cal H}$. Recall that we always take the activation $\sigma_{v}$ at the output vertices $v\in{\cal O}$ to be the identity. In this way, each coordinate of $S_{{\cal N}}(x)$ is a linear combination of the $x_{v}$’s at layer $L$ plus a bias term, which is a constant.

Thus, for a fully connected network ${\cal N}$, the output function $S_{\cal N}$ can be succinctly described by weight matrices and bias vectors

We will almost always consider only fully connected feed-forward NNs whose hidden layer widths are all the same, namely, $n_{1}=\dots=n_{L}=W$. Note that we can embed any fully connected feed-forward NN into a network with constant width $W:=\max_{j=1,\ldots,L}n_{j}$ by inserting $(W-n_{j})$ additional zero bias vertices to layer $j$ and adding new edges with weights set to between these vertices and those in the next layer if these vertices are in the first layer, we also add new edges with weights set to between them and the input vertices). We use this fact frequently in what follows, sometimes without mentioning it.

We refer to $W$ as the width of the network and to $L$ as its depth. In such networks, each vertex $v$ from a hidden layer can be associated with a pair of indices $(i,j)$, where $j$ is the layer index and $i$ is the row index of the location of $v$. We commonly refer to all vertices from a fixed row as a channel, and those from a fixed column as a layer. It is useful to introduce for every vertex $v$ from the hidden layers the function $z_{v}:=z_{i,j}$ which records how the value at this neuron depends on the original input $x=(x_{1},\dots,x_{d})$ before the activation $\sigma$ is applied. It follows that,

which is the value of the $i$-th coordinate of the vector $X^{(j)}$ defined in (4).

For a fully connected feed-forward network ${\cal N}$ with width $W$, depth $L$, activation function $\sigma$, input dimension $d$, and output dimension $d^{\prime}$, we define the set

Notice that $\Upsilon^{W,L}$ is closed under addition of weights and biases in the output layer. This follows immediately from (3). However, it is not closed under addition of functions because we can find two outputs $S_{{\cal N}_{1}},S_{{\cal N}_{2}}$ from $\Upsilon^{W,L}$ with $S_{{\cal N}_{1}}+S_{{\cal N}_{2}}\not\in\Upsilon^{W,L}$. This will become apparent even in our discussion of one layer ReLU networks, see §3. Therefore $\Upsilon^{W,L}$ is not a linear space. Each function $S_{\cal N}\in\Upsilon^{W,L}$ is determined by

parameters consisting of the entries of its weight matrices $W^{(1)},\ldots,W^{(L+1)}$ and bias vectors $b^{(1)},\ldots,b^{(L+1)}.$ We note in passing that it is possible that distinct choices of trainable parameters end up describing the same outputs.

Our main focus in this paper is to understand the approximation power of NNs and thus we work under the assumption that we have full access to the target function $f$. However, in the last two sections, we do make forays into the more realistic (numerical) settings where we are only provided (partial) information about $f$ in terms of data observations, or we are only allowed to query $f$ to gain information. This separation between the approximation setting and the numerical setting is important since it may be that we could approximate $f$ well if we had unlimited access to $f$, but in reality we are limited by the information provided to us.

Fully connected feed-forward NNs are an important approximation tool that is amenable to theoretical analysis. In practice, the most common choice of activation function $\sigma$ is the so-called rectified linear unit

This will constitute the main example of activation function studied in this article.

3 Fundamental Operations with Neural Networks

In this section, we discuss some fundamental operations that one can implement with NNs. Recall that $\Upsilon^{W,L}=\Upsilon^{W,L}(\sigma;d,d^{\prime})$ is the set of functions that are outputs of a NN with the activation function $\sigma$, input dimension $d$, output dimension $d^{\prime}$, and $L$ hidden layers each of fixed width $W$.

Let us begin by pointing out that deep neural networks naturally allow for two fundamental operations – parallelization and concatenation – which we will often use. Parallelization: If the NNs ${\cal N}_{j}$ have width $W_{j}$, depth $L$, input dimension $d$, output dimension $d^{\prime}$, and an activation function $\sigma_{j}$, $j=1,\dots,m$, then the parallelization of these networks is a new network ${\rm PAR}({\cal N}_{1},\ldots,{\cal N}_{m})$ with width $W=W_{1}+\ldots+W_{m}$, depth $L$, input dimension $d$ and output dimension $d^{\prime}$. Its graph is obtained by placing the hidden layers of ${\cal N}_{j}$ on the top of each other. The parallelized network can output any linear combination $S=\sum_{j=1}^{m}\alpha_{j}S_{j}$, where $S_{j}\in\Upsilon^{W_{j},L}(\sigma_{j};d,d^{\prime})$, $j=1,\dots,m$.

As described above, the network ${\rm PAR}(S_{{\cal N}_{1}},\ldots,S_{{\cal N}_{m}})$ does not have full connectivity since the nodes of ${\cal N}_{j}$ are not connected to the nodes of ${\cal N}_{i}$, $i\neq j$. However, we can view the resulting network as a fully connected network by completing it, that is, by adding the missing edges and assigning to them zero weights.

where the composition is performed $m-1$ times. Concatenation: If the NNs ${\cal N}_{j}$ have width $W_{0}$, depth $L_{j}$, input dimension $d_{j}$, output dimension $d_{j+1}$, and activation functions $\sigma_{j}$, $j=1,\dots,m$, then the concatenation of these networks is a network ${\rm CONC}({\cal N}_{1},\ldots,{\cal N}_{m})$ with width $W_{0}$, depth $L=\sum_{j=1}^{m}L_{j}$, input dimension $d_{1}$ and output dimension $d_{m+1}$. Its graph is obtained by placing the hidden layers of these networks side by side with full connectivity between the hidden layers of ${\cal N}_{j}$ and ${\cal N}_{j+1}$. The concatenated NN can output any composition $S=S_{m}\circ S_{m-1}\circ\cdots\circ S_{1}$, where the functions $S_{j}\in\Upsilon^{W_{0},L_{j}}(\sigma_{j};d_{j},d_{j+1})$, $j=1,\ldots,m$. It does this by assigning weights and biases, associated to edges connecting the last hidden layer of an ${\cal N}_{j}$ to a node of the first hidden layer of the neighbor ${\cal N}_{j+1}$, using the output weights and biases of ${\cal N}_{j}$ and input weights and biases of ${\cal N}_{j+1}$.

Parallelization and concatenation of neural networks allow us to perform the following operations between their outputs. Addition by increasing width: It follows from Parallelization that for any $L\geq 1$ and $S_{j}\in\Upsilon^{W_{j},L}(\sigma;d,d^{\prime})$, $j=1,\dots,m$, the linear combination

Composition: It follows from Concatenation that for any $W\geq 1$ and $S\in\Upsilon^{W,L_{1}}(\sigma;m,d^{\prime})$, $T\in\Upsilon^{W,L_{2}}(\sigma;d,m)$, the composition

In particular, it follows from Parallelization and Concatenation that given the outputs $T_{j}\in\Upsilon^{W_{j},L_{1}}(\sigma;d,1)$, $j=1,\ldots,m$, and $S\in\Upsilon^{W,L_{2}}(\sigma;m,d^{\prime})$, with $W=\sum_{j=1}^{m}W_{j}$, then the function

4 One Layer Neural Networks

The function $S_{\cal N}\in\Upsilon^{W,1}(\sigma;d,1)$ produced by a single hidden layer, fully connected feed-forward neural network $\mathcal{N}$ with activation function $\sigma$, $d$ inputs and one output has the representation

where $W$ is the width of the first (and only) hidden layer and $b_{0}$ is the bias of the output node. The above can equivalently be written as

where for the current discussion the distance is measured in the uniform norm $\|f\|_{C(\Omega)}:=\sup_{x\in\Omega}|f(x)|$. This question is discussed in detail in (?), see also (?), (?). Here we only point out some key results.

is not identically zero, then the density condition (12) holds. This condition can be used to prove the following examples of activation functions for which (12) holds.

For each such $\sigma$ the density statement (12) holds, see (?) for one of the first proofs in this case.

ReLU Networks

In this section, we summarize what is known about the outputs of NNs with ReLU activation (ReLU networks). We begin by making general remarks that hold for any ReLU network and then turn to special cases, especially those that form our main interest of study in this paper.

Perhaps the most important structural property of ReLU networks is that any output of such a network is a continuous piecewise linear function. To describe this precisely, we start with the following definitions.

Each such convex polytope is the intersection of a finite number of closed half spaces. We refer to the polytopes $P_{j}$ of such a partition as cells.

We then say that $S$ is subordinate to the polytope partition ${\cal P}.$

Let ${\cal N}$ be a ReLU network with $d$ inputs, one output node, and $m$ hidden neurons. Then, the output $S_{\cal N}$ of ${\cal N}$ is a CPwL function subordinate to a partition ${\cal P}_{\cal N}$ with at most $3^{m}$ cells, i.e., $\#{\cal P}_{\cal N}\leq 3^{m}$.

Proof: Let us denote by $z_{1}(x),\ldots,z_{m}(x)$ the pre-activations of the network’s neurons, that is the values stored at the neurons for input $x$ before ReLU is applied. For every activation pattern

Our purpose in the remainder of this section is to explore the properties of both the polytope partitions created by ReLU networks and the complexity of the CPwL functions that they output. We start in §3.1 by studying in detail ReLU networks with input and output dimension $1$, postponing a discussion of higher input dimensions to §3.2.

For the set $\Upsilon^{W,1}:=\Upsilon^{W,1}({\rm ReLU};1,1)$, we have the simple inclusion, see (?),

$H_{\bf p}$ is a CPwL function that takes the value one at $p_{2}$, zero at $p_{1}$ and $p_{3}$, is linear on $[p_{1},p_{2}]$ and $[p_{2},p_{3}]$, and vanishes outside of $[p_{1},p_{3}]$. Note that since $H_{\bf p}\equiv 0$ outside $[p_{1},p_{3}]$, we have

and hence $H_{\bf p}\in\Upsilon^{3,1}({\rm ReLU};1,1)$. In particular, the hat function $H$, defined as $H:=H_{(0,1/2,1)}$ and viewed as a function on $$ has the representation

Thus, $H\in\Upsilon^{2,1}({\rm ReLU};1,1)$ when considered only on $$.

1.2 Deep Univariate ReLU Networks

When $L=1$, any selection of weights and biases produces as output a CPwL function $S$ with at most $W$ breakpoints. Indeed, $S$ can be expressed as $S=b_{0}+\sum_{j=1}^{W}a_{j}\eta_{j}(t)$, where the functions $\eta_{j}(t)=(\pm t+b_{j})_{+}$. Obviously the bound $W$ cannot be improved. Although $\Upsilon^{W,1}$ does not contain all of $\Sigma_{W+1,1}$, it does contain all of $\Sigma_{W,1}$, see (14).

When $L>1$, the situation gets much more complicated. Even though there is no precise characterization of the set of outputs, we can provide some important insight. When $L$ grows, two important things happen:

the number breakpoints of functions from $\Upsilon^{W,L}$ can be exponential in $L$;

not every CPwL function with this large number of breakpoints is in $\Upsilon^{W,L}$, in fact, far from it.

There is a set $\Lambda$, $\#(\Lambda)\leq m(L)$ such that each of the $S_{k}$ have their breakpoints in $\Lambda$. Fix $k$ and consider the function $[S_{k}]_{+}$. It has two types of breakpoints. One are those it inherited from $\Lambda$ and the second is the set $\Lambda_{k}^{\prime}$ of new breakpoints that arose after the application of ReLU. We have $\#(\Lambda_{k}^{\prime})\leq\#(\Lambda)+1\leq m(L)+1$, $k=1,\dots,W$. Hence, $S$ has at most $m(L)+W(m(L)+1)$ breakpoints.It follows that

This recursion with the starting value $m(1)=W$ gives the bound

This bound can be improved somewhat at the expense of a more involved argument.

2 Multivariate ReLU Networks

We now turn to studying the properties of ReLU networks with input dimension $d>1$, starting with those networks that have one hidden layer. Deeper multivariate ReLU networks are discussed in §3.2.2.

A ReLU network $\mathcal{N}$ with input dimension $d>1$, output dimension $1,$ and one hidden layer of width $W$ outputs a function of the form

and the collection ${\cal H}:=\left\{H_{1},\ldots,H_{W}\right\}$, associated to the network ${\cal N}$. This collection is an example of a hyperplane arrangement, a classical subject in combinatorics (?).

In fact, Zaslavsky’s theorem shows that away from a co-dimension $1$ set of weights and biases (i.e. when the hyperplanes are in general position), this upper bound is attained.

where $L$ is an affine function. Since $L$ and $\eta_{j}$ are linearly independent on $B_{j}$, this implies $a_{j}=0$. Hence, all $a_{j}$ are zero and $S$ is a constant. Since $S$ was assumed to have compact support, this constant is zero.

However, when $d>1$, Zaslavsky’s theorem shows that the number of cells in ${\cal P}$ can grow as fast as $CW^{d}$ when $W\geq d$. Hence, in general, the set of all CPwL functions subordinate to ${\cal P}$ is a linear space with dimension much larger than $2W+1$.

if and only if the following condition holds:

2.2 Deep Multivariate ReLU Networks

which have a nonempty interior. We continue to write $z_{j}(x)$ for the CPwL function computed by the $j^{th}$ neuron in $\mathcal{N}$ before ReLU is applied, and we have assumed for simplicity that for every neuron $z_{j}$ the sets

have co-dimension at least $1$. It is important to note that the $H_{j}$’s are no longer hyperplanes since the functions $x\mapsto z_{j}(x)$ are not affine. Instead, $H_{j}$ is the zero level set of $z_{j}$ and, following the language in (?), we refer to the $H_{j}$ as bent hyperplanes and

as a bent hyperplane arrangement. We can now describe the cells in the partition ${\cal P}$,

To understand this setting more clearly, let us consider a neuron $z$ in the second hidden layer of $\mathcal{N}$. Note that the function $x\mapsto z(x)$ is the output of an element of $\Upsilon^{W,1}$. Hence, it is CPwL subordinate to the partition defined by the hyperplane arrangement

created by the neurons $z_{1},\ldots,z_{W}$ in the first hidden layer of $\mathcal{N}$. On each cell ${\cal C}$ of the arrangement ${\cal H}^{(1)}$, the function $x\mapsto z(x)$ is affine. Let $H_{z}$ denote the bent hyperplane associated with this neuron $z$ from the second layer. We see that $H_{z}\cap{\cal C}$ is given by the (possibly empty) intersection of a single hyperplane with ${\cal C}$. However, because $x\mapsto z(x)$ is a different affine function on different cells, its zero set $H_{z}$ may “bend” at the boundary between two cells and is not given globally by a single hyperplane. More is true: while in every cell $H_{z}$ coincides with a single hyperplane, globally, it may have several connected components.

Just as in the case of univariate ReLU networks considered in §3.1, the number of cells defined by the bent hyperplane arrangement $H$ in a deep ReLU network with any input dimension can grow exponentially with depth. In fact, see (?, Theorem 5), there are ReLU networks of depth $L$ and width $W\geq d$ giving rise to partitions with at least

cells. Similarly to the univariate case, the exponential growth in the number of pieces of the CPwL functions produced by deep networks is a consequence of composition.

3 Properties of deep ReLU networks

In a special network, we reserve the top $d$ channels to simply push forward the input values of $x$. Namely, channel $i\in\{1,\dots,d\}$ has

where $x=(x_{1},\dots,x_{d})$ is the initial input. This allows us to use $x$ as an input to the computation performed at any later layer of the network. We refer to such channels as source channels (SC).

In a special network, we also designate some channels, called collation channels (CC), to simply aggregate the value of certain intermediate computations. In a collation channel the ReLU activation may or may not be applied. The key point here is that nodes from both collation and source channels may be ReLU free. Therefore, such networks are not true ReLU networks.

We denote the set of functions $S$ which are the outputs of a special network of width $W$ and depth $L$ by $\overline{\underline{\Upsilon}}^{W,L}$. A useful observation made in (?) is that the functions that are outputs of a special network, when restricted to a bounded domain, are in $\Upsilon^{W,L}$, that is,

This is proved using the following observations.

Given any configuration of weights and biases in any collation channel of a special network and any fixed compact set $K$ of inputs, we may choose a sufficiently large value $b_{i,j}$ associated to the $(i,j)^{th}$ node so that $z_{i,j}(x)+b_{i,j}>0$ for all $x\in K$. Then, we construct the true ReLU network by assigning to this node the function $\eta^{\prime}_{i,j}$, given by $\eta^{\prime}_{i,j}(x)=[z_{i,j}(x)+b_{i,j}]_{+}=z_{i,j}(x)+b_{i,j}$. The effect of $b_{i,j}$ on any subsequent computation is then eliminated by adding an extra bias (to the bias present from the special network) for any neuron from the next layer to which the output $\eta^{\prime}_{i,j}(x)$ is passed to. We perform this procedure to every ReLU free node from the collation channels.

Similar treatment as above is done for all nodes in all source channels.

The ReLU network that has been constructed has the same output as that of the special network we started with.

This specific trick works only when $K$ is compact. Alternatively, at the expense of increasing the width, we can create a true ReLU network of width $W=W_{0}+2d+2k$, where $W_{0}+d+k$ is the width of the special network with $d$ source and $k$ collation channels by using the identity $t=t_{+}-(-t)_{+}$. This approach works for arbitrary inputs but at the expense of increasing the width of the network.

In what follows, we use extensively special networks to derive some important properties of deep networks since they facilitate many constructions.

3.2 Some important properties of deep ReLU networks

As noted in Observation from §3.2 in the case of one layer networks, the vector $w$ consisting of all incoming weights into any hidden node of a ReLU network, if nonzero, can be taken to be of Euclidean norm $\|w\|_{2}=1$. Indeed, this follows from the equality

and the fact that the factor $\|w\|_{2}$ can be absorbed by the outgoing weights.

Next, we return to the Addition Property. Earlier we have shown that we can add functions in $\Upsilon^{W,L}(\sigma;d,d^{\prime})$ by increasing the width of the network using the method of parallelization. Here, we want to observe that addition can also be performed by increasing depth and not significantly enlarging width. Here is a statement to that effect.

We discuss the case of minimum only, since the case of maximum is almost the same. We start with proving (25). In our construction, we will use the fact that

For general $m$, we let $k:=\lceil\log_{2}m\rceil$ and define $\hat{x}_{j}=x_{j}$, $1\leq j\leq m$ and $\hat{x}_{j}:=x_{m}$, $m<j\leq 2^{k}$. Applying the above to this new sequence gives the result (25). To show (24), we feed $z_{j}(x)$ into the first hidden layer of ${\cal N}_{k}$ by assigning appropriate input weights and node biases.

At the end, if we want to output the ReLU of min/max, we just add another hidden layer to perform the ReLU.

Another way to compute the above min/max is via increasing the depth and keeping the width relatively small by utilizing a recursive formula, first used in (?). Minimization/Maximization 2 (MM2): Let

We discuss the case of maximum only (the case of minimum is treated likewise). Let $\mu_{1}(x):=z_{1}(x)$ and $\mu_{k}(x):=\max\{z_{1}(x),\dots,z_{k}(x)\}$, $k\geq 2$. We use the recursion formula

and discuss the case ${\cal R}=^{d}$. For the case of a general rectangle ${\cal R}$ one needs to add appropriate biases. Our construction is the following:

the first $d$ channels of the network push forward the variables $x_{1},\ldots,x_{d}$. Their nodes can be viewed as ReLU nodes since $t_{+}=t$ for $t\geq 0$.

the $(d+1)^{st}$ channel computes in its first node $(z_{1}(x)-z_{2}(x))_{+}$. Note that if we wanted to, we could stop and output $\mu_{2}(x)$ at this stage. The $j^{th}$ node of this channel, $j=2,\ldots,m-2$, computes $(\mu_{j}(x)-z_{j+1}(x))_{+}$, which is then given as an input to the $(j+1)^{st}$ node. The final layer $L=m-1$ will hold $\mu_{m-1}(x)$ and hence can output $\mu_{m}(x)$.

To show the last statement, we add a hidden layer after the last hidden layer of the NN from the construction above to perform the ReLU of the max/min. Of course, we could augment the resulting network by adding nodes and connections so that we have a fully connected feed-forward NN.

More general statements hold when instead of computing the min/max of affine functions we have to find the min/max of outputs of neural networks.

where $W:=\max\{W_{1}+W_{2}+\cdots+W_{m},3\cdot 2^{\lceil\log_{2}m\rceil-1}\}$, $L=L_{0}+\lceil\log_{2}m\rceil$. If $W_{j}\geq 3$, $j=1,\ldots,m$, we have $W=W_{1}+W_{2}+\cdots+W_{m}$. The same statement holds for $\max\{S_{1},\dots,S_{m}\}$. We use Parallelization to construct the first $L_{0}$ hidden layers of the network ${\cal N}$ that outputs $S$ Then, from the $L_{0}$-th layer we can output any of the $S_{j}$, $j=1,\dots,m$. We concatenate this with the network in MM1 which has $\lceil\log_{2}m\rceil$ hidden layers to complete the construction of ${\cal N}$. Clearly, the resulting network has varying width, where the first $L_{0}$ layers are with width $W_{1}+W_{2}+\cdots+W_{m}$, while the last $\lceil\log_{2}m\rceil$ layers have width $3\cdot 2^{\lceil\log_{2}m\rceil-1}$. We augment this network by adding extra nodes and edges. At the end, our network has width $W=\max\{W_{1}+W_{2}+\cdots+W_{m},3\cdot 2^{\lceil\log_{2}m\rceil-1}\}.$ In the case of $W_{j}\geq 3$, $W_{1}+\ldots+W_{m}\geq 3\cdot 2^{\lceil\log_{2}m\rceil-1}$, which gives that $W=W_{1}+\ldots+W_{m}$.

In order to construct a neural network ${\cal N}$ which shows that $S\in\Upsilon^{W,L}$, we utilize Concatenation in place of Parallelization and we use special networks. Let ${\cal N}_{j}$ be a network of width $W_{0}$ and depth $L_{j}$ which outputs $S_{j}$, $j=1,\dots,m$. To each of the networks ${\cal N}_{j}$, we add $d$ source channels to push forward the original inputs $x_{1},\dots,x_{d}$ and one collation channel that we will use to update computations towards outputting $S$. Let us denote these special networks by ${\cal N}_{j}^{\prime}$. We now explain how to construct ${\cal N}$. The first $L_{1}$ hidden layers of ${\cal N}$ consist of those of ${\cal N}_{1}^{\prime}$. The collation channel simply pushes forward zero for these layers. We concatenate ${\cal N}_{1}^{\prime}$ with ${\cal N}_{2}^{\prime}$ by placing $S_{1}$ in the collation channel of ${\cal N}_{2}^{\prime}$ and then pushing it forward, and by placing the outputs of the source channels of ${\cal N}_{1}^{\prime}$, multiplied by appropriate weights (those that enter the first layer of ${\cal N}_{2}$), into the first hidden layer of ${\cal N}_{2}^{\prime}$. If $m=2$, we can complete the construction by placing a last hidden layer which takes $S_{1}$ from the collation channel and $S_{2}$ as an output from ${\cal N}_{2}^{\prime}$ and computes $(S_{1}-S_{2})_{+}$, $(S_{2})_{+}$, and $(-S_{2})_{+}$. We augment the resulting network with additional nodes, if necessary, so that we have a special network with width $W$. This network outputs $S$, has depth $L=L_{1}+L_{2}+1$, and width $W$. If $m>2$, we continue by concatenating with ${\cal N}_{3}^{\prime}$. The collation channel is now occupied by $T_{2}:=\min\{S_{1},S_{2}\}$. If $m=3$, then we complete as before by adding a layer to compute $(T_{2}-S_{3})_{+}$, $(S_{3})_{+}$, and $(-S_{3})_{+}$. Continuing this way we obtain the desired network.

3.3 General CPwL functions

This is proved in (?) by using the fact that any CPwL function $S$ can be written as a linear combination of piecewise linear convex functions, each with at at most $(d+1)$ affine pieces, that is,

with $s_{j}:=\#(S_{j})\leq d+1,$ for some affine functions $z_{1},\ldots,z_{k}$.

For the proof, we can assume that $s_{j}=d+1$ for all $j$ by artificially writing an index already in $S_{j}$ several times, so that we end up with networks with the same depth $L=\lceil\log_{2}(d+1)\rceil$. Using Parallelization, we then stack these networks to produce $S\in\Upsilon^{W^{\prime},L}({\rm ReLU};d,1)$ with $L=\lceil\log_{2}(d+1)\rceil$ and $W^{\prime}=3p2^{\lceil\log_{2}(d+1)\rceil-1}$.

A result along these lines was proven in (?), where it was shown that the output of any ReLU network can be generated by a sufficiently deep ReLU network with fixed width $W=d+3$.

3.4 Finite element spaces

Rather than address this question in its full generality, we consider only a very special setting that will be sufficient for our discussion of NN approximation given in later sections of this paper. The reader can consult (?) and (?) for a more far reaching exposition of the relation between FEM and NNs.

For our special example, we begin with $\Omega=^{d}$, and given $n\geq 1$, we consider the uniform partition ${\cal Q}_{n}$ of $\Omega$ into $n^{d}$ cubes with sidelength $1/n$. We denote by $V_{n}$ the set of vertices of the cubes in ${\cal Q}_{n}$. There are $(n+1)^{d}$ such vertices. Each cube $Q\in{\cal Q}_{n}$ can in turn be partitioned into $d!$ simplices using the so-called Kuhn triangulation with northwest diagonal. This gives a partition ${\cal K}$ of $\Omega$ into $n^{d}d!$ simplices. Let $X({\cal K})$ be the space of all CPwL functions defined on $\Omega$ and subordinate to ${\cal K}$. This is a linear space of dimension $N=(n+1)^{d}$. A basis for $X({\cal K})$ is given by the nodal functions $\{\phi_{v},\,v\in V_{n}\}$, which are the CPwL functions defined on $\Omega$, subordinate to ${\cal K}$, and satisfy

where $\delta$ is the usual Kronecker delta function. Each $S\in X({\cal K})$ has the representation

FEM spaces: Let $X({\cal K})$ be the finite element space in $d$ dimensions described above, and let $d^{*}:=(d+1)!$. Then the following holds:

To prove these statements, we first observe that each nodal basis function $\phi_{v}$ can be expressed as

where $D_{v}$ is the set of simplices in ${\cal K}$ that have $v$ as one of their vertices. There are $(d+1)!$ such simplices when $v$ is an internal vertex and less than that for vertices on the boundary of $\Omega$. The function $z_{\Delta}$ is the linear function which is one at $v$ and vanishes on the facet of $\Delta$ opposite to $v$. If $|D_{\nu}|<(d+1)!$, we add artificially some of the functions $z_{\Delta}$ that are already in $D_{\nu}$ so that we end up with $(d+1)!$ not necessarily different functions, since later we will do parallelization that requires the depth of certain networks to be the same.

Several remarks are in order concerning this result. First, note that the number of parameters used in both NNs is comparable (up to a factor depending on $d$ ) to the dimension $(n+1)^{d}$ of $X({\cal K})$. The most important point to stress is that when using the set $\Upsilon^{W,L}$ in place of a piecewise linear FEM space, we are using a much larger nonlinear family as an approximation tool. Indeed, the set $\Upsilon^{W,L}$ not only contains the FEM space $X({\cal K})$ based on the initial choice of partitioning, but it also contains an infinite number of such FEM spaces corresponding to an infinite number of possible ways to partition $\Omega$. In fact, the NN approach is even more than a simple generalization of the Adaptive Finite Element Method (AFEM), where one is allowed to adaptively choose partitions (from a restricted family of partitions). It will be shown in §8.7 that these NNs provide a provably better approximation rate to various Sobolev and Besov classes than that provided by the FEM spaces. While this seems like a tremendous advantage for NNs over FEMs, one must address (in the specific problem setting) how one (near) optimally chooses the parameters of these NNs.

In the case when FEMs are used to numerically solve linear elliptic PDEs, one can employ the Galerkin method which finds a (near) best approximation to the solution to the PDE by projecting onto $X({\cal K})$. This is well understood and quantified in both theory and practice through theorems that bound error and establish stable numerical implementation.

When we eventually discuss quantitative theorems for NN approximation, we shall see that the known results point to a tremendous potential increase in approximation efficiency (error versus number of parameters needed) when using NNs for the numerical solution of elliptic problems. Whether this advantage can be maintained in concrete stable numerical implementation is less clear.

4 Width versus depth

An underlying issue when choosing a NN architecture to be used in a numerical setting is whether to increase the width or the depth of the NN when one is willing to allocate more parameters to improve accuracy. Suppose, we fix a bound $n$ on the number of parameters to be used and ask which of the sets $\Upsilon^{W,L}$ depending on at most $n$ parameters should we employ in designing a numerical algorithm. All other issues being the same, the general consensus is that in practice deeper networks are preferable. We make some comments to explain this preference from the point of view of the enhanced approximation capacities of deeper networks.

First, we have shown that addition of the output functions of a NN can be implemented by either increasing width (parallelization) or depth (concatenation) with a controlled increase in the number of parameters. However, certain operations like composition and forming minimums can only be implemented by increasing depth. So, for example, if we fix a width $W=W_{0}$ sufficiently large to accommodate $d$ source channels and a couple of collation channels, then we can seemingly implement as outputs from $\Upsilon^{W_{0},L}$ all functions that occur as outputs of shallower networks with a comparable number of parameters. The only rigorous statement given to this effect was for ReLU networks with $d=1$. In this case, it was proved in (?) that for any fixed $W_{0}\geq 4$, we have

5 Interpolation by neural network outputs

A common strategy for approximating a given target function $f$ is to interpolate some of its point values. Although this is often not a good method for approximation, it is important to understand when we can interpolate a given set of data, and how stable is this process. A satisfactory understanding of interpolation using NNs is far from complete. The purpose of this section is to frame the interpolation problem and point out what is known. We begin by considering interpolation by NNs with an arbitrary activation function $\sigma$ and later specialize to ReLU activations.

This is the existence question for data interpolation. In the case interpolants exist, let us denote by ${\cal S}_{I}:={\cal S}_{I}(W,L;\sigma,d)$ the set of functions $S\in\Upsilon^{W,L}(\sigma;d,1)$ which satisfy the interpolation conditions (33).

Given that we are going to use the set $\Upsilon^{W,L}(\sigma;d,1)$ for interpolation, the first question to ask is: Question: Determine the largest value $D^{*}:=D^{*}(W,L;\sigma,d)$ such that the interpolation problem has a solution from $\Upsilon^{W,L}(\sigma;d,1)$ for all data sets of size $D^{*}$. One expects that $D^{*}$ should be closely related to the number of parameters used to describe $\Upsilon^{W,L}(\sigma;d,1)$.

There seems to be only one general theorem addressing the interpolation problem for general activation functions $\sigma$. It applies to the case of single hidden layer networks, that is, $L=1$, and is discussed in detail in the survey article (?), see Theorem 5.1 in that paper.

The following sections discuss the interpolation problem for ReLU activation where more results are known.

We take any points $\xi_{j}$ that satisfy the interlacing property

We establish that interpolation is possible by induction on $W$. When $W=1$, we choose $c:=y_{1}$ and $a_{1}$ so that $c+a_{1}(t^{(2)}-\xi_{1})=y_{2}$. For the induction step, let $S_{0}(t):=c+\sum_{j=1}^{W-1}a_{j}(t-\xi_{j})_{+}$ satisfy the first $W$ interpolation conditions. We define $a_{W}$ so that we have

Finally, we want to see that interpolation at $W+2$ points is generally not possible. For this, we use the following proposition which will also be useful when we discuss the Vapnik–Chervonenkis (VC) dimension of NNs.

if $\xi\leq t^{(1)}$, then $S$ is linear on $[t^{(1)},\infty)$, and therefore cannot satisfy the three interpolation conditions.

if $\xi\in(t^{(1)},t^{(2)})$, then in order for $S$ to satisfy the first two interpolation conditions, we would need $c>0$ and $a<0$. So, the function $S$ is then a non-increasing function of $t$ and thus $S(t^{(3)})\leq S(t^{(2)})$, which shows that $S$ cannot satisfy the third interpolation condition.

if $\xi\geq t^{(2)}$ then $S$ cannot satisfy the first two interpolation conditions, since $S$ is constant on $(-\infty,\xi]$.

if $t^{(n)}<\xi_{n-1}$, then $S_{0}(t^{(j)})=S(t^{(j)})=y_{j}$, $j=1,\ldots,n,$ and $S_{0}$ would contradict the induction hypothesis. Hence this case is not possible.

This completes the proof of the proposition. $\Box$

It is also possible to produce an interpolant to given data by using deep networks with a fixed width. This of course follows from (32) together with what we have just proved. However, we wish to give a direct construction because it will be used later in this paper.

Proof: We have shown above that there is an $S$ of the form (36) with $W:=D-1$, that satisfies the interpolation conditions (37). We view $S$ as a function on $$and construct a special network that outputs any such$S$. The first channel of this special network is a source channel that pushes forward the input$t$and the last channel is a collation channel. This last channel is initialized with at layer$1$and then successively collects the sums$\sum_{i=1}^{j-1}a_{i}(t-\xi_{i})_{+}$at layers$2,\ldots,D-1$, respectively, while the middle channel successively produces the terms$a_{j}(t-\xi_{j})_{+}$, at layers$j=1,\dots,D-1$, using the inputs$t$from the source channel. We can then output$S$from layer$D-1$.$\Box$

We turn now to results that hold for general $d\geq 1$. There is a simple way to derive interpolation results for arbitrary $d>1$ from those for $d=1$. Let

then the ridge function $f(x):=g(v\cdot x)$ satisfies $f(x^{(j)})=y^{j}$, $j=1,\dots,D$. We utilize this observation to prove the following.

While the above proposition is of theoretical interest, it is not used in practice because the ridge function interpolant does not reflect the local flavor of the data. A more common scenario is to construct via ReLU networks a dual basis $\{\phi_{j}\}$, $j=1,\dots,D$, for the data sites, that is, a basis that satisfies the conditions

The goal is to construct a locally supported dual basis. In that case, the interpolation operator

is a bounded projection onto ${\rm span}\{\phi_{j}\}$ whenever the data sites are in $\Omega$. The norm of this projector,

to a large extent determines the approximation properties of interpolation at these sites.

where we insert the best approximation $S$ to $f$ from $X({\cal K})$ to obtain the last inequality. This allows one to deduce estimates for NN approximation from those known in FEM and also to exhibit simple linear operators which achieve these bounds.

6 VC dimension of ReLU outputs

The maximum value of $n$ for which there exists such a collection of $n$ points that are shattered by ${\cal F}$ is called the Vapnik-Chervonenkis (VC) dimension of ${\cal F}$ and is denoted by ${\rm VC}({\cal F})$, see (?).

Let us note that the definition of VC dimension of ${\cal F}$ only requires the existence of one set of points where shattering takes place. When proving upper bounds on the error of approximation, it is useful to know precisely which collections of points can be shattered. The reader will see how this issue arises when we use VC dimension in proving approximation results.

We are interested in describing the VC dimension of the set ${\cal F}$ of outputs of ReLU networks in terms of the number $n(W,L)$ of their parameters. Let us now consider what is known in the special cases of interest to us.

We first consider the space $\Upsilon^{W,1}$ of function of $d$ variables which is described by $n(W,1)=(d+2)W+1$ parameters.

will be positive on the points in $\Lambda$ and zero on the rest of the points $P_{j}$, provided we take $\varepsilon$ small enough. It follows that

and hence the lower bounds stated in (iii), follow from the lower bounds on VC dimension of ${\cal C}$ given in (?). (iv) The lower bounds in this case follow from the fact that we can interpolate any data at any $(W+1)$ data sites, see Propostion 3.7 and (35). $\Box$

Next, we consider the case where $W_{0}$ is fixed but sufficiently large, depending only on $d$, and $L$ is allowed to vary. Note that in this case the number of parameters of the network $n(W_{0},L)\asymp W_{0}^{2}L$. The following theorem gives bounds on the VC dimension of such networks.

Let $W_{0}$ be fixed, and sufficiently large depending only on $d$. There are fixed constants $c_{1},C_{1}$, depending only on $d$, such that

The upper bound in this theorem follows from Theorem 8 in (?). The remainder of this section will provide a proof of the lower bound in a form which will be used later in this paper to prove certain approximation results. Related lower bounds are stated in Theorem 3 of (?).

6.3 Bit extraction using ReLU networks

In order to avoid certain technicalities, we present this result only in the case $d=1$. The full implementation for $d\geq 2$ can be found in (?) and (?).

Let $N:=n^{2}$ with $n\geq 4$ be an even integer. Define $t_{i}:=i/N$, $i=0,1,\dots,N$, and consider any data $y_{i}$, $i=0,\dots,N$, with the properties:

$y_{i+1}=y_{i}+\varepsilon_{i}$, with $\varepsilon_{i}\in\{-1,1\}$ for all $i=0,\dots,N-1$.

Before we present the proof of Theorem 3.10, which is a bit laborious, we introduce some notation, make several observations, and present the general idea of the proof.

First, note that for each $i=0,1,\dots,N-1,$ there is a unique representation

Next, recall that any $t\in$, can be represented as

where the bits $B_{k}(t)\in\{-1,1\}$ of $t$ are found using the familiar quantizer function

with $\chi_{I}$ denoting the characteristic function of a set $I$. The first bit of $t$, $B_{1}(t)=Q(t)$ and has the residual $R_{1}(t):=2t-B_{1}(t)\in$. We find the later bits and residuals recursively as

Given our assigned bit sequence $\{\varepsilon_{i}\}$, $i=0,\ldots,N-1$, available to us from the values $y_{i}$, $i=0,1,\dots,N$, we define the numbers

Note that $Y_{j}\in[-1+2^{-n},-2^{-n}]\cup[2^{-n},1-2^{-n}]\subset$, and the bits $B_{\nu}(Y_{j})=\varepsilon_{jn+\nu-1}$, $\nu=1,\dots,n$.

that in addition satisfies (40). We construct $S$ by showing that each of the functions

are each outputs of ReLU networks of an appropriate size.

To do this, let $\delta=2^{-N}$ and define:

the CPwL function $J=J_{N}$ which has breakpoints at each of the points

and no other breakpoints, and takes the value $j$ on the interval $[\xi_{j},\xi_{j}^{\prime}]$. We also require $J(1)=n$. Note that $J$ has the property

the CPwL function $K(t)=K_{N}(t):=J(nt-J(t))$. Observe that the key property of $K$ is

Next, we would like to implement quantization by a neural network. However, the function $Q$ is not continuous, and so we cannot exactly reproduce $Q$. Instead, we use a surrogate

We define the surrogate bits $\hat{B}_{\nu}(t)$ for $t\in$ by using $\hat{Q}$ in place of $Q$ in the recursive definition of $B_{\nu}$, described in (41). Because of the choice of $\delta$, $\hat{B}_{\nu}$ can be used in place of $B_{\nu}$ to compute the bits of $t$ whenever $t$ has the representation

For such a $t$, we have $\hat{B}_{\nu}(t)=B_{\nu}(t)$, $\nu=1,\dots,N-1$.

the CPwL function $Y$ which has exactly the same breakpoints as $J$, see (44), and satisfies

with $Y_{j}$ defined in (42) and $Y(1)=0$.

The function $S$ will be the output of a special neural network of width $W=11$ and depth $L=15n+2$, which is a concatenation of four special networks that we describe below. The top channel of each of these networks is a source channel which simply passes forward the input $t$. Some of the other channels are collation channels and are occupied by zeros in their first layers so that they can be used later for passing forward certain function values.

We want to point out that our construction is probably not optimal in the sense that it does not provide a NN with the best possible minimal width and depth that outputs $S$. In addition, some of the channels in our NN are ReLU free. We have discussed earlier how we can construct a true ReLU network with the same outputs as a network that has ReLU free nodes.

First NN: This network, which we denote by ${\cal N}_{1}$, has depth $4n-2$ and for any input $t\in$ outputs the function value $K(t)$. From our remarks on interpolation, see Proposition 3.6, we know that $J(t)$ is the output of a special ReLU network ${\cal N}_{0}$ of width $W=3$ and depth $2n-1$, where channel three is a collation channel. The CPwL function $K$ is the output of a ReLU network ${\cal N}_{1}$ of width $W=3$ and depth $4n-2$, which is obtained by concatenating the network ${\cal N}_{0}$ for $J$ with itself and using $nt-J(t)$ as the input to the second of these networks. The third channel is a collation channel, used first to build $J(t)$. Once $J(t)$ is computed, it sends this value as an input to the $2n$-th layer. Then, it is zeroed out by assigning a weight , and subsequently used as a collation channel to build $K(t)$. It follows from (45) that the output of this network is $k(i)$ when the input is $t_{i}$. We add eight other channels with zero parameters. These channels will be used later.

and is zero otherwise, since $3(\nu-k)_{+}=0$ when $\nu\leq k$ and $3(\nu-k)_{+}\geq 3$ when $\nu>k$. It follows from (49) that

Now, for $i=0,\dots,N-1$, consider one of our points $t_{i}$ which is not a multiple of $n$, that is, $k(i)\neq 0$. Then $Y(t_{i})=Y_{j(i)}$, $K(t_{i})=k(i)$, and

Since we cannot produce $B_{\nu}$ with a ReLU network, we use the surrogate $\hat{B}_{\nu}$ in its place. This leads us to define the following function

This function satisfies the interpolation conditions (43) since the bits $\hat{B}_{\nu}(Y(t))=B_{\nu}(Y(t))$, $\nu=1,\dots,n$, whenever $t$ is one of the points $t_{i}$, $i=0,\dots,N$, where interpolation is to take place. In addition, since for $j=0,\ldots,n-1$, $K(t_{jn})=0$ and $K(1)=J(0)=0$, we have

We verify this property when $t\in[0,1/n)\cap\Omega_{N}=[0,1/n-\delta]$ since the verification on the intervals $[j/n,(j+1)/n)\cap\Omega_{N}$, $j=1,\dots,n-1$, is the same. For $t\in[0,1/n-\delta]=[0,t_{n}-\delta]$ we have, see (48), $Y(t)=Y_{0}$, and therefore for $\nu=1,\dots,n$, $\hat{B}_{\nu}(Y(t))=\hat{B}_{\nu}(Y_{0})=B_{\nu}(Y_{0})=\varepsilon_{\nu-1}$. Thus, see (50), we have