Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis

Introduction

Infinite-dimensional (non-linear) dynamical systems play a central role in several sciences, especially for disciplines driven by stochastic analytic modeling. However, despite this fact, the causal neural network approximation theory for most relevant dynamical systems in stochastic analysis is lacking. Indeed, we currently only comprehend neural network approximations of stochastic differential equations (SDEs) with deterministic coefficients (e.g., G2021FS) and time-invariant random dynamical systems with the fading memory and echo state property/unique solution property (e.g., JaegerFMP; Gonon2022NNs). A significant problem is causal neural network approximation of solution operators to non-Markovian SDEs. Moreover, the understanding of how sequential DL models work is still not fully developed, even in the classical finite-dimensional setting. For instance, the seemingly elementary empirical fact that a sequential DL model’s expressiveness increases when one utilizes a high-dimensional latent state space is understood qualitatively for general dynamical systems on Euclidean spaces (as in the reservoir computing literature (e.g., Lukas_StochInputReservoir)).

However, the quantitative understanding of the relationship between a sequential learning model’s state and its expressiveness remains an open problem. One notable exception to this fact is the approximation of linear state-space dynamical systems by a stylized class of Recurrent Neural Networks (RNNs, henceforth); see helmut2022metric; LiJMLR2022.

Our paper provides a simple quantitative solution to a far reaching generalization of the above problem of constructing neural network approximation of infinite-dimensional (generalized) dynamical systems on “good” linear metric spaces. More precisely, we construct a neural network approximation of any function $f$ that “causally” and “regularly” maps sequences $(x_{t_{n}})_{n=-\infty}^{\infty}$ to sequences $(y_{t_{n}})_{n=-\infty}^{\infty}$, where each $x_{t_{n}}$ and every $y_{t_{n}}$ lives in a suitable linear metric space. In particular, we construct our causal neural network approximation framework on the following desiderata:

Predictions are causal, i.e., each $y_{t_{n}}$ is predicted independently of $(x_{t_{m}})_{m>n}$.

Each $y_{t_{n}}$ is predicted with a small neural network specialized at time $t_{n}$.

Only one of these specialized networks is stored in working memory at a time.

We first begin by describing our causal neural network model’s design. Subsequently, we will discuss our approximation theory’s implications in computational stochastic analysis.

Our neural network model, which we call the Causal Neural Operator (CNO, henceforth) is illustrated in Figure 1 and works in the following way. At any given time $t_{n}$, it predicts an instance of the output time-series at that time $t_{n}$ using an immediate time-window from the input time-series (e.g., it predicts each $y_{t_{n}}$ using only $(x_{t_{i}})_{i=n-10}^{n}$). At each time $t_{n}$, this prediction is generated by a non-linear operator defined by a finitely parameterized neural network model, called a neural filter (the vertical black arrows in Figure 1). Our neural network model stores only one neural filter’s parameters in working memory at the current time by using an auxiliary deep ReLU neural network, called a hypernetwork in the machine learning literature (e.g., HDQICLR2017; VOHSG2020ICLR), to generate the next neural filter specialized at $t_{n+1}$ using only the parameters of the current “active” neural filter specialized at time $t_{n}$ (the blue box in Figure 1). Thus, a dynamical system (i.e., the hypernetwork) on the neural filter’s parameter space interpolating between each neural filter’s parameters encodes our entire model.

The principal approximation-theoretic advantage of this approach lies in the fact that the hypernetwork is not designed to approximate anything, but rather, it only needs to memorize/interpolate a finite number of finite-dimensional (parameter) vectors. Since memorization (e.g., VershinynMemorization; KDDWP2022; Ruiyang) requires only a polynomial number of parameters to achieve zero approximation error on a finite set, while approximation (e.g., yarotsky2017error; KP2022JMLR; ZHS2022JMPA; BehnooshAnastasisTMLR) requires an exponential number of parameters to achieve a possibly non-zero error over a large set containing the finite set of interest, then, leveraging memorization yields both lighter (fewer parameters) and more accurate deep learning models; that is, the constructed neural network model is exponentially more efficient. In particular, using a neural network for memorization allows the trained DL model to generalize beyond the data it is interpolating, a capability that a simple list does not possess. When both the input and output spaces are finite-dimensional, our models effectively reduce to RNNs, which are known for their ability to generalize beyond their training data WangGenRNN. This generalization is attributed to factors such as having a finite VC (Vapnik-Chervonenkis) dimension KoiSontRNN; RNNGeneralize or finite Rademacher complexity JRadComplexRNN. Thus, this neural network design allows us to successfully encode all the parameters required to approximate long stretches of time $\{t_{0},\dots,t_{N}\}$ (for large $N$) with far fewer parameters (i.e., at the cost of $O(\log(N))$ additional layers in the hypernetwork). Thus, we successfully achieve desiderata (D1)–(D3) provided that each neural filter relies on only a small number of parameters. We show that this is the case whenever $f$ is “sufficiently smooth”; the rigorous formulation of all these outlined ideas are expressed in Lemma 5 and Theorem 3.2.

Though we are focused on the approximation theoretic properties of our modeling framework, we have designed our CNO by accounting for practical considerations. Namely, we intentionally designed the CNO model so that, like transformer networks VawaniTransformers2017, it can be trained non-recursively (via our federated training algorithm, see Algorithm 1 below). This design choice is motivated by the main reasons why the transformer network model (e.g., VawaniTransformers2017) has replaced residual (e.g., H2020CVPR) and RNN (especially Long Short-Term Memory (LSTMs, henceforth) HochNeurCom1997) counterparts in practice (e.g., Hop1982NA; Wi1986NATURE); namely, not back-propagating through time during training. The reason is that omitting any recurrence relation between a model’s prediction in sequential prediction tasks, at-least during the model’s construction, has been empirically confirmed to yield more reliable and accurate models trained faster and without vanishing or exploding gradient problems; see, e.g., HochVanishing; Pa2013ICML. Nevertheless, our model does ultimately reap the benefits of recursive models even if we construct it non-recursively, using our parallelizable training procedure.

The neural filter, illustrated in Figure 2, is a neural operator with quantitative universal approximation guarantees far beyond the Hilbert space setting. It works by first encoding infinite-dimensional problems into finite-dimensions problems. It then predicts outputs by passing the truncated basis coefficients through a feed-forward neural network with trainable (P)ReLU activation function. Finally, it reassembles them in the output space by interpreting the network’s outputs as the coefficients of a pre-specified Schauder basis or if both spaces are reproducing kernel Hilbert spaces then the first few basis functions can learned from data using principal component analysis Or a robust version thereof, e.g. RobustPCAinfinite2017 and then normalizing and orthogonalizing via Gram-Schmidt., e.g. as with PCA–Net PCANET2023. A similar encoding-MLP-decoding scheme was also used in C2023AA for approximately solving nonlinear Kolmogorov equations on Hilbert spaces. We also note that some infinite-dimensional deep learning models between function spaces on Euclidean domains, such as the DeepONet architecture of DeepONet2021Nature, replace the basis vectors with trainable deep neural networks; however, this technique does not readily apply to general Fréchet spaces. Our “static” approximation theorems provides quantitative approximation guarantees for several “neural operators” used in practice, especially in the numerical Partial Differential Equations (PDEs), e.g., Kar2021PR, and in the inverse-problem literature, e.g., An2020IOP; Bub2021IOP; Al2021Neurips; Bub2021ISIAMIS; deHoop2020MSL. In the static case, the same argument is valid also for the general qualitative (rate-free) approximation theorems of Sithcompact1996; LucaUATPaperI; Korolev2022SMA. We now describe more in detail the different areas in which the present paper contributes.

To the best of our knowledge, our dynamic result is the only quantitative universal approximation theorem guaranteeing that a recurrent neural network model can approximate any suitably regular infinite-dimensional non-linear dynamical systems. Likewise, our static result is to the best of our knowledge the only general infinite-dimensional guarantee showing that a neural operator enjoys favourable approximation rates when the target map is smooth enough.

In the finite-dimensional context, CNOs become strict sub-structures of full RNNs, where the internal parameters are updated/generated via an auxiliary hypernetwork. Noticing this structural inclusion, our results rigorously support the folklore that RNNs may be more suitable when approximating causal maps, than feedforward neural network (FFNN, henceforth), see Section 5. This is because our theory yields expression rates for RNN approximations of causal maps between finite-dimensional spaces, which are more efficient than currently available comparable rates for FFNNs.

This research project answers theoretical deep learning questions by combining tools from approximation theory, functional analysis, and stochastic analysis. Therefore, we provide a concise exposition of each of the relevant tools from these areas in our “preliminaries” Section 2. Section 3 contains our quantitative universal approximation theorems. In the static case, we derive expression rates for the static component of our model, namely the neural filters, which depend on the regularity of the target operator being approximated; from Hölder trace-class to smooth trace-class and on the usual quantities Such as the compact set’s diameter.. Our main approximation theorem in the dynamic case additionally encodes the target causal map’s memory decay rate.

Section 4.2 applies our main results to derive approximation guarantees for the solution operators of a broad range of SDEs with stochastic coefficients, possibly having jumps (“stochastic discontinuities”) at times on a pre-specified time-grid and with initial random noise. Section 5, examines the implication of our approximation rates for RNNs, in the finite-dimensional setting, where we find that RNNs are strictly more efficient than FFNN when approximating causal maps. Section 6 concludes. Finally, Appendix A contains any background material required in the derivations of our main results whose derivations are relegated to Appendix B and Appendix D contains auxiliary background material on Fréchet spaces and generalized inverses.

1 Notation

For the sake of the reader, we collect and define here the notations we will use in the rest of the paper, or we indicate the exact point where the first appearance of a symbol occurs:

Given a topological vector space $(F,\tau)$, $F^{\prime}$ will denote its topological dual, namely the space of continuous linear forms on $F$.

Given two topological vector spaces $(E,\sigma)$ and $(F,\tau)$, $L(E,F)$ denotes the space of continuous linear operators from $E$ into $F$; if $E=F$, then we will write $L(E)=L(E,E)$.

Given a Fréchet space $F$, we use $\langle\cdot,\cdot\rangle$ to denote the canonical pairing of $F$ with its topological dual $F^{\prime}$,

We denote the open ball of radius $r>0$ about a point $x$ in a metric space $(X,d)$ by $\operatorname{Ball}_{(X,d)}(x,r)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\{u\in X:\,d(x,u)<r\}$,

We denote the closure of a set $A$ in a metric space $(X,d)$ by $\overline{A}$.

$\beta_{k}^{F}$ with $F$= Fréchet space: (7)

$P_{F:n},I_{F:n}$ where $F$ is a Fréchet space: (11) and (12); furthermore, $A_{F:n}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}I_{F:n}\circ P_{F:n}$

$C_{tr}^{k,\lambda}(K,B)$ and $C_{\alpha,tr}^{\lambda}(K,B)$: 4 and 5

$\mathcal{NF}^{\text{(P)ReLU}}_{[n]}$: The set of neural filters from $B$ to $E$,

$V$: the ‘‘special function’’, defined as the inverse of the map The map $u\mapsto u^{4}\,\log_{3}(u+2)$ is a continuous and strictly increasing surjection of $[0,\infty)$ onto itself; whence, $V$ is well-defined. $u\mapsto u^{4}\,\log_{3}(u+2)$ on $[0,\infty)$.

Preliminaries

In this section, we remind some preparatory material for the derivations of the main results of this paper. Finally, we remark that the notation in each of the subsequent subsections is self-contained and it is the one used on the cited paper: it will be up to the reader to contextualize it in the next sections.

A Fréchet space $F$ is a complete metrizable locally convex topological vector space.

Evidently, every Banach space $(F,\|\cdot\|_{F})$ is a Fréchet space; in this case, simply $\mathcal{P}=\{\|\cdot\|_{F}\}$. A canonical choice for the metric $d_{F}$ on a Fréchet space $F$ (that generates the pre-existing topology) is given by:

We now remind the concept of directional derivative of a function between two Frećhet spaces. This notion of differentiation is significantly weaker than the concept of the derivative of a function between two Banach spaces. Nevertheless, it is the weakest notion of differentiation for which many of the familiar theorems from calculus hold. In particular, the chain rule is true (cfr. H1982AMS). Let $F$ and $G$ be Fréchet spaces, $U$ an open subset of $F$, and $P:U\subseteq F\rightarrow G$ a continuous map.

The derivative of $P$ at the point $x\in U$ in the direction $h\in F$ is defined by:

In particular, $P$ is said to be differentiable at $x$ in the direction $h$ if the previous limit exists. $P$ is said to be $C^{1}$ on $U$ if the limit in Equation(3) exists for all $x\in U$ and all $h\in F$, and $DP:(U\subseteq F)\times F\rightarrow G$ is continuous (jointly as a function on a subset of the product).

As anticipated, the Definition 2 of a $C^{1}$ map disagrees with the usual definition for a Banach space in the sense that the derivative will be the same map, but the continuity requirement is weaker. The previous definition can be generalized and applied to higher-order derivatives. For instance, if $P:U\subseteq F\rightarrow G$, then:

Analogously, $P$ is said to be $C^{2}$ on $U$ if $DP$ is $C^{1}$, which happens if and only if $D^{2}P$ exists and is continuous. If $P:U\subset F\to G$ we require $D^{2}P$ to be continuous jointly as a function on the product space

Similarly, the $k$-th derivative $D^{k}P(x)\{h_{1},h_{2},\ldots,h_{k}\}$ will be regarded as a map

$P$ is of class $C^{k}$ on $U$ if $D^{k}P$ exists and is continuous (jointly as a function on the product space).

We will say that $P$ is $C^{k}$-Dir if $P$ satisfies the previous definition.

where the series converges in $F$ (in the ordinary sense). It is immediate to see from the definition that the maps

are continuous linear functionals. We remind that if a Fréchet space admits a Schauder basis, it is separable. However, the converse does not hold in general; whether every separable Banach space has a basis appeared in 1931 for the first time in the Polish edition of Banach’s book (B1932) and was solved in the negative by Enflo (E1973AM). Additional background on Fréchet spaces is included in Appendix D.1.

2 Feedforward Neural Networks with ReLU and PReLU activation functions

With the previous identification, the recursive representation function of a $[d]$-dimensional deep feed-forward network is given by

We will refer to $J$ as $\hat{f}_{\theta}$’s depth. We will denote by $\mathcal{N}\mathcal{N}^{\operatorname{(P)ReLU}}_{[d]}$ a deep ReLU FFNN with complexity $[d]$.

Main Results

We begin by treating the “static case” wherein we show that CNO’s neural filters, illustrated in Figure 3, are universal approximators of (non-linear) Hölder class operators between “good” linear spaces. We note that the application of the CNO only requires us to customize its neural filters to the relevant input and outputs’ geometries.

We first fix our working setting for this section

Analogous definitions hold for $P_{B:n}$ and $I_{B:n}$.

Before proceeding, we make the following trivial, yet useful remark

We now state and prove the following lemma.

Let $E$ and $B$ be two Fréchet spaces. Let $f:E\rightarrow B$ be a (non-linear) operator between these two spaces which is $C^{k}$-$Dir.$ (see Subsection 2.1, below Equation (5)). Then, $f$ is $C^{k}$ stable as in Definition 3.

The restriction of any $C^{k}$-stable (non-linear) operator $f:E\rightarrow B$ between two Fréchet spaces $E$ and $B$ to any non-empty compact subset $K\subseteq E$ extends to a $C^{k}$-stable (non-linear) operator defined on all $E$, namely the function $f$ itself. However, because our approximation theorems will hold for a pair $(f,K)$ of a (non-linear) operator $f:E\rightarrow B$ and compact set $K$, then $f$ does not need to be smooth on $K$ but only indistinguishable from a smooth operator on $K$. That is, our main results focus on non-linear operators belonging to the following trace class.

Let $E$ and $B$ be two Fréchet spaces and let $\lambda>0$ be a constant. Let $K\subseteq E$ be a non-empty compact set. We say that a (non-linear and possibly discontinuous) operator $f:E\rightarrow B$ belongs to the trace class $C^{k,\lambda}_{\operatorname{tr}}(K,B)$ if there exists a $\lambda$-Lipschitz By $\lambda$-Lipschitz we mean that the optimal Lipschitz constant is $\lambda$. Notice that the case $\lambda=0$ corresponds to the trivial case of a constant $f$ which is not treated in the present work. $C^{k}$-stable (non-linear) operator $F:E\rightarrow B$ satisfying

The following Example 1, pictorially represented in Figure 4, highlights our main interest in trace class maps. Precisely, these maps can be globally poorly behaved, even discontinuous, but indistinguishable from smooth functions “locally” (i.e. on a particular compact subset of the input space $E$).

At this point, some remarks are in order. In general, the problem of identifying when a map belongs to $C^{k,\lambda}_{\operatorname{tr}}(K,B)$ is a well-studied and independent area of research dating back to the beginning of the previous century (e.g., W1992). Nonetheless, by virtue of Lemma 1 a full characterization of the pairs of functions and sets $(f,K)$ that belongs to $C^{k,\lambda}_{\operatorname{tr}}(K,B)$ in the special case that $E$ and $B$ are Euclidean spaces has been derived only (relatively) recently in a series of articles starting with F2005AOM. The interested reader may consult BRUE2021JFA where the $C^{1,\lambda}_{\operatorname{tr}}(K,B)$ case is treated in the case that $B$ is Banach and $K$ is finite-dimensional (in a suitable metric-theoretic sense), for some $\lambda>0$ depending on $K$ and on $f$. The case where $K$ is a subset of a separable Hilbert space is explicitly solved in Azagra2018JFA.

Moreover, we provide results for the following trace class.

Let $E$ and $B$ be two Fréchet spaces, $\alpha\in(0,1]$ and $\lambda>0$ be two constants. Let $K\subseteq E$ be a non-empty compact set. We say that a (non-linear and possibly discontinuous) operator $f:E\rightarrow B$ belongs to the trace class $C_{\alpha,\text{tr}}^{\lambda}(K,B)$ if there exists an Hölder continuous (non-linear) operator $F:E\rightarrow B$ of order $\alpha$ and constant $\lambda$ satisfying

Functions with Hölder extensions are also actively studied. For example, (Lindenstrauss2000AMS, Theorem 1.12) guarantees any Lipschitz function defined on a closed subset of a separable Hilbert space with values in a separable Hilbert space can be extended with the same Lipschitz constant. However, in general, the existence of Hölder extensions between Fréchet spaces, as well as quantitative estimates on the extension’s Hölder constant, can be subtle Naor2001Mathematika.

When it is clear from the context, we suppress the index $n$ and write $\psi$ instead of $\psi_{n}$ (resp. $\varphi$ instead of $\varphi_{n}$). Second, we introduce our first building block, which is the following neural operator, which we call a neural filter since it filters out the part of the input not encoded in the first few Schauder basis vectors.

Let $E$ and $B$ be two Fréchet spaces. A non-linear operator $\hat{f}:E\rightarrow B$ is called a neural filter if it can be represented as

whereas: $I_{B:n^{out}}$ and $P_{E:n^{in}}$ are the functions defined in setting ( A 1 ) , $\psi_{n}$ and $\varphi_{n}$ are defined by (14) and (15), and $\hat{f}_{\theta}\in\mathcal{NN}_{[n]}^{\text{(P)ReLU}}$ See Subsection 2.2., with the multi-index $[n]\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}(d_{0},\ldots,d_{J})$ where $d_{0}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}n^{in},d_{J}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}n^{out}$ are positive integers. The set of all neural filters with representation (16) is denoted by $\mathcal{NF}^{\text{(P)ReLU}}_{[n]}$.

where $[n_{\varepsilon_{D}}]=(d_{0},\dots,d_{J})$ is a multi-index such with $d_{0}=n^{in}_{\varepsilon_{D}}$ and $d_{J}=n^{out}_{\varepsilon_{D}}$ defined as in Table 1. The approximation error $\varepsilon_{A}$ is due to the fact that we will use approximation results for neural networks in a finite-dimensional setting See Equation (60). The “model complexity” of $\hat{f}_{\theta}$ is reported in Table 1 and is a function of $f$’s regularity and the spaces $E$ and $B$.

The rates in Table 3.1 are optimal for finite-dimensional Banach space input spaces and one-dimensional output space. To see this, we only need to consider the case where $E$ is a finite-dimensional Euclidean space and $B$ is the real-line with Euclidean distance. In this setting, neural filter model is a deep feedforward neural network with ReLU activation function. In which case, a direct inspection of the approximation rates in Table 1 reveal that they coincide with the approximation rates for Hölder functions derived in ZHS2022JMPA which are optimal, as they achieve the Vapnik–Chervonenkis (VC) lower-bound on a real-valued model class’ approximation rate (see (ZHS2022JMPA, Theorem 2.4)) determined by its VC-dimension See Bartless2018JMLR for details on the VC-dimension and near sharp computation of the VC-dimension of deep ReLU networks..

We emphasize that in the following, $\langle\cdot,\cdot\rangle$ denotes the Euclidean inner product NB, this notation coincides with our earlier use of the notation $\langle\cdot,\cdot\rangle$ for the pairing of a TVS with its topological dual space by the Riesz representation theorem.. In particular, in the first column of Table 1, the functions $\bar{f}_{i}$ are defined by

The inability to extend higher-regularity (Lipschitz or smooth) functions while preserving their regularity, is precisely the obstruction lying at the heart of any quantitative approximation theorem between general infinite-dimensional Fréchet spaces. More precisely, a qualitative guarantee for continuous functions would require a version of McShane’s extension theorem Beer2020McShane for $B$-valued continuous maps but, to the best of our knowledge, such a result is only available when both $E$ and $B$ are separable Hilbert space (Lindenstrauss2000AMS, Theorem 1.12). However, such a result would not provide control on the target function’s regularity. Thus, without assuming that the target function belongs to a given trace-class, e.g. Hölder or smooth trace classes, as considered here, there is no a-priori way to clearly relate the complexity of a deep learning model, such as our neural filters, which depend on the regularity of the extension to regularity of the target function restricted to $K$.

Even in finite-dimensions highly-regular extensions, such as smooth extensions, see W1992 and F2005AOM, need not exist. Moreover, it is not even clear if a uniformly continuous function can be extended to a uniformly continuous function with a proportional modulus of continuity (see GutevJMAAExtension_2020 for details).

2 Dynamic Case: Sequential Universal Approximation Causal Operators

Theorem 3.1 was a static result certifying that suitable non-linear operators between infinite-dimensional linear metric spaces can be approximated by our “neural filter” operator network. By training several neural filters, independently on separate time-windows, and then re-assembling then via a “central” hypernetwork we can causally approximate “any” (generalized) dynamical system between such infinite-dimensional spaces.

The construction of a finitely-parameterized causal neural network approximator for these types of dynamical systems is our main result, and the main focus of this section. Our main result (Theorem 3.2) effectively certifies its ability to construct a CNO approximating any noiseless target function in this idealized approximation-theoretic framework. By a $\delta$-packing of a set, we mean the maximum number of points which can be placed in that set which are each at a distance of $\delta>0$ apart See Appendix A.2 for details..

In what follows, we will refer to each element $t_{n}$ in the non-degenerate time grid as “time". We give now the following

Our main class of causal maps of finite virtual memory is the main deep learning model of this paper, namely, the causal neural operator outlined in Figure 1.

satisfying the representation for all $x\in\mathcal{X}$

where See Definition 6. $\hat{f}_{t_{i}}\in\mathcal{NF}_{[n_{\varepsilon_{D}}]}^{(P)ReLU}$, $\hat{f}_{t_{i}}=I_{B_{t_{i}}:n_{\varepsilon_{D}^{out}}}\circ\varphi_{n_{\varepsilon_{D}^{out}}}\circ\hat{f}_{\theta_{t_{i}}}\circ\psi_{n_{\varepsilon_{D}^{out}}}\circ P_{E_{(t_{i-M},t_{i}]}:n_{\varepsilon_{D}}^{in}}$ where each $\hat{f}_{\theta_{t_{i}}}$ is a (P)ReLU FFNN in $\mathcal{NN}_{[n_{\varepsilon_{D}}]}^{(P)ReLU}$ with multi-index $[n_{\varepsilon_{D}}]=(d_{0},\dots,d_{J})$ with $d_{0}=n^{in}_{\varepsilon_{D}}$ and $d_{J}=n^{out}_{\varepsilon_{D}}$.

We will typically require our causal maps to possess a certain degree of regularity to deduce quantitative approximation rates. The most regular maps considered in this manuscript are causal maps of finite virtual memory which smooth trace-class maps can approximate at each instance in time.

We also derive approximation guarantees for the low-regularity analogue of smooth causal maps.

Let $f:\mathcal{X}\rightarrow\mathcal{Y}$ be a causal map, in the notation of Definition 8. If there are an $\alpha\in(0,1]$ and a $\lambda>0$ such that $f_{t_{i}}\in C_{\alpha,\text{tr}}^{\lambda}(\mathcal{X}_{(t_{i-M},t_{i}]},B_{t_{i}}),\,i\in[[I]]$, then we say that $f$ is $(r,\alpha,\lambda)$-Hölder.

We now present the main result of the paper. Our causal universal approximation theorem guarantees that the CNO model can approximate any causal map while “preserving its forward flow of information through time”. The quantitative approximation rates, describing the complexity of the CNO model implementing the approximation are recorded in Table 2 below.

where See Definition 6. $\hat{f}_{t_{i}}\in\mathcal{NF}_{[n_{\varepsilon_{D}}]}^{(P)ReLU}$, $\hat{f}_{t_{i}}=I_{B_{t_{i}}:n_{\varepsilon_{D}^{out}}}\circ\varphi_{n_{\varepsilon_{D}^{out}}}\circ\hat{f}_{\theta_{t_{i}}}\circ\psi_{n_{\varepsilon_{D}^{out}}}\circ P_{E_{(t_{i-M},t_{i}]}:n_{\varepsilon_{D}}^{in}}$ where each $\hat{f}_{\theta_{t_{i}}}$ is a (P)ReLU FFNN in $\mathcal{NN}_{[n_{\varepsilon_{D}}]}^{(P)ReLU}$ with multi-index $[n_{\varepsilon_{D}}]=(d_{0},\dots,d_{J})$ with $d_{0}=n^{in}_{\varepsilon_{D}}$ and $d_{J}=n^{out}_{\varepsilon_{D}}$ defined as in Table 1. The model complexity of the hypernetwork $\hat{h}$ is recorded in Table 2.

For brevity, we do not repeat the complexities of the neural filters approximating the target function on any time window and recall that the neural filters’ approximation rates have previously been recorded in Table 1.

The complexity bounds of the CNO model, guaranteed by Theorems 1 and 3.2 concern the approximation of relatively general functions between rather general Fréchet spaces for arbitrary compact path spaces. In particular, in this setting, the target function may be incompatible with the compact path space. However, in the case of Hilbert spaces, one can identify classes of compact sets over which a given smooth function can be efficiently approximated. As one may expect, all rates becomes much simpler if the all involved quantities are more structured. The following set of assumptions illustrates a broad class of compact sets where the CNO does not experience the curse of dimensionality, and a favorable choice of a Schauder basis becomes evident. Furthermore, the bounds in Tables 1 and 2 become notably simpler. We now motivate our definition of these well-behaved compact sets. We start by considering the following finite-dimensional example.

Our well-behaved compact sets are an infinite-dimensional extension of our finite-dimensional thought experiment, in Example 2, where we require that the coefficients of the higher-order basis vectors decay exponentially rapidly. Before formally defining them, let us continue the previous example

By the Grothendieck’s compactness principle, see (DJSeq2012, Exercises 1.6), the set $\mathcal{X}$ is relatively compact in $L^{2}()$.

We abstract Example 3 into the following generally applicable condition. An additional example of the exponential decay condition in (19), which we now generalize, will be provided in the context of mathematical finance, and will later be given in Section 4.1.2 below. We additionally ask that our causal maps being approximated have a Markov-like property, in the sense that they only depend on the current state of the input sequence and not on the past.

Furthermore, the following complexity estimates hold for each neural filter $\hat{f}_{t_{i}}$:

Decoding Dimension: n_{\varepsilon_{D}}^{\text{out}}{\color[rgb]{0,0,0}=1},

Corollary 1 is stated for $(r,\infty,\lambda)$-smooth causal maps, but a similar result can also be obtained for causal maps that exhibit Lipschitz regularity by appropriately adjusting the proof. The primary difference is that the constant $C$ in inequality (20) would decrease at a much faster rate along with the “total approximation error” $\varepsilon>0$. This adjustment is necessary for the CNO to maintain dimension-free algebraic approximation rates in the associated compact path space. A similar technique was recently applied in the static low-regularity setting between Sobolev spaces, as noted in (KraLassasTakashi, Theorem 1). Thus, while the shape of the compact path spaces $\mathcal{X}$ regarding their exponential decay coefficient remains unchanged, the dependence on the diameter—indicated by the constant $C$—is what varies.

2.2 Discussion: How the CNO could be implemented

This paper mainly examines the approximation capabilities of infinite-dimensional RNN architectures, specifically our CNO. We discuss what these structures can approximate when provided with sufficient noiseless data and ideal training algorithms. However, a natural question arises regarding their practical implementation. To address this, we present an idealized training procedure in Algorithm 1, which serves as a guide for implementing the CNO.

In particular, we find it beneficial to share insights from a recent implementation of a mild variant of the CNO described in RezaTime. In that research, the objective was to learn causal maps on finite-dimensional manifolds of non-positive curvature instead of infinite-dimensional linear spaces. Instead of utilizing neural filters, a non-Euclidean readout layer, as introduced in PaponAnnie, was employed. This layer is compatible with the geometry of the space in which the dynamical system operates. Nevertheless, the core hypernetwork structure was preserved, which dynamically updates the model parameters over time. The training procedure for this structure was nearly identical to Algorithm 1, with only the necessary modifications. That work emphasizes a strong experimental focus, aiming to demonstrate the practicality of a training procedure such as Algorithm 1. The most accurate, stable, and rapid training method involved first training the model $\hat{f}_{\theta{t_{1}}}$ using empirical risk minimization, as outlined in Algorithm 1, until achieving nearly zero training loss. By ensuring the network was sufficiently large, we successfully avoided overfitting due to the double-descent phenomenon, as documented in studies on overparameterized neural networks MeiMontanariDD; ChengTSKernelChar. Our findings confirmed this holds true in our context as well, suggesting that similar results can be expected in the future when exploring the statistical properties of the CNO in an infinite-dimensional framework. After training the initial layer to achieve satisfactory predictions at time one, we discovered that the most stable and efficient training approach was to utilize transfer learning. This involved initializing the training of each model, denoted as $\hat{f}_{\theta_{t_{i+1}}}$, using the parameters obtained from the previous training round, specifically $\theta_{t_{i}}$. We then conducted only a few epochs of stochastic gradient descent. In general, the parameters $\theta_{t_{1}},\ldots,\theta_{t_{I_{\delta,Q}}}$ showed minimal variation from one another. Moreover, using $\theta_{t_{i}}$ as a starting point for training $\hat{f}_{\theta_{t_{i+1}}}$ facilitated the training of the hypernetwork. This is because $\hat{f}_{\theta_{t_{i+1}}}$ has multiple parametric representations that yield the same functional representation. Thus, initializing at $\theta_{t_{i}}$ ensured that we were not only learning the correct function within the function space but also remaining within the same region of the joint parameter space of the neural filters. This approach had the added benefit of requiring fewer training iterations, as the difference $|\theta_{t_{i}}-\theta_{t_{i+1}}|$ remained small. However, it is important to note, as discussed in PetWojLim, that the mapping from a deep learning model’s parameter space to its function space is typically only locally Lipschitz, with an extremely large local Lipschitz constant. Therefore, even if $|\theta_{t_{i}}-\theta_{t_{i+1}}|\approx 0$, the corresponding functions $\hat{f}_{\theta_{t_{i}}}$ and $\hat{f}_{\theta_{t_{i+1}}}$ may still be significantly different in the function space. Lastly, once we obtained each $\theta_{1},\dots,\theta_{I_{\delta,Q}}$, we learned the relevant recurrence relation by training the hypernetwork to minimize the mean squared error between the sequential parameters:

In RezaTime, we did not empirically need the theoretically necessary augmentation from $\theta_{t_{i}}$ to $z_{t_{i}}=(\theta_{t_{i}},u_{i})$ using some $\delta$-packing $\{u_{i}\}_{i=1}^{I_{\delta,Q}}$ of the Euclidean unit ball. The loss (21) was numerically optimized using stochastic gradient descent, and in our companion paper RezaTime, we found that this provided satisfactory performance. The advantage of using a hypernetwork is particularly evident at this stage, as it enabled us to train a recurrent neural operator – the CNO – without relying on backpropagation through time, a method known for its numerical instability. Instead, minimizing the loss function in equation (21) follows the standard approach of empirical risk minimization, which does not involve real-time components and does not present the same numerical issues. Additionally, a second benefit of the hypernetwork becomes apparent: once trained, the CNO can generate predictions for future time points that extend beyond the training data it was optimized with.

We now use our results to approximate solution operators arising in stochastic analysis and pricing functional arising naturally in mathematical finance.

Applications to Mathematical Finance and Stochastic Analysis

We now provide some examples of how our static approximation theorems are naturally amenable to pricing problems in mathematical finance. Our aim is both to showcase the need for the general Fréchet setting, as well as the naturality of Assumption 2.

When considering forwards and options on these, the set $D$ is typically a contractually specified week, month, quarter or year. Due to the continuity result above, we are in the context of our neural networks on a Fréchet space. Next, we show that under additional, realistic structural conditions the functional $F$, introduced above, can be efficiently approximated. We do this by verifying the conditions of Corollary 1.

1.2 Efficient Approximation Rates for Pricing with Smooth Rapidly Decaying Functions

2 Dynamic Examples: Solution Operators of Stochastic Differential Equations

We apply our results to show that several solution operators from stochastic analysis can be approximated by the CNO. Our neural network model can approximate stochastic processes without assuming strong structural conditions describing their evolution.We illustrate our result’s implications for obtaining numerical solutions to SDEs, and we discuss the implications for more general stochastic processes, e.g. processes with jumps, towards the end of this section.

3 A primer on Wiener Chaos

and, for instance, $h_{0}(x)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}1$, $h_{1}(x)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}x$, and so on.

We leverage the symmetrized tensor product of elements $f_{1},\dots,f_{q}\in L^{2}([0,t])$ defined by

where $S^{q}$ is the set of permutations of the indices $\{1,\dots,q\}$. More concretely, the Hilbert space generated by the symmetrized tensor product See (Bourbaki1998Algebre1a3, Chapter IV page 43). is identified See (PecattiTaqqu_2011_ChaosDiagrams, Lemma 8.4.2). with the set of symmetric functions A “function” $f\in L^{2}([0,t]^{q})$ is symmetric if $f(s_{1},\dots,s_{q})=f(s_{\pi(1)},\dots,s_{\pi(q)})$, for all $\pi\in S^{q}$, outside a set of $q$-dimensional Lebesgue measure $0$. in $L^{2}([0,t]^{q})$ which we denote by $L^{2}_{\operatorname{sym}}([0,t]^{q})$. Since the $q$-fold symmetrized tensor product is a subspace of the (usual) $q$-fold tensor product then the identification of the $q$-fold symmetric tensor product of $L^{2}([0,t])$ with $L^{2}_{\operatorname{sym}}([0,t]^{q})$ may be further simplified to

The connection between the symmetrized tensor product and the $q^{th}$ Wiener Chaos is that the $q^{th}$ Wiener Chaos $\mathcal{H}_{t}^{q}$ is structurally identical to $L^{2}_{\operatorname{sym}}([0,t]^{q})$ (identified with the $q$-fold symmetrized tensor product of $L^{2}_{\operatorname{sym}}([0,t])$ with itself). The map realizing this identification sends any $f\in L^{2}_{\operatorname{sym}}([0,t]^{q})$ to its $q$-fold multiple stochastic integral

Moreover, the map (27) is linear isometric isomorphism preserving inner products See (PecattiTaqqu_2011_ChaosDiagrams, Proposition 8.4.6 (1)).. Consequentially, any orthogonal basis of $L^{2}_{\operatorname{sym}}([0,t]^{q})$ is sent to an orthogonal basis of $\mathcal{H}_{t}^{q}$ under this identification. Since an orthogonal basis of $L^{2}_{\operatorname{sym}}([0,t]^{q})$ is given by the set

where $\{f_{i}\}_{i=1}^{\infty}$ is an orthogonal basis See (PecattiTaqqu_2011_ChaosDiagrams, page 153, point (iii)). of $L^{2}([0,t])$ then the identification (27) implies that the corresponding set of random variables

is an orthogonal basis of the $q^{th}$ Wiener Chaos $\mathcal{H}_{t}^{q}$. Such an orthogonal basis of $L^{2}([0,t])$ is given by the Fourier basis whose elements are

4 Simultaneous Approximation of SDEs with Different Initial Conditions using CNOs

Monte Carlo methods allow for the efficient solution to stochastic differential equations (SDEs) with a convergence rate of $\mathcal{O}(1/\sqrt{S})$ Typically in the $L^{2}$-sense. to the true solution, where $S$ is the number of samples, plus a comparable discretization error when resorting to a tamed Euler scheme JenztenTamedEulre2011. It is known that deep learning can provide a suitable alternative to Monte Carlo schemes by learning the SDE’s solution map given deterministic initial conditions, for a fixed terminal time, by approximating the solutions to their associated PDEs Beck2018 given by the Feynman-Kac Theorem.

In this section, we show how a single CNO can be used for simultaneously solving SDEs with various noisy initial conditions across different time-horizons, by simultaneously approximately learning solve a family of stochastic differential equations with many different stochastic initial conditions and different initial times.

This section’s application shows that the CNO can approximate causal maps with stochastic inputs on arbitrarily long time horizons. This extends the known guarantees for recurrent neural networks, specifically reservoir computers, which can approximate time-invariant causal maps Lukas_StochInputReservoir.

where each $\operatorname{SDE-Solve}_{t_{i}:t_{i+1}}(\eta_{t_{i}})$ is defined as in Equation (32). The typical example which we have in mind, in the following, are input sequences which are orbits of square-integrable random variables under the an SDE’s solution operator; i.e.

Consider the setting of this section and fix the path space

is $(0,1,\sqrt{3}e^{\frac{3}{2}M^{2}(\Delta^{+}+1)\Delta^{+}})$-Hölder.

with $i$ belongs to $[[I]]\cup\{0\}$ provided by the definition of causal maps See Definition 8., satisfies to the following uniform estimates:

where We recall, Definition 6, stating that $\hat{f}_{t_{i}}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}I_{B_{t_{i}}:n_{\varepsilon_{D}^{out}}}\circ\varphi_{n_{\varepsilon_{D}^{out}}}\circ\hat{f}_{\theta_{t_{i}}}\circ\psi_{n_{\varepsilon_{D}^{out}}}\circ P_{E_{t_{i}}:n_{\varepsilon_{D}}^{in}}$. $\hat{f}_{t_{i}}\in\mathcal{NF}_{[n_{\varepsilon_{D}}]}^{(P)ReLU}$. Moreover, for the hyperparameter $n_{\varepsilon_{D}}^{in}$ it holds

5 Discussion - Corollary 2: Jumps, Path-Dependence, and Accelerated Approximation Rates Under Smoothness

We briefly discuss some points surrounding Corollary 2. For instance, how the result allows for stochastic discontinuity-type jumps. We also discuss how the scope of Theorem 3.1 allows for Corollary 2 to be easily generalized; but we opt not to do that in this manuscript, rather opting for a less technical illustration of our general framework.

If, in addition to conditions (30) and (29), the drift and diffusion coefficients $\alpha$ and $\beta$ are sufficiently differentiable The precise conditions are formalized in (RosestolatoBook2017, Assumption 3.7)., then (RosestolatoBook2017, Theorem 3.9) implies that each of the maps $\operatorname{SDE-Solve}_{t_{i}:t_{i+1}}$ are $C^{k}$. Whence, the operator $\operatorname{SDE-Solve}$ is a smooth causal map of finite virtual memory. Thus, in this case, Theorem 3.2 implies improved approximation rates by the CNO model.

In financial applications, the possibility of a stochastic process’ to jump at predetermined times (called stochastic discontinuities in that context) are an essential ingredient of accurately modeling interest rates; for example, European reference interest rates typically exhibit jumps directly after monetary policy meetings of ECB Font2020FinStoch.

One could consider SDEs driven with path dependant random drift and diffusion coefficients, since all that is needed to apply Theorem 3.2 is the regularity of the $\operatorname{SDE-Solve}$ operator; which is guaranteed by results such as DePrato2016AnnProb or RosestolatoBook2017. However, we instead opted for a simple first presentation, explicitly illustrating the scope of our results in this easier case.

The Benefit of Causal Approximation: Super-Optimal Approximation Rates for Causal Maps

We now illustrate the quantitative advantage of causal approximation, i.e. using our CNO architecture, when the target function is causal. For illustrative purposes, we consider the simplest case where all involved spaces are finite-dimensional and Euclidean. By considering this setting, we can juxtapose our approximation rates derived from Theorem 3.2 against the best upper-bounds on the approximation rates for ReLU networks ZHS2022JMPA which apply to our class of causal maps, which match the well-known lower bounds for Lipschitz maps without the additional causality constraint DVL1993; Grohs2021IEEETransInfoTheory; however, there are currently no available lower bounds on this causal class.

In KN1998NN, the authors investigate the problem of approximating a dynamical system on a Euclidean space by a RNN. In their most general form, RNNs – sometimes also called “fully RNN", or fRNNs - are given for times $t>0$ by

Moreover, by pre-composing each $\hat{f}_{\theta_{t}}$ in Equation (38) with the following linear projection

and by noting that $\hat{f}_{\theta_{t}}\circ A$ is a FFNN because of the invariance with respect the pre-composition by affine functions, we have that the CNO becomes

NB, this is of-course satisfied by any autonomous dynamical system; namely when $g_{t}=g_{0}$ for all integers $t$, with $g_{0}$ smooth.

defines a $(0,\infty,\lambda)$-smooth causal map.

Conclusion

We presented a first universal approximation theorem which is both causal, quantitative, compatible with infinite-dimensional operator learning, and which is not restricted to “function spaces” but is compatible with general “good” infinite-dimensional linear metric spaces. Our main contributions, Theorem 3.1 and Theorem 3.2, provided approximation guarantees for any smooth or Hölder (non-linear) operator between Fréchet spaces in the “static” or “causal” case, where temporal structure is or is not present in the approximation problem, respectively.

We showed how the CNO model can approximate a variety of solution operators, and infinite dimensional dynamical systems, arising in stochastic analysis. Moreover, in the Euclidean case, we showed that our neural filter’s approximation rates are optimal. We then showed that, when the target operator being approximated is a dynamical system, then the CNO’s approximation rates are super-optimal. Optimality is quantified in terms of the number of parameters required to approximate any arbitrary map belonging to some broad class as in constructive approximation theory of DVL1993.

We believe the observations made in this work open up avenues for future literature. As a prime example, we would like to further optimize our CNO for the stochastic filtering problem assuming additional structural conditions. As future work, we aim to build on these results in the context of robust finance.

Acknowledgments

The authors would like to thank Alessio Spagnoletti for his helpful feedback. This research was funded by the NSERC Discovery grant (RGPIN-2023-04482) and was partially supported by the Research Council of Norway via the Toppforsk project Waves and Nonlinear Phenomena (250070).

Appendix A Background material for proofs

In an effort to keep the paper as self-contained as possible, this appendix contains any background material required in the derivations of our main results but not required for their formulation. We cover various properties of deep ReLU neural networks, covering and packing results, and we overview some properties of finite-dimensional “linear dimension reduction” techniques in well-behaved Fréchet spaces. We also include a list of some useful properties of generalized inverses.

This section contains auxiliary results on neural network approximation, parallelization, and memorization.

Theorem 1.1 in JZHS2021SIAM proves that ReLU FFNNs with width $\mathcal{O}(N\,\log(N))$ and depth $\mathcal{O}(L\,\log(L)+d)$ can approximate a function $f\in C^{s}(^{d})$ with a nearly optimal approximation error $\mathcal{O}(\|f\|_{C^{s}(^{d})}\,N^{-2s/d}\,L^{-2s/d})$, where the norm $\|\cdot\|_{C^{s}(^{d})}$ is defined as:

More precisely, they state and prove the following

where $C_{1}=17s^{d+1}3^{d}d$, $C_{2}=18s^{2}$ and $C_{3}=85(s+1)^{d}8^{s}$.

In particular, note that the previous result does not privilege the width to the depth and vice versa because the exponent for both $N$ and $L$ on the right-hand side of Equation (43) is $-2s/d$. On the other hand, ZHS2022JMPA, as a consequence of their main theorem for explicit error characterization, state and prove the following.

where $C_{1}=16$ and $C_{2}=18$ if $p\in[1,\infty)$; $C_{1}=3^{d+3}$ and $C_{2}=18+2d$ if $p=\infty$.

A.1.2 Efficient parallelization of ReLU neural networks

CJR2020IEEE propose an efficient parallelization of neural networks with different depths for a special class of activation functions, namely the ones that have the so-called $c$-identity requirements. Before giving a formal definition of such activation functions, we remind some quantities introduced in CJR2020IEEE. More precisely, $\mathcal{N}$ denotes the set of neural network skeletons, i.e.,

We give now the following definition (cfr. CJR2020IEEE, Definition 4):

For our scopes, we note that the ReLU activation fulfills the 2-identity requirement with $I=[([1\,-1]^{T},[0\,\,\,0]^{T}),([1\,-1],0)]$. In addition, the following proposition hold (cfr. CJR2020IEEE, Proposition 5):

A.1.3 Memory Capacity of Deep ReLU regressor

We here report a very recent lemma (KDDWP2022, Lemma 20). appearing in the deep metric embedding paper of KDDWP2022; see Lemma 20 in the just cited reference.

For the sake of completeness, we remind that the aspect-ratio of the finite metric space $(\mathcal{X}_{N},\|\cdot\|_{2})$ is defined as the ratio of the maximum distance between any two points therein over the minimum separation between any two distinct points, i.e.:

We notice that KLMN2005GFA introduce the notion of an aspect ratio of a measure space as the ratio of total mass over the minimum mass at any point. The relevance of the aspect ratio to our analysis is that it quantifies the difficulty to memorize a dataset. This is because finite subset of a Euclidean space with large aspect ratio are logarithmically (in the aspect ratio) more difficult to memorize than subsets with a small aspect ratio.

for every $i=1,\ldots,N$. Furthermore, the following quantitative “model complexity estimates" hold

Width : $\mathcal{NN}$ has width $n(N-1)+\max\{d,12\}$,

Depth : $\mathcal{NN}$ has depth of the order of

where $\mathcal{X}_{N}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\{x_{1},\ldots,x_{N}\}$.

Number of non-zero parameters : The number of non-zero parameters in $\mathcal{NN}$ is at most

The “dimensional constant" $C_{d}$ is defined by

A.2 Covering and packing numbers

In what follows, $\Theta$ will always have at least two points. We recall the basic definitions of these objects here, and we refer the reader to, e.g. (VanderVaart2023v2, Section 2.2.2), for more details and relations between them.

Let $(V,\|\cdot\|)$ be a normed space, and $\Theta\subset V$. A subset $F\subset V$ is an $\varepsilon$-covering (or $\varepsilon$-net) of $\Theta$ if for any $\theta\in\Theta$ there exists $f\in F$ such that $\|\theta-f\|\leq\varepsilon$.

Let $(V,\|\cdot\|)$ be a normed space, and $\Theta\subset V$ a subset. $F\subset\Theta$ is an $\varepsilon$-packing of $\Theta$ if $\inf_{x,y\in F,\,x\neq y}\|x-y\|>\varepsilon$ (notice the inequality is strict).

Both of these definitions define the notion of packing and covering.

$N(\Theta,\|\cdot\|,\varepsilon)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\inf\{\#(F)\,:\,F\text{ is an-}\varepsilon\text{ covering for }\,\Theta\}.$

$M(\Theta,\|\cdot\|,\varepsilon)\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\sup\{\#(F)\,:\,F\text{ is an-}\varepsilon\text{ packing for }\,\Theta\}.$

We note that, $N(\Theta,\|\cdot\|,\varepsilon)\leq M(\Theta,\|\cdot\|,\varepsilon)\leq N(\Theta,\|\cdot\|,\varepsilon/2)$; see e.g. (VanderVaart2023v2, page 147).

A.3 Bounded Approximation Property in Fréchet spaces with Schauder basises

We now remind the following important definition (cfr. BONET2020WP Definition 1.6) and proposition (cfr. BONET2020WP Proposition 1.16 (2)).

A locally convex space $E$ has the bounded approximation property (BAP, henceforth) if there exists an equi-continuous net $(A_{j})_{j\in I}\subset L(E)$, with $\text{dim}(A_{j}(E))<\infty$ for every $j\in E$ and $\lim_{j\in I}A_{j}(x)=x$ for every $x\in E$. In other words, the net $(A_{j})_{j\in I}$ converges to the identity for the topology of point-wise or simple convergence. In all the previous expressions, $I$ denotes a generic directed indexing set.

If $F$ is a barreled locally convex space with a Schauder basis, then $F$ has the BAP.

A finite-dimensional vector space $F$ can have just one vector space topology up to homeomorphism.

We observe the following characterization for an equi-continuous family $H\subset L(E,F)$, with $E,F$ Fréchet spaces.

$H\subset L(E,F)$ is an equi-continuous family if and only if

for any $V\subset F$ open neighborhood of the origin, $\cap_{T\in H}T^{-1}(V)$ is an open neighborhood of the origin (BONET2020WP page 1), if and only if

for any $V\subset F$ open neighborhood of the origin, there exists $U\subset E$ open neighborhood of the origin such that $\cup_{T\in H}T(U)\subset V$.

In this last case, we call the family $H$ uniformly equi-continuous (see K1983, page 169).

Appendix B Proofs

By assumption, $f:E\rightarrow B$ is $C^{k}$-Dir. This means that

Before proceeding, we state and prove the following Lemma.

Let $(X,d)$ and $(Y,\varrho)$ be two metric spaces and let $\mathcal{F}\subset C(X,Y)$ be a family of maps from $X$ to $Y$ such that $\forall\varepsilon>0$ $\exists\delta>0\,:\,d(x,x^{\prime})\leq\delta$, then $\varrho(f(x),f(x^{\prime}))\leq\varepsilon$, $f\in\mathcal{F}$. Then, the family $\mathcal{F}$ has a common modulus of continuity.

Le $\omega\,:\,[0,\infty)\rightarrow[0,\infty]$ be defined as:

In particular it holds for $x=z$ and $x^{\prime}=z^{\prime}$, i.e. $\varrho(f(z),f(z^{\prime}))\leq\omega(d(z,z^{\prime})),\,f\in\mathcal{F}$. Notice that if $z=z^{\prime}$, than the statement is trivial.

We observe that, in view of Remark 5 and the fact that the metric of a Fréchet space is translation-invariant, an equi-continuous family $H\subset L(E,F)$, with $E,F$ Fréchet spaces, satisfies the assumption of Lemma 4.

B.2 Proof of Theorem 3.1

The proof of Theorem 3.1 proceeds in three main steps. First, the target nonlinear operator is replaced by a finite-dimensional surrogate that preserves its regularity properties—precisely, uniform continuity and a prescribed degree of smoothness. This finite-dimensional surrogate is then approximated using a (P)ReLU MLP. Finally, an infinite-dimensional approximator—our neural filter—is constructed by projecting any infinite-dimensional input onto a finite-dimensional subspace, passing the result through the (P)ReLU MLP, and interpreting the MLP’s outputs as coefficients in a Schauder basis, which are then reassembled into an infinite-dimensional prediction. Tracking and controlling the approximation errors introduced at each step completes the proof.

In order to outline the ideas behind Theorem 3.1, we draw the diagram chase in Figure 7. Moreover, in order not to burden the notations, we will use the following abbreviations for any “encoding error" $\varepsilon_{D}$: $n^{in}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}n^{in}_{\varepsilon_{D}}$ and $n^{out}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}n^{out}_{\varepsilon_{D}}$. In what follows, we detail the proof for the case that See Definition 4. $f\in C^{k,\lambda}_{\operatorname{tr}}(K,B)$. The case where $f$ belongs to $C_{\alpha,\text{tr}}^{\lambda}(K,B)$ will be treated at the end of the Proof for the sake of clarity, and we will highlight the main differences with respect to the $C^{k,\lambda}_{\operatorname{tr}}(K,B)$ case.

Moreover, analogously as above, we derive the following inequality, because $F(K)$ is compact: $\max_{x\in F(K)}d_{B}(A_{B:n}(x),x)<\frac{\varepsilon_{D}}{2}$. Thus, the following positive integers

are finite. At this point, we remind that $\psi$ and $\varphi$ are the following two set-theoretic identity maps

where the equality in Equation (52) follows from the fact that on the compact $K$ the maps $f$ and $F$ coincides, the inequality in Equation (53) follows from the triangular inequality by using the diagram chase in Figure 7, and the equality in Equation (54) from the definition of $\bar{F}$. We now bound each of the above terms (54), (55) and (56). We start from the last one: it is controlled, by using the definition of $n^{out}$ as:

We now bound the second term, i.e., the term $\max_{x\in K}\,d_{B}\big(I_{B}\circ P_{B}\circ F\circ I_{E}\circ P_{E}(x),I_{B}\circ P_{B}\circ F(x)\big)$. Recall that $F$ is $\lambda$-Lipschitz. By using the definition of $n^{in}$ in (49), we have for $x\in K$:

and hence $\max_{x\in K}d_{B}\big(I_{B}\circ P_{B}\circ F\circ I_{E}\circ P_{E}(x),I_{B}\circ P_{B}\circ F(x)\big)\leq\varepsilon_{D}/2$.

We now control the term (54). In order to do so, we make the following observations: ( 1 ) $(R^{n^{in}},d_{E:n^{in}})$ is a topological vector space in which the topology coincides with the standard one; see Lemma 7; ( 2 ) therefore, the identity map and its inverse are continuous. ( 3 ) Being linear, it is also uniform continuous; see S1971, Page 74. These observations allow us to define $\omega_{\varphi}\,:\,[0,+\infty)\rightarrow[0,+\infty)$ the modulus of continuity of the map $\varphi$ which we may assume to be, without loss of generality See the argument done above for $\omega_{A,E}$., continuous and strictly monotone; $\omega_{\varphi}^{\dagger}$ will denote, as usual, its generalized inverse. This allows us to compute:

where the second line of (59) holds since $I_{B}$ is an isometric embedding, and thus in particular $\operatorname{Lip}(I_{B})=1$.

where $\varepsilon_{A}$ is the “approximation error" as in the statement of the theorem; we will prove later on the existence of such $\hat{f}_{\theta}$. Meanwhile, we note that the bound in Equation (59) becomes:

Putting together the previous equation with the estimates in Equations (57) and (58), we have that:

Now, let $\text{diam}_{E}(\cdot)$, $\text{diam}_{2}(\cdot)$ and $\text{diam}_{E:n^{in}}(\cdot)$ denote the diameter computed with respect to the metric $d_{E}$, the Euclidean distance and the distance $d_{E:n^{in}}$ respectively. It holds that:

We now identify a hypercube “nestling" $\psi\circ P_{E:n^{in}}(K)$, and we explicit the dependence on $n^{in}$. To this end, let

which is well-defined and invertible, and maps $\psi\circ P_{E:n^{in}}(K)$ to $^{n^{in}}$. In particular, the map

In the notation of Theorem A.1, if we set, $C_{3}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\,\max_{i=1,\ldots,n^{out}}\,\|\bar{f}_{i}\|_{C^{k}(^{n^{in}})}\,N^{-2k/n^{in}}\,L^{-2k/n^{in}}=\delta/(n^{out})^{1/2}$ and we also set $N=L$ then, the same result implies that the width and the depth of each $\hat{f}_{\theta}^{(i)}$ is provided in the same reference and, upon recalling the definition of $\delta$ in (60) we find that it is given by:

where $C_{1}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}17k^{n^{in}+1}3^{n^{in}}n^{in}$, $C_{2}=18k^{2}$, $C_{3}=85(k+1)^{n^{in}}8^{k}$ and $C_{\bar{f}}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\max_{i=1,\ldots,n^{out}}\|\bar{f}_{i}\|_{C^{k}(^{n^{in}})}$.

where $\text{\emph{Width}}(\hat{f}^{(1)}_{\theta})$ denotes the width of $\hat{f}_{\theta}^{(1)}$, and where we have used the fact that $\text{\emph{Width}}(\hat{f}^{(1)}_{\theta})=\text{\emph{Width}}(\hat{f}^{(i)}_{\theta})$ for every $i=1,\ldots,n^{in}$.

where $\text{\emph{Depth}}(\hat{f}^{(1)}_{\theta})$ denotes the width of $\hat{f}_{\theta}^{(1)}$, and where we have used the fact that $\text{\emph{Depth}}(\hat{f}^{(1)}_{\theta})=\text{\emph{Depth}}(\hat{f}^{(i)}_{\theta})$ for every $i=1,\ldots,n^{out}$.

which is nothing but (60). The Theorem is whence proved for $f\in C_{\operatorname{tr}}^{k,\lambda}(K,B)$.

We report to the reader the main changes of the proof.

The quantity $n^{in}$ in Equation (49) is instead given by:

In this way, the estimate in Equation (58) continues to hold with $F\in C_{\alpha,\text{tr}}^{\lambda}(K,B)$.

The inequality in Equation (60) is now guaranteed by Theorem A.2, instead of by Theorem A.1. Note, that the pre/post-composition of an $\alpha$-Hölder function with a Lipschitz function is again an $\alpha$-Hölder function.

The function $\bar{F}\circ W^{-1}$ in Equation (62) is $C_{\alpha,\text{tr}}^{\lambda}(K,B)$, and so, we may apply Theorem A.2 to deduce that there are $n^{in}$ ReLU FFNN satisfying to the estimates in Equation (64).

The width and the depth of each $\hat{f}_{\theta}^{i}$ are thus provided by Theorem A.2. Setting $N=L$ in that result yields

The considerations on the existence of an “efficient parallelization" continue to hold with the width and depth appropriately defined by using ( v ).

B.3 Proof of Corollary 1

where the outer infimum is taken over all $i$-dimensional linear subspaces $X_{i}$ of $X$. Both of these notions of “width”, i.e. linear complexity, of a subset coincide when the space $X$ is a Hilbert space, see e.g. (Pin1985Book, Proposition II.5.2); however, we introduce both notions since some results are formulate for general Banach spaces using one width rather than the other, in most parts of the literature.

for all $x\in r\cdot Z_{n}$; where $S_{1/r}:E_{t_{n}}\ni x\mapsto\frac{1}{r}\cdot x$. Note that

Consider the Kolmogorov $i$-width $\delta_{i}(Z_{n,i},E_{t_{n}})$ is optimized by the linear subspace spanned by $\{e_{n,j}\}_{j=0}^{i-1}$ and satisfies

Moreover, since $\{e_{n,j}\}_{j=0}^{i-1}$ is an orthonormal set then the orthogonal projection operator $A_{E_{t_{n}},i}:E_{t_{n}}\mapsto\operatorname{span}\{e_{n,j}\}_{j=0}^{i-1}$, given by $x\mapsto\sum_{j=0}^{i-1}\,\langle x,e_{n,j}\rangle_{E_{t_{n}}}\,e_{n,j}$ is optimal; whence,

By (79), setting $r=\varepsilon_{D}$ implies that (78) holds while

Since the target space is one dimensional then $n_{\varepsilon_{D}}^{out}=1$ for all $\varepsilon_{D}>0$. Thus, when approximating $f$ on $r\cdot K$, for $r=\varepsilon_{D}$, both $n_{\varepsilon_{D}}^{\text{in}}$ and $n_{\varepsilon_{D}}^{\text{out}}$ are constants.

Finally, Table 1 implies that the neural filters defining the CNO have width at most

Consequently, the number of non-zero (trainable) parameters is almost the width squared times the depth; whence $\mathcal{O}(\varepsilon^{-3/2}\log(1/\varepsilon)^{3})$.

B.4 The Dynamic Weaving Lemma

We now present our main technical tool for “weaving together” several neural filters approximating a causal map on distinct time windows. The key technical insight here is that each neural filter is approximated while the hypernetwork “weaving together” these neural filter memorizes, and memorization requires exponentially fewer parameters than approximation. The reason for this is that memorizing $N$ points requires between $N$ and $N^{2}$ trainable (non-zero) parameters, as demonstrated in sources like VershinynMemorization and Ruiyang. Notably, only $\mathcal{O}(1)$ neurons are necessary to memorize a function’s value at a single point. In contrast, approximating a function’s value on each sub-cube of $^{d}$ with side length $\delta$ requires $\mathcal{O}(1)$ neurons for each sub-cube, with a total of $\Theta(\delta^{-d})$ such sub-cubes. As a result, any uniform approximator needs an exponential number of neurons to uniformly approximate a function over any hypercube, whereas a memorizer of $N$ points does not have that same requirement.

for every “time” $t=0,\dots,\big\lfloor\delta^{-Q}\big\rfloor=:T_{\delta,Q}-1$. Moreover, the “model complexity” of $\hat{h}$ is specified by

Width: $\mathcal{NN}$ has width at-most $(P([d])+Q)T+12$;

Depth: $\mathcal{NN}$ has depth at-most of the order of

Number of non-zero parameters: The number of non-zero parameters in $\mathcal{NN}$ is at-most

where the constant $C_{d}>0$ is defined by

In the previous expressions $(i)$, $(ii)$ and $(iii)$ we set, for simplicity of notation, $T\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}T_{\delta,Q}-1$.

for every $t=0,\ldots,T_{R,\delta,Q}-1$. Furthermore, the following quantitative “model complexity estimates" hold

Number of non-zero parameters: The number of non-zero parameters in $\mathcal{NN}$ is at most

The “dimensional constant" $C_{d}>0$ is defined by

for every $t=0,\dots,T_{R,\delta,Q}$. Setting $R\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}1$ and $T\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}T_{R,\delta,Q}$ we conclude.

B.5 Proof of Theorem 3.2

where $M=M(\varepsilon_{A},I)=O(\varepsilon_{A}^{-r})$. Now, for every $i\in[[I]]$, for a fixed “encoding error" $\varepsilon_{D}>0$ (and “approximation error" $\varepsilon_{A}$), Theorem 3.1 ensures the existence of a neural filter See Definition 6. $\hat{f}_{t_{i}}\in\mathcal{NF}_{[n_{\varepsilon_{D}}]}^{\text{(P)ReLU}}$ satisfying to the following uniform estimates

Moreover, the “model complexity" of each $\hat{f}_{\theta_{t_{i}}}$ Refer to equation (16) is reported in Table 1. In particular, for $i\in[[I]]$, let $[d^{(i)}]\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}(d_{0}^{(i)},\ldots,d_{J_{i}}^{(i)})$ be the complexity of $\hat{f}_{\theta_{t_{i}}}$, and let $J^{\star,I}$ be the maximum depth of the networks $\{\hat{f}_{\theta_{t_{i}}}\}_{i=1}^{I}$, i.e. $J^{\star,I}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\max_{i\in[[I]]}J_{i}$. In addition, for each $j\in[[J^{\star,I}]]$, set

and let $d_{j}^{\star}$ be the maximum width among the $j^{th}$ layers, i.e. $d_{j}^{\star}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\max_{i\in[[I]]_{j}}d_{j}^{(i)}$.

Define $A\oplus 0_{0}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}A$ for any matrix $A$. Finally, let $[d^{\star}]\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}(d_{0}^{\star},\ldots,d_{J^{\star,I}}^{\star})$. Now, for each $i\in[[I]]$ and $j\in[[d^{\star}_{J^{\star,I}}]]$ we define:

for every “time" $i=1,\ldots,I_{\delta,Q}-1$, where $I_{\delta,Q}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}\lfloor\delta^{-Q}\rfloor$.

The depth and the width of the network are provided by the same lemma with $T_{\delta,Q}\stackrel{{\scriptstyle\mbox{\tiny def.}}}{{=}}I_{\delta,Q}$. Equations (90) and (91) imply that

for every $i\in[[I]]$. At this point, combining Equations (88) and (89), we have:

Appendix C Technical Lemmata

endowed with the product topology is still a Fréchet space carrying a Schauder basis: a choice for this one is provided by $(b_{t})_{t=1}^{\infty}\subset G$, where

From elementary results from functional analysis and topology, it is clear that $G$ endowed with the product topology is a topological vector space. This topology can be induced also by a metric, e.g.

where $d_{E}$ (respectively $d_{F}$) is a compatible metric for $E$ (respectively $F$). Evidently, $(G,d)$ is also complete. This topology is locally convex because it can be induced by the following countable collection of seminorms

We claim that $(b_{t})_{t=1}^{\infty}$ is a Schauder basis for $G$. Indeed, let $x=(e,f)$, with