Infinite-dimensional reservoir computing

Introduction

Reservoir computing (RC) [Jaeg 10, Maas 02, Jaeg 04, Maas 11] and in particular echo state networks (ESNs) [Matt 92, Matt 93, Jaeg 04] have gained much popularity in recent years due to their excellent performance in the forecasting of dynamical systems [Grig 14, Jaeg 04, Path 17, Path 18, Lu 18, Wikn 21, Arco 22] and due to the ease of their implementation. RC aims at approximating nonlinear input/output systems using randomly generated state-space systems (called reservoirs) in which only a linear readout is estimated. It has been theoretically established that this is indeed possible in a variety of deterministic and stochastic contexts [Grig 18b, Grig 18a, Gono 20c, Gono 21b, Gono 23] in which RC systems have been shown to have universal approximation properties.

In this paper, we focus on deriving error bounds for a variant of the architectures that we just cited and consider as approximants randomly generated linear systems with readouts given by randomly generated neural networks in which only the output layer is trained. Thus, from a learning perspective, we combine linear echo state networks and what is referred to in the literature as random features [Rahi 07] /extreme learning machines (ELMs) [Huan 06]. We develop explicit and readily computable approximation and estimation bounds for a newly introduced concept class whose elements we refer to as recurrent (generalized) Barron functionals since they can be viewed as a dynamical analog of the (generalized) Barron functions introduced in [Barr 92, Barr 93] and extended later in [E 20b, E 20a, E 19]. The main novelty in this concept class with respect to others already available in the literature, like the fading memory class [Boyd 85] in deterministic setups or $L^{p}$ functionals in the stochastic case, is that it consists of elements that admit an explicit infinite-dimensional state-space representation, which makes them analytically tractable. As we shall see later on, many interesting families of input/output systems belong to this class that, additionally, is universal in the $L^{p}$-sense.

From an approximation-theoretical point of view, the universality properties of linear systems with readouts belonging to dense families have been known for a long time both in deterministic [Boyd 85, Grig 18b] and in stochastic [Gono 20c] setups. Their corresponding dynamical and memory properties have been extensively studied (see, for instance, [Herm 10, Coui 16, Tino 18, Tino 20, Gono 20a, Li 22] and references therein). Our contribution in this paper can hence be considered as an extension of those works to the recurrent Barron class.

All the dynamic learning works cited so far use exclusively finite-dimensional state spaces. Hence, one of the main novelties of our contribution is the infinite-dimensional component in the concept class that we propose. It is worth mentioning that, even though in static setups, there exist numerous neural functional approximation results (see, for instance, [Chen 95, Stin 99, Krat 20, Bent 22, Cuch 22, Neuf 22], in addition to the works on Barron functions cited above), the use of infinite-dimensional state-space systems has not been much exploited, and it is only very recently that it is being seriously developed. See [Herm 12, Bouv 17b, Bouv 17a, Kira 19, Li 20, Kova 21, Acci 22, Gali 22, Hu 22, Salv 22] for a few examples. Dedicated hardware realizations of RC systems using quantum systems are a potential application of these extensions for which, in the absence of adequate tools, most of the theoretical developments have been carried out so far in exclusively finite-dimensional setups [Chen 19, Chen 20, Tran 20, Tran 21, Chen 22, Mart 23].

In order to introduce our contributions in more detail, we start by recalling that an echo state network (ESN) is an input/output system defined by the two equations:

The second step consists in obtaining generalization error bounds that match the parameter restrictions emerging from the approximation results for these systems. The key challenge here is that the observational data is non-IID and so classical statistical learning techniques [Bouc 13] can not be employed directly, see [Gono 20b]. By combining these bounds, we then obtain learning error bounds for such random recurrent neural networks for learning a general class of maps $\mathcal{C}$. As a by-product, we obtain new universality results for reservoir systems, see [Grig 18b, Grig 18a, Gono 20c, Gono 23, Gono 21b], with generically randomly generated coefficients. It is worth emphasizing that the construction that we describe in this paper yields a fully implementable recurrent neural network-based learning algorithm with provable convergence guarantees not suffering from the curse of dimensionality.

A dynamic analog of the generalized Barron functions

2 Definition and properties of recurrent Barron functionals

In this section we define the class of recurrent generalized Barron functionals and study some properties of these functionals.

and, writing $\mathbf{x}_{t}=\mathbf{x}_{t}({\bf z})$,

We denote by $\mathcal{C}$ the class of all recurrent generalized Barron functionals or just recurrent Barron functionals.

Note that the readout (2.2) is built in such a way that the output is constructed out of $\mathbf{x}_{-1}$ and ${\bf z}_{0}$ instead of exclusively $\mathbf{x}_{0}$, like it is customary in reservoir computing (see (1.2)). The reason for this choice is that it will allow us later on in Section 3.6 to recover the static situation as a particular case of the recurrent setup in a straightforward manner.

The unique solution property (USP). The existence and uniqueness of solutions property of state equations of the type (2.1), which is part of the definition of the recurrent Barron functionals, is a well-studied problem. This property is referred in the literature as the echo state property (ESP) [Jaeg 10, Yild 12, Manj 13] or the unique solution property (USP) [G Ma 20, Manj 22a] and it has been much tackled in deterministic (see references above and [Grig 18a], for instance), and stochastic setups [Manj 22b]. The following proposition is an infinite-dimensional adaptation of a commonly used sufficient condition for the USP to hold in the case of (2.1), as well as a full characterization of it when the activation function $\sigma_{1}$ is the identity, and hence (2.1) is a linear system. In both cases, the inputs are assumed to be bounded; possible generalizations to the unbounded case can be found in [Grig 19].

Consider the state equation given by (2.1) and the two following cases:

The proof of part (i) is a straightforward generalization of [Grig 18a, Theorem 3.1(ii)] together with the observation that the hypothesis $L_{\sigma_{1}}{\left|\kern-1.07639pt\left|\kern-1.07639pt\left|A\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}<1$ implies that the state equation is a contraction on the state entry. Part (ii) can be obtained by mimicking the proof of [Grig 21b, Proposition 4.2(i)]; a key element in that proof is the use of Gelfand’s formula for the spectral radius, which holds for operators between infinite-dimensional Banach spaces [Lax 02, Theorem 4, page 195]. ∎

The concept class $\mathcal{C}$ of recurrent Barron functionals is a vector space.

Next, define $\phi_{i}\colon\mathcal{X}\to\mathcal{X}$ as $\phi_{i}(w,{\bf a},{\bf c},b)=(w(\lambda_{1}+\lambda_{2}),\pi_{i}({\bf a}),{\bf c},b)$ for $i=1,2$ and set $\mu=\frac{\lambda_{1}}{\lambda_{1}+\lambda_{2}}(\mu^{(1)}\circ\phi_{1}^{-1})+\frac{\lambda_{2}}{\lambda_{1}+\lambda_{2}}(\mu^{(2)}\circ\phi_{2}^{-1})$, where $\mu^{(i)}\circ\phi_{i}^{-1}$ denotes the pushforward of $\mu^{(i)}$ under $\phi_{i}$. Then the integral transformation theorem shows that

and, by (2.5), for all ${\bf z}\in\mathcal{I}_{d}$,

Hence, $\lambda_{1}H_{1}+\lambda_{2}H_{2}\in\mathcal{C}$, as claimed. ∎

and, writing $\mathbf{x}_{t}^{N}=\mathbf{x}_{t}^{N}({\bf z})$,

We shall refer to the elements in $\mathcal{C}_{N}$ as finite-dimensional recurrent Barron functionals. In the following proposition, we show that the set of finite-dimensional recurrent Barron functionals can be naturally seen as a subspace of $\mathcal{C}$. In the statement, we use the notion and properties of system morphisms that are recalled, for instance, in [Grig 21b].

that is, every finite-dimensional recurrent Barron functional admits a realization as a recurrent Barron functional.

First, we notice that $\iota$ is system equivariant because

which proves that $H$ in (2.2) defines an element in $\mathcal{C}$. Finally, we show that the map $\iota$ is readout invariant using that the first $N$ components of each solution $\mathbf{x}_{t}$ are given by $\mathbf{x}_{t}^{N}$ and applying again the change-of-variables theorem:

for ${\bf z}\in\mathcal{I}_{d}$, as required. In particular, $H^{N}$ and $H$ are in fact the same functional, which proves the inclusion (2.8). ∎

3 Examples of recurrent Barron functionals

We now show that large classes of input/output systems naturally belong to the recurrent Barron class.

Finite-memory functionals. The following proposition shows that finite-memory causal and time-invariant systems that have certain regularity properties can be written as finite-dimensional recurrent Barron functionals with a linear state equation.

But $h$ is continuous in $(D_{d})^{T}$ by [Evan 10, Theorem 6(ii) in Chapter 5.6] and also the right-hand side in (2.11) is continuous in ${\bf u}$, hence the representation (2.11) holds for all ${\bf u}\in(D_{d})^{T}$.

which shows that $H$ is a finite-dimensional recurrent Barron functional, that is $H\in\mathcal{C}_{N}$. ∎

Linear functionals. The following propositions show that certain linear functionals are also in the recurrent Barron class $\mathcal{C}$.

Let $\lambda\in(0,1)$ be as in (i) or, in case (ii) holds, let $\lambda=1$.

since $\|\cdot\|_{p}\leq\|\cdot\|$ for $p\geq 2$ and $\|\cdot\|_{p}\leq d^{\frac{1}{p}-\frac{1}{2}}\|\cdot\|$ for $p\in$. Furthermore, for any ${\bf z}\in\mathcal{I}_{d}$,

which proves that $H\in\mathcal{C}$ in the case $\sigma_{2}(x)=x$.

For the case $\sigma_{2}(x)=\max(x,0)$, let $\varphi\colon\mathcal{X}\to\mathcal{X}$ be defined by $\varphi(\bar{w},{\bf a},{\bf c},b)=(-\bar{w},-{\bf a},-{\bf c},-b)$ and consider $\bar{\mu}=\frac{1}{2}\mu+\frac{1}{2}(\mu\circ\varphi^{-1})$. Then the change-of-variables theorem and the fact that $I_{\mu,p}<\infty$ (as established above) show that $I_{\bar{\mu},p}<\infty$ and by (2.14) and $\sigma_{2}(x)-\sigma_{2}(-x)=x$ we obtain

hence $\frac{1}{2}H\in\mathcal{C}$ and, by Proposition 2.3, also $H\in\mathcal{C}$. ∎

The next proposition also covers the linear case under slightly different hypotheses.

As a further example, we now see that any functional associated with a linear state-space system on a separable Hilbert space admits a realization as an element in $\mathcal{C}$ for $p=2$.

This shows that the functional $H$ can be represented as a readout (2.2) of a system (2.1), hence $H\in\mathcal{C}$. It also proves that $T$ is a system isomorphism between the system determined by $\bar{A}$, $\bar{C}$, $\bar{B}$, and $\bar{\mu}$, and the system in $\mathcal{C}$ determined by ${A}$, ${C}$, ${B}$, and ${\mu}$. ∎

The next proposition is a generalization of a universality result in [Gono 20c, Theorem III.13] to the context of recurrent Barron functionals.

This shows that $H^{RC}\in\mathcal{C}_{N}$ and $\|H-H^{RC}\|_{L^{\bar{p}}(\mathcal{I}_{d},\gamma)}<\varepsilon$. Combining this with Proposition 2.4 yields the claim.

In the case where $\sigma$ is bounded, continuous and non-constant we may directly work with $\sigma$ (instead of $\bar{\sigma}$) and proceed similarly. ∎

Generalized convolutional functionals. We conclude by showing that the class $\mathcal{C}$ contains functionals that generalize convolutional filters, an important family of transforms.

Notice that standard convolutional filters are a special case of the functional in (2.21) when $\mu$ is a Dirac measure and $\sigma_{2}$ is the identity.

Approximation

In this section, we establish approximation results for the concept class $\mathcal{C}$ of recurrent Barron functionals. As approximating systems, we use a modification of the finite-dimensional echo state networks (1.1)–(1.2) in which the linear readout $W$ in (1.2) is replaced by an extreme learning machine (ELM) / random features neural network, that is, a neural network of the type (1.3) whose parameters are all randomly generated, except for the output layer given by the weights $(W_{i})_{i=1,\ldots,N}$ which are trainable. To derive such bounds, we proceed in several approximation steps and (a) approximate the infinite system (2.1) by a finite system of type (1.1), (b) apply a Monte Carlo approximation of the integral in (2.2), and (c) use an importance sampling procedure to guarantee that the neural network weights can be generated from a generic distribution rather than the (unknown) measure $\mu$.

For each ${\bf z}\in\mathcal{I}_{d}$ the system

This can now be used to estimate the difference between $H$ and its truncation:

and, writing $\mathbf{x}_{t}^{N}=\mathbf{x}_{t}^{N}({\bf z})$,

This proves that $H^{N}\in\mathcal{C}_{N}$. ∎

2 Normalized realizations

The next result shows that a large class of elements in the concept class $\mathcal{C}$ can be transformed into what we call a normalized realization in which the solution map for (2.1) is a multiplicative scaling operator.

since ${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|A^{k_{0}}\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}<1$ and $D_{d}$ is bounded. The second term in (3.12) can be rewritten as

Combining (3.13) and (3.12) and inserting in (2.2) then yields

Let $\bar{\phi}\colon\mathcal{X}\to\mathcal{X}$ be given as $\bar{\phi}(w,{\bf a},{\bf c},b)=(w,\bar{\mathcal{L}}({\bf a}),{\bf c},{\bf a}\cdot\bar{\bf B}_{0}+b)$ and denote by $\bar{\mu}=\mu\circ\bar{\phi}^{-1}$ the pushforward measure of $\mu$ under $\bar{\phi}$. Then the change-of-variables formula and (3.11) show that

and similarly we obtain from (3.14) that for any ${\bf z}\in\mathcal{I}_{d}$,

3 Echo state network approximation of normalized, truncated functionals

Consider now the setting of reservoir computing, where we aim to approximate an unknown element in our class $\mathcal{C}$ using a finite-dimensional ELM-ESN pair, that is, a randomly generated echo state network-like state equation with a neural network as readout in which only the output layer is trained. As a first step, we consider in this section the situation when the target functional is in the class $\mathcal{C}_{N}$ of finite-dimensional recurrent Barron functionals.

More specifically, consider the echo state network

Let $T=\left\lceil\frac{N}{d}\right\rceil$ and define the Kalman controlability matrix

where, in case $T>\frac{N}{d}$, the map $\pi_{N\times N}$ removes the last $dT-N$ columns.

In order to obtain the last inequality in the previous expression, we decomposed $T=\lfloor\frac{T}{k_{0}}\rfloor k_{0}+r$, with $0\leq r<k_{0}$, which allowed us to write

4 Main approximation result

This section contains our main approximation result, in which we extend the possibility of approximating the elements in the class $\mathcal{C}_{N}$ of finite-dimensional recurrent Barron functionals, using finite-dimensional ELM-ESN pairs, to the full class $\mathcal{C}$ of recurrent Barron functionals.

Using this notation, we shall now formulate a result that shows that the elements in $H\in\mathcal{C}$ that admit a representation in which the state equation (2.4) is linear and has the unique solution property, can be approximated arbitrarily well by randomly generated ESN-ELM pairs in which only the output layer of the ELM is trained. We provide an approximation bound where the dependence on the dimensions $N$ of the ESN state space and $d$ of the input is explicitly spelled out.

holds. Notice that from the proof of Proposition 2.6 we obtain that $(\bar{\Lambda}{\bf z})_{d(k-1)+j}=\lambda^{k-1}(z_{-k})_{j}=\bar{{x}}_{-1,d(k-1)+j}$, where $\bar{\mathbf{x}}$ is the unique solution to

satisfies $H^{N}\in\mathcal{C}_{N}$ and that

for all ${\bf z}\in\mathcal{I}_{d}$ and by (3.11)

Choose $M>0$ such that $\|{\bf u}\|\leq M$ for all ${\bf u}\in D_{d}$. Furthermore, note that ${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|\bar{A}\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}\leq\lambda$ and $\bar{{x}}_{-1-l,(k-1)d+j}=\lambda^{k-1}[z_{-l-k}]_{j}$. Hence, using $\|\cdot\|_{q}^{q}\leq\max(d^{1-\frac{q}{2}},1)\|\cdot\|^{q}$ and $\|{\bf u}\|\leq M$ for all ${\bf u}\in D_{d}$ we obtain from (3.33)

satisfies for all ${\bf z}\in\mathcal{I}_{d}$ the bound (3.22), that is,

Combining this with (3.40), (3.34), (3.37) and (3.38) then yields

A bound similar to the one in (3.27) can be obtained if the ESN-ELM used as approximant satisfies the more general condition $\rho({A}^{{\rm ESN}})<1$. The theorem is proved in that case by replacing in (3.37) the use of the inequality (3.22) by its more general version (3.21).

Implementation. From a practical point of view, the bounds obtained in Theorem 3.7 and in Corollary 3.9 apply to two different learning procedures: in the first case, when the chosen ELM weight distribution $\nu$ satisfies the absolute continuity assumption in Theorem 3.7, only the ELM output layer needs to be trained. In the second case, when that condition is not available, all the ELM weights also need to be trained. However, note that these are not recurrent weights; consequently, the vanishing gradient problem does not occur here. In contrast, this problem sometimes occurs for standard recurrent neural networks in which all weights are trained by backpropagation through time.

We conclude this section by considering another implementation scenario in which we impose the measure that we use for the sampling of the ELM, and show that it is still possible to sample using measures that are arbitrarily close to it in the sense of the $1$-Wasserstein metric in exchange for potentially increasing an error term that can be compensated by increasing the dimension of the ESN. Let $\mathcal{W}_{1}$ be the $1$-Wasserstein metric. Suppose that we use the measure $\nu_{0}$ to generate the hidden layer weights. The next corollary shows how the error increases when we sample “almost” from the given measure $\nu_{0}$.

We now derive an alternative bound (regardless of whether $\mathcal{J}_{\nu_{0}}$ is finite or not). Combining both bounds then yields the bound (3.45).

From (3.39) and the estimate $\|\pi(\bar{\mathcal{L}}({\bf a}))\|\leq\max(1,d^{\frac{1}{2}-\frac{1}{p}})c_{0}\|{\bf a}\|_{p}$ we obtain

5 Universality

As an additional result, we obtain the following universality result, which shows that any square-integrable functional (not necessarily in the recurrent Barron class) can be well approximated by the ESN-ELM (3.16)-(3.17) with suitably chosen readout ${\bf W}$ and hidden weights generated from a distribution $\nu$ arbitrarily close to a prescribed measure $\nu_{0}$.

Firstly, by Proposition 2.10 there exists $H^{RC}\in\mathcal{C}\cap L^{2}(\mathcal{I}_{d},\gamma)$ such that $\|H-H^{RC}\|_{L^{2}(\mathcal{I}_{d},\gamma)}<\varepsilon/2$. From the proof of Proposition 2.10 and the construction in [Gono 20c] it follows that $H^{RC}$ is of the form

Let $\phi\colon\mathcal{X}^{\bar{N}}\to\mathcal{X}$ be defined by $\phi(w,{\bf a},{\bf c},b)=(w,\iota\Lambda^{-1}{\bf a},{\bf c},b)$ and denote by $\mu=\mu^{\bar{N}}\circ\phi^{-1}$ the pushforward measure of $\mu^{\bar{N}}$ under $\phi$. Then

Thus, $H^{RC}\in\mathcal{C}$ has a representation of the type (2.2) with ${\left|\kern-1.07639pt\left|\kern-1.07639pt\left|A\right|\kern-1.07639pt\right|\kern-1.07639pt\right|}={\bar{\lambda}}<1$. Furthermore, since $\mu^{\bar{N}}$ is atomic it follows directly that $I_{\mu,p}^{(2)}<\infty$ and $\int_{\mathcal{X}}[|w|+\|{\bf a}\|_{p}+\|{\bf c}\|+|b|]\mu(dw,d{\bf a},d{\bf c},db)<\infty$.

which does not depend on $N$. Similarly, the bound

the assumptions on the ESN matrices, and (3.47) can be used to obtain an upper bound $\bar{C}_{H,1}$ on $\mathcal{W}_{1}(\nu_{0},\mu_{A,B,C})$ that only depends on $\mu$, $d$, $p$, $\lambda$, $\bar{c}$, $\underline{l}$, $\bar{l}$ and $\nu_{0}$, but not on $N$. Set $\bar{c}_{H}=\bar{C}_{H,0}\max(1,\bar{C}_{H,1})^{\frac{1}{2}}$.

We now apply Corollary 3.12 to $H=H^{RC}$. We select $\delta\in(0,1)$ with $\delta<\varepsilon$ and

6 Special case: static situation

In this situation, we show that such static elements $H$ of the recurrent Barron class can be approximated by ELMs, that is, by feedforward neural networks

As noted above, $H$ is a recurrent generalized Barron functional with

for any $f$ for which the integrand in the last line is integrable with respect to $\mu_{0}$. This shows that $\bar{H}^{N}$ in (3.36) coincides with the map

and furthermore $\bar{H}^{N}=H^{N}$. Therefore, denoting by $H$ also the functional ${\bf z}\mapsto H({\bf z}_{0})$, in the first step of the error estimate (3.42) only the last term is non-zero.

Furthermore, $\psi_{A,B,C}(w,{\bf a},{\bf c},b)=(w,0,{\bf c},b)$ and thus $\mu_{A,B,C}(dw,d{\bf a},d{\bf c},db)=\delta_{0}(d{\bf a})\mu_{0}(dw,d{\bf c},db)$. Therefore, the hypothesis $\mu_{0}\ll\nu_{0}$ implies $\mu_{A,B,C}\ll\delta_{0}(d{\bf a})\nu_{0}$.

Choosing $W_{i}={\displaystyle\frac{w^{(i)}}{N}\frac{d\mu_{0}}{d\nu_{0}}(w^{(i)},{\bf c}^{(i)},b_{i})}$, from (3.40) we thus obtain

The expectation in (3.57) can be estimated by

with $c_{1}=2\max(2L_{\sigma_{2}}^{2},|\sigma_{2}(0)|^{2})\max(1,\sup_{{\bf v}\in D_{d}}\|{\bf v}\|^{2})$. ∎

Learning from a single trajectory

where we denote ${\bf Z}_{-i}^{-n+1}=(\ldots,0,0,{\bf Z}_{-n+1},\ldots,{\bf Z}_{-i-1},{\bf Z}_{-i})$.

The problem (4.2) admits an explicit solution (see, for instance, [Gono 21a, Section 4.3]). Once computed, the learned functional is $\widehat{H}_{\widehat{W}}$. The learning performance of the algorithm can now be evaluated by assessing its learning error (or generalization error)

where $\bar{\bf Z}$ is an IID copy of ${\bf Z}$ and $\bar{\bf Z}$ is independent of all other random variables introduced so far. An analysis of the learning error is carried out below in Theorem 4.2, in which we assume, inspired by the developments in [Gono 20b], that the input and the output processes are of the type introduced in the following definition.

where we used in the last step that (4.2) implies $\mathcal{R}_{n}(\widehat{H}_{\widehat{\bf W}})\leq\mathcal{R}_{n}(\widehat{H}_{\bf W})$.

Altogether, the hypotheses of [Gono 20b, Corollary 8(ii)] are satisfied. Writing $\mathcal{R}_{n}^{\infty}(G)=\frac{1}{n}\sum_{i=0}^{n-1}|G({\bf Z}_{-i}^{-\infty})-{Y}_{-i}|^{2}$ for the idealized empirical risk and applying this result proves that

Recall that $(w^{(i)},{\bf a}^{(i)},{\bf c}^{(i)},b_{i})_{i=1,\ldots,N}$ was considered fixed; now we take expectation also with respect to its distribution $\nu$ and obtain in the same way as we obtained (4.12)

The expectations in (4.14) can be estimated using Jensen’s inequality and (3.23):

Combining this with (4.6), (4.7) and (4.8) we thus obtain

and inserting (3.28) then shows (4.3) with (4.4) and (4.5), as claimed. ∎

The bound in Theorem 4.2 can be directly extended to the multivariate case $m>1$ as follows: if $H=(H_{1},\ldots,H_{m})$ and each $H_{i}$ satisfies the hypotheses formulated in Theorem 4.2, then

Conclusions

In this paper we have provided reservoir computing approximation and generalization bounds for a new concept class of input/output systems that extends to a dynamical context the so-called generalized Barron functionals. We call the elements of the new class recurrent generalized Barron functionals. Its elements are characterized by the availability of readouts with a certain integral representation built on infinite-dimensional state-space systems. We have also shown that they form a vector space and that the class contains as a subset systems of this type with finite-dimensional state-space representations. This class has been shown to be very rich: it contains all sufficiently smooth finite-memory functionals, linear and convolutional filters, as well as any input-output system built using separable Hilbert state spaces with readouts of integral Barron type. Moreover, this class is dense in the $L^{2}$-sense.

The reservoir architectures used for the approximation/estimation of elements in the new class are randomly generated echo state networks with either linear (for approximation) or ReLU (estimation) activation functions, and with readouts built using randomly generated neural networks, in which only the output layer is trained (extreme learning machines or random feature neural networks). The results in the paper yield a fully implementable recurrent neural network-based learning algorithm with provable convergence guarantees that does not suffer from the curse of dimensionality.

Acknowledgments. The authors acknowledge partial financial support coming from the Swiss National Science Foundation (grant number 200021_175801/1). L. Grigoryeva and J.-P. Ortega wish to thank the hospitality of the University of Munich and L. Gonon that of the Nanyang Technological University during the academic visits in which some of this work was developed. We thank Christa Cuchiero and Josef Teichmann for interesting exchanges that inspired some of the results in this paper.

References