Deep Neural Network Structures Solving Variational Inequalities

Introduction

A powerful tool from fixed point theory to analyze and solve optimization and inclusion problems in a real Hilbert space ${\mathcal{H}}$ is the class of averaged nonexpansive operators, which was introduced in . Recall that an operator $T\colon{\mathcal{H}}\to{\mathcal{H}}$ is nonexpansive if it is $1$-Lipschitzian, and $\alpha$-averaged for some $\alpha\in\left]0,1\right]$ if there exists a nonexpansive operator $Q\colon{\mathcal{H}}\to{\mathcal{H}}$ such that $T=(1-\alpha)\operatorname{Id}+\alpha Q$; if $\alpha=1/2$, $T$ is firmly nonexpansive. The importance of firmly nonexpansive operators in convex optimization and variational methods has long been recognized . More generally, averaged operators were shown in to play a prominent role in the analysis of convex feasibility problems. In this context the underlying problem is to find a common fixed point of averaged operators. In , it was shown that many convex minimization and monotone inclusion problems reduce to the more general problem of finding a fixed point of compositions of averaged operators, which provided a unified analysis of various proximal splitting algorithms. Along these lines, several fixed point methods based on various combinations of averaged operators have since been devised, see for recent work. Motivated by deep neural network structures with thus far elusive asymptotic properties, we investigate in the present paper a novel averaged operator model involving a mix of nonlinear and linear operators.

Artificial neural networks have attracted considerable attention as a tool to better understand, model, and imitate the human brain . In a Hilbertian setting , an $(n+1)$-layer feed-forward neural network architecture acting on real Hilbert spaces $({\mathcal{H}}_{i})_{0\leqslant i\leqslant n}$ is defined as the composition of operators $R_{n}\circ(W_{n}\cdot+b_{n})\circ\cdots\circ R_{1}\circ(W_{1}\cdot+b_{1})$ where, for every $i\in\{1,\ldots,n\}$, $R_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ is a nonlinear operator known as an activation operator, $W_{i}\colon{\mathcal{H}}_{i-1}\to{\mathcal{H}}_{i}$ is a linear operator, known as a weight operator, and $b_{i}\in{\mathcal{H}}_{i}$ is a so-called bias parameter. Deep neural networks feature a (possibly large) number $n$ of layers. In recent years, they have been found to be quite successful in a wide array of classification, recognition, and prediction tasks; see and its bibliography. Despite their success, the operational structure and properties of deep neural networks are not yet well understood from a mathematical viewpoint. In the present paper, we propose to analyze them within the following iterative model. We emphasize that our purpose is not to study the training of the network, which consists of optimally setting the weight operators and bias parameters from data samples, but to analyze mathematically such a structure once it is trained. Our model is also of general interest in constructive fixed point theory.

Our contributions are articulated around the following findings.

We show that a wide range of activation operators used in neural networks are actually proximity operators, which paves the way to the analysis of such networks via fixed point theory.

We provide a new analysis of compositions of proximity and affine operators, establishing mild conditions that guarantee that the resulting operator is averaged.

We show that, under suitable assumptions, the asymptotic output of the network converges to a point defined via a variational inequality. Furthermore, in general, this variational inequality does not derive from a minimization problem.

The remainder of the paper is organized as follows. In Section 2, we bring to light strong connections between the activation functions employed in neural networks and the theory of proximity operators in convex analysis. In Section 3 we derive new results on the averagedness properties of compositions of proximity and affine operators acting on different spaces. In Section 4, we investigate the asymptotic behavior of a class of deep neural networks structures and show that their fixed points solve a variational inequality. The main assumption on this subclass of Model 1.1 is that the structure of the network is periodic in the sense that a group of layers is repeated. Finally, in Section 5, the same properties are established for a broader class of networks.

Proximal activation in neural networks

Let $\varphi\in\Gamma_{0}({\mathcal{H}})$. Then the following hold:

[8, Proposition 12.29] $\text{\rm Fix}\,\text{\rm prox}_{\varphi}=\text{\rm Argmin}\,\varphi$.

[8, Corollary 24.5] Let $g\in\Gamma_{0}({\mathcal{H}})$ be such that $\varphi=g-\|\cdot\|^{2}/2$. Then $\text{\rm prox}_{\varphi}=\nabla g^{*}$.

which was initially proposed in . As this discontinuous activation model may lead to unstable neural networks, various continuous approximations have been proposed. Our key observation is that a vast array of activation functions used in neural networks belong to the following class.

Remarkably, we can precisely characterize this class of activation functions as that of proximity operators.

The most basic activation function is $\varrho=\operatorname{Id}=\text{\rm prox}_{0}$. It is in particular useful in dictionary learning approaches, which correspond to the linear special case of Model 1.1 .

can be written as $\varrho=\text{\rm prox}_{\phi}$, where $\phi$ is the indicator function of $$.

The rectified linear unit (ReLU) activation function

can be written as $\varrho=\text{\rm prox}_{\phi}$, where $\phi$ is the indicator function of $\left[0,+\infty\right[$.

Let $\alpha\in\left]0,1\right]$. The parametric rectified linear unit activation function is

We have $\varrho=\text{\rm prox}_{\phi}$, where

Proof. This follows from [23, Lemma 2.6 and Example 2.18].

Proof. Let $\xi\in\left]-1,1\right[=\text{\rm dom}\,\nabla\phi=\text{\rm dom}\,\partial\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\xi/\sqrt{1-\xi^{2}}$ and therefore $\text{\rm prox}_{\phi}=(\operatorname{Id}+\phi^{\prime})^{-1}\colon\mu\mapsto\mu/\sqrt{1+\mu^{2}}$.

The inverse square root linear unit activation function

can be written as $\varrho=\text{\rm prox}_{\phi}$, where

Proof. Let $\xi\in\left]-1,{+\infty}\right[=\text{\rm dom}\,\nabla\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\xi$ if $\xi\geqslant 0$, and $\xi+\phi^{\prime}(\xi)=\xi/\sqrt{1-\xi^{2}}$ if $\xi<0$. Hence, $\varrho=(\operatorname{Id}+\phi^{\prime})^{-1}$ is given by (2.8).

The arctangent activation function $(2/\pi)\text{arctan}$ is the proximity operator of

Proof. Let $\xi\in\left]-1,1\right[=\text{\rm dom}\,\nabla\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\text{tan}(\pi\xi/2)$ and therefore $\varrho=(\operatorname{Id}+\phi^{\prime})^{-1}=(2/\pi)\text{arctan}$.

The hyperbolic tangent activation function tanh is the proximity operator of

Proof. Let $\xi\in\left]-1,1\right[=\text{\rm dom}\,\nabla\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\text{arctanh}(\xi)$ and therefore $\varrho=(\operatorname{Id}+\phi^{\prime})^{-1}=\text{tanh}$.

Proof. Let $\xi\in\left]-1/2,1/2\right[=\text{\rm dom}\,\nabla\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\ln((1+2\xi)/(1-2\xi))$ and therefore $\text{\rm prox}_{\phi}=(\operatorname{Id}+\phi^{\prime})^{-1}\colon\mu\mapsto(1/2)(e^{\mu}-1)/(e^{\mu}+1)=1/(1+e^{-\mu})-1/2$.

Examples 2.12 and 2.13 are closely related in the sense that the function of (2.12) can be written as $\varrho=(1/2)\text{tanh}(\cdot/2)$.

Proof. Let $\xi\in\left]-1,1\right[=\text{\rm dom}\,\nabla\phi=\text{\rm ran}\,\text{\rm prox}_{\phi}$. Then $\xi+\phi^{\prime}(\xi)=\xi/(1-|\xi|)$ and therefore $\text{\rm prox}_{\phi}=(\operatorname{Id}+\phi^{\prime})^{-1}\colon\mu\mapsto\mu/(1+|\mu|)$.

The inverse hyperbolic sine activation function arcsinh is the proximity operator of $\phi=\text{cosh}-|\cdot|^{2}/2$.

Proof. We have $\phi^{\prime}\colon\xi\mapsto\text{\rm sign}(\xi)(e^{|\xi|}-1)-\xi$. Hence $(\operatorname{Id}+\phi^{\prime})\colon\xi\mapsto\text{\rm sign}(\xi)(e^{|\xi|}-1)$ and, in turn, $\text{\rm prox}_{\phi}=(\operatorname{Id}+\phi^{\prime})^{-1}\colon\xi\mapsto\text{\rm sign}(\xi)\ln(1+|\xi|)$.

Proof. (i)–(iii): This follows at once from Definition 2.2.

(iv)–(v): The fact that the resulting operators are proximity operators is established in [21, Section 3.3]. The fact that they are proximity operators of a function $\phi\in\Gamma_{0}({\mathcal{H}})$ that is minimal at is equivalent to the fact that $\text{\rm prox}_{\phi}0=0$ Lemma 2.1(i). This identity is easily seen to hold in each instance.

(vi): Set $\varrho=\varrho_{1}\circ(2\varrho_{2}-\operatorname{Id})+\operatorname{Id}-\varrho_{2}$. Then $\varrho$ is firmly nonexpansive [8, Proposition 4.31(ii)]. It is therefore increasing and nonexpansive. Finally, $\varrho(0)=0$.

Using Proposition 2.18, the above examples can be combined to obtain additional activation functions. For instance, it follows from Example 2.5 and Proposition 2.18(iv) that the soft thresholder

2 Activation operators

In Section 2.1, we have described activation functions which model neuronal activity in terms of a scalar function. In this section, we extend this notion to more general activation operators.

Let ${\mathcal{H}}$ be a real Hilbert space and let $R\colon{\mathcal{H}}\to{\mathcal{H}}$. Then $R$ belongs to the class $\mathcal{A}({\mathcal{H}})$ if there exists a function $\varphi\in\Gamma_{0}({\mathcal{H}})$ which is minimal at the zero vector and such that $R=\text{\rm prox}_{\varphi}$.

Property (ii) below shows that activation operators in $\mathcal{A}({\mathcal{H}})$ have strong stability properties. On the other hand, the boundedness property (iv) is important in neural network-based functional approximation .

Let ${\mathcal{H}}$ be a real Hilbert space and let $R\in\mathcal{A}({\mathcal{H}})$. Then the following hold:

Let $x$ and $y$ be in ${\mathcal{H}}$. Then $\|Rx-Ry\|^{2}\leqslant\|x-y\|^{2}-\|x-y-Rx+Ry\|^{2}$.

Let $x\in{\mathcal{H}}$. Then $\|Rx\|\leqslant\|x\|$.

Let $\varphi\in\Gamma_{0}({\mathcal{H}})$ be such that $R=\text{\rm prox}_{\varphi}$. Then $\text{\rm ran}\,R$ is bounded if and only if $\text{\rm dom}\,\varphi$ is bounded.

Proof. (i): This follows from Lemma 2.1(i).

(ii): This follows from the firm nonexpansiveness of proximity operators [8, Proposition 12.28].

(iv): We have $\text{\rm ran}\,R=\text{\rm ran}\,(\operatorname{Id}+\partial\varphi)^{-1}=\text{\rm dom}\,(\operatorname{Id}+\partial\varphi)=\text{\rm dom}\,\partial\varphi$. On the other hand, $\text{\rm dom}\,\partial\varphi$ is a dense subset of $\text{\rm dom}\,\varphi$ [8, Corollary 16.39].

Let ${\mathcal{H}}$ and ${\mathcal{G}}$ be real Hilbert spaces. Then the following hold:

Let $L\in\mathcal{B}\,({\mathcal{H}},{\mathcal{G}})$ be such that $\|L\|\leqslant 1$ and let $R\in\mathcal{A}({\mathcal{H}})$. Then $L^{*}\circ R\circ L\in\mathcal{A}({\mathcal{H}})$.

Let $(R_{i})_{i\in I}$ be a finite family in $\mathcal{A}({\mathcal{H}})$ and let $(\omega_{i})_{i\in I}$ be real numbers in $\left]0,1\right]$ such that $\sum_{i\in I}\omega_{i}=1$. Then $\sum_{i\in I}\omega_{i}R_{i}\in\mathcal{A}({\mathcal{H}})$.

Let $R\in\mathcal{A}({\mathcal{H}})$. Then $\operatorname{Id}-R\in\mathcal{A}({\mathcal{H}})$.

Let $R_{1}\in\mathcal{A}({\mathcal{H}})$ and $R_{2}\in\mathcal{A}({\mathcal{H}})$. Then $(R_{1}-R_{2}+\operatorname{Id})/2\in\mathcal{A}({\mathcal{H}})$.

Proof. The fact that the resulting operators are proximity operators is established in [21, Section 3.3]. In addition, is clearly a fixed point of the resulting operators. In view of Lemma 2.1(i), the proof is complete.

Then $\psi=g-\|\cdot\|^{2}/2$ and [40, Section 16] asserts that

Since $\nabla g^{*}=R+u$, according to Lemma 2.1(ii), $R=\text{\rm prox}_{\psi}-u$. We complete the proof by invoking the shift properties of proximity operators [8, Proposition 24.8(iii)].

Separable activation operators supply another important instance of activation operators.

Compositions of firmly nonexpansive and affine operators

Our analysis will revolve around the following property for a family of linear operators $(W_{i})_{1\leqslant i\leqslant m+1}$.

Let $m\geqslant 0$ be an integer, let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m}$ be real Hilbert spaces, set ${\mathcal{H}}_{m+1}={\mathcal{H}}_{0}$, and let $\alpha\in[1/2,1]$. For every $i\in\{1,\ldots,m+1\}$, let $W_{i}\in\mathcal{B}\,({\mathcal{H}}_{i-1},{\mathcal{H}}_{i})$ and set

It is required that, for every $\boldsymbol{x}=(x_{i})_{0\leqslant i\leqslant m}\in{\mathcal{H}}_{0}\times\cdots\times{\mathcal{H}}_{m}$ such that

In Condition 3.1, we take $\alpha\geqslant 1/2$ because, if $\boldsymbol{x}=(x_{i})_{0\leqslant i\leqslant m}\in({\mathcal{H}}_{0}\smallsetminus\{0\})\times{\mathcal{H}}_{1}\times\cdots\times{\mathcal{H}}_{m}$ satisfies (3.3), then $2^{m+1}(1-\alpha)\|x_{0}\|\leqslant\|L_{m+1}\boldsymbol{x}-2^{m+1}(1-\alpha)x_{0}\|+\|L_{m+1}\boldsymbol{x}\|\leqslant 2^{m+1}\alpha\|x_{0}\|$.

We establish some preliminary results before providing properties that imply Condition 3.1.

Let $m\geqslant 1$ be an integer, let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m}$ be real Hilbert spaces, and set $\theta_{0}=1$. For every $i\in\{1,\ldots,m\}$, let $W_{i}\in\mathcal{B}\,({\mathcal{H}}_{i-1},{\mathcal{H}}_{i})$ and set

Let $(x_{i})_{0\leqslant i\leqslant m}\in{\mathcal{H}}_{0}\times\cdots\times{\mathcal{H}}_{m}$ be such that (3.2) is satisfied. Then the following hold:

$(\forall i\in\{1,\ldots,m\})$ $\theta_{i}=\sum_{k=0}^{i-1}\theta_{k}\|W_{i}\circ\cdots\circ W_{k+1}\|$.

$(\forall i\in\{1,\ldots,m\})$ $\|x_{i}\|\leqslant\theta_{i}\|x_{0}\|$.

Proof. (i): This follows recursively from (3.4).

(ii): For every $i\in\{1,\ldots,m\}$, let $L_{i}$ be as in (3.1). We proceed by induction on $m$. We first observe that the inequality is satisfied if $m=1$ since $\|x_{1}\|\leqslant\|L_{1}x_{0}\|=\|W_{1}x_{0}\|\leqslant\|W_{1}\|\,\|x_{0}\|=\theta_{1}\|x_{0}\|$. Now assume that $m\geqslant 2$ and that the inequalities hold for $(x_{1},\ldots,x_{m-1})$. Then, since (i) yields

Let ${\mathcal{H}}$ be a real Hilbert space, and let $x$ and $y$ be in ${\mathcal{H}}$. Then

Proof. Since $\|x+y\|^{2}-2\|x+y\|(\|x\|+\|y\|)+(\|x\|+\|y\|)^{2}\geqslant 0$, we have

Let $m\geqslant 0$ be an integer, and let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m}$ be real Hilbert spaces. Let ${\boldsymbol{\mathcal{X}}}$ be the standard vector space ${\mathcal{H}}_{0}\times\cdots\times{\mathcal{H}}_{m}$ equipped with the norm $\|\cdot\|_{{\boldsymbol{\mathcal{X}}}}\colon\boldsymbol{x}=(x_{i})_{0\leqslant i\leqslant m}\mapsto\max_{0\leqslant i\leqslant m}\|x_{i}\|$ and let ${\boldsymbol{\mathcal{Y}}}$ be the standard vector space ${\mathcal{H}}_{0}\times{\mathcal{H}}_{0}$ equipped with the norm $\|\cdot\|_{{\boldsymbol{\mathcal{Y}}}}\colon\boldsymbol{y}=(y_{1},y_{2})\mapsto\|y_{1}\|+\|y_{2}\|$. Henceforth, the norm of $\boldsymbol{M}\in\mathcal{B}\,({\boldsymbol{\mathcal{X}}},{\boldsymbol{\mathcal{Y}}})$ is denoted by $\|\boldsymbol{M}\|_{{\boldsymbol{\mathcal{X}}},{\boldsymbol{\mathcal{Y}}}}$.

Let $m\geqslant 0$ be an integer, let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m}$ be nonzero real Hilbert spaces, set ${\mathcal{H}}_{m+1}={\mathcal{H}}_{0}$, and use Notation 3.5. For every $i\in\{1,\ldots,m+1\}$, let $W_{i}\in\mathcal{B}\,({\mathcal{H}}_{i-1},{\mathcal{H}}_{i})$. Further, let $\alpha\in[1/2,1]$, let $\theta_{0}=1$, let $(\theta_{i})_{1\leqslant i\leqslant m+1}$ be as in (3.4), and set

There exists $i\in\{1,\ldots,m+1\}$ such that $W_{i}=0$.

$\|\boldsymbol{M}\|_{{\boldsymbol{\mathcal{X}}},{\boldsymbol{\mathcal{Y}}}}\leqslant 1$.

$\|W-2^{m+1}(1-\alpha)\operatorname{Id}\|-\|W\|+2\theta_{m+1}\leqslant 2^{m+1}\alpha$.

$\alpha\neq 1$, for every $i\in\{1,\ldots,m+1\}$ $W_{i}\neq 0$, and there exists $\eta\in[0,\alpha/((1-\alpha)\theta_{m+1})]$ such that

Then $(W_{i})_{1\leqslant i\leqslant m+1}$ satisfies Condition 3.1.

Proof. We use the operators $(L_{i})_{1\leqslant i\leqslant m+1}$ introduced in Condition 3.1. Per Notation 3.5 and (3.9d),

Now let $\boldsymbol{x}\in{\boldsymbol{\mathcal{X}}}$ be such that

(i): We assume that $m\geqslant 1$. For every $k\in\{i,\ldots,m\}$, it follows from (3.4) that $\theta_{k}=0$ and in turn from Lemma 3.3(ii) and (3.13) that $x_{k}=0$. Therefore,

(ii): In view of (i), we assume that, if $m\geqslant 1$, $(\forall i\in\{1,\ldots,m\})$ $W_{i}\neq 0$. We then derive from (3.4) that $(\forall i\in\{1,\ldots,m\})$ $\theta_{i}\geqslant\prod_{k=1}^{i}\|W_{k}\|>0$. If $x_{0}=0$, (3.3) trivially follows from Lemma 3.3(ii), we therefore assume otherwise. Now set

According to Lemma 3.3(ii), $(\forall i\in\{0,\ldots,m\})$ $\|y_{i}\|\leqslant 1$. On the other hand, it follows from (3.9c), (3.15), and (3.1) that $M\boldsymbol{y}=L_{m+1}\boldsymbol{x}/\|x_{0}\|$. Altogether, we deduce from (3.12) that (3.3) holds.

(iii)$\Rightarrow$(ii): Take $\boldsymbol{y}\in{\boldsymbol{\mathcal{X}}}$ such that $\|\boldsymbol{y}\|_{{\boldsymbol{\mathcal{X}}}}\leqslant 1$. Then it follows from (3.9c) and Lemma 3.3(i) that

In turn, (3.11) yields $\|\boldsymbol{M}\|_{{\boldsymbol{\mathcal{X}}},{\boldsymbol{\mathcal{Y}}}}\leqslant 1$.

(iv)$\Rightarrow$(ii): Let $\boldsymbol{y}=(y_{0},\ldots,y_{m})\in{\boldsymbol{\mathcal{X}}}$ be such that $\|y_{0}\|=\cdots=\|y_{m}\|=1$, and set

Therefore, since (3.21) implies that $\alpha-\eta(1-\alpha)\|u\|\geqslant 0$, it results from (3) that

However, since (3.20) implies that $\alpha-\eta(1-\alpha)\|W\|\geqslant 0$, while (3.17) implies that $\|u\|\leqslant\theta_{m+1}-\|W\|$, we derive from (3) that

Hence, using (3), (3.27), (3.9c), (3.9a), and (3.9d) we obtain

Now set \boldsymbol{C}=\big{\{}{\boldsymbol{y}\in{\boldsymbol{\mathcal{X}}}}~{}\big{|}~{}{\|y_{0}\|=\cdots=\|y_{m}\|=1}\big{\}}. Then, in view of (3.11), (3), and [8, Proposition 11.1(ii)], we conclude that $\|\boldsymbol{M}\|_{{\boldsymbol{\mathcal{X}}},{\boldsymbol{\mathcal{Y}}}}=\sup_{\boldsymbol{y}\in\text{\rm conv}\,\boldsymbol{C}}\|\boldsymbol{M}\boldsymbol{y}\|_{{\boldsymbol{\mathcal{Y}}}}=\sup_{\boldsymbol{y}\in\boldsymbol{C}}\|\boldsymbol{M}\boldsymbol{y}\|_{{\boldsymbol{\mathcal{Y}}}}\leqslant 1$.

The next result establishes a link between deep neural network structures and the operators introduced in (3.1).

Let $m\geqslant 1$ be an integer and let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m+1}$ be nonzero real Hilbert spaces. For every $i\in\{1,\ldots,m+1\}$, let $W_{i}\in\mathcal{B}\,({\mathcal{H}}_{i-1},{\mathcal{H}}_{i})$ and let $L_{i}$ be as in (3.1). Further, for every $i\in\{1,\ldots,m\}$, let $P_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ be firmly nonexpansive. Set

let $x$ and $y$ be distinct points in ${\mathcal{H}}_{0}$, and set $v_{0}=(x-y)/\|x-y\|$. Then there exists $(v_{1},\ldots,v_{m})\in{\mathcal{H}}_{1}\times\cdots\times{\mathcal{H}}_{m}$ such that

Proof. For every $i\in\{1,\ldots,m\}$, since $P_{i}$ is firmly nonexpansive, there exists a nonexpansive operator $Q_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ such that

We proceed by induction on $m$. Suppose that $m=1$ and set

which implies that $\|v_{1}\|\leqslant{\|W_{1}(x-y)\|}/{\|x-y\|}=\|L_{1}v_{0}\|$. Then

Thus, (3.30) holds for $m=1$. Next, we assume that $m>1$ and that there exists $(v_{1},\ldots,v_{m-1})\in{\mathcal{H}}_{1}\times\cdots\times{\mathcal{H}}_{m-1}$ such that

In addition, it follows from (3.34) and (3.35) that

We now establish connections between Condition 3.1 for linear operators and the concept of averagedness for composite nonlinear operators.

Let $m\geqslant 1$ be an integer, let $({\mathcal{H}}_{i})_{0\leqslant i\leqslant m-1}$ be nonzero real Hilbert spaces, set ${\mathcal{H}}_{m}={\mathcal{H}}_{0}$, and let $\alpha\in[1/2,1]$. For every $i\in\{1,\ldots,m\}$, let $W_{i}\in\mathcal{B}\,({\mathcal{H}}_{i-1},{\mathcal{H}}_{i})$ and let $P_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ be firmly nonexpansive. Suppose that $(W_{i})_{1\leqslant i\leqslant m}$ satisfies Condition 3.1. Then $P_{m}\circ W_{m}\circ\cdots\circ P_{1}\circ W_{1}$ is $\alpha$-averaged.

Proof. Set $T=P_{m}\circ W_{m}\circ\cdots\circ P_{1}\circ W_{1}$. We must show that

is nonexpansive. By assumption, for every $i\in\{1,\ldots,m\}$, there exists a nonexpansive operator $Q_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ such that (3.31) holds. Let $(L_{i})_{1\leqslant i\leqslant m}$ be as in (3.1) and let $x$ and $y$ be distinct points in ${\mathcal{H}}_{0}$. According to Lemma 3.7, there exists $\boldsymbol{v}=(v_{0},\ldots,v_{m-1})\in{\mathcal{H}}_{0}\times\cdots\times{\mathcal{H}}_{m-1}$ such that

In turn, we derive from (3.38) and (3.31) that

which establishes the nonexpansiveness of $Q$.

Consider Theorem 3.8 with $m=2$. In view of Proposition 3.6(iii), $P_{2}\circ W_{2}\circ P_{1}\circ W_{1}$ is $\alpha$-averaged if $\|W_{2}\circ W_{1}-4(1-\alpha)\operatorname{Id}\|+\|W_{2}\circ W_{1}\|+2\|W_{2}\|\,\|W_{1}\|\leqslant 4\alpha$. In particular, if $\alpha=1$, this condition is obviously less restrictive than requiring that $W_{1}$ and $W_{2}$ be nonexpansive.

A variational inequality model

In this section, we first investigate an autonomous version of Model 1.1.

where $\boldsymbol{x}=(x_{1},\ldots,x_{m})$ denotes a generic element in ${\boldsymbol{\mathcal{H}}}$.

We start with a property of the compositions of the operators $(T_{i})_{1\leqslant i\leqslant m}$ of (4.1).

Consider the setting of Model 4.1, let $i$ and $j$ be integers such that $1\leqslant j\leqslant i\leqslant m$, and let $x\in{\mathcal{H}}_{j-1}$. Then

Proof. In view of (4.1), the property is satisfied when $i=j$. We now assume that $i>j$. Since $R_{i}\in\mathcal{A}({\mathcal{H}}_{i})$, Proposition 2.21(i) yields

Next, we establish a connection between Model 4.1 and a variational inequality.

In the setting of Model 4.1, consider the variational inequality problem

The set of solutions to (4.5) is $\boldsymbol{F}$.

$\boldsymbol{F}=\text{\rm zer}\,({\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\psi})=\text{\rm Fix}\,(\text{\rm prox}_{\boldsymbol{\psi}}\circ\boldsymbol{W}\circ\boldsymbol{S})$.

\boldsymbol{F}=\big{\{}{(T_{1}\overline{x}_{m},(T_{2}\circ T_{1})\overline{x}_{m},\ldots,(T_{m-1}\circ\cdots\circ T_{1})\overline{x}_{m},\overline{x}_{m})}~{}\big{|}~{}{\overline{x}_{m}\in F}\big{\}}.

Suppose that $(W_{i})_{1\leqslant i\leqslant m}$ satisfies Condition 3.1 for some $\alpha\in[1/2,1]$. Then $F$ is closed and convex.

Suppose that $(W_{i})_{1\leqslant i\leqslant m}$ satisfies Condition 3.1 for some $\alpha\in[1/2,1]$ and that one of the following holds:

$\text{\rm ran}\,(T_{m}\circ\cdots\circ T_{1})$ is bounded.

There exists $j\in\{1,\ldots,m\}$ such that $\text{\rm dom}\,\varphi_{j}$ is bounded.

Then $F$ and $\boldsymbol{F}$ are nonempty.

Suppose that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ is monotone. Then $\boldsymbol{F}$ is closed and convex. In addition, $F$ and $\boldsymbol{F}$ are nonempty if any of the following holds:

${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi}$ is surjective.

$\partial\boldsymbol{\varphi}-\boldsymbol{W}\circ\boldsymbol{S}$ is maximally monotone.

$\max_{1\leqslant i\leqslant m}\|W_{i}\|\leqslant 1$, $\boldsymbol{S}^{*}-\boldsymbol{W}$ has closed range, and $\ker(\boldsymbol{S}-\boldsymbol{W}^{*})=\{\boldsymbol{0}\}$.

$\max_{1\leqslant i\leqslant m}\|W_{i}\|\leqslant 1$ and, for every $i\in\{1,\ldots,m\}$, $\text{\rm dom}\,\varphi_{i}^{*}={\mathcal{H}}_{i}$.

For every $i\in\{1,\ldots,m\}$, $\text{\rm dom}\,\varphi_{i}={\mathcal{H}}$ and $\text{\rm dom}\,\varphi_{i}^{*}={\mathcal{H}}_{i}$.

$\boldsymbol{S}^{*}-\boldsymbol{W}$ has closed range, $\ker(\boldsymbol{S}-\boldsymbol{W}^{*})=\{\boldsymbol{0}\}$, and, for every $i\in\{1,\ldots,m\}$, $\text{\rm dom}\,\varphi_{i}={\mathcal{H}}_{i}$.

For every $i\in\{1,\ldots,m\}$, $\text{\rm dom}\,\varphi_{i}$ is bounded.

Proof. We first observe that $\boldsymbol{S}\in\mathcal{B}\,({\boldsymbol{\mathcal{H}}},\overset{\rightarrow}{{\boldsymbol{\mathcal{H}}}})$, $\boldsymbol{W}\in\mathcal{B}\,(\overset{\rightarrow}{{\boldsymbol{\mathcal{H}}}},{\boldsymbol{\mathcal{H}}})$, $\boldsymbol{\varphi}\in\Gamma_{0}({\boldsymbol{\mathcal{H}}})$, and $\boldsymbol{\psi}\in\Gamma_{0}({\boldsymbol{\mathcal{H}}})$.

(i): Let $\boldsymbol{x}\in{\boldsymbol{\mathcal{H}}}$. Then

(ii): Let $\boldsymbol{x}\in{\boldsymbol{\mathcal{H}}}$. Using (4.2), we obtain

(iii): Clear from the definitions of $F$ and $\boldsymbol{F}$.

(iv): Define $m$ firmly nonexpansive operators by $(\forall i\in\{1,\ldots,m\})$ $P_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}\colon y\mapsto R_{i}(y+b_{i})$. Then it follows from (4.1) and Theorem 3.8 applied to $(P_{i})_{1\leqslant i\leqslant m}$ that $T_{m}\circ\cdots\circ T_{1}$ is nonexpansive. In turn, we derive from [8, Corollary 4.24] that its fixed point set $F$ is closed and convex.

(v): Thanks to (iii), it is enough to show that $F\neq{\varnothing}$. Set $T=T_{m}\circ\cdots\circ T_{1}$ and recall that it is nonexpansive by virtue of Theorem 3.8.

(v)(a): Let $C$ be a closed ball such that $\text{\rm ran}\,T\subset C$ and set $S=T|_{C}$. Then $S\colon C\to C$ is nonexpansive and therefore [8, Proposition 4.29] asserts that $\text{\rm Fix}\,T=\text{\rm Fix}\,S\neq{\varnothing}$.

(v)(b)$\Rightarrow$(v)(a): We have $\text{\rm ran}\,T_{j}\subset\text{\rm ran}\,R_{j}=\text{\rm ran}\,\text{\rm prox}_{\varphi_{j}}=\text{\rm dom}\,(\operatorname{Id}+\partial\varphi_{j})=\text{\rm dom}\,\partial\varphi_{j}\subset\text{\rm dom}\,\varphi_{j}$. Hence $\text{\rm ran}\,T_{j}$ is bounded and Proposition 4.2 (with $i=m$) implies that

(vi): Set $\boldsymbol{A}={\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\psi}$. Since ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ is monotone and continuous, it is maximally monotone [8, Corollary 20.28], with ${\boldsymbol{\mathcal{H}}}$ as its domain. Since $\partial\boldsymbol{\psi}$ is also maximally monotone [8, Theorem 20.25], $\boldsymbol{A}$ is likewise [8, Corollary 25.5(i)] and hence $\boldsymbol{F}=\text{\rm zer}\,\boldsymbol{A}$ is closed and convex [8, Proposition 23.39]. Next, we note that, in view of (iii), $F\neq{\varnothing}$ $\Leftrightarrow$ $\boldsymbol{F}\neq{\varnothing}$.

(vi)(a): The hypothesis implies that $(b_{i})_{1\leqslant i\leqslant m}\in\text{\rm ran}\,({\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi})$ and therefore that (4.5) has a solution, i.e., $\boldsymbol{F}\neq{\varnothing}$.

(vi)(b)$\Rightarrow$(vi)(a): The claim follows from Minty’s theorem [8, Theorem 21.1].

(vi)(c)$\Rightarrow$(vi)(a): We have $\|\boldsymbol{W}\circ\boldsymbol{S}\|=\|\boldsymbol{W}\|=\max_{1\leqslant i\leqslant m}\|W_{i}\|\leqslant 1$. Therefore, $-\boldsymbol{W}\circ\boldsymbol{S}$ is nonexpansive, which implies that $({\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S})/2$ is firmly nonexpansive [8, Corollary 4.5], that is $(\forall\boldsymbol{x}\in{\boldsymbol{\mathcal{H}}})$ ${\left\langle{{\boldsymbol{x}-\boldsymbol{W}(\boldsymbol{S}\boldsymbol{x})}\mid{\boldsymbol{x}}}\right\rangle}\geqslant\|\boldsymbol{x}-\boldsymbol{W}(\boldsymbol{S}\boldsymbol{x})\|^{2}/2$. Consequently, ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ is $3^{*}$ monotone [8, Proposition 25.16], while $\partial\boldsymbol{\varphi}$ is also $3^{*}$ monotone [8, Example 25.13]. Finally, since $\boldsymbol{S}$ is unitary,

which shows that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ is surjective. Altogether, since [8, Corollary 25.5(i)] implies that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi}$ is maximally monotone, it follows from [8, Corollary 25.27(i)] that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi}$ is surjective.

(vi)(d)$\Rightarrow$(vi)(a): We have $\text{\rm dom}\,\boldsymbol{\varphi}^{*}={\boldsymbol{\mathcal{H}}}$. Hence since $\text{int\,dom}\,\boldsymbol{\varphi}^{*}\subset\text{\rm dom}\,\partial\boldsymbol{\varphi}^{*}$ [8, Proposition 16.27], we have $\text{\rm ran}\,\partial\boldsymbol{\varphi}=\text{\rm dom}\,(\partial\boldsymbol{\varphi})^{-1}=\text{\rm dom}\,\partial\boldsymbol{\varphi}^{*}={\boldsymbol{\mathcal{H}}}$. Hence, $\partial\boldsymbol{\varphi}$ is surjective. We conclude using the same arguments as in (vi)(c): $\partial\boldsymbol{\varphi}$ and ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ are both $3^{*}$ monotone and their sum is maximally monotone, which allows us to invoke [8, Corollary 25.27(i)].

(vi)(e)$\Rightarrow$(vi)(a): As seen in (vi)(d), $\partial\boldsymbol{\varphi}$ is surjective. We have ${\boldsymbol{\mathcal{H}}}=\text{int\,dom}\,\boldsymbol{\varphi}\subset\text{\rm dom}\,\partial\boldsymbol{\varphi}$ [8, Proposition 16.27]. Consequently, ${\boldsymbol{\mathcal{H}}}=\text{\rm dom}\,({\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S})\subset\text{\rm dom}\,\partial\boldsymbol{\varphi}$. Altogether, since $\partial\boldsymbol{\varphi}$ is $3^{*}$ monotone, it follows from [8, Corollary 25.27(ii)] that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi}$ is surjective.

(vi)(f)$\Rightarrow$(vi)(a): As seen in (vi)(c), ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ is surjective and $\partial\boldsymbol{\varphi}$ is $3^{*}$ monotone. In addition, $\text{\rm dom}\,({\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S})\subset\text{\rm dom}\,\partial\boldsymbol{\varphi}$ since ${\boldsymbol{\mathcal{H}}}=\text{int\,dom}\,\boldsymbol{\varphi}\subset\text{\rm dom}\,\partial\boldsymbol{\varphi}$ [8, Proposition 16.27]. Altogether, it follows from [8, Corollary 25.27(ii)] that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}+\partial\boldsymbol{\varphi}$ is surjective.

(vi)(g): Here \text{\rm dom}\,\boldsymbol{A}=\text{\rm dom}\,\partial\boldsymbol{\varphi}\subset\text{\rm dom}\,\boldsymbol{\varphi}=\raisebox{-1.42262pt}{\mbox{\LARGE{\times}}}_{\!i=1}^{\!m}\text{\rm dom}\,\varphi_{i} is bounded. Hence, $\boldsymbol{F}=\text{\rm zer}\,\boldsymbol{A}\neq{\varnothing}$ [8, Proposition 23.36(iii)].

In Proposition 4.3(vi), it is required that ${\boldsymbol{\operatorname{Id}}\,}-\boldsymbol{W}\circ\boldsymbol{S}$ be monotone, or equivalently, that its self-adjoint part ${\boldsymbol{\operatorname{Id}}\,}-(\boldsymbol{W}\circ\boldsymbol{S}+\boldsymbol{S}^{*}\circ\boldsymbol{W}^{*})/2$ be positive. In a finite-dimensional setting, this just means that the eigenvalues of the matrix $\boldsymbol{W}\boldsymbol{S}+\boldsymbol{S}^{*}\boldsymbol{W}^{*}$ are in $\left]{-\infty},2\right]$.

where, given $x\in{\mathcal{H}}_{i-1}$, $[W_{i}x]_{k}$ is the $k$th component of $W_{i}x$ and

Altogether, we conclude that $\boldsymbol{F}$ is a closed convex polyhedron.

2 Asymptotic analysis

Next, we investigate the asymptotic behavior of (1.2) in the context of Model 4.1.

In the setting of Model 4.1, set $T=T_{m}\circ\cdots\circ T_{1}$, let $\alpha\in[1/2,1]$, and suppose that the following hold:

$(W_{i})_{1\leqslant i\leqslant m}$ satisfies Condition 3.1 with parameter $\alpha$.

$\lambda_{n}\equiv 1/\alpha=1$ and $Tx_{n}-x_{n}\to 0$.

For every $i\in\{1,\ldots,m-1\}$, $R_{i}$ is weakly sequentially continuous.

For every $i\in\{1,\ldots,m-1\}$, $R_{i}$ is a separable activation operator in the sense of Proposition 2.24.

For every $i\in\{1,\ldots,m-1\}$, ${\mathcal{H}}_{i}$ is finite-dimensional.

Proof. We first derive from (1.2) and Model 4.1 that

Now set $(\forall i\in\{1,\ldots,m\})$ $P_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}\colon y\mapsto R_{i}(y+b_{i})$. Then (4.1) yields $T=P_{m}\circ W_{m}\circ\cdots\circ P_{1}\circ W_{1}$ and, since the operators $(R_{i})_{1\leqslant i\leqslant m}$ are firmly nonexpansive, the operators $(P_{i})_{1\leqslant i\leqslant m}$ are likewise. Hence, it follows from (b), Theorem 3.8, and (4.2) that

We now prove the convergence of the individual sequences under each assumption.

(iii): We have already established that $x_{n}\>\rightharpoonup\>\overline{x}_{m}$. Since $W_{1}$ is weakly continuous as a bounded linear operator, so is $T_{1}$ in (4.1). Hence, (1.2) implies that $x_{1,n}=T_{1}x_{n}\>\rightharpoonup\>T_{1}\overline{x}_{m}=\overline{x}_{1}$. Likewise, we obtain successively $x_{2,n}=T_{2}x_{1,n}\>\rightharpoonup\>T_{2}\overline{x}_{1}=\overline{x}_{2}$, $x_{3,n}=T_{3}x_{2,n}\>\rightharpoonup\>T_{3}\overline{x}_{2}=\overline{x}_{3}$,…, $x_{m,n}=T_{m}x_{m-1,n}\>\rightharpoonup\>T_{m}\overline{x}_{m-1}=\overline{x}_{m}$.

(iv)$\Rightarrow$(iii): See [8, Proposition 24.12(iii)].

(v)$\Rightarrow$(iii): A proximity operator is nonexpansive and therefore continuous, hence weakly continuous in a finite-dimensional setting.

(vi): As shown above, $x_{n}\>\rightharpoonup\>\overline{x}_{m}\in F$. It follows from Proposition 3.6(iii) and Theorem 3.8 (applied with $m=1$) that, for every $i\in\{1,\ldots,m\}$, $T_{i}$ is $\beta_{i}$-averaged. Hence, upon applying [24, Theorem 3.5(ii)] with $\alpha$ as an averaging constant of $T$, we infer that

Thus, $x_{1,n}-x_{n}=T_{1}x_{n}-x_{n}\to T_{1}\overline{x}_{m}-\overline{x}_{m}$, which implies that $x_{1,n}=(x_{1,n}-x_{n})+x_{n}\>\rightharpoonup\>(T_{1}\overline{x}_{m}-\overline{x}_{m})+\overline{x}_{m}=T_{1}\overline{x}_{m}$. However, since $x_{2,n}-x_{1,n}=(T_{2}\circ T_{1})x_{n}-T_{1}x_{n}\to(T_{2}\circ T_{1})\overline{x}_{m}-T_{1}\overline{x}_{m}$, we obtain $x_{2,n}\>\rightharpoonup\>(T_{2}\circ T_{1})\overline{x}_{m}$. Continuing this telescoping process yields the claim.

The next result covers the case when the variational inequality problem (4.5) has no solution.

To model closely existing deep neural networks, we have chosen the activation operators in Definition 2.20 and Model 4.1 to be proximity operators. However, as is clear from the results of Section 3 and in particular the central Theorem 3.8, an activation operator $R_{i}\colon{\mathcal{H}}_{i}\to{\mathcal{H}}_{i}$ could more generally be a firmly nonexpansive operator that admits as a fixed point. By [8, Corollary 23.9], this means that $R_{i}$ is the resolvent of some maximally monotone operator such $A_{i}\colon{\mathcal{H}}_{i}\to 2^{{\mathcal{H}}_{i}}$ (i.e., $R_{i}=(\operatorname{Id}+A_{i})^{-1}$) such that $0\in A_{i}0$. In this context, the variational inequality (4.5) assumes the more general form of a system of monotone inclusions, namely,

Analysis of nonperiodic networks

We analyze the deep neural network described in Model 1.1 in the following scenario.

In the setting of Model 1.1, suppose that Assumption 5.1 is satisfied, let $i\in\{1,\ldots,m\}$, and set

We can now present the main result of this section on the asymptotic behavior of Model 1.1. The proof of this result relies on Theorem 4.7, which it extends.

Consider the setting of Model 1.1 and let $\alpha\in[1/2,1]$. Suppose that Assumption 5.1 is satisfied as well as the following:

$F=\text{\rm Fix}\,T\neq{\varnothing}$, where $T=T_{m}\circ\cdots\circ T_{1}$.

$(W_{i})_{1\leqslant i\leqslant m}$ satisfies Condition 3.1 with parameter $\alpha$.

$\lambda_{n}\equiv\alpha=1$ and $Tx_{n}-x_{n}\to 0$.

For every $i\in\{1,\ldots,m-1\}$, $R_{i}$ is weakly sequentially continuous.

For every $i\in\{1,\ldots,m-1\}$, $R_{i}$ is a separable activation function in the sense of Proposition 2.24.

For every $i\in\{1,\ldots,m-1\}$, ${\mathcal{H}}_{i}$ is finite-dimensional.

In (c)(i)–(c)(ii) above, Proposition 4.3(iii) ensures that $(T_{1}\overline{x}_{m},(T_{2}\circ T_{1})\overline{x}_{m},\ldots,(T_{m-1}\circ\cdots\circ T_{1})\overline{x}_{m},\overline{x}_{m})$ solves (4.5).

(vi): For every $i\in\{1,\ldots,m\}$, set

Thus, since Proposition 3.6(iii) and Theorem 3.8 imply that the operators $(T_{i})_{1\leqslant i\leqslant m}$ are averaged, the proof can be completed as that of Theorem 4.7(vi) since [24, Theorem 3.5(ii)] asserts that (4.15) remains valid under (5.21).

References