Pathological spectra of the Fisher information metric and its variants in deep neural networks

Introduction

Deep neural networks (DNNs) have outperformed many standard machine-learning methods in practical applications . Despite their practical success, many theoretical aspects of DNNs remain to be uncovered, and there are still many heuristics used in deep learning. We need a solid theoretical foundation for elucidating how and under what conditions DNNs and their learning algorithms work well.

The Fisher information matrix (FIM) is a fundamental metric tensor that appears in statistics and machine learning. An empirical FIM is equivalent to the Hessian of the loss function around a certain global minimum, and it affects the performance of optimization in machine learning. In information geometry, the FIM defines the Riemannian metric tensor of the parameter manifold of a statistical model . The natural gradient method is a first-order gradient method in the Riemannian space where the FIM works as its Riemannian metric . The FIM also acts as a regularizer to prevent catastrophic forgetting ; a DNN trained on one dataset can learn another dataset without forgetting information if the parameter change is regularized with a diagonal FIM.

However, our understanding of the FIM for neural networks has so far been limited to empirical studies and theoretical analyses of simple networks. Numerical experiments empirically confirmed that the eigenvalue spectra of the FIM and those of the Hessian are highly distorted; that is, most eigenvalues are close to zero, while others take on large values . Focusing on shallow neural networks, Pennington and Worah theoretically analyzed the FIM’s eigenvalue spectra by using random matrix theory, and Fukumizu derived a condition under which the FIM becomes singular. Liang et al. have connected FIMs to the generalization ability of DNNs by using model complexity, but their results are restricted to linear networks. Thus, theoretical evaluations of deeply nonlinear cases seem to be difficult mainly because of iterated nonlinear transformations. To go one step further, it would be helpful if a framework that is widely applicable to various DNNs could be constructed.

Investigating DNNs with random weights has given promising results. When such DNNs are sufficiently wide, we can formulate their behavior by using simpler analytical equations through coarse-graining of the model parameters, as is discussed in mean field theory and random matrix theory . For example, Schoenholz et al. proposed a mean field theory for backpropagation in fully-connected DNNs. This theory characterizes the amplitudes of gradients by using specific quantities, i.e., order parameters in statistical physics, and enables us to quantitatively predict parameter regions that can avoid vanishing or explosive gradients. This theory is applicable to a wide class of DNNs with various non-linear activation functions and depths. Such DNNs with random weights are substantially connected to Gaussian process and kernel methods . Furthermore, the theory of the neural tangent kernel (NTK) explains that even trained parameters are close enough to the random initialization in sufficiently wide DNNs and the performance of trained DNNs is determined by the NTK on the initialization .

Karakida et al. focused on the FIM corresponding to the mean square error (MSE) loss and proposed a framework to express certain eigenvalue statistic by using order parameters. They revealed that when fully-connected networks with random initialization are sufficiently wide, the FIM’s eigenvalue spectrum asymptotically becomes pathologically distorted. As the network width increases, a small number of the eigenvalues asymptotically take on huge values and become outliers while the others are much smaller. The distorted shape of the eigenvalue spectrum is consistent with empirical reports . While LeCun et al. implied that such pathologically large eigenvalue might appear in multi-layered networks and affect the training dynamics, its theoretical elucidation has been limited to a data covariance matrix in a linear regression model. The results of can be regarded as a theoretical verification of this large eigenvalue suggested by . The obtained eigenvalue statistics have given insight into the convergence of gradient dynamics , mechanism of batch normalization to decrease the sharpness of the loss function , and generalization measure of DNNs based on the minimum description length .

In this paper, we extend the framework of the previous work and reveal that various types of FIMs and variants show pathological spectra. Our main contribution is the following:

FIM for classification tasks with softmax output: While the previous works analyzed the FIM for regression based on the MSE loss, we typically use the cross-entropy loss with softmax output in classification tasks. We analyze this FIM for classification tasks and reveal that its spectrum is pathologically distorted as well. While the FIM for regression tasks has unique and degenerated outliers in the infinite-width limit, the softmax output can make these outliers disperse and remove the degeneracy. Our theory shows that there are number-of-classes outlier eigenvalues, which is consistent with experimental reports . Experimental results demonstrate that the eigenvalue density has a tail of outliers spreading form the bulk.

Furthermore, we also give a unified perspective on the variants:

Diagonal Blocks of FIM: We give a detailed analysis of the diagonal block parts of the FIM for regression tasks. Natural gradient algorithms often use a block diagonal approximation of the FIM . We show that the diagonal blocks also suffer from pathological spectra.

Connection to NTK: The NTK and FIM inherently share the same non-zero eigenvalues. Paying attention to a specific re-scaling of the parameters assumed in studies of NTK, we clarify that NTK’s eigenvalue statistics become independent of the width scale. Instead, the gap between the average and maximum eigenvalues increases with the sample size. This suggests that, as the sample size increases, the training dynamics converge non-uniformly and that calculations with the NTK become ill-conditioned. We also demonstrate a simple normalization method to make eigenvalue statistics that are independent of both the width and the sample size.

Metric tensors for input and feature spaces: We consider metric tensors for input and feature spaces spanned by neurons in input and hidden layers. These metric tensors potentially enable us to evaluate the robustness of DNNs against perturbations in the input and feedforward propagated signals. We show that the spectrum is pathologically distorted, similar to FIMs, in the sense that the outlier of the spectrum is much far from most of the eigenvalues. The softmax output makes the outliers disperse as well.

In summary, this study sheds light into the asymptotical eigenvalue statistics common to various wide networks.

Preliminaries

We investigated the fully-connected feedforward neural network shown in Fig. 1. The network consists of one input layer, $L-1$ hidden layers ($l=1,...,L-1$), and one output layer. It includes shallow nets ($L=2$) and arbitrary deep nets ($L\geq 3$). The network width is denoted by $M_{l}$. The pre-activations $u^{l}_{i}$ and activations of units $h_{i}^{l}$ in the $l$-th layer are defined recursively by

and consider the limiting case of a sufficiently large $M$ with constant coefficients $\alpha_{l}>0$. The number of output units is taken to be a constant $C$, as is usually done in practice. We denote the linear output of the last layer by

We also investigate DNNs with softmax outputs in Section 3.3. The $C$-dimensional softmax function is given by

To avoid complicating the notation, we will omit the index $k$ of the output unit, i.e., $\delta_{i}^{l}=\delta_{k,i}^{l}$. To evaluate the above feedforward and backward signals, we assume the following conditions.

Random weights and biases: Suppose that the parameter set is an ensemble generated by

and thus is fixed, where $\mathcal{N}(0,\sigma^{2})$ denotes a Gaussian distribution with zero mean and variance $\sigma^{2}$. Treating the case in which different layers have different variances is straightforward. Note that the variances of the weights are scaled in the order of $1/M$. In practice, the learning of DNNs usually starts from random initialization with this scaling .

Activation functions: Suppose the following two conditions: (i) the activation function $\phi(x)$ has a polynomially bounded weak derivative. (ii) the network is non-centered, which means a DNN with bias terms ($\sigma_{b}\neq 0$) or activation functions satisfying a non-zero Gaussian mean. The definition of the non-zero Gaussian mean is $\int Dz\phi(z)\neq 0$. The notation $Du=du\exp(-u^{2}/2)/\sqrt{2\pi}$ means integration over the standard Gaussian density.

Condition (i) is used to obtain recurrence relations of backward order parameters . Condition (ii) plays an essential role in our evaluation of the FIM . The two conditions are valid in various realistic settings, because conventional networks include bias terms, and widely used activation functions, such as the sigmoid function and (leaky-) ReLUs, have bounded weak derivatives and non-zero Gaussian means. Different layers may have different activation functions.

2 Overview of metric tensors

We will analyze two types of metric tensors (metric matrices) that determine the responses of network outputs, i.e., the response to a local change in parameters and the response to a local change in the input and hidden neurons. They are summarized in Fig. 2. One can systematically understand these tensors from the perspective of perturbations of variables.

where we have abbreviated the network outputs as $f_{k}(n)=f_{k}(x(n);\theta)$ to avoid complicating the notation. This is an empirical FIM in the sense that the average is computed over empirical input. We can express it in the matrix form shown in Fig. 2(a). The Jacobian $\nabla_{\theta}f$ is a $P\times CN$ matrix whose each column corresponds to $\nabla_{\theta}f_{k}(n)$ ($k=1,...,C$, $n=1,...,N$). We investigate this type of empirical metric tensor for arbitrary $N$. One can set $N$ as a constant value or make it increase depending on $M$. The empirical FIM (10) converges to the expected FIM as $N\rightarrow\infty$. In addition, the FIM can be partitioned into $L^{2}$ layer-wise block matrices. We denote the $(l,l^{\prime})$-th block as $F^{ll^{\prime}}$ ($l,l^{\prime}=1,...,L$). We take a closer look at the eigenvalue statistics of diagonal blocks in Section 3.2.

In this paper, we also investigate another FIM denoted by $F_{cross}$ which corresponds to classification tasks with cross-entropy loss. As shown in Fig. 2(a), one can represent $F_{cross}$ as a modification of $F$. A specific coefficient matrix $Q$ is inserted between the Jacobian $\nabla_{\theta}f$ and its transpose. $Q$ is defined by (25) and composed of nothing but softmax functions. Its mathematical definition is given in Section 3.3. One more interesting quantity is a left-to-right reversed product of $\nabla_{\theta}f$ described in Fig. 2 (b). This matrix is known as the neural tangent kernel (NTK). The FIM and NTK share the same non-zero eigenvalues by definition, although we need to be careful in the change of parameterization used in the studies of NTK. The details are shown in Section 4.

We refer to $A$ as the metric tensor for the input and feature spaces because each $h^{l}$ acts as the input to the next layer and corresponds to the features realized in the network. We can also deal with a layer-wise diagonal block of $A$. Let us denote the $(l,l^{\prime})$-th block by $A^{ll^{\prime}}$ ($l,l^{\prime}=0,...,L-1$). In particular, the first diagonal block $A^{00}$ indicates the robustness of the network output against perturbation of the input:

Robustness against input noise has been investigated by similar (but different) quantities, such as sensitivity , and robustness against adversarial examples .

3 Order parameters for wide neural networks

where $h_{i}^{l}(n)$ is the output of the $l$-th layer generated by the $n$-th input sample $x(n)$ ($n=1,...,N$). The variable $\hat{q}^{l}_{{1}}$ describes the total activity in the $l$-th layer, and the variable $\hat{q}^{l}_{{2}}$ describes the overlap between the activities for different input samples $x(n)$ and $x(m)$. These variables have been utilized to describe the depth to which signals can propagate from the perspective of order-to-chaos phase transitions . In the large $M$ limit, these variables can be recursively computed by integration over Gaussian distributions :

for $l=0,...,L-1$. Because the input samples generated by Eq. (7) yield $\hat{q}^{0}_{{1}}=1$ and $\hat{q}_{{2}}^{0}=0$ for all $n$ and $m$, $\hat{q}_{{2}}^{l}$ in each layer takes the same value for all $n\neq m$; so does $\hat{q}_{{1}}^{l}$ for all $n$. A two-dimensional Gaussian integral is given by

with $c=b/a$. One can represent this integral in a bit simpler form, i.e., $I_{\phi}[a,b]=\int Dy(\int Dx\phi(\sqrt{a-b}x+\sqrt{b}y))^{2}.$

Next, let us define the following variables for backpropagated signals:

These order parameters depend only on the type of activation function, depth, and the variance parameters $\sigma_{w}^{2}$ and $\sigma_{b}^{2}$. The recurrence relations for the order parameters require $L$ iterations of one- and two-dimensional numerical integrals. Moreover, we can obtain explicit forms of the recurrence relations for some of the activation functions .

Eigenvalue statistics of FIMs

This section shows the asymptotic eigenvalue statistics of the FIMs. When we have an $P\times P$ metric tensor whose eigenvalues are $\lambda_{i}$ ($i=1,...,P$), we compute the following quantities:

The obtained results are universal for any sample size $N$ which may depend on $M$.

This subsection overviews the results obtained in the previous studies . The metric tensor $F$ is equivalent to the Fisher information matrix (FIM) , originally defined by

Basically, there are two types of FIM for supervised learning, depending on the definition of the statistical model. One type corresponds to the mean squared error (MSE) loss for regression tasks; the other corresponds to the cross-entropy loss for classification tasks. The latter is discussed in Section 3.3. Let us consider the following statistical model for the regression:

The previous studies uncovered the following eigenvalue statistics of the FIM (10):

When $M$ is sufficiently large, the eigenvalue statistics of $F$ can be asymptotically evaluated as

where $\alpha:=\sum_{l=1}^{L-1}\alpha_{l}\alpha_{l-1}$, and positive constants $\kappa_{1}$ and $\kappa_{2}$ are obtained using order parameters,

The average of the eigenvalue spectrum asymptotically decreases in the order of $1/M$, while the variance takes a value of $O(1)$ and the largest eigenvalue takes a huge value of $O(M)$. It implies that most of the eigenvalues are asymptotically close to zero, while the number of large eigenvalues is limited. Thus, when the network is sufficiently wide, one can see that the shape of the spectrum asymptotically becomes pathologically distorted. This suggests that the parameter space of the DNNs is locally almost flat in most directions but highly distorted in a few specific directions.

In particular, regarding the large eigenvalues, we have

When $M$ is sufficiently large, the eigenspace corresponding to $\lambda_{max}$ is spanned by $C$ eigenvectors,

When $N=\rho^{-1}M$ with a constant $\rho>0$, under the gradient independence assumption, the second largest eigenvalue $\lambda_{max}^{\prime}$ is bounded by

for non-negative constants $c_{1}$ and $c_{2}$.

The subtraction (22) means eliminating the $C$ largest eigenvalues from $F$. Numerical experiments confirmed that the largest eigenvalue of $\bar{F}$ is of order 1. Note that when $N\propto M$, the sample size is sufficiently large but the network satisfies $P\gg N$ and keeps overparameterized.

Figure 3 (a; left) shows a typical spectrum of the FIM. We computed the eigenvalues by using random Gaussian weights, biases, and inputs. We used deep Tanh networks with $L=3$, $M=200$, $C=10$, $\alpha_{l}=1$ and $(\sigma_{w}^{2},\sigma_{b}^{2})=(3,0.64)$. The sample size was $N=100$. The histograms were made from eigenvalues over $100$ different networks with different random seeds. The histogram had two populations. The red dashed histogram was made by eliminating the largest $C$ eigenvalues. It coincides with the smaller population. Thus, one can see that the larger population corresponds to the $C$ largest eigenvalues. The larger population in experiments can be distributed around $\lambda_{max}$ because $M$ is large but finite.

Remark : Loss landscape and gradient methods. The empirical FIM (10) is equivalent to the Hessian of the loss, i.e., $\nabla_{\theta}^{2}Loss(\theta)$, around the global minimum with zero training loss. Karakida et al. referred to the steep shape of the local loss landscape caused by $\lambda_{max}$ as pathological sharpness. The sharpness of the loss landscape is connected to an appropriate learning rate of gradient methods for convergence. The previous work empirically confirmed that a learning rate $\eta$ satisfying

is necessary for the steepest gradient method to converge . In fact, this $2/\lambda_{max}$ acts as a boundary of neural tangent kernel regime . Because $\lambda_{max}$ increases depending on the width and depth, we need to carefully choose an appropriately scaled learning rate to train the DNNs.

2 Diagonal blocks of FIM

It is helpful to investigate the relation between $F$ and its diagonal blocks when one considers the diagonal block approximation of $F$. For example, use of a diagonal block approximation can decrease the computational cost of natural gradient algorithms . When a matrix is composed only of diagonal blocks, its eigenvalues are given by those of each diagonal block. $F$ approximated in this fashion has the same mean of the eigenvalues as the original $F$ and the largest eigenvalue $\max_{l}\lambda_{max}^{l}$, which is of $O(M)$. Thus, the diagonal block approximation also suffers from a pathological spectrum. Eigenvalues that are close to zero can make the inversion of the FIM in natural gradient unstable, whereas using a damping term seems to be an effective way of dealing with this instability .

3 FIM for multi-label classification tasks

The cross-entropy loss is typically used in multi-label classification tasks. It comes from the log-likelihood of the following statistical model:

The contribution of softmax output appears only in $Q_{n}$. $F_{cross}$ is linked to $F$ through the matrix representation shown in Fig. 2(a). One can view $F$ as a matrix representation with $Q=I$, that is, the identity matrix. In contrast, $F_{cross}$ corresponds to a block-diagonal $Q$ whose $n$-th block is given by the $C\times C$ matrix $Q_{n}$. In a similar way to Eq. (8), we can see $F_{cross}$ as the metric tensor for the parameter space. Using the softmax output $g$, we have

where the square root is taken entry-wise.

We obtain the following result of $F_{cross}$’s eigenvalue statistics:

When $M$ is sufficiently large, the eigenvalue statistics of $F_{cross}$ are asymptotically evaluated as

where the constant coefficients are given by

The derivation is shown in Appendix B.1. We find that the eigenvalue spectrum shows the same width dependence as the FIM for regression tasks. Although the evaluation of $\lambda_{max}$ in Theorem 3.3 is based on inequalities, one can see that $\lambda_{max}$ linearly increases as the width $M$ or the depth $L$ increase. The softmax functions appear in the coefficients $\beta_{k}$. It should be noted that the values of $\beta_{k}$ generally depend on the index $k$ of each softmax output. This is because the values of the softmax functions depend on the specific configuration of $W^{L}$ and $b^{L}$.

If a relatively loose bound is acceptable, we can use the following simpler evaluation. Let us denote the eigenvalue statistics of $F$ shown in Theorem 3.1 by $(m_{\lambda}^{lin},s_{\lambda}^{lin},\lambda_{max}^{lin})$. They corresponds to the contribution of linear output $f$ before putting it into the softmax function. Taking into account the contribution of the softmax function, we have

Note that $F_{cross}$’s eigenvalues satisfy $\lambda_{i}(F_{cross})\leq\lambda_{max}(Q)\lambda_{i}(F)$ ($i=1,...,P$; $\lambda_{1}\geq\lambda_{2}\geq...\geq\lambda_{P}$). The inequality (28) comes from $\lambda_{max}(Q)\leq 1$.

Figure 4 shows that our theory predicts experimental results rather well for artificial data. We computed the eigenvalues of $F_{cross}$ with random Gaussian weights, biases, and inputs. We set $L=3$, $M=1000$, $C=10$, $\alpha_{l}=1$ and ($\sigma_{w}^{2},\sigma_{b}^{2}$) = ($3,0.64$) in the tanh case, ($2,0.1$) in the ReLU case, and ($1,0.1$) in the linear case. The sample size was set to $N=100$. The predictions of Theorem 3.3 coincided with the experimental results for sufficiently large widths.

Exhaustive experiments on the cross-entropy loss have recently confirmed that there are $C$ dominant large eigenvalues (so-called outliers) . Consistent with the results of this experimental study, we found that there are $C$ large eigenvalues:

$F_{cross}$ has the first $C$ largest eigenvalues of $O(M)$.

The theorem is proved in Appendix B.2. These $C$ large eigenvalues are reminiscent of the $C$ largest eigenvalues of $F$ shown in Theorem 3.1.

These $C$ largest eigenvalues can act as outliers. Note that because $\lambda_{i}(F_{cross})\leq\lambda_{i}(F)$, we have $\lambda_{C+1}\leq\lambda_{max}^{\prime}$. $\lambda_{C+1}$ denotes the $(C+1)$-th largest eigenvalue of $F_{cross}$. Let us suppose that $N\propto M$ and the assumptions of Theorem 3.2 hold. In this case, $\lambda_{max}^{\prime}$ is upper-bounded by (21) and then $\lambda_{C+1}$ is of $O(\sqrt{M})$ at most. This means that the first $C$ largest eigenvalues of $F_{cross}$ can become outliers. It would be interesting to extend the above results and theoretically quantify more precise values of these outliers. One promising direction will be to analyze a hierarchical structure of $F_{cross}$ empirically investigated by Papyan . It is also noteworthy that our outliers disappear under the mean subtraction of $f$ in the last layer mentioned in (22). This is because we have $\lambda_{i}(F_{cross})\leq\lambda_{i}(\bar{F})$ under the mean subtraction.

Figure 3(b; left) shows a typical spectrum of $F_{cross}$. We set $M=1000$ and other settings were the same as in the case of $F$. We found that compared to $F$, $F_{cross}$ had the largest $C(=10)$ eigenvalues which were widely spread from the bulk of the spectrum. Naively speaking, this is because the coefficient matrix $Q$ has distributed eigenvalues as is shown in Fig. 3(b; right). Compared to the FIM for regression, which corresponds to $Q=I$, the distributed $Q$’s eigenvalues can make $F_{cross}$’s eigenvalues disperse. Note that the black histogram is obtained as the summation over $100$ trials (different random initializations). The largest eigenvalues in a single trial are shown as blue boxes. The diagonal block $F_{cross}$ also has the same characteristics of the spectrum as the original $F_{cross}$ in Fig. 3(b; middle).

Although this work mainly focuses on DNNs with random weights, it also gives some insight into the training of sufficiently wide neural networks. It is known that the whole training dynamics of gradient descent can be explained by NTK in sufficiently wide neural networks. See Section 4 for more details. In the case of the cross-entropy loss, its functional gradient is given by $Q\Theta(y-g)$ . $\Theta$ is the NTK at random initialization described in (29). $Q$ depends on the softmax function $g$ at each time step. One can numerically solve the training dynamics of $g$ and obtain the theoretical value of the training loss . In Fig. 5(left), we confirmed that the theoretical line coincided well with the experimental results of gradient descent training. We set random initialization by $(\sigma_{w}^{2},\sigma_{b}^{2})=(2,0)$. We used artificial data with Gaussian inputs ($N=100$) and generated their labels by a teacher network whose architecture was the same as the trained network. In addition, Figure 5(right) shows the largest eigenvalues during the training. As is expected from NTK theory, $\lambda_{max}^{lin}$ kept unchanged during the training. In contrast, the $F_{cross}$’s largest eigenvalue dynamically changed because $Q$ depends on the softmax output $g$ which changes with the scale of $O(1)$. We calculated theoretical bounds of $\lambda_{max}$ (blue lines) by substituting $g$ at each time step into Theorem 3.3. Note that the bounds obtained in Theorem 3.3 are available even in the NTK regime because the coefficients $\beta_{k}$ admit any value of $g$ and are not limited to the random initialization. These theoretical bounds explained well the experimental results of $\lambda_{max}$ during the training.

Because the global minimum is given by $g=y$, all entries of $Q$ and $\lambda_{max}$ approach zero after a large enough number of steps. In the MSE case, we can explain the critical learning rate for convergence as is remarked in (23). In contrast, it is challenging to estimate such a critical learning rate in the cross-entropy case since $Q$ and $F_{cross}$ dynamically change.

Connection to Neural Tangent Kernel

The empirical FIM (10) is essentially connected to a recently proposed Gram matrix, i.e., the Neural Tangent Kernel (NTK). Jacot et al. defined the NTK by

Note that the Jacobian $\nabla_{\theta}f$ is a $P\times CN$ matrix whose each column corresponds to $\nabla_{\theta}f_{k}(n)$ ($k=1,...,C$, $n=1,...,N$). Under certain conditions with sufficiently large $M$, the NTK at random initialization governs the whole training process in the function space by

where the notation $t$ corresponds to the time step of the parameter update and $\eta$ represents the learning rate. Specifically, NTK’s eigenvalues determine the speed of convergence of the training dynamics. Moreover, one can predict the network output on the test samples by using the Gaussian process with the NTK .

The NTK and empirical FIM share essentially the same non-zero eigenvalues. It is easy to see that one can represent the empirical FIM (10) by $F=\nabla_{\theta}f\nabla_{\theta}f^{\top}/N$. This means that the NTK (29) is the left-to-right reversal of $F$ up to the constant factor $1/N$. Karakida et al. introduced $F^{*}=\nabla_{\theta}f^{\top}\nabla_{\theta}f/N$, which is essentially the same as the NTK, and referred to $F^{*}$ as the dual of $F$. They used $F^{*}$ to derive Theorem 3.1.

It should be noted that the studies of NTK typically suppose a special parameterization different from the usual setting. They consider DNNs with a parameter set $\theta=\{\omega_{ij}^{l},\beta_{i}^{l}\}$ which determines weights and biases by

This NTK parameterization changes the scaling of Jacobian. For instance, we have $\nabla_{\omega^{l}_{ij}}f=\frac{\sigma_{w}}{\sqrt{M_{l-1}}}\nabla_{W^{l}_{ij}}f$. This makes the eigenvalue statistics slightly change from those of Theorem 3.1. When $M$ is sufficiently large, the eigenvalue statistics of $\Theta$ under the NTK parameterization are asymptotically evaluated as

The positive constants $\kappa_{1}$ and $\kappa_{2}$ are obtained using order parameters,

The derivation is given in Appendix C. The NTK parameterization makes the eigenvalue statistics independent of the width scale. This is because the NTK parameterization maintains the scale of the weights but changes the scale of the gradients with respect to the weights. It also makes ($\kappa_{1},\kappa_{2}$) shift to ($\kappa_{1}^{\prime},\kappa_{2}^{\prime}$). This shift occurs because the NTK parameterization makes the order of the weight gradients $\nabla_{\omega}f$ comparable to that of the bias gradients $\nabla_{\beta}f$. The second terms in $\kappa_{1}^{\prime}$ and $\kappa_{2}^{\prime}$ correspond to a non-negligible contribution from $\nabla_{\beta}f$.

While $m_{\lambda}$ is independent of the sample size $N$, $\lambda_{max}$ depends on it. This means that the NTK dynamics converge non-uniformly. Most of the eigenvalues are relatively small and the NTK dynamics converge more slowly in the corresponding eigenspace. In addition, a prediction made with the NTK requires the inverse of the NTK to be computed . When the sample size is large, the condition number of the NTK, i.e., $\lambda_{max}/\lambda_{min}$, is also large and the computation with the inverse NTK is expected to be numerically inaccurate.

2 Scale-independent NTK

A natural question is under what condition do NTK’s eigenvalue statistics become independent of both the width and the sample size? As indicated in Eq. (22), the mean subtraction in the last layer with $N\propto M$ is a simple way to make the FIM’s largest eigenvalue independent of the width. Similarly, one can expect that the mean subtraction makes the NTK’s largest eigenvalue of $O(N)$ disappear and the eigenvalue spectrum take a range of $O(1)$ independent of the width and sample size.

Figure 5 empirically confirms this speculation. We set $L=3$, $C=2$, $\alpha_{l}=1$ and used the Gaussian inputs and weights with $(\sigma_{w}^{2},\sigma_{b}^{2})=(2,0)$. As shown in Fig. 5 (left), NTK’s eigenvalue spectrum becomes pathologically distorted as the sample size increases. To make an easier comparison of the spectra, the eigenvalues in this figure are normalized by $1/N$. As the sample size increases, most of the eigenvalues concentrate close to zero while the largest eigenvalues become outliers. In contrast, Figure 5 (right) shows that the mean subtraction keeps NTK’s whole spectrum in the range of $O(1)$ under the condition $N\propto M$. The spectrum empirically converged to a fixed distribution in the large $M$ limit.

Metric tensor for input and feature spaces

When $M$ is sufficiently large, the eigenvalue statistics of $A_{k}$ are asymptotically evaluated as

Figure 7(a; left) shows typical spectra of $A$ and Figure 7(a; right) those of $A^{00}$. We used deep Tanh networks with $M=500$ and $N=1000$. The other experimental settings are the same as those in Fig. 3(a). The pathological spectra appear as the theory predicts. Similarly, we show spectra of softmax output in Fig. 7(b). The softmax output made the outliers widely spread in the same manner as in $F_{cross}$.

Let us remark on some related works in the literature of deep learning. First, Pennington et al. investigated similar but different matrices. Briefly speaking, they used random matrix theory and obtained the eigenvalue spectrum of $\nabla_{u^{l}}f$ with $N=1$, $M_{l}=C=M$. They found that the isometry of the spectrum is helpful to solve the vanishing gradient problem. Second, DNNs are known to be vulnerable to a specific noise perturbation, i.e., the adversarial example . One can speculate that the eigenvector corresponding to $\lambda_{max}$ may be related to adversarial attacks, although such a conclusion will require careful considerations.

Discussion

We evaluated the asymptotic eigenvalue statistics of the FIM and its variants in sufficiently wide DNNs. They have pathological spectra in the conventional setting of random initialization and activation functions. This suggests that we need to be careful about the eigenvalue statistics and their influence on the learning when we use large-scale deep networks in naive settings.

The current work focused on fully-connected neural networks and it will be interesting to explore the spectra of other architectures such as ResNets and CNN. It will be also fundamental to explore the eigenvalue statistics that the current study cannot capture. While our study captured some of the basic eigenvalue statistics, it remains to derive the whole spectrum analytically. In particular, after the normalization excludes large outliers, the bulk of the spectrum becomes dominant. Random matrix theory enables us to analyze the FIM’s eigenvalue spectrum in a shallow and centered network without bias terms . Extending the random matrix theory to more general cases including deep neural networks seems to be a prerequisite for further progress. Furthermore, we assumed a finite number of network output units. In order to deal with multi-label classifications with high dimensionality, it would be helpful to investigate eigenvalue statistics in the wide limit of both hidden and network output layers. Finally, although we focused on the finite depth and regarded order parameters as constants, they can exponentially explode on extremely deep networks in the chaotic regime . The NTK in such a regime has been investigated in .

It would also be interesting to explore further connections between the eigenvalue statistics and learning. Recent studies have yielded insights into the connection between the generalization performance of DNNs and the eigenvalues statistics of certain Gram matrices including FIM and NTK . We expect that the theoretical foundation of the metric tensors given in this paper will lead to a more sophisticated understanding and development of deep learning in the future.

Appendices

where $\otimes$ represents the Kronecker product. The variables $h$ and $\delta$ are functions of $x$, and the expectation is taken over $x$. This expression is not easy to treat in the analysis, and we introduce a dual expression of the FIM, which is essentially the same as NTK, as follows.

First, we briefly overview the derivation of Theorem 3.1 shown in . The essential point is that a Gram matrix has the same non-zero eigenvalues as its dual. One can represent the empirical FIM (10) as

Its columns are the gradients on each input, i.e., $\nabla_{\theta}f_{k}(n)$ $(n=1,...,N)$. Let us refer to a $CN\times CN$ matrix $F^{*}:=R^{\top}R$ as the dual of FIM. Matrices $F$ and $F^{*}$ have the same non-zero eigenvalues by definition. This $F^{*}$ can be partitioned into $N\times N$ block matrices. The $(k,k^{\prime})$-th block is given by

for $k,k^{\prime}=1,...,C$. In the large $M$ limit, the previous study showed that $F^{*}$ asymptotically satisfies

where $\delta_{k,k^{\prime}}$ is the Kronecker delta. As is summarized in Lemma A.1 in , the second term of Eq. (A.4) is negligible in the large $M$ limit. In particular, it is reduced to $O(\sqrt{M})/N$ under certain condition. The matrix $K$ has entries given by

Note that $\kappa_{1}$ is positive by definition and $\kappa_{2}$ is positive under the condition (ii) of activation functions. The other eigenvalues of $K$ are given by $\kappa_{1}-\kappa_{2}$.

A.2 Diagonal blocks

We can immediately derive eigenvalue statistics of diagonal blocks in the same way as Theorem 3.1. We can represent the diagonal blocks as $F^{ll}:=R^{l}R^{l\top}$ with

where the parameter set $\theta^{l}$ means all parameters in the $l$-th layer. The $CN\times CN$ matrix $F^{ll*}$ can be partitioned into $N\times N$ block matrices whose $(k,k^{\prime})$-th block is given by

for $k,k^{\prime}=1,...,C$. As one can see from the additivity of $\sum_{l=1}^{L}F^{ll*}(k,k^{\prime})=F^{*}(k,k^{\prime})$, the following evaluation is part of Eq. (A.4):

where $Q$ is a $CN\times CN$ matrix. One can rearrange the columns and rows of $Q$ and partition it into $N\times N$ block matrices $Q(k,k^{\prime})$ whose entries are given by

for $k,k^{\prime}=1,...,C$. Each block is a diagonal matrix. Note that the non-zero eigenvalues of $RQR^{\top}$ are equivalent to those of $QR^{\top}R$. Since we have $F^{*}=R^{\top}R$, we should investigate the eigenvalues of the following matrix:

The third line holds asymptotically, since the order of $F(k,k)$ in Eq. (A.4) is higher than that of $F^{*}(k,k^{\prime})$ ($k\neq k^{\prime}$). The fourth line comes from $\sum_{k}g_{k}(n)=1$.

where $\sum_{m\setminus\{n\}}$ means a summation over $m$ excluding the $n$-th sample.

Finally, we derive the largest eigenvalue. Let us denote the eigenvectors of $F$ as

Because we have asymptotically $F^{*}\nu_{k}=\lambda_{max}(F)\nu_{k}$, the lower bound is given by

Taking the index $k$ that maximizes the right-hand side, we obtain the lower bound of $\lambda_{max}$. The upper bound of $\lambda_{max}$ immediately comes from a simple inequality for non-negative variables, i.e., $\lambda_{max}\leq\sqrt{\sum_{i}\lambda^{2}_{i}}=\sqrt{Ps_{\lambda}}$. Thus, we obtain Theorem 3.3.

Note that we immediately have $\lambda_{i}(F^{*}_{cross})\leq\lambda_{max}(Q)\lambda_{i}(F^{*})$ from (B.3). This means that $F_{cross}$’s eigenvalues satisfy $\lambda_{i}(F_{cross})\leq\lambda_{max}(Q)\lambda_{i}(F)$ ($i=1,...,P$). In addition, we have $\lambda_{max}(Q)\leq 1$ because $\lambda_{max}(Q)\leq\max_{n}\lambda_{max}(Q_{n})$ and $\lambda_{max}(Q_{n})\leq\max_{k}g_{k}(n)\leq 1$. Therefore, $\lambda_{i}(F_{cross})\leq\lambda_{i}(F)$ holds and we obtain inequalities (28).

B.2 Derivation of Theorem 3.4

for $r=1,...,P$. Define $V_{k}$ to be a linear subspace spanned by $k$ eigenvectors of $F$ corresponding to $\lambda_{max}(F)$, i.e., $\{v_{i_{1}},...,v_{i_{k}}\}$. The indices $\{i_{1},...,i_{k}\}$ are chosen from $\{1,...,C\}$ without duplication.

where $\lambda_{max}(F)$ is of $O(M)$ from Theorem 3.1. This holds for all of $r=1,...,C$ and we can say that there exist $C$ large eigenvalues of $O(M)$.

C Derivation of NTK’s eigenvalue statistics

The NTK is defined as $\Theta=NF^{*}$ under the NTK parameterization. In the same way as Eq. (A.4), the $(k,k^{\prime})$-th block of the NTK is asymptotically given by

for $k,k^{\prime}=1,...,C$. In contrast to Eq. (A.4), the NTK parameterization makes $F^{*}$ multiplied by $1/M$. The negligible term of $o(1)$ is reduced to $O(1/\sqrt{M})$ under the condition summarized in . The entries of $K^{\prime}$ are given by

In the same way as with the FIM, the trace of $K^{\prime}$ leads to $m_{\lambda}$, the Frobenius norm of $K^{\prime}$ leads to $s_{\lambda}$, and $K^{\prime}$ has the largest eigenvalue $((N-1)\kappa_{2}^{\prime}+\kappa_{1}^{\prime})$ for arbitrary $N$. The eigenspace of $\Theta$ corresponding to $\lambda_{max}$ is also the same as $F^{*}$. It is spanned by eigenvectors $\nu_{k}$ ($k=1,...,C$).

D Eigenvalue statistics of A𝐴A

The metric tensor $A_{k}$ can be represented by $A_{k}=\nabla_{h}f_{k}\nabla_{h}f_{k}^{\top}/N$, where $\nabla_{h}f_{k}$ is an $M_{h}\times N$ matrix and its columns are the gradients on each input, i.e., $\nabla_{h}f_{k}(n)$ $(n=1,...,N)$. Let us introduce the $N\times N$ dual matrix of $A_{k}$, i.e., $A^{*}_{k}:=\nabla_{h}f_{k}^{\top}\nabla_{h}f_{k}/N$. It has the same non-zero eigenvalues as $A_{k}$ by definition. Its $nm$-th entry is given by

in the large $M$ limit. Accordingly, we have

D.2 Diagonal blocks

In the same way as in the FIM, we can also evaluate the eigenvalue statistics of the diagonal blocks $A^{ll}_{k}$ ($l=0,...,L-1$). The metric tensor $A^{ll}_{k}$ can be represented by $A^{ll}_{k}=\nabla_{h^{l}}f_{k}\nabla_{h^{l}}f_{k}^{\top}/N$. Consider its dual, i.e., $A^{ll*}_{k}=\nabla_{h^{l}}f_{k}^{\top}\nabla_{h^{l}}f_{k}/N$. Note that we partitioned $A_{k}$ into $L^{2}$ block matrices whose $(l,l^{\prime})$-th block is expressed by an $M_{l}\times M_{l^{\prime}}$ matrix:

In the large $M$ limit, we have asymptotically

The eigenvalue statistics of $A^{ll}_{k}$ are asymptotically evaluated as

References