Similarity of Neural Network Representations Revisited

Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

Introduction

Across a wide range of machine learning tasks, deep neural networks enable learning powerful feature representations automatically from data. Despite impressive empirical advances of deep neural networks in solving various tasks, the problem of understanding and characterizing the neural network representations learned from data remains relatively under-explored. Previous work (e.g. Advani & Saxe (2017); Amari et al. (2018); Saxe et al. (2014)) has made progress in understanding the theoretical dynamics of the neural network training process. These studies are insightful, but fundamentally limited, because they ignore the complex interaction between the training dynamics and structured data. A window into the network’s representation can provide more information about the interaction between machine learning algorithms and data than the value of the loss function alone.

This paper investigates the problem of measuring similarities between deep neural network representations. An effective method for measuring representational similarity could help answer many interesting questions, including: (1) Do deep neural networks with the same architecture trained from different random initializations learn similar representations? (2) Can we establish correspondences between layers of different network architectures? (3) How similar are the representations learned using the same network architecture from different datasets?

We build upon previous studies investigating similarity between the representations of neural networks (Laakso & Cottrell, 2000; Li et al., 2015; Raghu et al., 2017; Morcos et al., 2018; Wang et al., 2018). We are also inspired by the extensive neuroscience literature that uses representational similarity analysis (Kriegeskorte et al., 2008a; Edelman, 1998) to compare representations across brain areas (Haxby et al., 2001; Freiwald & Tsao, 2010), individuals (Connolly et al., 2012), species (Kriegeskorte et al., 2008b), and behaviors (Elsayed et al., 2016), as well as between brains and neural networks (Yamins et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Sussillo et al., 2015).

Our key contributions are summarized as follows:

We discuss the invariance properties of similarity indexes and their implications for measuring similarity of neural network representations.

We motivate and introduce centered kernel alignment (CKA) as a similarity index and analyze the relationship between CKA, linear regression, canonical correlation analysis (CCA), and related methods (Raghu et al., 2017; Morcos et al., 2018).

We show that CKA is able to determine the correspondence between the hidden layers of neural networks trained from different random initializations and with different widths, scenarios where previously proposed similarity indexes fail.

We verify that wider networks learn more similar representations, and show that the similarity of early layers saturates at fewer channels than later layers. We demonstrate that early layers, but not later layers, learn similar representations on different datasets.

What Should Similarity Be Invariant To?

This section discusses the invariance properties of similarity indexes and their implications for measuring similarity of neural network representations. We argue that both intuitive notions of similarity and the dynamics of neural network training call for a similarity index that is invariant to orthogonal transformation and isotropic scaling, but not invertible linear transformation.

A similarity index is invariant to invertible linear transformation if $s(X,Y)=s(XA,YB)$ for any full rank $A$ and $B$ . If activations $X$ are followed by a fully-connected layer $f(X)=\sigma(XW+\beta)$ , then transforming the activations by a full rank matrix $A$ as $X^{\prime}=XA$ and transforming the weights by the inverse $A^{-1}$ as $W^{\prime}=A^{-1}W$ preserves the output of $f(X)$ . This transformation does not appear to change how the network operates, so intuitively, one might prefer a similarity index that is invariant to invertible linear transformation, as argued by Raghu et al. (2017).

However, a limitation of invariance to invertible linear transformation is that any invariant similarity index gives the same result for any representation of width greater than or equal to the dataset size, i.e. $p_{2}\geq n$ . We provide a simple proof in Appendix A.

Let $X$ and $Y$ be $n\times p$ matrices. Suppose $s$ is invariant to invertible linear transformation in the first argument, i.e. $s(X,Z)=s(XA,Z)$ for arbitrary $Z$ and any $A$ with $\text{rank}(A)=p$ . If $\text{rank}(X)=\text{rank}(Y)=n$ , then $s(X,Z)=s(Y,Z)$ .

There is thus a practical problem with invariance to invertible linear transformation: Some neural networks, especially convolutional networks, have more neurons in some layers than there are examples the training dataset (Springenberg et al., 2015; Lee et al., 2018; Zagoruyko & Komodakis, 2016). It is somewhat unnatural that a similarity index could require more examples than were used for training.

A deeper issue is that neural network training is not invariant to arbitrary invertible linear transformation of inputs or activations. Even in the linear case, gradient descent converges first along the eigenvectors corresponding to the largest eigenvalues of the input covariance matrix (LeCun et al., 1991), and in cases of overparameterization or early stopping, the solution reached depends on the scale of the input. Similar results hold for gradient descent training of neural networks in the infinite width limit (Jacot et al., 2018). The sensitivity of neural networks training to linear transformation is further demonstrated by the popularity of batch normalization (Ioffe & Szegedy, 2015).

Invariance to invertible linear transformation implies that the scale of directions in activation space is irrelevant. Empirically, however, scale information is both consistent across networks and useful across tasks. Neural networks trained from different random initializations develop representations with similar large principal components, as shown in Figure 1. Consequently, Euclidean distances between examples, which depend primarily upon large principal components, are similar across networks. These distances are meaningful, as demonstrated by the success of perceptual loss and style transfer (Gatys et al., 2016; Johnson et al., 2016; Dumoulin et al., 2017). A similarity index that is invariant to invertible linear transformation ignores this aspect of the representation, and assigns the same score to networks that match only in large principal components or networks that match only in small principal components.

2 Invariance to Orthogonal Transformation

Rather than requiring invariance to any invertible linear transformation, one could require a weaker condition; invariance to orthogonal transformation, i.e. $s(X,Y)=s(XU,YV)$ for full-rank orthonormal matrices $U$ and $V$ such that $U^{\text{T}}U=I$ and $V^{\text{T}}V=I$ .

Indexes invariant to orthogonal transformations do not share the limitations of indexes invariant to invertible linear transformation. When $p_{2}>n$ , indexes invariant to orthogonal transformation remain well-defined. Moreover, orthogonal transformations preserve scalar products and Euclidean distances between examples.

Invariance to orthogonal transformation seems desirable for neural networks trained by gradient descent. Invariance to orthogonal transformation implies invariance to permutation, which is needed to accommodate symmetries of neural networks (Chen et al., 1993; Orhan & Pitkow, 2018). In the linear case, orthogonal transformation of the input does not affect the dynamics of gradient descent training (LeCun et al., 1991), and for neural networks initialized with rotationally symmetric weight distributions, e.g. i.i.d. Gaussian weight initialization, training with fixed orthogonal transformations of activations yields the same distribution of training trajectories as untransformed activations, whereas an arbitrary linear transformation would not.

Given a similarity index $s(\cdot,\cdot)$ that is invariant to orthogonal transformation, one can construct a similarity index $s^{\prime}(\cdot,\cdot)$ that is invariant to any invertible linear transformation by first orthonormalizing the columns of $X$ and $Y$ , and then applying $s(\cdot,\cdot)$ . Given thin QR decompositions $X=Q_{A}R_{A}$ and $Y=Q_{B}R_{B}$ one can construct a similarity index $s^{\prime}(X,Y)=s(Q_{X},Q_{Y})$ , where $s^{\prime}(\cdot,\cdot)$ is invariant to invertible linear transformation because orthonormal bases with the same span are related to each other by orthonormal transformation (see Appendix B).

3 Invariance to Isotropic Scaling

Comparing Similarity Structures

Our key insight is that instead of comparing multivariate features of an example in the two representations (e.g. via regression), one can first measure the similarity between every pair of examples in each representation separately, and then compare the similarity structures. In neuroscience, such matrices representing the similarities between examples are called representational similarity matrices (Kriegeskorte et al., 2008a). We show below that, if we use an inner product to measure similarity, the similarity between representational similarity matrices reduces to another intuitive notion of pairwise feature similarity.

A simple formula relates dot products between examples to dot products between features:

The elements of $XX^{\text{T}}$ and $YY^{\text{T}}$ are dot products between the representations of the $i^{\text{th}}$ and $j^{\text{th}}$ examples, and indicate the similarity between these examples according to the respective networks. The left-hand side of (1) thus measures the similarity between the inter-example similarity structures. The right-hand side yields the same result by measuring the similarity between features from $X$ and $Y$ , by summing the squared dot products between every pair.

Hilbert-Schmidt Independence Criterion.

Equation 1 implies that, for centered $X$ and $Y$ :

The Hilbert-Schmidt Independence Criterion (Gretton et al., 2005) generalizes Equations 1 and 2 to inner products from reproducing kernel Hilbert spaces, where the squared Frobenius norm of the cross-covariance matrix becomes the squared Hilbert-Schmidt norm of the cross-covariance operator. Let $K_{ij}=k(\mathbf{x}_{i},\mathbf{x}_{j})$ and $L_{ij}=l(\mathbf{y}_{i},\mathbf{y}_{j})$ where $k$ and $l$ are two kernels. The empirical estimator of HSIC is:

where $H$ is the centering matrix $H_{n}=I_{n}-\frac{1}{n}\mathbf{1}\mathbf{1}^{\text{T}}$ . For linear kernels $k(\mathbf{x},\mathbf{y})=l(\mathbf{x},\mathbf{y})=\mathbf{x}^{\text{T}}\mathbf{y}$ , HSIC yields (2).

Gretton et al. (2005) originally proposed HSIC as a test statistic for determining whether two sets of variables are independent. They prove that the empirical estimator converges to the population value at a rate of $1/\sqrt{n}$ , and Song et al. (2007) provide an unbiased estimator. When $k$ and $l$ are universal kernels, HSIC = 0 implies independence, but HSIC is not an estimator of mutual information. HSIC is equivalent to maximum mean discrepancy between the joint distribution and the product of the marginal distributions, and HSIC with a specific kernel family is equivalent to distance covariance (Sejdinovic et al., 2013).

Centered Kernel Alignment.

HSIC is not invariant to isotropic scaling, but it can be made invariant through normalization. This normalized index is known as centered kernel alignment (Cortes et al., 2012; Cristianini et al., 2002):

For a linear kernel, CKA is equivalent to the RV coefficient (Robert & Escoufier, 1976) and to Tucker’s congruence coefficient (Tucker, 1951; Lorenzo-Seva & Ten Berge, 2006).

Kernel Selection.

Below, we report results of CKA with a linear kernel and the RBF kernel $k(\textbf{x}_{i},\textbf{x}_{j})=\exp(-||\textbf{x}_{i}-\textbf{x}_{j}||_{2}^{2}/(2\sigma^{2}))$ . For the RBF kernel, there are several possible strategies for selecting the bandwidth $\sigma$ , which controls the extent to which similarity of small distances is emphasized over large distances. We set $\sigma$ as a fraction of the median distance between examples. In practice, we find that RBF and linear kernels give similar results across most experiments, so we use linear CKA unless otherwise specified. Our framework extends to any valid kernel, including kernels equivalent to neural networks (Lee et al., 2018; Jacot et al., 2018; Garriga-Alonso et al., 2019; Novak et al., 2019).

Related Similarity Indexes

In this section, we briefly review linear regression, canonical correlation, and other related methods in the context of measuring similarity between neural network representations. We let $Q_{X}$ and $Q_{Y}$ represent any orthonormal bases for the columns of $X$ and $Y$ , i.e. $Q_{X}=X(X^{\text{T}}X)^{-1/2}$ , $Q_{Y}=Y(Y^{\text{T}}Y)^{-1/2}$ or orthogonal transformations thereof. Table 1 summarizes the formulae and invariance properties of the indexes used in experiments. For a comprehensive general review of linear indexes for measuring multivariate similarity, see Ramsay et al. (1984).

A simple way to relate neural network representations is via linear regression. One can fit every feature in $X$ as a linear combination of features from $Y$ . A suitable summary statistic is the total fraction of variance explained by the fit:

We are unaware of any application of linear regression to measuring similarity of neural network representations, although Romero et al. (2015) used a least squares loss between activations of two networks to encourage thin and deep “student” networks to learn functions similar to wide and shallow “teacher” networks.

Canonical Correlation Analysis (CCA).

Canonical correlation finds bases for two matrices such that, when the original matrices are projected onto these bases, the correlation is maximized. For $1\leq i\leq p_{1}$ , the $i$ th canonical correlation coefficient $\rho_{i}$ is given by:

For the purpose of this work, we consider two summary statistics of the goodness of fit of CCA:

where $||\cdot||_{*}$ denotes the nuclear norm. The mean squared CCA correlation $R^{2}_{\text{CCA}}$ is also known as Yanai’s GCD measure (Ramsay et al., 1984), and several statistical packages report the sum of the squared canonical correlations $p_{1}R_{\text{CCA}}^{2}=\sum_{i=1}^{p_{1}}\rho_{i}^{2}$ under the name Pillai’s trace (SAS Institute, 2015; StataCorp, 2015). The mean CCA correlation $\bar{\rho}_{\text{CCA}}$ was previously used to measure similarity between neural network representations in Raghu et al. (2017).

SVCCA.

CCA is sensitive to perturbation when the condition number of $X$ or $Y$ is large (Golub & Zha, 1995). To improve robustness, singular vector CCA (SVCCA) performs CCA on truncated singular value decompositions of $X$ and $Y$ (Raghu et al., 2017; Mroueh et al., 2015; Kuss & Graepel, 2003). As formulated in Raghu et al. (2017), SVCCA keeps enough principal components of the input matrices to explain a fixed proportion of the variance, and drops remaining components. Thus, it is invariant to invertible linear transformation only if the retained subspace does not change.

Projection-Weighted CCA.

Morcos et al. (2018) propose a different strategy to reduce the sensitivity of CCA to perturbation, which they term “projection-weighted canonical correlation” (PWCCA):

where $\mathbf{x}_{j}$ is the $j^{\text{th}}$ column of $X$ , and $\mathbf{h}_{i}=X\mathbf{w}_{X}^{i}$ is the vector of canonical variables formed by projecting $X$ to the $i^{\text{th}}$ canonical coordinate frame. As we show in Appendix C.3, PWCCA is closely related to linear regression, since:

Neuron Alignment Procedures.

Mutual Information.

Among non-linear measures, one candidate is mutual information, which is invariant not only to invertible linear transformation, but to any invertible transformation. Li et al. (2015) previously used mutual information to measure neuronal alignment. In the context of comparing representations, we believe mutual information is not useful. Given any pair of representations produced by deterministic functions of the same input, mutual information between either and the input must be at least as large as mutual information between the representations. Moreover, in fully invertible neural networks (Dinh et al., 2017; Jacobsen et al., 2018), the mutual information between any two layers is equal to the entropy of the input.

Linear CKA versus CCA and Regression

Linear CKA is closely related to CCA and linear regression. If $X$ and $Y$ are centered, then $Q_{X}$ and $Q_{Y}$ are also centered, so:

When performing the linear regression fit of $X$ with design matrix $Y$ , $R_{\text{LR}}^{2}=||Q_{Y}^{\text{T}}X||_{F}^{2}/||X||_{F}^{2}$ , so:

When might we prefer linear CKA over CCA? One way to show the difference is to rewrite $X$ and $Y$ in terms of their singular value decompositions $X=U_{X}\Sigma_{X}V_{X}^{\text{T}}$ , $Y=U_{Y}\Sigma_{Y}V_{Y}^{\text{T}}$ . Let the $i^{\text{th}}$ eigenvector of $XX^{\text{T}}$ (left-singular vector of $X$ ) be indexed as $\textbf{u}_{X}^{\text{i}}$ . Then $R^{2}_{\text{CCA}}$ is:

Let the $i^{\text{th}}$ eigenvalue of $XX^{\text{T}}$ (squared singular value of $X$ ) be indexed as $\lambda_{X}^{i}$ . Linear CKA can be written as:

Linear CKA thus resembles CCA weighted by the eigenvalues of the corresponding eigenvectors, i.e. the amount of variance in $X$ or $Y$ that each explains. SVCCA (Raghu et al., 2017) and projection-weighted CCA (Morcos et al., 2018) were also motivated by the idea that eigenvectors that correspond to small eigenvalues are less important, but linear CKA incorporates this weighting symmetrically and can be computed without a matrix decomposition.

Comparison of (13) and (14) immediately suggests the possibility of alternative weightings of scalar products between eigenvectors. Indeed, as we show in Appendix D.1, the similarity index induced by “canonical ridge” regularized CCA (Vinod, 1976), when appropriately normalized, interpolates between $R^{2}_{\text{CCA}}$ , linear regression, and linear CKA.

Results

We propose a simple sanity check for similarity indexes: Given a pair of architecturally identical networks trained from different random initializations, for each layer in the first network, the most similar layer in the second network should be the architecturally corresponding layer. We train 10 networks and, for each layer of each network, we compute the accuracy with which we can find the corresponding layer in each of the other networks by maximum similarity. We then average the resulting accuracies. We compare CKA with CCA, SVCCA, PWCCA, and linear regression.

We first investigate a simple VGG-like convolutional network based on All-CNN-C (Springenberg et al., 2015) (see Appendix E) on CIFAR-10. Figure 2 and Table 2 show that CKA passes our sanity check, but other methods perform substantially worse. For SVCCA, we experimented with a range of truncation thresholds, but no threshold revealed the layer structure (Appendix F.2); our results are consistent with those in Appendix E of Raghu et al. (2017).

We also investigate Transformer networks, where all layers are of equal width. In Appendix F.1, we show similarity between the 12 sublayers of the encoders of Transformer models (Vaswani et al., 2017) trained from different random initializations. All similarity indexes achieve non-trivial accuracy and thus pass the sanity check, although RBF CKA and $R^{2}_{\text{CCA}}$ performed slightly better than other methods. However, we found that there are differences in feature scale between representations of feed-forward network and self-attention sublayers that CCA does not capture because it is invariant to non-isotropic scaling.

2 Using CKA to Understand Network Architectures

CKA is equally effective at revealing relationships between layers of different architectures. Figure 5 shows the relationship between different layers of networks with and without residual connections. CKA indicates that, as networks are made deeper, the new layers are effectively inserted in between the old layers. Other similarity indexes fail to reveal meaningful relationships between different architectures, as we show in Appendix F.5.

In Figure 6, we show CKA between networks with different layer widths. Like Morcos et al. (2018), we find that increasing layer width leads to more similar representations between networks. As width increases, CKA approaches 1; CKA of earlier layers saturates faster than later layers. Networks are generally more similar to other networks of the same width than they are to the widest network we trained.

3 Similar Representations Across Datasets

CKA can also be used to compare networks trained on different datasets. In Figure 7, we show that models trained on CIFAR-10 and CIFAR-100 develop similar representations in their early layers. These representations require training; similarity with untrained networks is much lower. We further explore similarity between layers of untrained networks in Appendix F.3.

4 Analysis of the Shared Subspace

Equation 14 suggests a way to further elucidating what CKA is measuring, based on the action of one representational similarity matrix (RSM) $YY^{\text{T}}$ applied to the eigenvectors $\mathbf{u}_{X}^{i}$ of the other RSM $XX^{\text{T}}$ . By definition, $XX^{\text{T}}\mathbf{u}_{X}^{i}$ points in the same direction as $\mathbf{u}_{X}^{i}$ , and its norm $||XX^{\text{T}}\mathbf{u}_{X}^{i}||_{2}$ is the corresponding eigenvalue. The degree of scaling and rotation by $YY^{\text{T}}$ thus indicates how similar the action of $YY^{\text{T}}$ is to $XX^{\text{T}}$ , for each eigenvector of $XX^{\text{T}}$ . For visualization purposes, this approach is somewhat less useful than the CKA summary statistic, since it does not collapse the similarity to a single number, but it provides a more complete picture of what CKA measures. Figure 8 shows that, for large eigenvectors, $XX^{\text{T}}$ and $YY^{\text{T}}$ have similar actions, but the rank of the subspace where this holds is substantially lower than the dimensionality of the activations. In the penultimate (global average pooling) layer, the dimensionality of the shared subspace is approximately 10, which is the number of classes in the CIFAR-10 dataset.

Conclusion and Future Work

Measuring similarity between the representations learned by neural networks is an ill-defined problem, since it is not entirely clear what aspects of the representation a similarity index should focus on. Previous work has suggested that there is little similarity between intermediate layers of neural networks trained from different random initializations (Raghu et al., 2017; Wang et al., 2018). We propose CKA as a method for comparing representations of neural networks, and show that it consistently identifies correspondences between layers, not only in the same network trained from different initializations, but across entirely different architectures, whereas other methods do not. We also provide a unified framework for understanding the space of similarity indexes, as well as an empirical framework for evaluation.

We show that CKA captures intuitive notions of similarity, i.e. that neural networks trained from different initializations should be similar to each other. However, it remains an open question whether there exist kernels beyond the linear and RBF kernels that would be better for analyzing neural network representations. Moreover, there are other potential choices of weighting in Equation 14 that may be more appropriate in certain settings. We leave these questions as future work. Nevertheless, CKA seems to be much better than previous methods at finding correspondences between the learned representations in hidden layers of neural networks.

Acknowledgements

We thank Gamaleldin Elsayed, Jaehoon Lee, Paul-Henri Mignot, Maithra Raghu, Samuel L. Smith, Alex Williams, and Michael Wu for comments on the manuscript, Rishabh Agarwal for ideas, and Aliza Elkin for support.

References

Appendix A Proof of Theorem 1

where $K_{X}$ is a basis for the null space of the rows of $X$ and $K_{Y}$ is a basis for the null space of the rows of $Y$ . Then let $A=X^{\prime-1}Y^{\prime}$ .

Because $X^{\prime}$ and $Y^{\prime}$ have rank $p$ by construction, $A$ also has rank $p$ . Thus, $s(X,Z)=s(XA,Z)=s(Y,Z)$ . ∎

Appendix B Orthogonalization and Invariance to Invertible Linear Transformation

Here we show that any similarity index that is invariant to orthogonal transformation can be made invariant to invertible linear transformation by orthogonalizing the columns of the input.

Let $X$ be an $n\times p$ matrix of full column rank and let $A$ be an invertible $p\times p$ matrix. Let $X=Q_{X}R_{X}$ and $XA=Q_{XA}R_{XA}$ , where $Q_{X}^{T}Q_{X}=Q_{XA}^{T}Q_{XA}=I$ and $R_{X}$ and $R_{XA}$ are invertible. If $s(\cdot,\cdot)$ is invariant to orthogonal transformation, then $s(Q_{X},Y)=s(Q_{XA},Y)$ .

Let $B=R_{X}AR_{XA}^{-1}$ . Then $Q_{X}B=Q_{XA}$ , and B is an orthogonal transformation:

Thus $s(Q_{X},Y)=s(Q_{X}B,Y)=s(Q_{XA},Y)$ . ∎

Appendix C CCA and Linear Regression

Consider the linear regression fit of the columns of an $n\times m$ matrix $C$ with an $n\times p$ matrix $A$ :

Let $A=QR$ , the thin QR decomposition of A. Then the fitted values are given by:

The residuals $E=C-\hat{C}$ are orthogonal to the fitted values, i.e.

Assuming that $C$ was centered by subtracting its column means prior to the linear regression fit, the total fraction of variance explained by the fit is:

Although we have assumed that $Q$ is obtained from QR decomposition, any orthonormal basis with the same span will suffice, because orthogonal transformations do not change the Frobenius norm.

C.2 CCA

Let $X$ be an $n\times p_{1}$ matrix and $Y$ be an $n\times p_{2}$ matrix, and let $p=\text{min}(p_{1},p_{2})$ . Given the thin QR decompositions of $X$ and $Y$ , $X=Q_{X}R_{X}$ , $Y=Q_{Y}R_{Y}$ such that $Q_{X}^{\text{T}}Q_{X}=I$ , $Q_{Y}^{\text{T}}Q_{Y}=I$ , the canonical correlations $\rho_{i}$ are the singular values of $A=Q_{X}^{\text{T}}Q_{Y}$ (Björck & Golub, 1973; Press, 2011) and thus the square roots of the eigenvalues of $A^{\text{T}}A$ . The squared canonical correlations $\rho_{i}^{2}$ are the eigenvalues of $A^{\text{T}}A=Q_{Y}^{\text{T}}Q_{X}Q_{X}^{\text{T}}Q_{Y}$ . Their sum is $\sum_{i=1}^{p}\rho_{i}^{2}=\text{tr}(A^{\text{T}}A)=||Q_{Y}^{\text{T}}Q_{X}||_{\text{F}}^{2}$ .

Now consider the linear regression fit of the columns of $Q_{X}$ with $Y$ . Assume that $Q_{X}$ has zero mean. Substituting $Q_{Y}$ for $Q$ and $Q_{X}$ for $C$ in Equation 16, and noting that $||Q_{X}||_{\text{F}}^{2}=p_{1}$ :

C.3 Projection-Weighted CCA

Let $X$ be an $n\times p_{1}$ matrix and $Y$ be an $n\times p_{2}$ matrix, with $p_{1}\leq p_{2}$ . Morcos et al. (2018) proposed to compute projection-weighted canonical correlation as:

where the $\mathbf{x}_{j}$ are the columns of $X$ , and the $\mathbf{h}_{i}$ are the canonical variables formed by projecting $X$ to the canonical coordinate frame. Below, we show that if we modify $\bar{\rho}_{\text{PW}}$ by squaring the dot products and $\rho_{i}$ , we recover linear regression.:

Our derivation begins by forming the SVD $Q_{X}^{\text{T}}Q_{Y}=U\Sigma V^{\text{T}}$ . $\Sigma$ is a diagonal matrix of the canonical correlations $\rho_{i}$ , and the matrix of canonical variables $H=Q_{X}U$ . Then $R^{2}_{\text{MPW}}$ is:

Because we assume $p_{1}\leq p_{2}$ , $U$ is a square orthogonal matrix and $UU^{\text{T}}=I$ . Further noting that $Q_{X}^{\text{T}}H=U$ and $U\Sigma=Q_{X}^{\text{T}}Q_{Y}V$ :

Substituting $Q_{Y}$ for $Q$ and $X$ for $C$ in Equation 16:

Appendix D Notes on Other Methods

Beyond CCA, we could also consider the “canonical ridge” regularized CCA objective (Vinod, 1976):

Unlike in the unregularized case, the singular values $\sigma_{i}$ do not measure the correlation between the canonical variables. Instead, they become arbitrarily small as $\kappa_{X}$ or $\kappa_{Y}$ increase. Thus, we need to normalize the statistic to remove the dependency on the regularization parameters.

Applying von Neumann’s trace inequality yields a bound:

Applying the Cauchy-Schwarz inequality to (25) yields the alternative bounds:

A normalized form of (22) could be produced by dividing by any of (25), (26), or (27).

If $\kappa_{X}=\kappa_{Y}=0$ , then (25) and (26) are equal to $p_{1}$ . In this case, (22) is simply the sum of the squared canonical correlations, so normalizing by either of these bounds recovers $R^{2}_{\text{CCA}}$ .

If $\kappa_{Y}=0$ , then as $\kappa_{X}\to\infty$ , normalizing by the bound from (25) recovers $R^{2}$ :

Moreover, setting $\kappa_{X}=\kappa_{Y}=\kappa$ and taking the limit as $\kappa\to\infty$ , the normalization from (27) leads to $\text{CKA}(XX^{\text{T}},YY^{\text{T}})$ :

Overall, the hyperparameters of the canonical ridge objective make it less useful for exploratory analysis. These hyperparameters could be selected by cross-validation, but this is computationally expensive, and the resulting estimator would be biased by sample size. Moreover, our goal is not to map representations of networks to a common space, but to measure the similarity between networks. Appropriately chosen regularization will improve out-of-sample performance of the mapping, but it makes the meaning of “similarity” more ambiguous.

D.2 The Orthogonal Procrustes Problem

The orthogonal Procrustes problem consists of finding an orthogonal rotation in feature space that produces the smallest error:

The solution is $\hat{Q}=UV^{\text{T}}$ where $U\Sigma V^{\text{T}}=X^{\text{T}}Y$ , the singular value decomposition. At the maximum of (39):

which is similar to what we call “dot product-based similarity” (Equation 1), but with the squared Frobenius norm of $Y^{\text{T}}X$ (the sum of the squared singular values) replaced by the nuclear norm (the sum of the singular values). The Frobenius norm of $Y^{\text{T}}X$ can be obtained as the solution to a similar optimization problem:

In the context of neural networks, Smith et al. (2017) previously proposed using the solution to the orthogonal Procrustes problem to align word embeddings from different languages, and demonstrated that it outperformed CCA.

Appendix E Architecture Details

All non-ResNet architectures are based on All-CNN-C (Springenberg et al., 2015), but none are architecturally identical. The Plain-10 model is very similar, but we place the final linear layer after the average pooling layer and use batch normalization because these are common choices in modern architectures. We use these models because they train in minutes on modern hardware.

Appendix F Additional Experiments

When applied to Transformer encoders, all similarity indexes we investigated passed the sanity check described in Section 6.1. We trained Transformer models using the tensor2tensor library (Vaswani et al., 2018) on the English to German translation task from the WMT18 dataset (Bojar et al., 2018) (Europarl v7, Common Crawl, and News Commentary v13 corpora) and computed representations of each of the 75,804 tokens from the 3,000 sentence newstest2013 development set, ignoring end of sentence tokens. In Figure F.2, we show similarity between the 12 sublayers of the encoders of 10 Transformer models (45 pairs) trained from different random initializations. Each Transformer sublayer contains four operations, shown in Figure F.2; results vary based which operation the representation is taken after. Table F.1 shows the accuracy with which we identify corresponding layers between network pairs by maximal similarity.

The Transformer architecture alternates between self-attention and feed-forward network (FFN) sublayers. The checkerboard pattern in representational similarity after the self-attention/feed-forward network operation in Figure F.2 indicates that representations of attention sublayers are more similar to other attention sublayers than to FFN sublayers, and similarly, representations of FFN sublayers are more similar to other FFN than to feed-forward network layers. CKA reveals a checkerboard pattern for activations after the channel-wise scale operation (before the attention/FFN operation) that other methods do not. Because CCA is invariant to non-isotropic scaling, CCA similarities before and after channel-wise scaling are identical. Thus, CCA cannot capture this structure, even though the structure is consistent across networks.