Learning Unitary Operators with Help From u(n)

Introduction

While recurrent neural networks (RNNs) are seeing widespread success across many tasks, the fundamental architecture presents challenges to typical training algorithms. In particular, the problem of ‘vanishing/exploding gradients’ (?) in gradient-based optimization persists, where gradients either vanish or diverge as one goes deeper into the network, resulting in slow training or numerical instability. The long short-term memory (LSTM) network (?) was designed to overcome this issue. Recently, the use of norm-preserving operators in the transition matrix - the matrix of weights connecting subsequent internal states - of the RNN have been explored (?; ?; ?). Using operators with bounded eigenvalue spectrum should, as demonstrated by ? (?), bound the norms of the gradients in the network, assuming an appropriate non-linearity is applied. Unitary matrices satisfy this requirement and are the focus of this work.

Imposing unitarity (or orthogonality) on this transition matrix is however challenging for gradient-based optimization methods, as additive updates typically do not preserve unitarity. Solutions include re-unitarizing after each batch, or using a parametrization of unitary matrices closed under addition. In this work we propose a solution in the second category, using results from the theory of Lie algebras and Lie groups to define a general parametrization of unitary matrices in terms of skew-Hermitian matrices (elements of the Lie algebra associated to the Lie group of unitary matrices). As explained in more detail below, elements of this Lie algebra can be identified with unitary matrices, while the algebra is closed under addition, forming a vector space over real numbers.

While we are motivated by the issues of RNNs, and we consider an application in RNNs, our primary focus here is on a core question: how can unitary matrices be learned? Assuming the choice of a unitary transition matrix is an appropriate modelling choice, the gradients on this operator should ultimately guide it towards unitarity, if it is possible under the parametrization or learning scheme used. It is therefore useful to know which approach is best. We distil the problem into its simplest form (a learning task described in detail later), so that our findings cannot be confounded by other factors specific to the RNN long-term memory task, before demonstrating our parametrization in that setting.

We draw most inspiration from the recent work of ? (?), who proposed a specific parametrization of unitary matrices and demonstrated its utility in standard long-term memory tasks for RNNs. We describe their parametrization here, as we use it later. Citing the difficulty of obtaining a general and efficient parametrization of unitary matrices, they use the fact that the unitary group is closed under matrix multiplication to form a composite operator:

where each component is unitary and easily parametrized:

$\mathcal{F}$ and $\mathcal{F}^{-1}$ are the Fourier and inverse Fourier transforms (or, in practice, their discrete matrix representations)

In total, this parametrization has $7n$ real learnable parameters ($2n$ for each reflection and $n$ for each diagonal operator), so describes a subspace of unitary matrices (which have $n^{2}$ real parameters). Nonetheless, they find that an RNN using this operator as its transition matrix outperforms LSTMs on the adding and memory tasks described first in ? (?). This prompted us to consider other parametrizations of unitary matrices which might be more expressive or interpretable.

? (?) constrain a part of the transition matrix to be close to the identity, acting as a form of long-term memory store, while ? (?) initialize it to the identity, and then use ReLUs as non-linearities. ? (?) study analytic solutions to the long-term memory task, supporting observations and intuitions that orthogonal (or unitary) matrices would be appropriate as transition matrices for this task. They also study initializations to orthogonal and identity matrices, and consider experiments where an additional term in the loss function encourages an orthogonal solution to the transition matrix, without using an explicit parametrization. ? (?) study exact solutions to learning dynamics in deep networks and find that orthogonal weight initializations at each layer lead to depth-independent learning (thus escaping the vanishing/exploding gradient problem). Interestingly, they attribute this to the eigenvalue spectrum of orthogonal matrices lying on the unit circle. They compare with weights initialized to random, scaled Gaussian values, which preserve norms in expectation (over values of the random matrix) and find orthogonal matrices superior. It therefore appears that preserving norms is not sufficient to stabilize gradients over network depth, but that the eigenvalue spectrum must also be strictly controlled.

In a related but separate vein, ? (?) penalize the difference of difference of norms between subsequent hidden states in the network. This is not equivalent to imposing orthogonality of the transition matrix, as the norm of the hidden state may be influenced by the inputs and non-linearities, and their method directly addresses this norm.

The theory of Lie groups and Lie algebras has seen most application in machine learning for its use in capturing notions of invariance. For example, ? (?), learn infinitesimal Lie group generators (elements of the Lie algebra) associated with affine transformations of images, corresponding to visual perceptual invariances. This is different to our setting as our generators are already known (we assume the Lie group $U(n)$) and wish to learn the coefficients of a given transformation relative to that basis set of generators. However, our approach could be extended to the case where the basis of $\mathfrak{u}(n)$ is unknown, and must be learned. As we find later (appendix B), the choice of basis can impact performance, and so may be an important consideration. ? (?) learn commutative subgroups of $SO(n)$ (known as toroidal subgroups), motivated by learning the irreducible representations of the symmetry group corresponding to invariant properties of images. Their choice of group parametrization is equivalent to selecting a particular basis of the corresponding Lie algebra, as they describe, but primarily exploit the algebra to understand properties of toroidal subgroups.

? (?) perform motion estimation by defining a regression function in terms of a function on the Lie algebra of affine transformations, and then learning this. This is similar to our approach in the sense that they do optimization in the Lie algebra, although as they consider two-dimensional affine transformations only, their parametrization of the Lie algebra is straight forward.

Finally, ? (?) describe an online learning algorithm for orthogonal matrices – which are the real-valued equivalent to unitary matrices. They also claim that the approach is extends easily to unitary matrices.

Structure of this paper

We begin with an introduction to the relevant facts and definitions from the theory of Lie groups and Lie algebras, to properly situate this work in its mathematical context. Further exposition is beyond the scope of this paper, and we refer the interested reader to any of the comprehensive introductory texts on the matter.

We explain our parametrization in detail and describe a method to calculate the derivative of the matrix exponential - a quantity otherwise computationally intractable. Then, we describe a simple but clear experiment designed to test our core question of learning unitary matrices. We compare to an approach using the parametrization of ? (?) and one using polar decomposition to ‘back-project’ to the closest unitary matrix. We use this experimental set-up to probe aspects of our model, studying the importance of the choice of basis (appendix B), and the impact of the restricted parameter set used by one of the alternate approaches. We additionally implement our parametrization in a recurrent neural network as a ‘general unitary RNN’, and evaluate its performance on standard long-memory tasks.

The Lie algebra 𝔲​(n)𝔲𝑛\mathfrak{u}(n)

A Lie group is a group which is also a differentiable manifold, with elements of the group corresponding to points on the manifold. The group operations (multiplication and inversion) must be smooth maps (infinitely differentiable) back to the group. In this work we consider the group $U(n)$: the set of $n\times n$ unitary matrices, with matrix multiplication. These are the complex-valued analogue to orthogonal matrices, satisfying the property

where $\dagger$ denotes the conjugate transpose (or Hermitian conjugate). Unitary matrices preserve matrix norms, and have eigenvalues lying on the (complex) unit circle, which is the desired property of the transition matrix in a RNN.

The elements $\dot{U}(0)$ belong to the Lie algebra. We refer to this Lie algebra as $\mathfrak{u}(n)$, and an arbitrary element as $L$. Then Equation 4 defines the properites of these Lie algebra elements; they are $n\times n$ skew-Hermitian matrices: $L^{\dagger}=-L$.

Lie algebras are also endowed with an operation known as the Lie bracket, which has many interesting properties, but is beyond the scope of this work. Lie algebras are interesting algebraic objects and have been studied deeply, but in this work we use $\mathfrak{u}(n)$ because of the exponential map.

Above, it was shown that elements of the algebra can be derived from the group (considering infinitesimal steps away from the identity). There is a reverse operation, allowing elements of the group to be recovered from the algebra: this is the exponential map. In the case of matrix groups, the exponential map is simply the matrix exponential:

Very simply, $L\in\mathfrak{u}(n)$, then $\exp(L)\in U(n)$. While this map is not in general surjective, it so happens that $U(n)$ is a compact, connected group and so $\exp$ is indeed surjective (?). That is, for any $U\in U(n)$, there exists some $L\in\mathfrak{u}(n)$ such that $\exp(L)=U$. Notably, while orthogonal matrices also form a Lie group $O(n)$, with associated Lie algebra $\mathfrak{o}(n)$ consisting of skew-symmetric matrices, $O(n)$ is not connected, and so the exponential map can only produce special orthogonal matrices - those with determinant one - $SO(n)$ being the component of $O(n)$ containing the identity.

Parametrization of U​(n)𝑈𝑛U(n) in terms of 𝔲​(n)𝔲𝑛\mathfrak{u}(n)

The dimension of $\mathfrak{u}(n)$ as a real vector space is $n^{2}$. This is readily derived from noting that an arbitrary $n\times n$ complex matrix has $2n^{2}$ free real parameters, and the requirement of $L^{\dagger}=-L$ imposes $n^{2}$ constraints. So, a set of $n^{2}$ linearly-independent skew-Hermitian matrices defines a basis for the space; $\{T_{j}\}_{j=\{1,\dots,n^{2}\}}$. Then any element $L$ can be written as

where $\{\lambda_{j}\}_{j=1,\dots,n^{2}}$ are $n^{2}$ real numbers; the coefficients of $L$ with respect to the basis. Using the exponential map,

we see that these $\{\lambda_{j}\}_{j=1,\dots,n^{2}}$ suffice as parameters of $U$ (given the basis $T_{j}$). This is the parametrization we propose. It has two attractive properties:

It is a fully general parametrization, as the exponential map is surjective

Gradient updates on $\{\lambda_{j}\}_{j=1,\dots,n^{2}}$ preserve unitarity automatically, as the algebra is closed under addition

This parametrization means gradient steps are taken in the vector space of $\mathfrak{u}(n)$, rather than the manifold of $U(n)$, which may provide a flatter cost landscape - although confirming this intuition would require further analysis. This work is intended to explore the use of this parametrization for learning arbitrary unitary matrices.

There are many possible choices of basis for $\mathfrak{u}(n)$. We went for the following set of sparse matrices:

$n$ diagonal, imaginary matrices: $T_{a}$ is $i$ on the $a$-th diagonal, else zero.

$\frac{n(n-1)}{2}$ symmetric, imaginary matrices with two non-zero elements, e.g., for $n=2$, $\left(\begin{matrix}0&i\\ i&0\end{matrix}\right)$

$\frac{n(n-1)}{2}$ anti-symmetric, real matrices with two non-zero elements, e.g., for $n=2$, $\left(\begin{matrix}0&1\\ -1&0\end{matrix}\right)$

We explore the effects of choice of basis in appendix B.

Derivatives of the matrix exponential

The matrix exponential appearing in Equation 7 poses an issue for gradient calculations. In general, the derivative of the matrix exponential does not have a closed-form expression, so computing gradients is intractable.

In early stages of this work, we used the method of finite differences to approximate gradients, which would prohibit its use in larger-scale applications (such as RNNs). In the appendix we describe an investigation into using random projections to overcome this limitation, which while promising turned out to yield minimal benefit.

We therefore sought mathematical solutions to this complexity issue, which we describe here and in further detail in the appendix. Exploiting the fact that $L$ is skew-Hermitian, we can derive an analytical expression for the derivative of $U$ with respect to each of its parameters, negating the need for finite differences.

where $W$ is a unitary matrix of eigenvectors obtained in the eigenvalue decomposition of $U$; $U=WDW^{\dagger}$, ($D$ = diag($d_{1}$, $\dots$, $d_{n^{2}}$); $d_{i}$ are the eigenvalues of $U$).

Each $V_{a}$ is a matrix defined component-wise

Where $T_{a}$ is the basis matrix of the Lie algebra in the $a$-th direction.

We provide the derivation, based on work from ? (?) and ? (?) in Appendix A.

We can simplify the expression $W^{\dagger}T_{a}W$ for each $T_{a}$, depending on the type of basis element. In these expressions, $\mathbf{w}_{a}$ refers to the $a$-th row of W.

These expressions follow from the sparsity of the basis and are derived in appendix A. Thus, we reduce the calculation of $W^{\dagger}T_{a}W$ from two matrix multplications to at most two vector outer products.

Overall, we have reduced the cost of calculating gradients to a single eigenvalue decomposition, and for each parameter two matrix multiplications (equation 8), one or two vector outer products, and element-wise multiplication of two matrices (equations 9, 10). As we see in the RNN experiments, this actually makes our approach faster than the (restricted)uRNN of (?) for roughly equivalent numbers of parameters.

Supervised Learning of Unitary Operators

We consider the supervised learning problem of learning the unitary matrix $U$ that generated a $\mathbf{y}$ from $\mathbf{x}$; $\mathbf{y}=U\mathbf{x}$, given examples of such $\mathbf{x}$s and $\mathbf{y}$s. This is the core learning problem that needs to be solved for the state-transformation matrix in RNNs. It is similar to the setting considered in ? (?) (they consider an online learning problem). We compare a number of methods for learning $U$ at different values of $n$. We further consider the case where we have artificially restricted the number of learnable variables in our parametrization (for the sake of comparison), and generate a pathological change of basis to demonstrate the relevance of selecting a good basis (appendix B).

While this problem is easily solved in the batch setting using least-squares, we wish to learn $U$ through mini-batch stochastic gradient descent, to emulate a deep learning scenario.

For each experimental run (a single $U$), we generate one million training $\{\mathbf{x}_{j},\mathbf{y}_{j}\}$ pairs, divided into batches of size 20. The test and validation sets both contain $100,000$ examples. In practice we set $\sigma^{2}=0.01$ and use a fixed learning rate of $0.001$. For larger dimensions, we run the model through the data for multiple epochs, shuffling and re-batching each time.

All experiments were implemented in Python. The code is available here: https://github.com/ratschlab/uRNN. For the matrix exponential, we use the scipy builtin expm, which uses Pade approximation (?). We make use of the fact that $iL$ is Hermitian to use eigh (also in scipy) to perform eigenvalue decompositions.

Generating the ground-truth unitary matrix

The $U$ we wish to recover is generated by one of three methods:

QR decomposition: we create a $n\times n$ complex matrix with normally-distributed entries and then perform a QR decomposition, producing a unitary matrix $U$ and an upper triangular matrix (which is discarded). This approach is also used to sample orthogonal matrices in ? (?), noting a result from ? (?) demonstrating that this is equivalent to sampling from the appropriate Haar measure.

Lie algebra: given the standard basis of $\mathfrak{u}(n)$, we sample $n^{2}$ normally-distributed real $\lambda_{j}$ to produce $U=\exp{\left(\sum_{j}\lambda_{j}T_{j}\right)}$

Unitary composition: we compose parametrized unitary operators as in ? (?) (Equation 1). The parameters are sampled as follows: angles in $D$ come from $\mathcal{U}(-\pi,\pi)$. The complex reflection vectors in $\mathbf{R}$ come from $\mathcal{U}(-s,s)$ where $s=\sqrt{\frac{6}{2n}}$.

We study the effects of this generation method on test-set loss in a later section. While we find no significant association between generation method and learning approach, in our experiments we nonetheless average over an equal number of experiments using each method, to compensate for possible unseen bias.

Approaches

We compare the following approaches for learning $U$:

projection: $U$ is represented as an unconstrained $n\times n$ complex matrix, but after each gradient update we project it to the closest unitary matrix, using polar decomposition (?). This amounts to $2n^{2}$ real parameters.

arjovsky: $U$ is parametrized as in Equation 1, which comes to $7n$ real parameters.

lie_algebra: (we refer to this as $\mathfrak{u}(n)$) $U$ is parametrized by its $n^{2}$ real coefficients $\{\lambda_{j}\}$ in the Lie algebra, as in Equation 7.

As baselines we use the true matrix $U$, and a random unitary matrix $U_{R}$ generated by the same method as $U$ (in that experimental run).

We also implemented the algorithm described in ? (?) and considered both unitary and orthogonal learning tasks (our parametrization contains orthogonal matrices as a special case) but found it too numerically unstable and therefore excluded it from our analyses.

Comparison of Approaches

Table 1 shows the test-set loss for different values of $n$ and different approaches for learning $U$. We performed between 6 and 18 replicates of each experiment, and show bootstrap estimates of means and standard errors over these replicates. As we can see, the learning task becomes more challenging as $n$ increases, but our parametrization ($\mathfrak{u}(n)$) consistently outperforms the other approaches.

Restricting to 7​n7𝑛7n parameters

As mentioned, arjovsky uses only $7n$ parameters. To check if this difference accounts for the differences in loss observed in Table 1, we ran experiments where we fixed all but $7n$ (selected randomly) of the $\{\lambda_{j}\}$ in the lie_algebra parametrization. The fixed parameters retained their initial values throughout the experiment. We observe that, as suspected, restricting to $7n$ parameters results in a performance degradation equivalent to that of arjovsky.

Table 2 shows the results for $n=8,14,20$. The fact that the restricted case is consistently within error of the arjovsky model supports our hypothesis that the difference in learnable parameters accounts for the difference in performance. This suggests that generalising the model of ? to allow for $n^{2}$ parameters may result in performance similar to our approach. However, how to go about such a generalisation is unclear, as a naive approach would simply use a composition of $n^{2}$ operators, and this would likely become computationally intractable.

Method of generating U𝑈U

As described, we used three methods to generate the true $U$. One of these produces $U$ in the subspace available to the composition parametrization (Equation 1), so we were curious to see if this parametrization performed better on experiments using that method. We were also concerned that generating $U$ using the Lie algebra parametrization might make the task too ‘easy’ for our approach, as its random initialization could lie close to the true solution.

Figure 1 shows box-plots of the distribution of test losses from these approaches for the three methods, comparing our approach ($\mathfrak{u}(n)$) with that of ? (?), denoted arjovsky. To combine results from experiments using different values of $n$, we first scaled test-set losses by the performance of rand (the random unitary matrix), so the y-axis ranges from 0 (perfect) to 1 (random performance). The dotted line denotes the average (over methods) of the test-set loss for true, similarly scaled. The right panel in Figure 1 shows a zoomed-in version of the $\mathfrak{u}(n)$ result where the comparison with true is more meaningful, and a comparison with the case where we have restricted to $7n$ learnable parameters (see earlier).

We do not observe a difference (within error) between the methods, which is consistent between $\mathfrak{u}(n)$ and arjovsky. Our concern that using the Lie algebra to generate $U$ would make the task ‘too easy’ for $\mathfrak{u}(n)$ was seemingly unfounded.

Unitary Recurrent Neural Network for Long Memory Tasks

To demonstrate that our approach is practical for use in deep learning, we incorporate it into a recurrent neural network to solve standard long-memory tasks. Specifically, we define a general unitary RNN with recurrence relation

where $f$ is a nonlinearity, $\beta$ is a free scaling factor, $U$ is our unitary matrix parametrised as in equation 7, $\mathbf{h}_{t}$ is the hidden state of the RNN and $\mathbf{x}_{t}$ is the input data at ‘time-point’ $t$. We refer to this as a ‘general unitary RNN’ (guRNN), to distinguish it from the restricted uRNN of ? (?).

We use the guRNN on two tasks: the ‘adding problem’ and the ‘memory problem’, first described in (?). For the sake of brevity we refer to (?) for specific experimental details, as we use an identical experimental setup (reproduced in TensorFlow; see above github link for code). We compare our model (guRNN) with the restricted uRNN (ruRNN) parametrised as in equation 1, a LSTM (?), and the IRNN of ? (?). Figure 2 shows the results for each task where the sequence length or the memory duration is $T=100$.

While our model guarantees unitarity of $U$, this is not sufficient to prevent gradients from vanishing. Consider the norm of the gradient of the cost $C$ with respect to the data at time $\tau$, and use submultiplicativity of the norm to write;

where $f^{\prime}$ is a diagonal matrix giving the derivatives of the nonlinearity. Using a unitary matrix fixes $\|U\|=1$, but beyond further restrictions (on $V$ and $\mathbf{b}$) does nothing to control the norm of $f^{\prime}$, which is at most 1 for common nonlinearities. Designing a nonlinearity to better preserve gradient norms is beyond the scope of this work, so we simply scaled $U$ by a constant multiplicative factor $\beta$ to counteract the tendency of the nonlinearity to shrink gradients. In Figure 2 we denote this setup by guRNNβ. Confirming our intuition, this simple modification greatly improves performance on both tasks.

Perhaps owing to our efficient gradient calculation (appendix A) and simpler recurrence relation, our model runs faster than that of (?) (in our implementation), by a factor of 4.8 and 2.6 in the adding and memory tasks shown in Figure 2 respectively. This amounts to the guRNN processing 61.2 and 37.0 examples per second in the two tasks, on a GeForce GTX 1080 GPU.

Discussion

Drawing from the rich theory of Lie groups and Lie algebras, we have described a parametrization of unitary matrices appropriate for use in deep learning. This parametrization exploits the Lie group-Lie algebra correspondence through the exponential map to represent unitary matrices in terms of real coefficients relative to a given basis of the Lie algebra $\mathfrak{u}(n)$. As this map from $\mathfrak{u}(n)$ to $U(n)$ is surjective, the parametrization can describe any unitary matrix.

We have demonstrated that unitary matrices can be learned with high accuracy using simple gradient descent, and that this approach outperforms a recently-proposed parametrization (from ? (?)) and significantly outperforms the approach of ‘re-unitarizing’ after gradient updates. This experimental design is quite simple, designed to probe a core problem, before considering the broader setting of RNNs.

Our experiments with general unitary RNNs using this parametrization showed that this approach is practical for deep learning. With a fraction of the parameters, our model outperforms LSTMs on the standard ‘memory problem’ and attains comparable (although inferior) performance on the adding problem (?). Further work is required to understand the difference in performance between our approach and the ruRNN of (?) - perhaps the $7n$-dimensional subspace captured by their parametrization is serendipitously beneficial for these RNN tasks - although we note that the results presented here are not the fruit of exhaustive hyperparameter exploration. Of particular interest is the impressive performance of both uRNNs on the memory task, where the LSTM and IRNN appear to fail to learn.

While our RNN experiments have demonstrated the utility of using a unitary operator for these tasks, we believe that the role of the nonlinearity in the vanishing and exploding gradient problem must not be discounted. We have shown that a simple scaling factor can help reduce the vanishing gradient problem induced by the choice of nonlinearity. More analysis considering the combination of nonlinearity and transition operator must be performed to better tackle this problem.

The success of our parametrization for unitary operator learning suggests that the approach of performing gradient updates in the Lie algebra is particularly effective. As Lie groups describe many naturally-occuring symmetries, the Lie group-Lie algebra correspondence could be rich for further exploitation to enhance performance in tasks beyond our initial motivation of recurrent neural networks.

Appendix A: Derivation of derivative of the matrix exponential

This derivation draws elements from ? (?) and ? (?).

We assume we can calculate: $dL$, $W$, and $D$ and seek an expression for $dU$.

Pre-multiplying with $W^{\dagger}$ and post-multiplying with $W$:

We can then say that $dU=WVW^{\dagger}$ where

We use the convention that repeated indices denote summation over that index, unless otherwise stated.

Using 19 we get $A_{ii}=dD_{ii}=(W^{\dagger}dLW)_{ii}$

This produces Equation 9 in the main paper.

Off-diagonal case (i≠j𝑖𝑗i\neq j): (no summation over i,j𝑖𝑗i,j)

In this case, the purely diagonal part vanishes. We get:

Remembering that this is all component-wise multiplication (no summation over $i$ and $j$), we can rearrange expressions to get:

This section is specific to our work, as it relies on the choice of basis for $\mathfrak{u}(n)$.

In our case, $dL$ is simple. $L$ is a linear combination of the parameters $\lambda_{i}$;

Where $T_{i}$ are the basis matrices of $\mathfrak{u}(n)$.

We need $W^{\dagger}T_{a}W$ for all $a$. Since the $T_{a}$s are sparse, this is cheaper than performing $n^{2}$ full matrix multiplications, as we demonstrate now.

$T_{a}$ is zero except for a $i$ in the $a$-th position on the diagonal.

where $\mathbf{w}_{a}$ is the $a$-th row of $W$.

$T_{rs}$ is zero except for $i$ in position $(r,s)$ and $(s,r)$.

$T_{rs}$ is zero except for $1$ in position $(r,s)$ and $-1$ in position $(s,r)$.

These reproduce the expressions in the main paper. The outer product of two $n$-dimensional vectors is an $O(n^{2})$ operation, and so this provides a (up to) factor $n$ speed-up on matrix multiplication.

Appendix B: Changing the basis of 𝔲​(n)𝔲𝑛\mathfrak{u}(n)

The Lie group parametrization assumes a fixed basis of $\mathfrak{u}(n)$. Our intuition is that this makes some regions of $U(n)$ more ‘accessible’ to the optimization procedure, elements whose coefficients are small given this basis. Learning a matrix $U$ which came from elsewhere in $U(n)$ may therefore be more challenging. We emulated this ‘change of basis‘ without needing to explicitly construct a new basis by generating a change of basis matrix, $M$. That is, if $V_{j}$ is the $j$-th element of the new basis, it is given by

A change of basis matrix must be full-rank. We generate one by sampling a square, $n^{2}\times n^{2}$ matrix from a continuous uniform distribution $\mathcal{U}(-c,c)$ ($c$ is a constant we vary in experiments, see Figure 3). This is very unlikely to be singular. We choose the $c$ range of the distribution such that $M$ will have ‘large’ values relative to the true matrix $U$, whose parameters $\lambda$ (relative to $T$) are drawn from $\mathcal{N}(0,0.01)$.

Preliminary experiments suggested that the learning rate must be adjusted to compensate for the change of scale - evidence for this is visible in the first column of Figure 3, where changing the basis without changing the learning rate results in an unstable validation set trace. Poor performance resulting from an inappropriate learning rate is not our focus here, so we performed experiments for different values of the learning rate. Figure 3 shows a grid of validation set losses as we vary the learning rate (columns) and the value of $c$ (rows).

Our intuition is that if the performance under the change of basis is purely driven by the difference in scale, using an appropriately-scaled learning rate should negate its affect. Each parameter $\lambda_{j}$ is scaled by a variable uniformly distributed between $(-c,c)$. The expectation value of the absolute value of this quantity is $c^{2}/2$, so we consider learning rates normalised by this factor.

As seen in Figure 3, the graphs on the diagonal are not identical, suggesting that merely scaling the learning rate does not account for the change of learning behavior given a new basis - at least in expectation. Nonetheless, it is reassuring to observe that for all choices of $c$ explored, there exists a learning rate which facilitates learning, even if it markedly slower than the ‘ideal’ case. While having a ‘misspecified’ basis does appear to negatively impact learning, it can be largely overcome with choice of learning rate.

References