Structured Transforms for Small-Footprint Deep Learning

Introduction

Non-linear vector-valued transforms of the form, $f(\mathbf{x},\mathbf{M})=s(\mathbf{M}\mathbf{x})$, where $s$ is an elementwise nonlinearity, $\mathbf{x}$ is an input vector, and $\mathbf{M}$ is an $m\times n$ matrix of parameters are building blocks of complex deep learning pipelines and non-parametric function estimators arising in randomized kernel methods . When $\mathbf{M}$ is a large general dense matrix, the cost of storing $mn$ parameters and computing matrix-vector products in $O(mn)$ time can make it prohibitive to deploy such models on lightweight mobile devices and wearables where battery life is precious and storage is limited. This is particularly relevant for “always-on” mobile applications, such as continuously looking for specific keywords spoken by the user or processing a live video stream onboard a mobile robot. In such settings, the models may need to be hosted on specialized low-power digital signal processing components which are even more resource constrained than the device CPU.

A parsimonious structure typically imposed on parameter matrices is that of low-rankness . If $\mathbf{M}$ is a rank $r$ matrix, with $r\ll\min(m,n)$, then it has a (non-unique) product representation of the form $\mathbf{M}=\mathbf{G}\mathbf{H}^{T}$ where $\mathbf{G},\mathbf{H}$ have only $r$ columns. Clearly, this representation reduces the storage requirements to $(mr+nr)$ parameters, and accelerates the matrix-vector multiplication time to $O(mr+nr)$ via $\mathbf{M}\mathbf{x}=\mathbf{G}(\mathbf{H}^{T}\mathbf{x})$. Another popular structure is that of sparsity typically imposed during optimization via zero-inducing $l_{0}$ or $l_{1}$ regularizers. Other techniques include freezing $\mathbf{M}$ to be a random matrix as motivated via approximations to kernel functions , storing $\mathbf{M}$ in low fixed-precision formats , using specific parameter sharing mechanisms , or training smaller models on outputs of larger models (“distillation”) .

Structured Matrices: An $m\times n$ matrix which can be described in much fewer than $mn$ parameters is referred to as a structured matrix. Typically, the structure should not only reduce memory requirements, but also dramatically accelerate inference and training via fast matrix-vector products and gradient computations. Below are classes of structured matrices arising pervasively in many contexts with different types of parameter sharing (indicated by the color).

Toeplitz matrices have constant values along each of their diagonals. When the same property holds for anti-diagonals, the resulting class of matrices are called Hankel matrices. Toeplitz and Hankel matrices are intimately related to one-dimensional discrete convolutions , and arise naturally in time series analysis and dynamical systems. A Vandermonde matrix is determined by taking elementwise powers of its second column. A very important special case is the complex matrix associated with the Discrete Fourier transform (DFT) which has Vandermonde structure with $v_{j}=\omega_{n}^{j},j=1\ldots n$ where $\omega_{n}=\exp{\frac{-2\pi i}{n}}$ is the primitive $n^{th}$ root of unity. Similarly, the entries of $n\times n$ Cauchy matrices are completely defined by two length $n$ vectors. Vandermonde and Cauchy matrices arise naturally in polynomial and rational interpolation problems.

“Superfast” Numerical Linear Algebra: The structure in these matrices can be exploited for faster linear algebraic operations such as matrix-vector multiplication, inversion and factorization. In particular, the matrix-vector product can be computed in time $O(n\log n)$ for Toeplitz and Hankel matrices, and in time $O(n\log^{2}n)$ for Vandermonde and Cauchy matrices.

Generalizations of Structured Matrices: Consider deriving a matrix by taking arbitrary linear combinations of products of structured matrices and their inverses, e.g. $\alpha_{1}\mathbf{T}_{1}\mathbf{T}_{2}^{-1}+\alpha_{2}\mathbf{T}_{3}\mathbf{T}_{4}^{-1}\mathbf{T}_{5}$ where each $\mathbf{T}_{i}$ is a Toeplitz matrix. The parameter sharing structure in such a derived matrix is by no means apparent anymore. Yet, it turns out that the associated displacement operator remarkably continues to expose the underlying parsimony structure, i.e. such derived matrices are still mapped to relatively low-rank matrices! The displacement rank approach allows fast linear algebra algorithms to be seamlessly extended to these broader classes of matrices. The displacement rank parameter controls the degree of structure in these generalized matrices.

Technical Preview, Contributions and Outline: We propose building deep learning pipelines where parameter matrices belong to the class of generalized structured matrices characterized by low displacement rank. In Section 2, we attempt to give a self-contained overview of the displacement rank approach , drawing key results from the relevant literature on structured matrix computations (proved in our supplementary material for completeness). In Section 3, we show that the proposed structured transforms for deep learning admit fast matrix multiplication and gradient computations, and have rich statistical modeling capacity that can be explicitly controlled by the displacement rank hyperparameter, covering, along a continuum, an entire spectrum of configurations from highly structured to unstructured matrices. While our focus in this paper is on Toeplitz-related transforms, our proposal extends to other structured matrix generalizations. In Section 4, we study inference and training-time acceleration with structured transforms as a function of displacement rank and dimensionality. We find that our approach compares highly favorably with numerous other techniques for learning size-constrained models on several benchmark datasets. Finally, we demonstrate our approach on mobile speech recognition applications where we are able to match the performance of much bigger state of the art models with a fraction of parameters.

Displacement Operators associated with Structured Matrices

We begin by providing a brisk background on the displacement rank approach. Unless otherwise specified, for notational convenience we will henceforth assume squared transforms, i.e., $m=n$, and discuss rectangular transforms later. Proofs of various assertions can be found in our self-contained supplementary material or in .

By carefully choosing $\mathbf{A}$ and $\mathbf{B}$ one can instantiate Sylvester and Stein displacement operators with desirable properties. In particular, for several important classes of displacement operators, $\mathbf{A}$ and/or $\mathbf{B}$ are chosen to be an $f$-unit-circulant matrix defined as follows.

For a real-valued scalar $f$, the $(n\times n)$ f-circulant matrix, denoted by $\mathbf{Z}_{f}$, is defined as follows,

The $f$-unit-circulant matrix is associated with a basic downward shift-and-scale transformation, i.e., the matrix-vector product $\mathbf{Z}_{f}\mathbf{v}$ shifts the elements of the column vector $\mathbf{v}$ “downwards”, and scales and brings the last element $v_{n}$ to the “top”, resulting in [{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}f}v_{n},v_{1},\ldots v_{n-1}]^{T}. It has several basic algebraic properties (see Proposition 1.1 ) that are crucial for the results stated in this section

Figure 2 lists the rank of the Sylvester displacement operator in Eqn 16 when applied to matrices belonging to various structured matrix classes, where the operator matrices $\mathbf{A},\mathbf{B}$ in Eqn. 16 are chosen to be diagonal and/or $f$-unit-circulant. It can be seen that despite the difference in their structures, all these classes are characterized by very low displacement rank. Figure 2 shows how this low-rank transformation happens in the case of a $4\times 4$ Toeplitz matrix (also see section 1, Lemma 1.2 ). Embedded in the $4\times 4$ Toeplitz matrix $\mathbf{T}$ are two copies of a $3\times 3$ Toeplitz matrix shown in black and red boxes. The shift and scale action of $\mathbf{Z}_{1}$ and $\mathbf{Z}_{-1}$ aligns these sub-matrices. By taking the difference, the Sylvester displacement operator nullifies the aligned submatrix leaving a rank 2 matrix with non-zero elements only along its first row and last column. Note that the negative sign introduced by $\mathbf{T}\mathbf{Z}_{-1}$ term prevents the complete zeroing out of the value of $t$ (marked by red star) and is hence critical for invertibility of the displacement action.

In order to express structured matrices with low-displacement rank directly as a function of its low-displacement generators, we need to invert $L$ and obtain a learnable parameterization. For Stein type displacement operator, the following elegant result is known (see proof in ):

where krylov$(\mathbf{A},\mathbf{v})$ is defined by:

Henceforth, our focus in this paper will be on Toeplitz-like matrices for which the displacement operator of interest (see Table 2) is of Sylvester type: $\nabla_{\mathbf{Z}_{1},\mathbf{Z}_{-1}}$. In order to apply Theorem 2.2, one can switch between Sylvester and Stein operators, setting $\mathbf{A}=\mathbf{Z}_{1}$ and $\mathbf{B}=\mathbf{Z}_{-1}$ which both satisfy the conditions of Theorem 2.2 (see property 3, Proposition 1.1 ). The resulting expressions involve Krylov matrices generated by $f$-unit-circulant matrices which are called $f$-circulant matrices in the literature.

Given a vector $\mathbf{v}$, the f-Circulant matrix, $\mathbf{Z}_{f}(\mathbf{v})$, is defined as follows:

Two special cases are of interest: $f=1$ corresponds to Circulant matrices, and $f=-1$ corresponds to skew-Circulant matrices.

Finally, one can obtain an explicit parameterization for Toeplitz-like matrices which turns out to involve taking sums of products of Circulant and skew-Circulant matrices.

Learning Toeplitz-like Structured Transforms

Motivated by Theorem 2.4, we propose learning parameter matrices of the form in Eqn. 32 by optimizing the displacement factors $\mathbf{G},\mathbf{H}$. First, from the properties of displacement operators , it follows that this class of matrices is very rich from a statistical modeling perspective.

The set of all $n\times n$ matrices that can be written as,

All $n\times n$ Circulant and Skew-Circulant matrices for $r\geq 1$.

All $n\times n$ Toeplitz matrices for $r\geq 2$.

Inverses of Toeplitz matrices for $r\geq 2$.

All products of the form $\mathbf{A}_{1}\ldots\mathbf{A}_{t}$ for $r\geq 2t$.

All linear combinations of the form $\sum_{i=1}^{p}\beta_{i}\mathbf{A}^{(i)}_{1}\ldots\mathbf{A}^{(i)}_{t}$ where $r\geq 2tp$.

where each $\mathbf{A}_{i}$ above is a Toeplitz matrix or the inverse of a Toeplitz matrix.

When we learn a parameter matrix structured as Eqn. 33 with displacement rank equal to 1 or 2, we also search over convolutional transforms. In this sense, structured transforms with higher displacement rank generalize (one-dimensional) convolutional layers. The displacement rank provides a knob on modeling capacity: low displacement matrices are highly structured and compact, while high displacement matrices start to contain increasingly unstructured dense matrices.

Next, we show that associated structured transforms of the form $f(\mathbf{x})=\mathbf{M}(\mathbf{G},\mathbf{H})\mathbf{x}$ admit fast evaluation, and gradient computations with respect to $\mathbf{G},\mathbf{H}$. First we recall the following well-known result concerning the diagonalization of $f$-Circulant matrices.

This result implies that for the special cases of $f=1$ and $f=-1$ corresponding to Circulant and Skew-circulant matrices respectively, the matrix-vector multiplication can be computed in $O(n\log n)$ time via the Fast Fourier transform:

where $\bm{\eta}=[1,\eta,\eta^{2}\ldots\eta^{n-1}]^{T}~{}~{}\textrm{where}~{}~{}\eta=(-1)^{\frac{1}{n}}=\exp(i\frac{\pi}{n})$, the root of negative unity.

In particular, a single matrix-vector product for Circulant and Skew-circulant matrices has the computational cost of $3$ FFTs. Therefore, for matrices of the form in Eqn. 33 comprising of $r$ products of Circulant and Skew-Circulant matrices, naively computing a matrix-vector product for a batch of $b$ input vectors would take $6rb$ FFTs. However, this cost can be significantly lowered to that of $2(rb+r+b)$ FFTs by making the following observation:

Given an $n\times b$ matrix $\mathbf{X}$, the matrix-matrix product, $\mathbf{Y}=\left(\sum_{i=1}^{r}\mathbf{Z}_{1}(\mathbf{g}_{i})\mathbf{Z}_{-1}(\mathbf{h}_{i})\right)\mathbf{X}$, can be computed at the cost of $2(rb+b+r)$ FFTs, using the following algorithm.

Set $\bm{\eta}=[1,\eta,\eta^{2}\ldots\eta^{n-1}]^{T}~{}~{}\textrm{where}~{}~{}\eta=(-1)^{\frac{1}{n}}=\exp(i\frac{\pi}{n})$

Set $\mathbf{Y}=\bf{ifft}\left(\mathbf{Y}\right)$

We now show that when our structured transforms are embedded in a deep learning pipeline, the gradient computation can also be accelerated. First, we note that the Jacobian structure of $f$-Circulant matrices has the following pleasing form.

The Jacobian of the map $f(\mathbf{x},\mathbf{v})=\mathbf{Z}_{f}(\mathbf{v})\mathbf{x}$ with respect to the parameters $\mathbf{v}$ is $\mathbf{Z}_{f}(\mathbf{x})$.

This leads to the following expressions for the Jacobians of the structured transforms of interest.

Consider parameterized vector-valued transforms of the form,

The Jacobians of $f$ with respect to the $j^{th}$ column of $\mathbf{G},\mathbf{H}$, i.e. $\mathbf{g}_{j},\mathbf{h}_{j}$, at $\mathbf{x}$, are as follows:

Based on Eqns. 40, 41 the gradient over a minibatch of size $b$ requires computing, $\sum_{i}^{b}[J_{\mathbf{g}_{j}}f|_{\mathbf{x_{i}}}]^{T}\bm{\delta}_{i}$ and $\sum_{i=1}^{b}[J_{\mathbf{h}_{j}}f|_{\mathbf{x_{i}}}]^{T}\bm{\delta}_{i}$ where, $\{\mathbf{x}_{i}\}_{i=1}^{b}$ and $\{\bm{\delta}_{i}\}_{i=1}^{b}$ are batches of forward and backward inputs during backpropagation. These can be naively computed with $6rb$ FFTs. However, as before, by sharing FFT of the forward and backward inputs, and the fft of the parameters, this can be lowered to $(4br+4r+2b)$ FFTs. Below we give matricized implementation.

Let $\mathbf{X},\mathbf{Z}$ be $n\times b$ matrices whose columns are forward and backward inputs respectively of minibatch size $b$ during backpropagation. The gradient with respect to $\mathbf{g}_{j},\mathbf{h}_{j}$ can be computed at the cost of $(4br+4r+2b)$ FFTs as follows:

Gradient wrt $\mathbf{g}_{j}$ ($2b+1$ FFTs)

Gradient wrt $\mathbf{h}_{j}$ ($2b+1$ FFTs)

Rectangular Transforms: Variants of Theorems 2.2, 2.4 exist for rectangular transforms, see . Alternatively, for $m<n$ we can subsample the outputs of square $n\times n$ transforms at the cost of extra computations, while for $m>n$, assuming $m$ is a multiple of $n$, we can stack $\frac{m}{n}$ output vectors of square $n\times n$ transforms.

Empirical Studies

Acceleration with Structured Transforms: In Figure 3, we analyze the speedup obtained in practice using $n\times n$ Circulant and Toeplitz-like matrices relative to a dense unstructured $n\times n$ matrix (fully connected layer) as a function of displacement rank and dimension $n$. Three scenarios are considered: inference speed per test instance, training speed as implicitly dictated by forward passes on a minibatch, and gradient computations on a minibatch. Factors such as differences in cache optimization, SIMD vectorization and multithreading between Level-2 BLAS (matrix-vector multiplication), Level-3 BLAS (matrix-matrix multiplication) and FFT implementations (we use FFTW: http://www.fftw.org) influence the speedup observed in practice. Speedup gains start to show for dimensions as small as $512$ for Circulant matrices. The gains become dramatic with acceleration of the order of 10 to 100 times for several thousand dimensions, even for higher displacement rank Toeplitz-like transforms.

Effectiveness for learning compact Neural Networks: Next, we compare the proposed structured transforms with several existing techniques for learning compact feedforward neural networks. We exactly replicate the experimental setting from the recent paper on HashedNets which uses several image classification datasets first prepared by . mnist is the original $10$-class MNIST digit classification dataset with $60000$ training examples and $10000$ test examples. bg-img-rot refers to a challenging version of mnist where digits are randomly rotated and placed against a random black and white background. rect ($1200$ training images, $50000$ test images) and convex ($8000$ training images, $50000$ test images) are $2$-class binary image datasets where the task is to distinguish between tall and wide rectangles, and whether the “on” pixels form a convex region or not, respectively. In all datasets, input images are of size $28\times 28$. Several existing techniques are benchmarked in for compressing a reference single hidden layer model with $1000$ hidden nodes.

Random Edge Removal (RER) where a fraction of weights are randomly frozen to be zero-valued.

Neural Network (NN) where the hidden layer size is reduced to satisfy a parameter budget.

Dark Knowledge (DK) : A small neural network is trained with respect to both the original labeled data, as well as soft targets generated by a full uncompressed neural network.

HashedNets (HN) : This approach uses a low-cost hash function to randomly group connection weights which share the same value.

HashedNets with Dark Knowledge (HNDK): Trains a HashedNet with respect to both the original labeled data, as well as soft targets generated by a full uncompressed neural network.

We consider learning models of comparable size with the weights in the hidden layer structured as a Toeplitz-like matrix. We also compare with the Fastfood approach of where the weight matrix is a product of diagonal parameter matrices and fixed permutation and Walsh-Hadamard matrices, also admitting $O(n\log n)$ multiplication and gradient computation time. The Circulant Neural Network approach proposed in is a special case of our framework (Theorem 3.1).

Results in Table 1 show that Toeplitz-like structured transforms outperform all competing approaches on all datasets, sometimes by a very significant margin, with similar or drastically lesser number of parameters. It should also be noted that while random weight tying in HashedNets reduces the number of parameters, the lack of structure in the resulting weight matrix cannot be exploited for FFT-like $O(n\log n)$ multiplication time. We note in passing that for HashedNets weight matrices whose entries assume only one of $B$ distinct values, the Mailman algorithm can be used for faster matrix-vector multiplication, with complexity $O(n^{2}\log(B)/(\log n))$, which still is much slower than matrix-vector multiplication time for Toeplitz-like matrices. Also note that the distillation ideas of are complementary to our approach and can further improve our results.

Mobile Speech Recognition: We now demonstrate the techniques developed in this paper on a speech recognition application meant for mobile deployment. Specifically, we consider a keyword spotting (KWS) task, where a deep neural network is trained to detect a specific phrase, such as “Ok Google” . The data used for these experiments consists of $10-15K$ utterances of selected phrases (such as “play-music”, “decline-call”), and a larger set of $396K$ utterances to serve as negative training examples. The utterances were randomly split into training, development and evaluation sets in the ratio of $80:5:15$. We created a noisy evaluation set by artificially adding babble-type cafeteria noise at 0dB SNR to the “play-music” clean data set. We will refer to this noisy data set as cafe0. We refer the reader to for more details about the datasets. We consider the task of shrinking a large model for this task whose architecture is as follows : the input layer consists of 40 dimensional log-mel filterbanks, stacked with a temporal context of 32, to produce an input of $32\times 40$ whose dimensions are in time and frequency respectively. This input is fed to a convolutional layer with filter size $32\times 8$, frequency stride 4 and $186$ filters. The output of the convolutional layer is of size $9\times 186=1674$. The output of this layer is fed to a $1674\times 1674$ fully connected layer, followed by a softmax layer for predicting $4$ classes constituting the phrase “play-music”. The full training set contains about $90$ million samples. We use asynchronous distributed stochastic gradient descent (SGD) in a parameter server framework , with $25$ worker nodes for optimizing various models. The global learning rate is set to $0.002$, while our structured transform layers use a layer-specific learning rate of $0.0005$; both are decayed by an exponential factor of $0.1$.

Results with 11 different models are reported in Figure 4 (left) including the state of the art keyword spotting model developed in . At an operating point of 1 False Alarm per hour, the following observations can be made: With just $3348$ parameters, a displacement rank=1 Toeplitz-like structured transform outperforms a standard low-rank bottleneck model with rank=$16$ containing $16$ times more parameters; it also lowers false reject rates from $10.2\%$ with Circulant and $14.2\%$ with Fastfood transforms to about $8.2\%$. With displacement rank $10$, the false reject rate is $6.2\%$, in comparison to $6.8\%$ with the $3$ times larger rank=$32$ standard low-rank bottleneck model. Our best Toeplitz-like model comes within $0.4\%$ of the performance of the $80$-times larger fully-connected and $3.6$ times larger reference models. In terms of raw classification accuracy as a function of training time, Figure 4 (right) shows that our models (with displacement ranks $1,2$ and $10$) come within $0.2\%$ accuracy of the fully-connected and reference models, and easily provide much better accuracy-time tradeoffs in comparison to standard low-rank bottleneck models, Circulant and Fastfood baselines. The conclusions are similar for other noise conditions (see supplementary material ).

Perspective

We have introduced and shown the effectiveness of new notions of parsimony rooted in the theory of structured matrices. Our proposal can be extended to various other structured matrix classes, including Block and multi-level Toeplitz-like matrices related to multidimensional convolution . We hope that such ideas might lead to new generalizations of Convolutional Neural Networks.

Acknowledgements: We thank Krzysztof Choromanski, Carolina Parada, Rohit Prabhavalkar, Rajat Monga, Baris Sumengen, Kilian Weinberger and Wenlin Chen for their contributions to this work.

References