A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels

Introduction

inline, caption=, color=pink] List of planned changes to the second ICLR submission.

Reveal author names / acknowledgments when the paper is finally accepted.

Don’t do anything since it’s rather clear from our exposition:

R4.1: Clarify more that we don’t “need” to consider physics, but that is is just a useful analogy and that both situations are just based on representastion theory. MY (Leon’s) OPINION: I think this is actually pretty clear, we emphasize often that this is just an analogy. Do you think this could be missunderstood?

Maurice: Zu CG-Nets in Related Work Section: Klar machen, dass er SO(3) Features und deren Tensor Produkte betrachtet, nicht SO(2) Features, und, dass die Operationen nicht lokal sind. Zu Orthogonalität: ”Falls ”orthogonal” heisst, das beide designs miteinander verbunden werden koennen bin ich nicht ganz sicher. Wahrscheinlcih geht das schon irgendwie, ist aber nicht ganz offensichtlich.”

Maurice: Mehr Credit an TFN geben, weil die bereits unsere Lösung (inklusive CG Koeffizienten) vorhergesehen haben.

R1: Clarify the expected impact in the applications section: For G-equivariant convolutions, one needs G-steerable kernels, and we provide the kernel solutions. O(3) and SU(N) for future results.

R4: Clarify that we don’t provide “full solutions”, but that we reduce the problem to finding CG coefficients, harmonic basis functions, and endomorphisms, at least in intro and conclusion (or frame it as “We find the general structure of the solution”). (This is more or less done, see the several to do notes with changes on this)

R4.2: Use notation $d_{j}$ instead of $[j]$, and the same for $[l]$ and $[J]$.

R4.3: Explain how to come from the solutions for irreps to the solutions for general representations.

R4.4: Explain early and clearly enough why endomorphisms are necessary to consider. Also cite Broecker/Dieck.

R4.5: Make Thm. C.7 precise by directly working with the limit.

R4.6: Replace “which is zero for almost all J” by “which is zero for all but finitely many”.

R4.7: Always write $\operatorname{SO}(2)$ instead of $\operatorname{U}(1)$, but then explain in the one section in the appendix why we think it is useful to switch.

Undoubtedly, symmetries play a central role in the formulation of physical theories. Any imposed symmetry greatly reduces the set of admissible physical laws and dynamics. Specifically in quantum mechanics, the Hilbert space of a system is equipped with a group representation which specifies the transformation law of system states. Quantum mechanical operators, which map between different states, are required to respect these transformation laws. That is, any symmetry transformation of a state on which they act should lead to a corresponding transformation of the resulting state after their action. This requirement imposes a symmetry constraint on the operators themselves – only specific operators can map between a given pair of states.

The situation in equivariant deep learning is remarkably similar to that in physics. Instead of a physical system, one considers in this case some learning task subject to symmetries. For instance, image segmentation is usually assumed to be translationally symmetric: a shift of the input image should lead to a corresponding shift of the predicted segmentation mask. Convolutional networks guarantee this property via their inherent translation equivariance. The role of the quantum states is in equivariant deep learning taken by the features in each layer, which are due to the enforced equivariance endowed with some transformation law. The analog of quantum mechanical operators, mapping between states, is the neural connectivity, mapping between features of consecutive layers. As in the case of operators, there is a symmetry (equivariance) constraint on the neural connectivity – only specific connectivity patterns guarantee a correct transformation law of the resulting features.

In this work we are considering group equivariant convolutional networks (GCNNs), which are convolutional networks that are equivariant w.r.t. symmetries of the space on which the convolution is performed. Typical examples are isometry equivariant CNNs on Euclidean spaces (Weiler & Cesa, 2019) or spherical CNNs (Cohen et al., 2018). Many different formulations of GCNNs have been proposed, however, it has recently been shown that $H$-equivariant GCNNs on homogeneous spaces $H/G$ can in a fairly general setting be understood as performing convolutions with $G$-steerable kernels (Cohen et al., 2019b). Convolutional weight sharing hereby guarantees the equivariance under “translations” of the space while $G$-steerability is a constraint on the convolution kernel that ensures its equivariance under the action of the stabilizer subgroup $G<H$. Although the space of $G$-steerable kernels has been characterized for specific choices of groups $G$ and feature transformation laws, i.e., group representations $\rho$, see Section 5, no general solution was known so far. This work characterizes the solution space for arbitrary compact groups $G$.

Our solution is motivated by the close resemblance of the $G$-steerability kernel constraint to the defining constraint of spherical tensor operators (or more general representation operators (Jeevanjee, 2011)) in quantum mechanics. The famous Wigner-Eckart theorem describes the general structure of these operators by Clebsch-Gordan coefficients, with the degrees of freedom given by reduced matrix elements. By generalizing this theorem, we find a general characterization and parameterization of $G$-steerable kernel spaces. For specific examples, like $G=\operatorname{SO}(3)$ or compact subgroups of $G=\operatorname{O}(2)$, our kernel space solution specializes to earlier work, e.g., Worrall et al. (2016); Thomas et al. (2018); Weiler & Cesa (2019). Our main contributions are the following:

We present a generalized Wigner-Eckart theorem 4.1 for $G$-steerable kernels. It describes the general structure of equivariant kernels in terms of 1) endomorphism bases, which generalize reduced matrix elements, 2) Clebsch-Gordan coefficients, and 3) harmonic basis functions on a suitable homogeneous space. In contrast to the usual formulation, we cover any compact group $G$ and both real and complex representations.

Corollary 4.2 explains how to parameterize $G$-steerable kernels and thus GCNNs.

We apply the theorem exemplarily to solve for the kernel spaces for the symmetry groups $\operatorname{SO}(2)\,$, $\mathds{Z}/2\,$, $\operatorname{SO}(3)\,$ and $\operatorname{O}(3)\,$, considering both real and complex representations. Thereby, we demonstrate that the endomorphism bases, Clebsch-Gordan coefficients, and harmonic basis functions can usually be determined for practically relevant symmetry groups.

This paper is organized as follows: Section 2 motivates our investigation by highlighting analogies between representation operators and $G$-steerable kernels. Section 3 concisely introduces mathematical concepts which are in Section 4 used to formulate our Wigner-Eckart theorem for steerable kernels. The following Sections 5 and 6 put our result in context to prior work and give a recipe for constructing steerable kernel bases in practice. Example applications of this recipe are found in Appendix E.

As the full and detailed proofs underlying our generalized Wigner-Eckart theorem are rather lengthy, the reader can find them together with all required background knowledge on representation theory in the appendix. The main part of this paper states the key concepts and results in a self-contained way and gives a short outline of the proofs.

Symmetry-constrained Operators and their Matrix Elements

To motivate our generalized Wigner-Eckart theorem, we review quantum mechanical representation operators and $G$-steerable kernels with an emphasis on the similarity of their underlying symmetry constraints. Due to their symmetries, the matrix elements of such operators and kernels are fully specified by a comparatively small number of reduced matrix elements or learnable parameters, respectively. This reduction is for representation operators described by the Wigner-Eckart theorem. For clarity, we discuss this theorem in its most popular form, i.e., for spherical tensor operators ($\operatorname{SO}(3)$-representation operators transforming under irreducible representations).

Consider a quantum mechanical system with symmetry under the action of some group $G$, for instance rotations. The action of this symmetry group on quantum states is modeled by some unitary $G$-representation Unitary representations are explained in Section 3. The notation $U$ for the operator is distinct from the notation U of the unitary group $\operatorname{U}(H)$. $U:G\to\operatorname{U}(\mathcal{H})$ on the Hilbert space $\mathcal{H}$. More specifically, $G$ acts on kets according to $|\psi\rangle\mapsto|\psi^{\prime}\rangle:=U(g)\,|\psi\rangle$ and on bras according to $\langle\psi|\mapsto\langle\psi^{\prime}|:=\langle\psi|\,U(g)^{\dagger}$, where $U(g)^{\dagger}$ is the adjoint of $U(g)$. Observables of the system correspond to self-adjoint operators $A=A^{\dagger}$. The expectation value of such an observable in some quantum state $|\psi\rangle$ is given by $\langle\psi|A|\psi\rangle\in\mathds{R}$.

The transformation behaviors of states and observables need to be consistent with each other. As an example, consider a system consisting of a single, free particle in $\mathds{R}^{3}$, which is (among other symmetries) symmetric under rotations $G=\operatorname{SO}(3)$. The momentum of the particle in the direction of the three frame axes is measured by the three momentum operators $(P_{1},P_{2},P_{3})$. Since the momentum of a classical particle transforms geometrically like a vector, one needs to demand the same for the momentum observable expectation values. If we denote by $p_{i}:=\langle\psi|P_{i}|\psi\rangle$ the expected momentum in $i$-direction, this means that the expected momentum of a rotated system is given by

where $R\in\operatorname{SO}(3)$ is an element of the rotation group. This result should agree with the expectation values for rotated system states, that is,

As this argument is independent from the particular choice of state $|\psi\rangle$, and making use of the linearity of the operations, this implies a consistency constraint

which identifies the collection $(P_{1},P_{2},P_{3})$ as a vector operator. Other geometric quantities are required to satisfy similar constraints: For instance, energy is a scalar (i.e., invariant) quantity and the Hamilton operator $H$ is a scalar operator, satisfying $H=U(R)^{\dagger}\mkern 1.0muH\mkern 1.0muU(R)$. Similarly, any matrix valued classical quantity corresponds to a rank $(1,1)$ Cartesian tensor operator $(M_{ij})_{i,j=1,2,3}$ subject to $\sum_{kl}R_{ik}\mkern 1.0muM_{kl}\mkern 1.0mu(R^{-1})_{lj}=U(R)^{\dagger}\mkern 1.0muM_{ij}\mkern 1.0muU(R)$. The overarching framework to study such situations is the notion of a representation operator, which we define as a family of operators $(A_{1},\dots,A_{N})$ which are required to satisfy the constraint

where $\pi:G\to\operatorname{U}(\mathds{C}^{N})$ is some unitary representation of the symmetry group under consideration. The examples above correspond to specific choices of representations, namely the trivial representation $\pi(R)=1$ for scalars, the “standard” representation $\pi(R)=R$ for vectors and the tensor product representation $\pi(R)=R\otimes(R^{-1})^{\!\top}$ for matrices. Spherical tensor operators, discussed below, correspond to the irreps (irreducible representations) of $\operatorname{SO}(3)$.

Convolution kernels of group equivariant CNNs are required to satisfy a very similar constraint to that in Eq. (1). Before coming to such GCNNs, consider the case of conventional CNNs, processing image-like signals on a Euclidean space $\mathds{R}^{d}$. Such signals are formalized as $c$-channel feature maps $f:\mathds{R}^{d}\to\mathds{K}^{c}$ that assign a $c$-dimensional feature vector $f(x)\in\mathds{K}^{c}$ to each point $x\in\mathds{R}^{d}$, where we allow for $\mathds{K}$ being either of the real or complex numbers $\mathds{R}$ or $\mathds{C}$. Each CNN layer maps its input feature map $f_{\textup{in}}:\mathds{R}^{d}\to\mathds{K}^{c_{\textup{in}}}$ via a convolution to an output feature map $f_{\textup{out}}:=K\star f_{\textup{in}}:\mathds{R}^{d}\to\mathds{K}^{c_{\textup{out}}}$. Since the convolution maps $c_{\textup{in}}$ input channels to $c_{\textup{out}}$ output channels, the kernel $K:\mathds{R}^{d}\to\mathds{K}^{c_{\textup{out}}\times c_{\textup{in}}}$ is matrix-valued.

Conventional CNNs are translation equivariant, however it is often desirable that the convolution is equivariant w.r.t. a larger symmetry group, for instance the isometries $\operatorname{E}(d)$ of $\mathds{R}^{d}$ (Weiler & Cesa, 2019). For simplicity, we consider semidirect product groups of the form $(\mathds{R}^{d},+)\rtimes G$, where $G\leq\operatorname{GL}(d)$ is any compact group. Group elements $tg\in(\mathds{R}^{d},+)\rtimes G$ are uniquely split into a translation $t\in(\mathds{R}^{d},+)$ and an element $g\in G$, stabilizing the origin. They act on $\mathds{R}^{d}$ according to $x\mapsto(tg)\cdot x:=gx+t$. The equivariance of a GCNN – which is the analog to the symmetry of a quantum mechanical system – requires the feature spaces to be endowed with a group action of the symmetry group. A natural choice is to model the feature spaces as spaces of feature fields, for instance scalar, vector or tensor fields (Cohen & Welling, 2016b).

Such feature fields are defined as functions $f:\mathds{R}^{d}\to V$, where the difference to conventional feature maps is that the space $V\cong\mathds{K}^{c}$ of feature vectors is equipped with a group representation $\rho:G\to\operatorname{GL}(V)$ of the stabilizer $G$. The full symmetry group acts on feature fields according to $f\mapsto(tg)\cdot f:=\rho(g)\circ f\circ(tg)^{-1}$, which is known as the induced representation of $\rho$. As proven in (Weiler et al., 2018a), the most general linear and equivariant map from an input field $f_{\textup{in}}:\mathds{R}^{d}\to V_{\textup{in}}$ to an output field $f_{\textup{out}}:\mathds{R}^{d}\to V_{\textup{out}}$ is a convolution with a $G$-steerable kernel $K:\mathds{R}^{d}\to\operatorname{Hom}_{\mathds{K}}(V_{\textup{in}},V_{\textup{out}})\cong\mathds{K}^{c_{\textup{out}}\times c_{\textup{in}}}$. Such kernels take values in the space of linear operators from $V_{\textup{in}}$ to $V_{\textup{out}}$ and are required to satisfy the $G$-steerability (equivariance) constraint

One can easily check that a convolution with a $G$-steerable kernel $K$ is indeed equivariant, i.e., satisfies $K\star((tg)\cdot f)=(tg)\cdot(K\star f)$ for any $tg\in(\mathds{R}^{d},+)\rtimes G$. This result was later generalized to feature fields on homogeneous spaces $H/G$ of unimodular locally compact groups $H$ (Cohen et al., 2019b) and on Riemannian manifolds with structure group $G$ (Cohen et al., 2019a). That the equivariance of the convolutional network requires $G$-steerable kernels in any of these settings underlines the great practical relevance of our results.

The two constraints, Eq. (1) and Eq. (2), are remarkably similar: the left-hand-sides are in both cases given by a $G$-transformation of the operator or kernel itself while the right-hand-sides are given by pre- and postcomposition of the operator or kernel with unitary representations. More details on this comparison can be found in Appendix C.1.3.

All information about a linear operator $A:\mathcal{H}\to\mathcal{H}$ is encoded by its matrix elements $A_{\mu\nu}:=\langle\mu|A|\nu\rangle\in\mathds{C}$ relative to a given basis, where $|\nu\rangle\in\mathcal{H}$ and $\langle\mu|\in\mathcal{H}^{*}$ denote basis elements of the Hilbert space and its dual. Similarly, all information about a convolution kernel $K$ is encoded by its matrix elements $K_{\mu\nu}(x):={\langle\mu|K(x)|\nu\rangle\in\mathds{K}}$, where $|\nu\rangle\in V_{\textup{in}}$ and $\langle\mu|\in V_{\textup{out}}^{*}$ are elements of chosen bases for the input representation and dual output representation. Considering general operators and kernels, i.e., ignoring the symmetry constraints in Eqs. (1) and (2), all matrix elements are independent degrees of freedom. In the case of convolution kernels, they correspond directly to the ${c_{\textup{out}}\mkern-2.5mu\cdot\mkern-1.5muc_{\textup{in}}}$ learnable parameters for every point of the kernel. However, if $A$ is a representation operator – or if $K$ is a $G$-steerable kernel – the symmetry constraints couple the matrix elements to each other such that they can not be chosen freely anymore. For representation operators, this statement is made precise by the Wigner-Eckart theorem.

The Wigner-Eckart theorem is best known in its classical form, which applies specifically to spherical tensor operators. These operators are the representation operators for the irreps of $\operatorname{SO}(3)$, i.e., the Wigner D-matrices ${D_{j}:\operatorname{SO}(3)\to\operatorname{U}(\mathds{C}^{2j+1})}$. As such, spherical tensor operators of rank $j$ are defined as families $\bm{T}_{j}=(T_{j}^{-j},\dots,T_{j}^{j})^{\top}$ of $2j+1$ operators $T_{j}^{m}$ that satisfy the constraint

In order to express the operators $T_{j}^{m}$ in terms of matrix elements, we need to fix a basis of $\mathcal{H}$. Due to the $\operatorname{SO}(3)$-symmetry of $\bm{T}_{j}$, a natural choice are the angular momentum eigenstates The system could in general have further quantum numbers, which we suppress here for simplicity. $|ln\rangle$, where $l\in\mathds{N}_{\geq 0}$ and $n=-l,\ldots,l$. For fixed quantum numbers $j$, $l$, and $J$, there are $2j+1$ components $T_{j}^{m}$ of $\bm{T}_{j}$, $2l+1$ basis kets $|ln\rangle$, and $2J+1$ basis bras $\langle JM|$. This implies that there are $(2J+1)(2j+1)(2l+1)$ different matrix elements $\langle JM|\,T_{j}^{m}\,|ln\rangle\in\mathds{C}$ for these quantum numbers. According to the Wigner-Eckart theorem, all of these matrix elements are fully specified by one single number (Jeevanjee, 2011):

Let $j,l,J\in\mathds{N}_{\geq 0}$ and let $\bm{T}_{j}$ be a spherical tensor operator of rank $j$. Then there is a unique complex number, the reduced matrix element $\lambda\in\mathds{C}$ (often written $\langle J\|\bm{T}_{j}\|l\rangle\in\mathds{C}$), that completely determines any of the ${(2J+1)}{(2j+1)}{(2l+1)}$ matrix elements $\langle JM|\mkern 2.0muT_{j}^{m}\mkern 2.0mu|ln\rangle$ by the relation

The coupling coefficients $\langle JM|jm;ln\rangle$, known as Clebsch-Gordan coefficients, are given by the projection of the tensor product basis $|jm;ln\rangle:=|jm\rangle\otimes|ln\rangle$ on $|JM\rangle$. They are purely algebraic and therefore independent of the spherical tensor operator $\bm{T}_{j}$.

This result generalizes to arbitrary representation operators of the form in Eq. (1) (Agrawala, 1980). The similarities between representation operators and $G$-steerable kernels suggests that a similar statement might hold for the matrix elements of $G$-steerable kernels as well. As proven below, this is indeed the case: our generalized Wigner-Eckart theorem separates their independent degrees of freedom from purely algebraic relations between mutually dependent matrix elements. It does therefore give an explicit parametrization of the space of $G$-steerable kernels.

Building Blocks of Steerable Kernels

This chapter gives a brief introduction to the mathematical concepts that are required to formulate our Wigner-Eckart theorem for $G$-steerable kernels. The first two of the following paragraphs explain why it is w.l.o.g. possible to restrict attention to steerable kernels on homogeneous spaces and to irreducible representations. The following three paragraphs discuss the building blocks of steerable kernels, which are endomorphisms, harmonic basis functions described by the Peter-Weyl theorem, and tensor product representations and their Clebsch-Gordan decomposition. An illustration of the concepts introduced in this chapter is given in Appendix A. The reader may jump back and forth between the technical definitions here and the running example in the appendix.

Convolution kernels are usually defined on a Euclidean space $\mathds{R}^{d}$, i.e., they are functions $K:\mathds{R}^{d}\to\operatorname{Hom}_{\mathds{K}}(V_{\operatorname{in}},V_{\operatorname{out}})$. The $G$-steerability constraint in Eq. (2) relates kernel values $K(x)$ at $x$ to kernel values $K(gx)$ at all other points $gx$ on the orbit ${Gx}:=\{gx\,|\,g\in G\}$ of $x$. To solve the constraint, it is therefore w.l.o.g. sufficient to consider restrictions of kernels to the individual orbits, from which the full solution on $\mathds{R}^{d}$ can be assembled (Weiler et al., 2018a). By construction, the orbits have the structure of a homogeneous space:

Let $\,\cdot:G\times X\to X$ be a continuous action of a compact group $G$ on a topological space $X$. Then $X$ is called a homogeneous space w.r.t. $G$ if $\emptyset\neq X$ and if for all $x,y\in X$ there is a $g\in G$ such that $gx=y$. The action is then called transitive.

We will in the following w.l.o.g. consider steerable kernels $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{\operatorname{in}},V_{\operatorname{out}})$ on such homogeneous spaces $X$.

The theorems below apply specifically to unitary representations, that is, representations for which the automorphisms $\rho(g)$ preserve distances (Knapp, 2002). As asserted by Theorem B.20, this is not really a restriction as every finite-dimensional linear representation can be considered as being unitary. Thus, we assume $\rho:G\to\operatorname{U}(V)$, where $\operatorname{U}(V)$ is the unitary group, i.e., the group of distance-preserving linear functions on $V$. In the case of $\mathds{K}=\mathds{R}$ we say orthogonal instead of unitary and write $\operatorname{O}(V)$.

Additionally, prior research has shown that it is sufficient to solve the kernel constraint in Eq. (2) for irreducible (unitary) input- and output representations instead of arbitrary finite-dimensional representations (Weiler & Cesa, 2019). This is possible due to the linearity of the constraint and the fact that any finite-dimensional unitary representation decomposes by Proposition B.38 into an orthogonal direct sum of irreps. The solution for general representations can thus be recovered from the solutions for irreps. More details on these considerations can be found in Section D.1.3.

If two unitary irreps are related by an isometric intertwiner, they are isomorphic; see Definition B.18. The set of isomorphism classes of unitary irreps of $G$ is denoted by $\widehat{G}$. We assume that for each isomorphism class $j\in\widehat{G}$ we have picked a representative irrep $\rho_{j}:G\to\operatorname{U}(V_{j})$. We denote by $d_{j}$ the dimension of $V_{j}$, so that we have $V_{j}\cong\mathds{K}^{d_{j}}$.

Overall, we can w.l.o.g. replace $\mathds{R}^{d}$ with $X$ and $\rho_{\operatorname{in}}$ and $\rho_{\operatorname{out}}$ by $\rho_{l}:G\to\operatorname{U}(V_{l})$ and $\rho_{J}:G\to\operatorname{U}(V_{J})$, where $X$ is a homogeneous space and $\rho_{l}$ and $\rho_{J}$ are (representatives of isomorphism classes of) irreducible unitary representations of $G$. This leads to our working definition of steerable kernels, to which we restrict from now on:

Let $X$ be a homogeneous space of $G$ and $\rho_{l}:G\to\operatorname{U}(V_{l})$ and $\rho_{J}:G\to\operatorname{U}(V_{J})$ be representatives of isomorphism classes of irreducible unitary representations of $G$. A $G$-steerable kernel (on a homogeneous space and w.r.t. unitary irreps) is any function $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$ such that the following $G$-steerability constraint holds:

An important concept, underlying the reduced matrix elements in the Wigner-Eckart theorem for spherical tensor operators, is that of endomorphisms of linear representations.

Let $\rho:G\to\operatorname{GL}(V)$ be a linear representation. An endomorphism of $\rho$ is a linear map $c:V\to V$ that commutes with $\rho$, i.e., which satisfies $c\circ\rho(g)=\rho(g)\circ c$ for all $g\in G$. The space of all endomorphisms of $\rho$ is written $\operatorname{End}_{G,\mathds{K}}(V)$.

Endomorphisms play a central role in our generalized Wigner-Eckart theorem for steerable kernels. To get an insight why this is the case, consider a given steerable kernel $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$. The post-composition $(c\circ K)(x)\coloneqq c\circ(K(x))$ of this kernel with any endomorphism $c\in\operatorname{End}_{G,\mathds{K}}(V_{J})$ is obviously still steerable, i.e., satisfies Eq. (3). A basis of the space of steerable kernels is therefore partly explained by bases of the endomorphism spaces, and thus occurs in our general solution. In the following, we write $\{c_{r}\mid r=1,\dots,E_{J}\}$ for the basis of $\operatorname{End}_{G,\mathds{K}}(V_{J})$, where $E_{J}\coloneqq\operatorname{dim}(\operatorname{End}_{G,\mathds{K}}(V_{J}))$ is the dimension of the endomorphism space.

How complicated can the space of endomorphisms be? For $\mathds{K}=\mathds{C}$, Schur’s Lemma D.8 tells us that the endomorphism spaces of irreducible representations are always $1$-dimensional, generated by the identity. In that case, one can omit considering endomorphisms in our final description of basis kernels. For $\mathds{K}=\mathds{R}$, however, one can show that the endomorphism spaces of irreducible representations have either $1$, $2$, or $4$ dimensions, and such representations are then correspondingly called of real type, complex type, and quaternionic type, see Bröcker & Dieck (2003), Theorem II.$6.3$.

inline]The paragraph above was not in the original ICLR submission and is supposed to make the reviewer happy who was originally confused that we need endomorphisms. inline, backgroundcolor=green] I like the sentence but it takes lots of space….

A cornerstone in our proof of the Wigner-Eckart theorem for steerable kernels is Theorem C.7. It states that the space of steerable kernels, which are $G$-equivariant maps $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$, is isomorphic to the space of linear $G$-equivariant maps of the form $\widehat{K}:L^{2}_{\mathds{K}}(X)\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$. We are therefore interested in the representation theory of $L^{2}_{\mathds{K}}(X)$, which is described by the Peter-Weyl theorem. Usually, the Peter-Weyl theorem uses $G$ itself as the homogeneous space and is formulated for complex representations (Knapp, 2002). However, generalizations to arbitrary homogeneous spaces and real representations are possible, as we explain in Appendix B.2

Let $G$ be a compact group and $X$ a homogeneous space. Let $\widehat{G}$ be the set of isomorphism classes of irreducible representations. For $j\in\widehat{G}$, let $\rho_{j}:G\to\operatorname{U}(V_{j})$ be a representative with dimension $d_{j}=\operatorname{dim}(V_{j})$. Then there are multiplicities $m_{j}\in\mathds{N}_{\geq 0}$ with $m_{j}\leq d_{j}$, and for each $i=1,\dots,m_{j}$ there are harmonic basis functions $Y_{ji}^{m}:X\to\mathds{K},\ m=1,\dots,d_{j}$, such that the following three properties hold:

The $Y_{ji}^{m}$, for fixed $j$ and $i$, are steerable (Freeman & Adelson, 1991; Hel-Or & Teo, 1998), i.e., transformation via $g\in G$ can be expressed by shifting basis coefficients with $\rho_{j}$:

Any square-integrable function $f:X\to\mathds{K}$ can be uniquely expanded in terms of harmonic basis functions, i.e.,

with coefficients $\lambda_{jim}\!\in\mathds{K}$.

The $Y_{ji}^{m}$ are an orthonormal system with respect to the scalar product given by integration:

Note the similarity of these properties to those encountered in usual Fourier analysis. Indeed, the Peter-Weyl theorem can be viewed as describing the harmonic analysis on arbitrary compact groups and their homogeneous spaces.

From a representation theoretic viewpoint, the functions $Y_{ji}^{m}$ for fixed $j$ and $i$ span an irreducible subrepresentation $V_{ji}$ of the unitary representation $\lambda:G\to\operatorname{U}(L^{2}_{\mathds{K}}(X))$ given by $\left[\lambda(g)f\right](x)\coloneqq f(g^{-1}x)$. $L^{2}_{\mathds{K}}(X)$ then splits into an orthogonal direct sum $L^{2}_{\mathds{K}}(X)=\widehat{\bigoplus}_{j\in\widehat{G}}\bigoplus_{i=1}^{m_{j}}V_{ji}$. This viewpoint is explained in the equivalent, more representation theoretic formulation of the Peter-Weyl theorem in Theorem B.22.

The last ingredients that we need to discuss are tensor product representations and Clebsch-Gordan coefficients. They appear, roughly speaking, in the following way: the kernel $K$ can be thought of as being built from harmonic basis functions $Y_{ji}^{m}$ which transform according to the corresponding irrep $\rho_{j}$. When a harmonic kernel component of type $\rho_{j}$ acts on an input feature field of type $\rho_{l}$, the combination will transform according to their tensor product $\rho_{j}\otimes\rho_{l}$. If the convolution should map to an output field of type $\rho_{J}$, not any harmonic component $Y_{ji}^{m}$ is admissible, but only those for which $\rho_{J}$ appears as a subrepresentation in the tensor product $\rho_{j}\otimes\rho_{l}$. The Clebsch-Gordan coefficients encode whether $\rho_{j}\otimes\rho_{l}$ contains $\rho_{J}$, and, if it does, in which way and how often $\rho_{J}$ is embedded in the tensor product.

The tensor product $\rho_{j}\otimes\rho_{l}$ of two irreps is itself in general not irreducible anymore. However, as it is again a unitary representation, it splits by Proposition B.38 into a direct sum of irreducible unitary subrepresentations. Thus, there is an equivariant isomorphism

The integer $[J(jl)]$ is the multiplicity of $V_{J}$ in $V_{j}\otimes V_{l}$, which is zero for all but finitely many $J$.

For fixed $l$ and $J$, we will be able to find a basis kernel of type $j$ that transforms input features of type $l$ to output features of type $J$ if and only if $[J(jl)]>0$.

The matrix elements of $\operatorname{CG}_{jl}$ are denoted as Clebsch-Gordan coefficients:

Let $Y_{j}^{m}\otimes Y_{l}^{n}$ be the basis tensors in $V_{j}\otimes V_{l}$ and let the basis element $Y_{Js}^{M}$ be the copy of $Y_{J}^{M}$ with index $s$ in $\bigoplus\nolimits_{J\in\widehat{G}}\bigoplus\nolimits_{s=1}^{[J(jl)]}V_{J}$. Then the Clebsch-Gordan coefficients are the matrix elements of $\operatorname{CG}_{jl}$ relative to these bases,

i.e., the scalar product of \operatorname{CG}_{jl}\!\big{(}Y_{j}^{m}\otimes Y_{l}^{n}\big{)} and $Y_{Js}^{M}$.

For more details on the definitions in this section see Appendix D.1.

A Wigner-Eckart Theorem for G-steerable Kernels

Now that we have discussed all of the required ingredients, we are ready for stating our main theorem. Intuitively, our Wigner-Eckart theorem identifies exactly those combinations of harmonics, Clebsch-Gordan coefficients and endomorphisms that, when being assembled together, yield a $G$-steerable kernel $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$. The kernel will thereby comprise all those harmonics $Y_{ji}^{m}$ for which the tensor product $V_{j}\otimes V_{l}$ contains $V_{J}$ as a factor. The number of possible combinations depends therefore on the number of different isomorphism classes $j\in\widehat{G}$ for which $V_{J}$ appears as a factor in the tensor product, the multiplicity $[J(jl)]$ with which it occurs, and the multiplicities $m_{j}$ of harmonics $Y_{ji}^{m}$ in the Peter-Weyl decomposition that transform according to $\rho_{j}$. In addition, each individual combination can subsequently be composed with an endomorphism in $\operatorname{End}_{G,\mathds{K}}(V_{J})$, which increases the number of combinations by a factor of $E_{J}=\operatorname{dim}(\operatorname{End}_{G,\mathds{K}}(V_{J}))$ to a total of

This number is finite, as we explain in Remark D.18.

How are such assembled steerable kernels parameterized? The learnable parameters correspond to the degrees of freedom in the individual components from which the kernel is built. While the Clebsch-Gordan coefficients and harmonic basis functions are fixed, the endomorphisms are elements of the $E_{J}$-dimensional vector spaces $\operatorname{End}_{G,\mathds{K}}(V_{J})$. The degrees of freedom of a $G$-steerable kernel are therefore identified with the choice of endomorphisms. This statement is made precise by the isomorphism $\operatorname{GKer}$, defined in Eq. (7) in Theorem 4.1. This gives a total of $\Lambda_{Jl}$ parameters which take values in $\mathds{K}$. Note that the choice of endomorphisms corresponds directly to the choice of reduced matrix elements of spherical tensor operators.

For a kernel $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$, we write $\left\langle JM\middle|K(x)\middle|ln\right\rangle$ for the matrix elements of $K(x)\in\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$ with indices $n\leq d_{l}$ and $M\leq d_{J}$, see also Definition D.9. Similarly, endomorphisms $c\in\operatorname{End}_{G,\mathds{K}}(V_{J})$ have matrix elements $\left\langle JM|c|JM^{\prime}\right\rangle$ with $M,M^{\prime}\leq d_{J}$. We furthermore write $\left\langle i,jm\middle|x\right\rangle\coloneqq\overline{Y_{ji}^{m}(x)}$. Finally, recall that the space of $G$-steerable kernels is denoted by $\operatorname{Hom}_{G}(X,\,\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J}))$.

Our main result is the following Wigner-Eckart theorem for $G$-steerable kernels. Other versions at different levels of abstraction can be found in Theorems D.13 and D.16.

Let $X$ be a homogeneous space of the compact group $G$, $\rho_{l}:G\to\operatorname{U}(V_{l})$ and $\rho_{J}:G\to\operatorname{U}(V_{J})$ irreducible input- and output representations, $(\rho_{j})_{j\in\widehat{G}}$ an enumeration of all unitary irreps, $\{Y_{ji}^{m}\mid j\in\widehat{G},\ \ i=1,\dots,m_{j},\ \ m=1,\dots,d_{j}\}$ the harmonic basis functions in $L^{2}_{\mathds{K}}(X)$, and $\left\langle s,JM\middle|jm;ln\right\rangle$ the Clebsch-Gordan coefficients. There is an isomorphism of vector spaces

A general steerable kernel $K=\operatorname{GKer}((c_{jis})_{jis})$ with $c_{jis}\in\operatorname{End}_{G,\mathds{K}}(V_{J})$ has matrix elements

We shortly sketch a proof of this theorem. We use the notation $\operatorname{Hom}_{G,\mathds{K}}$ to denote linear equivariant maps. The space of steerable kernels can be progressively transformed as follows:

In step $(1)$, we linearize the kernels such that they become (continuous) representation operators, as detailed in Theorem C.7. Step $(2)$ applies the representation-theoretic version of the Peter-Weyl Theorem B.22 to decompose $L^{2}_{\mathds{K}}(X)$ in harmonic basis functions. In $(3)$, we remove the topological closure, denoted by $\widehat{(\cdot)}$, using Lemma D.20. Step $(4)$ makes use of the well-known fact that linear maps can be described on each direct summand individually. In step $(5)$, we use the hom-tensor adjunction Proposition D.23. In step $(6)$, we use the Clebsch-Gordan decomposition Eq. (5), which provides us with Clebsch-Gordan coefficients. In step $(7)$, we use that nontrivial linear equivariant maps from $V_{J^{\prime}}$ to $V_{J}$ exist by Schur’s Lemma B.29 only for $J^{\prime}=J$ and, once again, that we can describe linear maps on each direct summand individually. Finally, in step $(8)$, we note that $\operatorname{Hom}_{G,\mathds{K}}(V_{J},V_{J})=\operatorname{End}_{G,\mathds{K}}(V_{J})$ is the space of endomorphisms. Steps $(9)$ and $(10)$ are explained in Corollary 4.2. The formula of the matrix coefficients Eq. (8) is fully proven in Theorem D.13 by carefully tracing back all the isomorphisms above.

Technically, step (1) is the main gap that we had to bridge: it establishes that non-linear kernels on $X$ can be seen as linear representation operators on $L^{2}_{\mathds{K}}(X)$. Steps $(2)$ to $(7)$ orient at the proof of the Wigner-Eckart theorem for representation operators by Agrawala (1980). However, it differs non-trivially from the reference by a) allowing the operator to be non-injective, b) topological considerations, since $L^{2}_{\mathds{K}}(X)$ is not simply a direct sum of irreps but its topological closure, and c) the possibility to allow for real representations, which is why we end up with endomorphisms. ∎

A direct consequence of Theorem 4.1 is the following corollary, which clarifies how steerable kernels can be parameterized:

The space $\operatorname{Hom}_{G}(X,\ \operatorname{Hom}_{\mathds{K}}(V_{l},V_{J}))$ of steerable kernels is spanned by basis kernels $\{K_{jisr}:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})\ |\ j\in\widehat{G},\ \ i\leq m_{j},\ \ s\leq[J(jl)],\ \ r\leq E_{J}\}$ with matrix elements

where $c_{r}$ is one of the $E_{J}$ basis endomorphisms of $\operatorname{End}_{G,\mathds{K}}(V_{J})$. This means that a general steerable kernel $K:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$ is of the form

with a total of $\Lambda_{Jl}=E_{J}\cdot\sum_{j\in\widehat{G}}[J(jl)]\cdot m_{j}$ learnable parameters $\lambda_{jisr}\in\mathds{K}$. Overall, the kernel space can therefore be parameterized with an isomorphism

which expands a parameter array into steerable kernels (Weiler et al., 2018a; Weiler & Cesa, 2019). Thereby, $\overline{\operatorname{GKer}}=\operatorname{GKer}\circ\,\Omega$, where

is an isomorphism that chooses the same basis for each copy of $\operatorname{End}_{G,\mathds{K}}(V_{J})$.

We simply choose $K_{jisr}\coloneqq\operatorname{GKer}((c^{jisr}_{j^{\prime}i^{\prime}s^{\prime}})_{j^{\prime}i^{\prime}s^{\prime}})$ with $c^{jisr}_{j^{\prime}i^{\prime}s^{\prime}}=\delta_{j^{\prime}j}\cdot\delta_{i^{\prime}i}\cdot\delta_{s^{\prime}s}\cdot c_{r}$. Clearly, the $\bm{c}^{jisr}$ are a basis of $\bigoplus\nolimits_{j\in\widehat{G}}\bigoplus\nolimits_{i=1}^{m_{j}}\bigoplus\nolimits_{s=1}^{[J(jl)]}\operatorname{End}_{G,\mathds{K}}(V_{J})$, and since $\operatorname{GKer}$ is an isomorphism, the $K_{jisr}$ form a basis of steerable kernels. The isomorphism $\Omega$ corresponds to steps $(9)$ and $(10)$ in the proof of Theorem 4.1. ∎

A matrix-expression of the basis kernels from Eq. (9) is given in Eq. (24).

We make three remarks about this theorem:

The endomorphism matrix elements $\left\langle JM\middle|c_{jis}\middle|JM^{\prime}\right\rangle$ relate to the reduced matrix elements $\lambda=\left\langle J\middle|\bm{T}_{j}\middle|l\right\rangle\in\mathds{C}$ of spherical tensor operators as follows: in the case of spherical tensor operators one deals with complex irreps, whose endomorphism spaces are according to Schur’s Lemma D.8 $1$-dimensional, generated by the identity. Consequently, such endomorphisms $c$ have matrix-elements $\left\langle JM\middle|c\middle|JM^{\prime}\right\rangle=\lambda\,\delta_{MM^{\prime}}$ for some scaling factor $\lambda\in\mathds{C}$, which parameterizes the endomorphism space. While not actually being a specific matrix element, $\lambda$ determines all endomorphism matrix elements and is therefore denoted as reduced matrix element of the spherical tensor operator. The direct analog to $\lambda$ in our generalized Wigner-Eckart theorem are the learnable parameters $\lambda_{jisr}\in\mathds{K}$, which parameterize $G$-steerable kernels. Note that the sum over $M^{\prime}$ in Eq. (8) disappears in the original Wigner-Eckart Theorem 2.1 since the endomorphisms of complex irreps are scaled identity matrices.

In Eq. (8) we see terms $\left\langle i,jm\middle|x\right\rangle$ which are not present in the original Wigner-Eckart Theorem 2.1. They appear through a process in which the steerable kernel is linearized to make them more similar to spherical tensor operators, as we explain in Theorem C.7. In that process, the domain $X$ gets replaced by the space of square-integrable functions $L^{2}_{\mathds{K}}(X)$ with basis $\{Y_{ji}^{m}\}$, and $\left\langle i,jm\middle|x\right\rangle$ can be interpreted as a coupling coefficient between such a basis function and the original point $x\in X$.

Related Work

Harmonic convolution kernels have a long history in classical image processing, dating back at least to the early ’80s (Hsu & Arsenault, 1982; Rosen & Shamir, 1988). The term steerable filter was coined in Freeman & Adelson (1991). The authors found a basis of steerable filters by expanding the filters in terms of a Fourier basis, which can be seen as a special case of the Peter-Weyl theorem. Hel-Or & Teo (1998) generalized steerable filters to general Lie groups and proposed their explicit construction for Abelian Lie groups. Reisert & Burkhardt (2007) proposed matrix valued steerable kernels between representation spaces, which are very similar to our $G$-steerable kernels. The fact that harmonic functions and kernels appear frequently throughout physics, signal processing, and related fields, reflects their fundamental nature and great practical relevance.

Steerable CNNs formulate group equivariant CNNs in the language of representation theory and feature fields, which leads automatically to steerable kernels. This design was proposed by Cohen & Welling (2016b), who specifically considered finite groups, for which the kernel constraint can be solved numerically. Weiler et al. (2018a) introduced the $G$-steerability constraint in the form in Eq. (2) for $G=\operatorname{SO}(3)$. The authors choose a slightly different approach to solve the constraint in which they decompose the space $\operatorname{Hom}_{\mathds{R}}(V_{l},V_{J})\cong V_{l}^{*}\otimes V_{J}$ instead of $V_{j}\otimes V_{l}$ via Clebsch-Gordan coefficients. An essentially equivalent design was simultaneously proposed by Thomas et al. (2018), who decomposed $V_{j}\otimes V_{l}$ as in the present work, however, specifically for $G=\operatorname{SO}(3)$; see Appendix E.5. The case of complex valued irreps of $\operatorname{SO}(2)$ was investigated by Worrall et al. (2016) and Wiersma et al. (2020); see Appendix E.1. Weiler & Cesa (2019) solve the constraint for any, not necessarily irreducible, representation of the groups $\operatorname{O}(2)$, $\operatorname{SO}(2)$, $\operatorname{D}_{\!N}$ and $\operatorname{C}_{\!N}$. Their solution strategy is based on an expansion of the kernel in the Fourier basis of $L^{2}_{\mathds{R}}(S^{1})$ and solving for the Fourier coefficients satisfying the constraint. This is a special case of the strategy that we employ in the proof of our Wigner-Eckart theorem. de Haan et al. (2020) solve for $\operatorname{SO}(2)$-steerable kernels by viewing them as invariants of the tensor product representation $L^{2}_{\mathds{R}}(S^{1})^{*}\otimes V_{l}^{*}\otimes V_{J}$. As they use real valued irreps, they can use that the duals are isomorphic to their original counterparts. Note that the solution strategies in all of these papers apply only for specific choices of groups $G$ and representations. Our Wigner-Eckart theorem unifies all of them in one general framework.

To which use cases does the proposed kernel space solution apply? As argued by Cohen et al. (2019b), any $H$-equivariant convolutional network on a homogeneous space $H/G$ needs to satisfy a $G$-steerability constraint — if $H$ is locally compact and unimodular. Furthermore, gauge equivariant convolutions on Riemannian manifolds rely on $G$-steerable kernels, where $G$ is in this case the structure group of the feature vector bundle over $M$ (Cohen et al., 2019a). While these works proved the necessity of steerable kernels, they did not solve the constraint – a gap which is filled by our Wigner-Eckart theorem for compact groups $G$, see also Remark D.15. We want to emphasize that steerable convolutions include in particular the popular group convolutions on flat spaces (Cohen & Welling, 2016a) and homogeneous spaces of compact groups (Kondor & Trivedi, 2018) and Lie groups (Bekkers, 2020), including for instance the sphere (Cohen et al., 2018). Specifically, if $\rho_{\operatorname{in}}$ and $\rho_{\operatorname{out}}$ are chosen to be regular representations $L^{2}_{\mathds{K}}(G)$, steerable convolutions are equivalent to group convolutions (Weiler & Cesa, 2019). While being equivalent in theory, an implementation in terms of harmonic basis kernels is more appropriate as it naturally allows for bandlimiting, which reduces aliasing artifacts in a discretized implementation and improves the model performance (Weiler et al., 2018b; Graham et al., 2020).

A related line of work are Clebsch-Gordan Networks (Kondor et al., 2018; Kondor, 2018; Anderson et al., 2019; Bogatskiy et al., 2020). As in our work, these models consider features that transform under irreducible representations. However, while our features live at specific points of the base space, their features are global features, e.g., Fourier coefficients of functions on a homogeneous base space like the sphere. This results in a network design that implements convolution implicitly using a fully-connected linear layer. They apply bilinear equivariant nonlinearities which compute the tensor products of irrep features. A subsequent Clebsch-Gordan decomposition disentangles the resulting product features back into irrep features. Note that in this network design, the Clebsch-Gordan coefficients are used in the nonlinear part of the network, which differs from our use of these coefficients in the parameterization of steerable basis kernels, i.e. in the linear part of the network.

Example Applications

Cohen et al. (2019b) showed in a fairly general setting that every GCNN is based on $G$-steerable kernels. In practice, a basis for the space of $G$-steerable kernels needs to be determined for parameterizing GCNNs. This work explains the general structure of these basis kernels for compact (point-)symmetry groups $G$ and their homogeneous spaces $X$ in terms of several ingredients. Corollary 4.2 explains that one needs to determine

the irreps $\rho_{l}$ of $G$, where $l\in\widehat{G}$,

harmonic basis functions $Y_{ji}^{m}$ in $L^{2}_{\mathds{K}}(X)$ according to the Peter-Weyl Theorem 3.4,

the Clebsch-Gordan decomposition of $V_{j}\otimes V_{l}$ for any $j,l\in\widehat{G}$, given by the Clebsch-Gordan coefficients $\left\langle s,JM|jm;ln\right\rangle$, and

an $E_{J}$-dimensional basis of endomorphisms $c_{r}\in\operatorname{End}_{G,\mathds{K}}(V_{J})$ for any $J\in\widehat{G}$.

Given these ingredients, they can in a fifth step be put together according to Eq. (9) to obtain a complete, $\Lambda_{Jl}$-dimensional basis of $G$-steerable kernels $K_{jisr}:X\to\operatorname{Hom}_{\mathds{K}}(V_{l},V_{J})$.

Appendix E demonstrates this procedure for the examples of $G$ being $\operatorname{SO}(2)$ with both real and complex irreps, $\operatorname{SO}(3)$ with both real and complex irreps, $\operatorname{O}(3)$ with both real and complex irreps, and the real irreps of the reflection group $\mathds{Z}/2$. In any of these cases, we derive the kernel bases following exactly the five steps outlined above. This procedure can easily be applied to further compact groups, for instance $\operatorname{SU}(2)$ or $\operatorname{SU}(3)$, which play an important role in physics applications of deep learning.

Conclusions and Future Work

Prior work revealed that group equivariant convolutions generally rely on $G$-steerable kernels. No general solution of the linear constraint defining such kernels was known so far. Our Wigner-Eckart theorem for $G$-steerable kernels characterizes the solution space for the practically relevant case of $G$ being any compact group. It gives a complete basis of steerable kernels in terms of harmonic basis functions, Clebsch-Gordan coefficients, and endomorphisms – ingredients which we determine for several exemplary symmetry groups. The degrees of freedom – or learnable parameters – correspond thereby precisely to the choice of endomorphisms. This mirrors the situation in quantum mechanics, where the degrees of freedom of spherical tensor operators, or more generally representation operators, are given by reduced matrix elements.

It would be desirable to extend this result to non-compact groups, where the Peter-Weyl Theorem does not hold anymore. One alternative might be Pontryagin duality (Reiter, 1968), which describes the Fourier transform on locally compact abelian groups. This might lead to a better theoretical understanding and generalizations of kernels that are, for example, scale-equivariant (Worrall & Welling, 2019; Bekkers, 2020; Sosnovik et al., 2020). Obviously, being abelian is a restriction, and as work on Lorentz group equivariant networks shows (Shutty & Wierzynski, 2020), an extension of our results to non-compact, non-abelian groups should in principle be possible. This is further motivated by the existence of a Wigner-Eckart Theorem for non-compact Lie groups (Sellaroli, 2015). One angle might be to consider groups $G$ that are of so-called type I, second-countable, and locally compact. In that case, the space of square-integrable functions on $G$ has a direct integral decomposition. This is a generalization of the Peter-Weyl theorem that can be found in Segal (1950) and Mautner (1955).

Finally, we hope that the analogies between steerable kernels and representation operators appearing in physics inspire further research in this fascinating crossdisciplinary domain. This could lead to applications of GCNNs for learning tasks with physical symmetries.

We thank Lucas Lang for discussions on the Wigner-Eckart Theorem and observables in physics and Patrick Forré for discussions on the link between steerable kernels and representation operators. Additionally, we are greatful for discussions with Gabriele Cesa on the connection between real and complex representations of compact groups. Furthermore, we thank Stefan Dawydiak and Terrence Tao for online discussions on aspects surrounding a real version of the Peter-Weyl theorem. Finally, we thank Roberto Bondesan, Miranda Cheng, Tom Lieberum, and Rupert McCallum for feedback on different aspects of our work.

References

List of Symbols

Numbers and Collections of Numbers

Groups

Basic Representation Theory

Vector Spaces and Hilbert Spaces

(Hilbert) Space Constructions from Other Spaces

Topological Spaces, Metric Spaces, Normed Spaces

Homogeneous Spaces and the Peter-Weyl Theorem

Kernels and Representation Operators

The Wigner-Eckart Theorem

Appendix A Building Blocks of SO(2)-Steerable Kernels – Running Example for Section 3

In this short chapter, we briefly explain the components of steerable kernels at the specific example of real valued irreps of the circle group $\operatorname{SO}(2)$. While this example is quite simple, it shows some non-trivial properties like $2$-dimensional endomorphism spaces $\operatorname{End}_{\operatorname{SO}(2),\mathds{R}}(V_{J})$ for $J>0$ and a Clebsch-Gordan decomposition in which the multiplicity $[J(jl)]$ can differ from or $1$. To give a quick overview: Example A.1 considers the circle as an orbit and homogeneous space of $\operatorname{SO}(2)$ while Example A.2 introduces the real valued irreps. Their endomorphisms are stated in Example A.3. As discussed in Example A.4, the Peter-Weyl theorem corresponds here to the usual Fourier series on $S^{1}$. The Clebsch-Gordan decomposition of tensor products of the irreps are discussed in Example A.5. With these ingredients, we are ready to instantiate the kernel spaces as described by our Wigner-Eckart Theorem 4.1 for steerable kernels, for which we refer, including proofs, to Section E.2.

$\operatorname{SO}(2)$-steerable kernels $K:\mathds{R}^{2}\to\operatorname{Hom}_{\mathds{R}}(V_{\operatorname{in}},V_{\operatorname{out}})\cong\mathds{R}^{c_{\operatorname{out}}\times c_{\operatorname{in}}}$ allow for rotation equivariant convolutions. For instance, a convolution with an $\operatorname{SO}(2)$-steerable kernel on $\mathds{R}^{2}$ is guaranteed to be $\operatorname{SE}(2)=(\mathds{R}^{2},+)\rtimes\operatorname{SO}(2)$ equivariant while a convolution with an $\operatorname{SO}(2)$-steerable kernel on $S^{2}$ will be $\operatorname{SO}(3)$-equivariant.

$\operatorname{SO}(2)$ acts on the kernel’s domain $\mathds{R}^{2}$ by rotating it. The orbits of the action are therefore given by 1) the origin $\{0\}$ and 2) circles of arbitrary radius. We know that the kernel constraint can be solved on each orbit individually, and so we can restrict to looking at those. Since $\{0\}$ is rather trivial, we specifically consider the circle $S^{1}$ as a more interesting homogeneous space.

Consider the circle $S^{1}$ and the rotation group $\operatorname{SO}(2)$. For convenience, we reparameterize both: we view $\operatorname{SO}(2)$ as the group of angles $\phi\in\mathds{R}/2\pi\mathds{Z}\cong[0,2\pi]/_{0\sim 2\pi}$ and $S^{1}$ as the space $\mathds{R}/2\pi\mathds{Z}$ as well. Then the action of $\operatorname{SO}(2)$ on $S^{1}$ is given by $\phi\cdot x\coloneqq(\phi+x)\!\mod 2\pi$. It is easy to see that this action is transitive, which makes the circle a homogeneous space of $\operatorname{SO}(2)$.

As it is sufficient to solve the kernel constraint for irreducible orthogonal input- and output representations, we now state a classification of those up to isomorphism.

The irreducible orthogonal representations ${\rho_{l}:\operatorname{SO}(2)\to\operatorname{O}(V_{l})}$ of $\operatorname{SO}(2)$ are labeled by indices (“quantum numbers”) $l\in\mathds{N}_{\geq 0}$. For $l=0$, one has the trivial representation with $V_{0}=\mathds{R}$ and $\rho_{0}(\phi)=\operatorname{id}_{\mathds{R}}$. For $l\geq 1$, one has $V_{l}=\mathds{R}^{2}$ and

i.e., rotation matrices of “frequency $l\,$”. The isomorphism classes of irreducible orthogonal representations are then given by $\widehat{\operatorname{SO}(2)}\cong\mathds{N}_{\geq 0}$.

We are thus in the following considering $\operatorname{SO}(2)$-steerable kernels of the form

Remember that if $c:V_{J}\to V_{J}$ is an endomorphism, i.e., commutes with $\rho_{J}$, that $c\circ K$ is then steerable as well. Thus, we now look at a classification of the endomorphisms of the irreducible orthogonal representations:

Let $G=\operatorname{SO}(2)$ with the irreducible representations $\rho_{J}$ as in Example A.2. Clearly, the endomorphism space $\operatorname{End}_{\operatorname{SO}(2),\mathds{R}}(V_{0})$ is $1$-dimensional, i.e., $E_{0}=1$. For all $J\geq 1$, the endomorphism space is two-dimensional ($E_{J}=2$) and given by combinations of scalings and rotations Another way to imagine this is to identify $V_{J}$ with the complex plane $\mathds{C}$. Then an endomorphism is given by multiplication with an arbitrary complex number. on $V_{J}=\mathds{R}^{2}$. A basis of this space is given by the following two matrices:

That $c_{1}$ is an endomorphism of $\rho_{J}$ for $J\geq 1$ is immediately clear. That the same holds for $c_{2}$ is checked by the following simple calculation:

The proof that there are no other endomorphisms is sketched in Proposition E.5.

Another ingredient that we need to construct $\operatorname{SO}(2)$-steerable kernels on $S^{1}$ is the decomposition of $L^{2}_{\mathds{R}}(S^{1})$ into its irreducible subrepresentations $V_{ji}$, which the Peter-Weyl theorems guarantees to exist. Less abstractly, we are interested in an orthonormal set of harmonic (steerable) basis functions on $S^{1}$ that span $L^{2}_{\mathds{R}}(S^{1})$ – which corresponds to the usual Fourier series on $S^{1}$.

As in Example A.1, we assume $G=\operatorname{SO}(2)$ and $X=S^{1}$. A standard result in harmonic analysis says that square-integrable functions $f:S^{1}\to\mathds{R}$, i.e., $f\in L^{2}_{\mathds{R}}(S^{1})$, can be uniquely written as an infinite sum of sine and cosine terms,

where $a_{0}$ and $a_{j},b_{j},\ j\geq 1$ are real-valued expansion coefficients.

How does this result relate to the harmonic basis functions in the Peter-Weyl theorem 3.4? As stated above, we have isomorphism classes $\widehat{G}=\widehat{\operatorname{SO}(2)}\cong\mathds{N}_{\geq 0}$ of irreps with representatives $\rho_{j}$. A comparison of the Fourier series in Eq. (10) with property 2 in the Peter-Weyl theorem 3.4 suggests the following identification of harmonic basis functions $Y_{j}^{m}$ and coefficients $\lambda_{jm}$,

where we introduced the shorthand notations $\cos_{j}(x):=\cos(jx)$ and $\sin_{j}(x):=\sin(jx)$. Note that we dropped the index $i=1,\dots,m_{j}$ since $m_{j}=1$ for any $j\in\widehat{\operatorname{SO}(2)}$. As expected, we have indices $m=1$ for $j=0$ with $d_{j}=\operatorname{dim}(V_{0})=1$ and indices $m=1,2$ for $j\geq 1$ with $d_{j}=\operatorname{dim}(V_{j})=2$. The orthogonality relations in property 3 of the Peter-Weyl theorem hold up to a simple normalization of these basis functions and are easily checked by explicitly computing the scalar products. Property 1, i.e., the $\operatorname{SO}(2)$-steerability of the harmonic bases, is trivial for $j=0$. For $j\geq 1$, the standard angle summation formulas for cosines and sines lead to the following expressions for harmonics that are translated by $\phi\in\operatorname{SO}(2)$:

which is just property 1 in the Peter-Weyl theorem. This is concisely summarized by

which shows that the basis functions $Y_{j}^{1}=\cos_{j}$ and $Y_{j}^{2}=\sin_{j}$ span an invariant subspace $V_{j}$ of $L^{2}_{\mathds{R}}(S^{1})$ under rotations. From a more abstract viewpoint, the Peter-Weyl theorem just states that $L^{2}_{\mathds{R}}(S^{1})$ splits into the orthogonal direct sum $\bigoplus_{j\in\widehat{\operatorname{SO}(2)}}V_{j}$.

Finally, we need to investigate the tensor products of irreducible representations and their decomposition via Clebsch-Gordan coefficients. They will be used to correctly assemble harmonic basis functions to steerable kernels.

Remember the irreducible representations $\rho_{l}:\operatorname{SO}(2)\to\operatorname{O}(V_{l})$ given in Example A.2. As we prove in Proposition E.4, including a description of the Clebsch-Gordan coefficients, the tensor products decompose as follows:

where the last isomorphism only holds if $j,l\geq 1$ and $j\neq l$. If $j,l\geq 1$ and $j=l$, then we obtain

i.e., $V_{0}$ here appears twice in the decomposition of a tensor product of irreducible representations. We therefore have multiplicities $[J(jl)]$ which are $1$ for $[0(00)]$, $[j(j0)]$, $[l(0l)]$, $[|j-l|(jl)]$, $[j+l(jl)]$ and $[2l(ll)]$ while $[0(ll)]=2$. Any other multiplicity is zero.

With these ingredients one can then determine all $\operatorname{SO}(2)$-steerable kernels. This is explained in Proposition E.6.

Appendix B Representation Theory of Compact Groups

In this chapter, we outline the main ingredients of the representation theory of compact groups that we need for our applications to steerable CNNs. Usually, this theory is only developed for representations over the complex numbers. However, since we want to apply it also to steerable CNNs using real representations, we need to be a bit more careful. In particular, we need to make sure that the Peter-Weyl theorem is correctly stated and proven.

The outline is as follows: In Section B.1, we start by stating all the important definitions and concepts from group theory and representation theory of (unitary) representations that are needed for formulating the Peter-Weyl theorem. After defining Haar measures both for compact groups and their homogeneous spaces and shortly discussing their square-integrable functions, we formulate the Peter-Weyl Theorem B.22. In Section B.2, then, we give a proof of this version of the Peter-Weyl theorem, carefully making sure to not use properties that are only true over $\mathds{C}$. In some essential steps, mainly the density of the matrix coefficients in the regular representation, we refer to the literature, since the proof clearly does not make use of $\mathds{C}$ per se. While we initially only give the proof for the regular representation, i.e., the space of square-integrable functions on the group itself, we end this section with a discussion of general unitary representations and, in particular, the space of square-integrable functions for an arbitrary homogeneous space.

In the whole chapter, let $\mathds{K}$ be the field of real or complex numbers.

In this section, we define preliminary concepts from topological groups and their actions. This material can, for example, be found in detail in Arkhangel’skii & Tkachenko (2008). For the topological concepts that we use, we refer to Chapter F.1.

A group $G=(G,\cdot,(\cdot)^{-1},e)$, most often simply written $G$, consists of the following data:

A multiplication $\cdot:G\times G\to G$, $(g,h)\mapsto g\cdot h$.

An inversion $(\cdot)^{-1}:G\to G$, $g\mapsto g^{-1}$.

A distinguished unit element $e\in G$. It is also called neutral element.

They are assumed to have the following properties for all $g,h,k\in G$:

The multiplication is associative: $g\cdot(h\cdot k)=(g\cdot h)\cdot k$.

The unit element is neutral with respect to multiplication: $e\cdot g=g=g\cdot e$.

The inversion of an element multiplied with itself is the neutral element: $g\cdot g^{-1}=g^{-1}\cdot g=e$.

A group is called abelian if, additionally, the multiplication is commutative: $g\cdot h=h\cdot g$ for all $g,h\in G$. If this is the case, a group is often written as $G=(G,+,-(\cdot),0)$.

If we consider several groups at once, say $G$ and $H$, then we often do not distinguish their multiplication, inversion, and neutral elements in notation. It will be clear from the context which group the operation belongs to.

Let $G$ be a group and $H\subseteq G$ a subset. $H$ is called a subgroup if: