Deep Scale-spaces: Equivariance Over Scale

Introduction

Scale is inherent in the structure of the physical world around us and the measurements we make of it. Ideally, the machine learning models we run on this perceptual data should have a notion of scale, which is either learnt or built directly into them. However, the state-of-the-art models of our time, convolutional neural networks (CNNs) (Lecun et al., 1998), are predominantly local in nature due to small filter sizes. It is not thoroughly understood how they account for and reason about multiscale interactions in their deeper layers, and empirical evidence (Chen et al., 2018; Yu and Koltun, 2015; Yu et al., 2017) using dilated convolutions suggests that there is still work to be done in this arena.

In computer vision, typical methods to circumvent scale are: scale averaging, where multiple scaled versions of an image are fed through a network and then averaged (Kokkinos, 2015); scale selection, where an object’s scale is found and then the region is resized to a canonical size before being passed to a network (Girshick et al., 2014); and scale augmentation, where multiple scaled versions of an image are added to the training set (Barnard and Casasent, 1991). While these methods help, they lack explicit mechanisms to fuse information from different scales into the same representation. In this work, we construct a generalized convolution taking, as input, information from different scales.

The utility of convolutions arises in scenarios where there is a translational symmetry (translation invariance) inherent in the task of interest (Cohen and Welling, 2016a). Examples of such tasks are object classification (Krizhevsky et al., 2012), object detection (Girshick et al., 2014), or dense image labelling (Long et al., 2015). By using translational weight-sharing (Lecun et al., 1998) for these tasks, we reduce the parameter count while preserving symmetry in the deeper layers. The overall effect is to improve sample complexity and thus reduce generalization error (Sokolic et al., 2017). Furthermore, it has been shown that convolutions (and various reparameterizations of them) are the only linear operators that preserve symmetry (Kondor and Trivedi, 2018). Attempts have been made to extend convolutions to scale, but they either suffer from breaking translation symmetry (Henriques and Vedaldi, 2017; Esteves et al., 2017), making the assumption that scalings can be modelled in the same way as rotations (Marcos et al., 2018), or ignoring symmetry constraints (Hilbert et al., 2018). The problem with the aforementioned approaches is that they fail to account for the unidirectional nature of scalings. In data there exist many one-way transformations, which cannot be inverted. Examples are occlusions, causal translations, downscalings of discretized images, and pixel lighting normalization. In each example the transformation deletes information from the original signal, which cannot be regained, and thus it is non-invertible. We extend convolutions to these classes of symmetry under noninvertible transformations via the theory of semigroups. Our contributions are the introduction of a semigroup equivariant correlation and a scale-equivariant CNN.

Background

This section introduces some key concepts such as groups, semigroups, actions, equivariance, group convolution, and scale-spaces. These concepts are presented for the benefit of the reader, who is not expected to have a deep knowledge of any of these topics a priori.

for $s,t>0$. It says that we can generate a scale-space from other levels of the scale-space, not just from $f_{0}$. Furthermore, since $s,t>0$ it also says that we cannot generate sharper images from blurry ones, using just a Gaussian convolution. Thus moving to blurrier levels encodes a degree of information loss. This property emanates from the closure of Gaussians under convolution, namely for multidimensional Gaussians with covariance matrices $\Sigma$ and $T$

We assume the initial image $f_{0}$ has a maximum spatial frequency content—dictated by pixel pitch in discretized images—which we model by assuming the image has already been convolved with a width $s_{0}$ Gaussian, which we call the zero-scale. Thus an image of bandlimit $s$ in the scale-space is found at GSS slice $f(s-s_{0},\cdot)$, which we see from Equation 3. There are many varieties of scale-spaces: the $\alpha$-scale-spaces (Pauwels et al., 1995), the discrete Gaussian scale-spaces (Lindeberg, 1990), the binomial scale-spaces (Burt, 1981), etc. These all have specific kernels, analogous to the Gaussian, which are closed under convolution (details in supplement).

Slices at level $t$ in the GSS correspond to images downsized by a dilation factor $0\leq a\leq 1$ ($1=\text{no downsizing}$, $0=\text{shrinkage to a point}$). An $a$-dilated and appropriately bandlimited image $p(a,x)$ is found as (details in supplement)

For clarity, we refer to decreases in the spatial dimensions of an image as dilation and increases in the blurriness of an image as scaling. For a generalization to anisotropic scaling we replace scalar scale $t$ with matrix $T$, zero-scale $s_{0}$ with covariance matrix $\Sigma_{0}$, and dilation parameter $a$ with matrix $A$, so

Semigroups The semigroup property of Equation 3 is the gateway between classical scale-spaces and the group convolution (Cohen and Welling, 2016a) (see end of section). Semigroups $(S,\circ)$ consist of a non-empty set $S$ and a (binary) composition operator $\circ:S\times S\to S$. Typically, the composition $s\circ t$ of elements $s,t\in S$ is abbreviated $st$. For our purposes, these individual elements will represent dilation parameters. For $S$ to be a semigroup, it must satisfy the following two properties

Associativity: $(st)r=s(tr)=str$ for all $s,t,r\in S$

Note that commutativity is not a given, so $st\neq ts$. The family of Gaussian densities under spatial convolution is a semigroupFor the Gaussians we find the identity element as the limit $\lim_{t\downarrow 0}G(\cdot,t)$, which is a Dirac delta. Note that this element is not strictly in the set of Gaussians, thus the Gaussian family has no identity element.. Semigroups are a generalization of groups, which are used in Cohen and Welling (2016a) to model invertible transformations. For a semigroup to be a group, it must also satisfy the following conditions

Identity element: there exists an $e\in S$ such that $es=se=s$ for all $s\in S$

Inverses: for each $s\in S$ there exists a $s^{-1}$ such that $s^{-1}s=ss^{-1}=e$.

Actions Semigroups are useful because we can use them to model transformations, also known as (semigroup) actions. Given a semigroup $S$, with elements $s\in S$ and a domain $X$, the action $L_{s}:X\to X$ is a map, also written $x\mapsto L_{s}[x]$ for $x\in X$. The defining property of the semigroup action (Equation 7) is that it is associative and closed under composition, essentially inheriting its compositional structure from the semigroup $S$. This action is in fact called a left actionA right action is $R_{st}[x]=R_{t}[R_{s}[x]]$. Notice $R_{st}$ applies $R_{s}$ then $R_{t}$, opposite in order to the left action.. Notice that for a composed action $L_{st}$, we first apply $L_{t}$ then $L_{s}$.

Actions can also be applied to functions $f:X\to Y$ by viewing $f$ as a point in a function space. The result of the action is a new function, denoted $L_{s}[f]$. Since the domain of $f$ is $X$, we commonly write $L_{s}[f](x)$. Say the domain is a semigroup $S$ then an example left actionThis is indeed a left action because $L_{s}[L_{t}[f]](x)=L_{t}[f](xs)=f(xst)=L_{st}[f](x)$. is $L_{s}[f](x)=f(xs)$. Another example, which we shall use later, is the action $S_{A,z}^{\Sigma_{0}}$ used to form scale-spaces, namely

Lifting We can also view actions from functions on $X$ to functions $X$ as maps from functions on $X$ to functions on the semigroup $S$. We call this a lift (Kondor and Trivedi, 2018), denoting lifted functions as $f^{\uparrow}$. One example is the scale-space action (Equation 8). If we set $x=0$, then

which is the expression for an anisotropic scale-space (parameterized by the dilation $A$ rather than the scale $T(A,\Sigma_{0})=A^{-1}\Sigma_{0}A^{-T}-\Sigma_{0}$). To lift a function on to a semigroup, we do not necessarily have to set $x$ equal to a constant, but we could also integrate it out. An important property of lifted functions is that actions become simpler. For instance, if we define $f^{\uparrow}(s)=L_{s}[f](0)$ then

The action on $f$ could be complicated, like a Gaussian blur, but the action on $f^{\uparrow}$ is simply a ‘shift’ on $f^{\uparrow}(s)\mapsto f^{\uparrow}(st)$. We can then define the action $L_{t}$ on lifted functions as $L_{t}[f^{\uparrow}](s)=f^{\uparrow}(st)$.

Given a signal $f$ and filter $\psi$, we interpret the correlation as the collection of inner products of $f$ with all $s$-translated versions of $\psi$. This basic correlation has been extended to transformations other than translation via the group correlation, which as presented in Cohen and Welling (2016a) are

where $H$ is the relevant group and $L_{s}$ is a group action e.g. for rotation $L_{s}[x]=R_{s}x$, where $R_{s}$ is a rotation matrix. Note how the summation domain is $H$, as are the domains of the signal and filter. Most importantly, the domain of the output is $H$. It truly highlights how this is an inner product of $f$ and $\psi$ under all $s$-transformations of the filter. The correlation exhibits a very special property. It is equivariant under actions of the group $H$. This means

That is $L_{s}$ and $\star_{H}$ commute—group correlation followed by the action is equivalent to the action followed by the group correlation. Note, the group action may ‘look’ different depending on whether it was applied before or after the group correlation, but it represents the exact same transformation.

Method

We aim to construct a scale-equivariant convolution. We shall achieve this here by introducing an extension of the correlation to semigroups, which we then tailor to scalings.

Semigroup Correlation There are multiple candidates for a semigroup correlation $\star_{S}$. The basic ingredients of such a correlation will be the inner product, the semigroup action $L_{s}$, and the functions $\psi$ and $f$. Furthermore, it must be equivariant to (left) actions on $f$. For a semigroup $S$, domain $X$, and action $L_{s}:X\to X$, we define:

It is the set of responses formed from taking the inner product between a filter $\psi$ and a signal $f$ under all transformations of the signal. Notice that we transform the signal and not the filter and that we write $\psi\star_{S}f$, not $f\star_{S}\psi$—it turns out that a similar expression where we apply the action to the filter is not equivariant to actions on the signal. Furthermore this expression lifts a function from $X$ to $S$, so we expect actions on $f$ to look like a ‘shift’ on the semigroup. A proof of equivariance to (left) actions on $f$ is as follows

where $x^{\prime}=x+s$. We can also recover the group correlation by setting $S=H$, $X=H$, where $H$ is a discrete group, and $L_{s}[f](x)=f(L^{\prime}_{s}x)$, where $L^{\prime}_{s}$ is the form of $L_{s}$ acting on the domain $X=H$:

where $x^{\prime}=L^{\prime}_{s}[x]$ and since $L^{\prime}$ is a group action inverses exist, so $x=L^{\prime}_{s^{-1}}[x^{\prime}]$. The semigroup correlation has a two notable differences from the group correlation: i) In the semigroup correlation we transform the signal and not the filter. When we restrict to the group and standard convolution, transforming the signal or the filter are equivalent operations since we can apply a change of variables. This is not possible in the semigroup case, since this change of variables requires an inverse, which we do not necessarily have. ii) In the semigroup correlation, we apply an action to the whole signal as $L_{s}[f]$, as opposed to just the domain ($f(L_{s}[x])$). This allows for more general transformations than allowed by the group correlation of Cohen and Welling (2016a), since transformations of the form $f(L_{s}[x])$ can only move pixel locations, but transformations of the form $L_{s}[f]$ can alter the values of the pixels as well, and can incorporate neighbourhood information into the transformation.

Equivariant Nonlinearities Not all nonlinearities commute with semigroup actions, but it turns out that pointwise nonlinearities $\nu$ commute with a special subset of actions of the form,

where $L_{s}^{\prime}$ is a representation of $L_{s}$ acting on the domain of $f$. For these sorts of actions, we cannot alter the values of the $f$, just the locations of the values. If we write function composition as $[\nu\bullet f](x)=\nu(f(x))$, then a proof of equivariance is as follows:

Equation 21 may at first glance seem overly restrictive, but it turns out that this is not the case. Recall that for functions lifted on to the semigroup, the action is $L_{s}[f](x)=f(xs)$. This satisfies Equation 21, and so we are free to use pointwise nonlinearities.

Batch normalization For batch normalization, we compute batch statistics over all dimensions of an activation tensor except its channels, as in Cohen and Welling (2016a).

Initialization Since our correlations are based on dilated convolutions, we use the initialization scheme presented in (Yu and Koltun, 2015). For pairs of input and output channels the center pixel of each filter is set to one and the rest are filled with random noise of standard deviation $10^{-2}$.

Boundary conditions In our semigroup correlation, the scale dimension is infinite—this is a problem for practical implementations. Our solution is to truncate the scale-space to finite scale. This breaks global equivariance to actions in $S$, but is locally correct. Boundary effects occur at activations with receptive fields covering the truncation boundary. To mitigate these effects we: i) use filters with a scale-dimension no larger than two, ii) interleave filters with scale-dimension 2 with filters of scale-dimension 1. The scale-dimension 2 filters enable multiscale interactions but they propagate boundary effects; whereas, the scale-dimension 1 kernels have no boundary overlap, but also no multiscale behavior. Interleaving trades off network expressiveness against boundary effects.

Scale-space implementation We use a 4 layer scale-space and zero-scale $1/4$ with dilations at integer powers of 2, the maximum dilation is 8 and kernel width 33 (4 std. of a discrete Gaussian). We use the discrete Gaussian of Lindeberg (1990). In 1D for scale parameter $t$, this is

where $I_{x}(t)$ is the modified Bessel function of integer order. For speed, we make use of the separability of isotropic kernels. For instance, convolution with a 2D Gaussian can we written as a convolution with 2 identical 1D Gaussians sequentially along the $x$ and then the $y$-axis. For an $N\times N$ input image and $M\times M$ blurring kernel, this reduces computational complexity of the convolution as $O(M^{2}N^{2})\to O(2MN^{2})$. With GPU parallelization, this saving is $O(M^{2})\to O(M)$, which is especially significant for us since the largest blur kernel we use has $M=33$.

Multi-channel features Typically CNNs use multiple channels per activation tensor, which we have not included in our above treatment. In our experiment we include input channels $i$ and output channels $o$, so a correlation layer is

Experiments and Results

Here we present our results on the Patch Camelyon (Veeling et al., 2018) and Cityscapes (Cordts et al., 2016) datasets. We also visualize the quality of scale-equivariance achieved.

Patch Camelyon The Patch Camelyon or PCam dataset (Veeling et al., 2018) contains 327 680 tiles from two classes, metastatic (tumorous) and non-metastatic tissue. Each tile is a $96\times 96$ px RGB-crop labelled as metastatic if there is at least one pixel of metastatic tissue in the central $32\times 32$ px region of the tile. We test a 4-scale DenseNet model (Huang et al., 2017), ‘S-DenseNet’, on this task (architecture in supplement). We also train a scale non-equivariant DenseNet baseline and the rotation equivariant model of (Veeling et al., 2018). Our training procedure is: 100 epochs SGD, learning rate 0.1 divided by 10 every 40 epochs, momentum 0.9, batch size of 512, split over 4 GPUs. For data augmentation, we follow the procedure of (Veeling et al., 2018; Liu et al., 2017), using random flips, rotation, and 8 px jitter. For color perturbations we use: brightness delta 64/255, saturation delta 0.25, hue delta 0.04, constrast delta 0.75. The evaluation metric we test on is accuracy. The results in Table 1 show that both the scale and rotation equivariant models outperform the baseline.

Cityscapes The Cityscapes dataset (Cordts et al., 2016) contains 2975 training images, 500 validation images, and 1525 test images of resolution $2048\times 1024$ px. The task is semantic segmentation into 19 classes. We train a 4-scale ResNet He et al. (2016), ‘S-ResNet’, and baseline. We train an equivariant network with and without multiscale interaction layers. We also train two scale non-equivariant models, one with the same number of channels, one with the same number of parameters. Our training procedure is: 100 epochs Adam, learning rate $10^{-3}$ divided by 10 every 40 epochs, batch size 8, split over 4 GPUs. The results are in Table 1. The evaluation metric is mean average precision. We see that our scale-equivariant model outperforms the baselines. We must caution however, that better competing results can be found in the literature. The reason our baseline underperforms compared to the literature is because of the parameter/channel-matching, which have shrunk its size somewhat due to our own resource constraints. On a like-for-like comparison scale-equivariance appears to help.

We validate the quality of equivariance empirically by comparing activations of a dilated image against the theoretical action on the activations. Using $\Phi$ to denote the deep scale-space (DSS) mapping, we compute the normalized $L2$-distance at each level $k$ of a DSS. Mathematically this is

Related Work

In recent years, there have been a number of works on group convolutions, namely continuous roto-translation in 2D (Worrall et al., 2017) and 3D (Weiler et al., 2018a; Kondor et al., 2018; Thomas et al., 2018) and discrete roto-translations in 2D (Cohen and Welling, 2016a; Weiler et al., 2018b; Bekkers et al., 2018; Hoogeboom et al., 2018) and 3D (Worrall et al., 2017), continuous rotations on the sphere (Esteves et al., 2018; Cohen et al., 2018b), in-plane reflections (Cohen and Welling, 2016a), and even reverse-conjugate symmetry in DNA sequencing (Lunter and Brown, 2018). Theory for convolutions on compact groups—used to model invertible transformations—also exists (Cohen and Welling, 2016a, b; Kondor and Trivedi, 2018; Cohen et al., 2018a), but to date the vast majority of work has focused on rotations.

For scale, there far fewer works. (Henriques and Vedaldi, 2017; Esteves et al., 2017) both perform a log-polar transform of the signal before passing to a standard CNN. Log-polar transforms reparameterize the input plane into angle and log-distance from a predefined origin. The transform is sensitive to origin positioning, which if done poorly breaks translational equivariance. (Marcos et al., 2018) use a group CNN architecture designed for roto-translation, but instead of rotating filters in the group correlation, they scale them. This seems to work on small tasks, but ignores large scale variations. (Hilbert et al., 2018) instead use filters of different sizes, but without any attention to equivariance.

Discussion, Limitations, and Future Works

We found the best performing architectures were composed mainly of correlations where the filters’ scale dimension is one, interleaved with correlations where the scale dimension is higher. This is similar to a network-in-network (Lin et al., 2013) architecture, where $3\times 3$ convolutional layers are interleaved with $1\times 1$ convolutions. We posit this is the case because of boundary effects, as were observed in Figure 4. We suspect working with dilations with smaller jumps in scale between levels of the scale-space would help, so $A_{k}=c^{-k}I$ for $1<c<2$. This is reminiscent of the tendency in the image processing community to use non-integer scales. This would, however, involve the development of non-integer dilations and hence interpolation. We see working on mitigating boundary effects as an important future work, not just for scale-equivariance, but CNNs as a whole.

Another limitation of the current model is the increase in computational overhead, since we have added an extra dimension to the activations. This may not be a problem long term, as GPUs grow in speed and memory, but the computational complexity of a correlation grows exponentially in the number of symmetries of the model, and so we need more efficient methods to perform correlation, either exactly or approximately.

We see semigroup correlations as an exciting new family of operators to use in deep learning. We have demonstrated a proof-of-concept on scale, but there are many semigroup-structured transformations left to be explored, such as causal shifts, occlusions, and affine transformations. Concerning scale, we are also keen to explore how multi-scale interactions can be applied on other domains as meshes and graphs, where symmetry is less well-defined.

Conclusion

We have presented deep scale-spaces a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. We outlined new theory for a generalization of the convolution operator on semigroups, the semigroup correlation. Then we showed how to derive the standard convolution and the group convolution (Cohen and Welling, 2016a) from the semigroup correlation. We then tailored the semigroup correlation to the scale-translation action used in classical scale-space theory and demonstrated how to use this in modern neural architectures.

References

Appendix A Scale-spaces

Here we provide some extra information on scale-spaces for the interested reader. For in depth literature, we suggest Florack et al. , Pauwels et al. , Lindeberg , Crowley et al. , Salden et al. , Duits et al. , Duits and Burgeth , Burgeth et al. [2005a, b].

We are given an initial 1D signal $f_{0}$ with intrinsic bandlimit, or zero-scale, defined by $s_{0}$ i.e. , we impose a maximum frequency, such that there is a correspondence with discretized signals. Note that $\sqrt{s_{0}}$ is inversely proportional to frequency content of the signal. We wish to downsize it isotropically by a factor $a$, which we call the dilation. For this we introduce the downsizing action $L_{a}[f](x)=f(a^{-1}x)$ for $a\leq 1$. We model the bandlimit of the initial signal as the result of convolving some other signal $f$ with a Gaussian of width $s_{0}$, so

Now the result of the downscaling action of $f_{0}$ is as follows

From the first to second lines we have performed a change of variables $z=ay$. So we see that the effect of downsizing a bandlimited signal by $a$ shifts the bandlimit from $s_{0}$ to $a^{2}s_{0}$. Since $a\leq 1$, this means the blurring Gaussian is narrower and so the frequency content of the signal has been shifted higher. The key relation to bear in mind is the shift $s_{0}\mapsto a^{2}s_{0}$.

For a proper scaling, we want the result of the downscaling to have the same bandlimit as the original signal $f_{0}$. This is because if we are representing the signal on a discrete grid, then we have a physically defined maximum frequency content we can store, given by the pixel separation. The solution is to convolve the signal $L_{a}[f_{0}]$ with a correcting Gaussian of width $s_{0}-a^{2}s_{0}$. Note that this is possible, since $a\leq 0$, so $s_{0}-a^{2}s_{0}>0$. Alternatively, we want to find a correcting Gaussian to blur before downsizing. Say this correcting Gaussian has bandlimit $t$, then we have

But we also want the bandlimit of the downscaled signal to be $s_{0}$, so we have the relation

Thus a 1D Gaussian scale-space, parameterized by dilation, can be built by setting

A.2 N𝑁ND Gaussian Scale-space

We now explore downscaling in $N$-dimensions. We first of all represent an initial image $f_{0}$ as

We use the Gaussian to represent the fact that $f_{0}$ should have an intrinsic bandlimit, usually defined by the resolution at which it is sampled. Now let’s introduce the affine action:

It simply applies an affine transformation to our signal. Now using a similar logic to in the 1D case, if we concatenate the affine action with bandlimiting we get

So we see that resizing a bandlimited signal shifts the bandlimit according to $\Sigma_{0}\mapsto A\Sigma_{0}A^{\top}$. Since we would like to have the same bandlimit on our signal before and after the resizing (since we can only represent the signal at constant resolution), we introduce a second bandlimiting by convolving with a Gaussian of width $\Sigma_{0}-A\Sigma_{0}A^{\top}$. To save space, we write $G_{\Sigma}=G(\cdot,\Sigma)$. So

From the first to the second equality, we have exchanged the order of the Gaussian convolution and the affine action and altered the bandwidth from $\Sigma_{0}-A\Sigma_{0}A^{\top}$ to $A^{-1}\Sigma_{0}A^{-\top}-\Sigma_{0}$, which comes from the relation established in Equation 35. Thus the affine action for an affine scale-pyramid is defined as

where we have defined $G_{A}^{\Sigma_{0}}=G_{A^{-1}\Sigma_{0}A^{-\top}-\Sigma_{0}}$.

For what values of $A$ and $z$ is this action valid? Let’s first focus on $A$. To maintain the zero-scale of $\Sigma_{0}$, we had to convolve with a Gaussian of width $\Delta=\Sigma_{0}-A\Sigma_{0}A^{\top}$. Now we know that covariance matrices have to be symmetric $\Delta=\Delta^{\top}$ and positive definite $\Delta\succ 0$. We see already that it is symmetric, but it is not necessary positive definite. If the base bandlimit is of the form $\Sigma_{0}=\sigma_{0}^{2}I$ (the original image is isotropically bandlimited), then we can rearrange to

This expression is only positive definite if $I-AA^{\top}\succ 0$; that is

This condition implies that $A^{\top}$ is a contraction because

Another way of phrasing this is the that the singular values of $A$ may not exceed unity. Note that rotations do not break this constraint. So we see this this model naturally aligns with our notion that we can only model image downscalings, and that upscalings are prohibited.

A.3 Other Scale-space variants

We have presented the Gaussian scale-space in $N$D, but there also exists a zoo of other scales-spaces. The most prominent are: the $\alpha$-scale-spaces [Pauwels et al., 1995], the discrete Gaussian scale-spaces [Lindeberg, 1990], and the binomial scale-spaces [Burt, 1981]. In the following, we give a brief introduction to each, exhibited in 1D.

$\alpha$-scale-spaces $\alpha$-scale-spaces Pauwels et al. are a generalization of the Gaussian space-space in the continuous domain. They are easiest to understand by considering their form in Fourier-space. We begin by considering the Fourier transform of the Gaussian space-space over the spatial dimension

where $\hat{f}$ is the Fourier transform of $f$. We are interested in finding a collection of filters like $G$, closed under convolution. In the Fourier domain this corresponds to finding a collection of filters, like $\hat{G}$ closed under multiplication. The Fourier transform of the Gauss-Weierstrass kernel is

The collection $\{\hat{G}(\omega,t)\}_{t>0}$ is indeed closed under multiplication and forms a semigroup. To form the $\alpha$-scale-spaces we notice that the Fourier kernel

is also closed under multiplication and defines a semigroup. The range of $\alpha$ is typically taken to be $(0,1]$, to make sure that higher levels in the $\alpha$-scale-space are blurrier. Notice that for $\alpha=1$ we return to the standard Gaussian scale-space.

where $N>0$ is the width of the kernel and $0\leq x\leq N$ is the spatial location of the filter tap. Thus the scale-space is

As $N$ grows $B(N,x)$ rapidly converges to a Gaussian kernel of variance $\sigma^{2}=N/4$. The Binomial filters are closed under convolution obeying the semigroup property

where the term $I_{k}(t)$ is a modified Bessel function of the first kind. These can be implemented easily using scipy.special.ive. The scale-space is formed in the usual way as

Appendix B Architectures In The Experiments

In the experiments, we use a DenseNet Huang et al. and a ResNet He et al. . The architectures are as follows. For the scale equivariant versions, we use 4 scales of a discrete Gaussian scale-space Lindeberg .

ResNet The residual network consists of a concatenation of residual blocks. A single residual block implements the following

where on the RHS we refer to $x$ as the skip connection and $\mathcal{F}(x)$ as the residual connection. If $x$ has fewer channels than $\mathcal{F}(x)$, then we pad the missing dimensions with zeros. Each residual connection uses a concatenation of two scale-equivariant correlation interleaved with batch normalization (BN) and a ReLU (ReLU) nonlinearity. These are composed as follows (input left, output right)

where corr[$k,h,w$] refers to a scale correlation with kernel size $[k,h,w]$ and where $k$ is the number of scale channels, $h$ is the spatial height of the filter, and $w$ is its spatial width. We denote the entire residual block as res[$k,h,w$].

The model we use is given in Table 2. It follows the practice of Yu et al. , who use a bilinear upsampling at the end of the network, since segmentations do not tend to contain high frequency details. In our experiments we use the models shown in Table 3

DenseNet The Dense network Huang et al. consists of a concatenation of 3 dense blocks. Each dense block is composed of layers of the form

where $[y_{1},y_{2},...,y_{N}]$ is the concatenation of all the previous layers’ outputs. Each layer $\mathcal{H}$ is the composition (input left, output right)

where corr[$k,3,3$] was described in the previous section. We use the notation $\text{dense}_{C}[k,h,w]\times N$ to denote a dense block with $N$ layers and $C$ output channels per layer. The number of channel outputs remains constant per layer within a dense block. Between dense blocks, we insert transition layer which have the form

Here we use a $1\times 1$ convolution to halve the number of output channels, and then perform a spatial average pooling with kernel size 2 and stride 2, followed by a second dense layer. We denote these as $\text{transition}[k,h,w]$. We also use long skip connection between transistion layers. The network we use is shown in Table 4.