Scattering Networks for Hybrid Representation Learning

Introduction

Natural image processing tasks are high dimensional problems that require introducing lower dimensional representations: in the case of image classification, they must reduce the non-informative image variabilities, whereas for image generation, it is desirable to parametrize them. For example, some of the main source of variability are often due to geometrical operations such as translations and rotations. Then, an efficient classification pipeline necessarily builds invariants to these variabilities, whereas mapping to those sources of variabilities is desirable in the context of image generation. Deep architectures build representations that lead to state-of-the-art results on image classification tasks . These architectures are designed as very deep cascades of non-linear end-to-end learned modules . When trained on large-scale datasets they have been shown to produce representations that are transferable to other datasets , which indicates they have captured generic properties of a supervised task that consequently do not need to be learned. Indeed several works indicate geometrical structures in the filters of the earlier layers of Deep CNNs . However, understanding the precise operations performed by those early layers is a complicated and possibly intractable task. In this work we investigate if it is possible to replace these early layers by simpler cascades of non-learned operators that reduce and parametrize variability while retaining all the discriminative information.

Indeed, there can be several advantages to incorporating pre-defined geometric priors via a hybrid approach of combining pre-defined and learned representations. First, end-to-end pipelines can be data hungry and ineffective when the number of samples is low. Secondly, it could lead to more interpretable classification pipelines, which are amenable to analysis, and permits the performance of parallel transport along the Euclidean group. Finally, it can reduce the spatial dimensions and the required depth of the learned modules, improving their computational and memory requirements.

A potential candidate for an image representation is the SIFT descriptor , which was widely used before 2012 as a feature extractor in classification pipelines . This representation was typically encoded via an unsupervised Fisher Vector (FV) and fed to a linear SVM. However, several works indicate that this is not a generic enough representation on top of which to build further modules . Indeed end-to-end learned features produce substantially better classification accuracy.

A Scattering Transform is an alternative that solves some of the issues with SIFT and other predefined descriptors. In this work, we show that contrary to other proposed descriptors , a Scattering Network can avoid discarding information. Indeed, a Scattering Transform is not quantized, and the loss of information is avoided thanks to a combination of wavelets and non linear operators. Furthermore, it is shown in that a Scattering Network provides a substantial improvement in classification accuracy over SIFT. A Scattering Transform also provides certain mathematical guarantees, which CNNs generally lack. Finally, wavelets are often observed in the initial layers, as in the case of AlexNet . Thus, combing the two approaches is natural.

This article is an extended version of . Our main contributions are as follows. First, we design and develop a fast algorithm to compute a Scattering Transform to use in a deep learning context. We demonstrate that using supervised local descriptors obtained by shallow $1\times 1$ convolutions with very small spatial window sizes obtains AlexNet accuracy on the ImageNet classification task (Subsection 4.1). We show empirically that these encoders build explicit invariance to local rotations (Subsection 4.3). Second, we propose hybrid networks that combine scattering with modern CNNs (Section 5) and show that using scattering and a ResNet of reduced depth, we obtain similar accuracy to ResNet-18 on ImageNet (Subsection 5.1). Then, we study adversarial examples to the Scattering Transform with a linear classifier. We then develop a procedure to reconstruct an image from its scattering representation in Section 3.4 and show that this can be used to incorporate the scattering transform in a hybrid Generativel Adversarial Network in Section 6.2. Finally, we demonstrate in Subsection 5.3 that scattering permits a substantial improvement in accuracy in the setting of limited data.

Our highly efficient GPU implementation of the scattering transform is, to our knowledge, orders of magnitude faster than any other implementations, and allows training very deep networks while applying scattering on the fly. Our scattering implementationhttp://github.com/edouardoyallon/pyscatwave and pre-trained hybrid modelshttp://github.com/edouardoyallon/scalingscattering are publicly available.

Related Work

Closely related to our work, proposed a hybrid representation for large scale image recognition combining a predefined representation and Neural Networks (NN), that uses a Fisher Vector (FV) encoding of SIFT and leverages NNs as scalable classifiers. In contrast we use the scattering transform in combination with convolutional architectures and show hybrid results that well exceed those of .

A large body of recent literature has also considered unsupervised and self-supervised learning for constructing discriminative image features that can be used in subsequent image recognition pipelines. However, to the best of our knowledge on complex datasets such as imagenet these representations do not yet approach the accuracy of supervised methods or hand-crafted unsupervised representations. In particular the FV encoding discussed above is an unsupervised representation that has outperformed any unsupervised learned representation on the imagenet dataset .

With regards to the algorithmic implementation of the Scattering Transform, former implementations were only scaled for CPU as they retain too many intermediate variables, which can be too large for GPU use. A major contribution of our work is to propose an efficient approach which fits in GPU memory, which subsequently allows a much faster computational time than the CPU implementations. This is essential for scaling to the ImageNet dataset.

Concurrent to our work the Scattering Transform was also recently used in a context of generative modeling : it is shown that by inverting the scattering transform, it is possible to generate images in a similar fashion as GANs. We however adopt a rather different approach by building hybrid GANs that directly learn to generate Scattering coefficients, which we reconstruct back into images.

Scattering: a baseline for image classification

We now describe the scattering transform and motivate its use as a generic input for supervised tasks. A scattering network belongs to the class of CNNs whose filters are fixed wavelets . The construction of this network has strong mathematical foundations , meaning it is well understood, relies on few parameters, and is stable to a large class of geometric transformations. In general, the parameters of a scattering transform do not need to be adapted to the bias of the dataset , making its output a suitable generic representation.

We then propose and motivate the use of supervised CNNs built on top of the scattering network. Finally we propose supervised encodings of scattering coefficients using 1x1 convolutions, which can retain interpretability and locality properties.

Observe that $j_{1}$ and $\theta_{1}$ have been discretized – the wavelet is chosen to be selective in angle and localized in the Fourier domain. With appropriate discretization , $\{A_{J}x,W_{1}x\}$ is approximatively an isometry on the set of signals with limited bandwidth, which implies that the energy of the signal is preserved. This operator then belongs to the category of multi-resolution analysis operators, each filter being excited by a specific scale and angle, but with the output coefficients not being invariant to translation. To achieve invariance we cannot apply $A_{J}$ directly to $W_{1}x$ since it would result in a trivial invariant, namely zero.

To tackle this issue, we first apply a non-linear point-wise complex modulus to $W_{1}x$, followed by an averaging $A_{J}$, and a downsampling of scale $2^{J}$, which builds a non-trivial invariant. Here, the mother wavelet is analytic, thus $|W_{1}x|$ is regular which implies that the energy of $|W_{1}x|$ in the Fourier domain is more likely to be contained in a lower frequency regime than $W_{1}x$. Thus, $A_{J}$ preserves more energy of $|W_{1}x|$. It is possible to define

these are the first-order scattering coefficients. Following deep-learning terminology, each $S^{1}x(j_{1},\theta_{1},\cdot)$ can be thought of as a one channel in a feature map. Again, the use of averaging builds an invariant to translation up to $2^{J}$. In our grayscale image example, $S^{1}x[p]$ is a feature map of resolution $\frac{N}{2^{J}}\times\frac{N}{2^{J}}$ with $JL$ channels.

To recover some of the high-frequencies lost due to the averaging applied on the first order coefficients, we apply again a second wavelet transform $W_{2}$ (with the same filters as $W_{1}$) to each channel of the first-order scatterings, before the averaging step. This leads to the second-order scattering coefficients

We only compute paths of increasing scale ($j_{1}<j_{2}$) because non-increasing paths have been shown to bear no energy . In our grayscale image example, $S^{2}x[p]$ is a feature map of resolution $\frac{N}{2^{J}}\times\frac{N}{2^{J}}$ with $\frac{1}{2}J(J-1)L^{2}$ channels (one per increasing path).

We do not compute higher order scatterings, because their energy is negligible . We call $Sx(u)$(or $S_{J}x(u)$) the final scattering coefficient corresponding to the concatenation of the order 0, 1 and 2 scattering coefficients, intentionally omitting the path index of each representation. A schematic diagram is shown in Figure 1. In the case of color images, we apply independently a scattering transform to each RGB channel of the image, which means $Sx(u)$ is a feature map with 3\times\big{(}1+JL+\frac{1}{2}J(J-1)L^{2}\big{)} channels, and the original image is down-sampled by a factor $2^{J}$ .

This representation has been proved to linearize small deformations of images , be non-expansive and almost complete , which makes it an ideal input to a deep network algorithm, which can build invariants to this local variability via a first linear operator. We discuss its use as an initialization of a deep network in the next sections.

2 Efficient Implementation of Scattering Transforms

The implementation of a Scattering Network must be re-thought to benefit from GPU acceleration. Indeed, a GPU is a device which has a limited memory size in comparison with a CPU, and thus it is not possible to store intermediate computations. In this section, we show how to solve this problem of memory. We first describe the naive tree implementation and then our efficient GPU based implementation.

We recall the algorithm to compute a Scattering Transform and its implementation in for order 2 Scattering with a scale of $J$ and $L$ different orientations for the wavelets. We explicitly show this algorithm is not appropriate to be scaled on a GPU. It corresponds to a level order traversal of the tree of computations of the Figure 2(a). Let us consider again a discretized input signal $x[p]$ of size $N^{2}$ which is a power of 2, and a spatial sampling of 1. For the sake of simplicity, we assume that an algorithm such as a symmetric padding has already been applied to $x$ in order to avoid boundary effects that are inherent to periodic convolutions. The filter bank corresponds to $JL+1$ filters:

We only consider periodized filters, e.g.:

The above approach is far too memory-consuming for a GPU implementation. For example, for $J=2,3,4,L=8,$ and $N=256$, which corresponds to the setting used on our ImageNet experiments, we numerically have approximately $2M,2.5M,2.6M$ parameters for a single tensor. A parameter is about 4 bytes, thus an image is about 8MB in the smallest case. In the case of batches of size 256 with color images, we thus need at least 6GB of memory simply to store the intermediate tensors used by the scattering, which does not take in account extra-memory used by libraries such as cuFFT for example. In particular, this reasoning demonstrates that a typical GPU with 12GB of memory can not efficiently process images in parallel with this naive approach.

2.2 Memory efficient implementation on GPUs

We now describe a GPU implementation which tries to minimize the memory usage during the computations. The procedures (a/a’), (b), (c) and (d) of the previous section can be efficiently implemented entirely on GPUs. They are fast, and can be implemented in batches, which permits parallel computations of the scattering representation. This is necessary for deep learning pipelines, which commonly use batches of data augmented samples.

We also note that in the case of training hybrid networks it is possible to store the computed scattering coefficients for a dataset via a cache. In this case, it is possible to obtain a speedup by a large factor since no extra computations are required to compute the earlier layers as optimization of the network proceeds. These early layers are often the most computationally expensive in comparison with deeper layers.

3 Reconstruction from the Scattering Coefficients

Reconstruction of an image from a scattering representation can be critical for permitting it’s use in applications such as image generation. It also permits to obtain insights into the representation. We describe a simple method to reconstruct an image from its order 2 scattering representation. Several works proposed to synthesize textures and stochastic processes from their expected scattering coefficients. In the case of stationary processes, the final local averaging of a scattering transform allows the building of an unbiased estimator of the expected scattering coefficients, and the smallest variance is achieved using the largest windows size of invariance, i.e. the full image. This does not hold in the case of natural images, which do not correspond to stationary processes, and thus, global invariance to translation is not desirable because it loses spatial localization information. We show a straightforward approach can yield competitive reconstruction.

This is achieved via a gradient descent, without however any (known) theoretical guarantees of convergence to the original signal. Computations are made possible thanks to the auto-differentiation tool of PyTorch. In this setting, we chose the optimizer Adam. The initial image is initialized as a white noise with variance $10^{-4}$ and is represented in the YUV space because it decorrelates approximatively the color channels and the intensity channels, and we observed it leads to better reconstruction. The algorithm converges to a visually reasonable solution after $200$ iterations, the loss reaching a plateau, and there is no extra-regularization or parametrization because empirically this has not yielded better reconstruction. Results are displayed in Figure 3 for different values of $J$ and an image $x$ of size $256^{2}$. For each reconstruction, we evaluate its quality by computing the relative error of reconstruction with the original signal $\text{err}(x)$, and its distance in the scattering space $\text{err}(S_{J})$,

We demonstrate good reconstruction in the case of $J=2,3,4$ and we show that numerically, by $J\geq 5$, the obtained images are rather different from the original image due to the averaging loss. The attributes that are not well reconstructed are blurry and not at the appropriate spatial localization, which seems to indicate they have been lost by the spatial averaging. For $J<7$, the Scattering coefficients are almost identical, however, for $J=5,6$ several corners and borders of the images are not well recovered, which indicates it is possible to find very different images with similar scattering coefficients. An open question is to understand if cascading more wavelet transforms could recover this information. For $J\geq 7$, the reconstructed signals are very different, only several textures seem to have been recovered and the color channels are decorrelated. Furthermore, the case $J=7$ exhibits strong artifacts from the large scale wavelet, which is linked to the implementation of the wavelet transform.

Due to this lack of localization and ability to discriminate, in the following sections we combine CNNs with a scattering transform with scales $J<5$, and therefore filters of width less than $2^{5}=32$ pixels.

4 Cascading a Supervised Architecture on Top of Scattering

We now motivate the use of a supervised architecture on top of a scattering network. Scattering transforms have yielded excellent numerical results on datasets where the variabilities are completely known, such as MNIST or FERET. In these task, the problems encountered are linked to sample and geometric variance and handling these variances leads to solving these problems. However, in classification tasks on more complex image datasets, such variabilities are only partially known as there are also non geometrical intra-class variabilities. Although applying the scattering transform on datasets like CIFAR-10 or CalTech leads to nearly state-of-the-art results in comparison to other unsupervised representations, there is a large gap in performance when comparing to supervised representations . CNNs fill in this gap. Thus we consider the use of deep neural networks utilizing generic scattering representations in order to learn more complex invariances than geometric ones alone.

Thus, we consider two types of cascaded deep networks on top of scattering. The first, referred to as the Shared Local Encoder (SLE), learns a supervised local encoding of the scattering coefficients. We motivate and describe the SLE in the next subsection as an intermediate representation between unsupervised local pipelines, widely used in computer vision prior to 2012, and modern supervised deep feature learning approaches. The second, referred to as a hybrid CNN, is a cascade of a scattering network and a standard CNN architecture, such as a ResNet . In the sequel we empirically analyse hybrid CNNs, which allow us to greatly reduce the spatial dimensions on which convolutions are learned and can reduce sample complexity.

Local Encoding of Scattering

First, we motivate the use of the Shared Local Encoder for natural image classifications. Then, we evaluate the supervised SLE on the Imagenet ILSVRC2012 dataset. This is a large and challenging natural color image dataset consisting of $1.2$ million training images and $50,000$ validation images, divided into $1000$ classes. We then show some unique properties of this network and evaluate its features on a separate task.

We now discuss the spatial support of different approaches, in order to motivate our local encoder for scattering. In CNNs constructed for large scale image recognition, the representations at a specific spatial location and depth depend upon large parts of the initial input image and thus mixes global information. For example, in , the effective spatial support of the corresponding filter is already 32 pixels (out of 224) at depth 2. The specific representations derived from CNNs trained on large scale image recognition are often used as representations in other computer vision tasks or datasets .

On the other hand prior to 2012 local encoding methods led to state of the art performance on large scale visual recognition tasks . In these approaches local neighborhoods of an image were encoded using method such as SIFT descriptors , HOG , and wavelet transforms . They were also often combined with an unsupervised encoding, such as sparse coding or Fisher Vectors (FVs) . Indeed, many works in classical image processing or classification suggest that local encodings of an image are efficient descriptions. Additionally for some algorithms that rely on local neighbourhoods, the use of local descriptors is essential . Observe that a representation based on local non overlapping spatial neighborhood is simpler to analyze, as there is no ad-hoc mixing of spatial information. Nevertheless, in large scale classification, this approach was surpassed by fully supervised learned methods .

We show that it is possible to apply a similarly local, yet supervised encoding algorithm to a scattering transform, as suggested in the conclusion of . First observe that at each spatial position $u$, a scattering coefficient $S(u)$ corresponds to a descriptor of a local neighborhood of spatial size $2^{J}$. As explained in the first Subsection 3.1, each of our scattering coefficients are obtained using a stride of $2^{J}$, which means the final representation can be interpreted as a non-overlapping concatenation of descriptors. Let $f$ be a cascade of fully connected layers that we identically apply on each $Sx(u)$. Then $f$ is a cascade of CNN operators with spatial support size $1\times 1$, thus we write $fSx\triangleq\{f(Sx(u))\}_{u}$. In the sequel, we do not make any distinction between the $1\times 1$ CNN operators and the operator acting on $Sx(u),\forall u$. We refer to $f$ as a Shared Local Encoder. We note that similarly to $Sx$, $fSx$ corresponds to non-overlapping encoded descriptors. To learn a supervised classifier on a large scale image recognition task, we cascade fully connected layers on top of the SLE.

Combined with a scattering network, the supervised SLE, has several advantages. Since the input corresponds to scattering coefficients whose channels are structured, the first layer of $f$ is structured as well. We further explain and investigate this first layer in Subsection 4.3. Unlike standard CNNs, there is no linear combination of spatial neighborhoods of the different feature maps, thus the analysis of this network need only focus on the channel axis. Observe that if $f$ was fed with raw images, for example in gray scale, it could not build any non-trivial operation except separating different level sets of these images.

In the next section, we investigate empirically this supervised SLE trained on the ILSVRC2012 dataset.

2 Shared Local Encoder on Imagenet

We first describe our training pipeline, which is similar to . We trained our network for 90 epochs to minimize the standard cross entropy loss, using SGD with momentum 0.9 and a batch size of 256. We used a weight decay of $1\times 10^{-4}$. The initial learning rate is $0.1$, and is decreased by a factor of 10 at epochs $30$, $50$, $70$, and $80$. During the training process, each image is randomly rescaled, cropped, and flipped as in . The final crop size is $224\times 224$. At testing, we rescale the image to a size of $256\times 256$, and extract a center crop of size $224\times 224$.

We use an architecture which consists of a cascade of a scattering network, a SLE $f$, followed by fully connected layers. Figure 4 describes our architecture. We select the parameter $J=4$ for our scattering network, which means the output representation has size $\frac{224}{2^{4}}\times\frac{224}{2^{4}}=14\times 14$ spatially and 1251 channels. $f$ is implemented as 3 layers of 1x1 convolutions $F_{1},F_{2},F_{3}$ with layer size 1024. There are 2 fully connected layers of ouput size 1524. For all learned layers we use batch normalization followed by a ReLU non-linearity. We compute the mean and variance of the scattering coefficients on the whole of ImageNet, and standardized each spatial scattering coefficients with them.

Table II reports our numerical accuracies obtained with a single crop at testing, compared with local encoding methods, and AlexNet, which was the state-of-the-art approach in 2012. We obtain 20.4% at Top 5 and 43.0% Top 1 errors. The performance is analogous to the AlexNet . In term of architecture, our hybrid model is analogous, and comparable to that of , for which SIFT features are extracted followed by FV encoding. Observe the FV is an unsupervised encoding compared to our supervised encoding. Two approaches are then used: the spatial localization is handled either by a Spatial Pyramid Pooling , which is then fed to a linear SVM, or the spatial variables are directly encoded in the FVs and classified with a stack of four fully connected layers. This last method is a major difference with ours, as the obtained descriptor does not have a spatial indexing anymore which are instead quantized. Furthermore, in both case, the SIFT are densely extracted which correspond to approximatively $2\times 10^{4}$ descriptors, whereas in our case, only $14^{2}=196$ scattering coefficients are extracted. Indeed, we tackle the non-linear aliasing (due to the fact that the scattering transform is not oversampled) via random cropping during training, enabling invariance to small translations. In Top 1, and obtain error rates of 44.4% and 45.7%, respectively. Our method brings a substantial improvement of 1.4% and 2.7%, respectively.

The BVLC AlexNethttps://github.com/BVLC/caffe/wiki/Models-accuracy-on-ImageNet-2012-val obtains a of 43.1% single-crop Top 1 error, which is nearly equivalent to the 43.0% of our SLE network. The AlexNet has 8 learned layers and as explained before, large receptive fields. On the contrary, our training pipeline consists in 6 learned layers with constant receptive field of size $16\times 16$, except for the fully connected layers that build a representation mixing spatial information from different locations. This is a surprising result, as it seems to suggest contextual information is only necessary at the very last layers, to reach AlexNet accuracy.

We study briefly the local SLE, which only has a spatial extent of $16\times 16$, as a generic local image descriptor. We use the Caltech-101 benchmark which is a dataset of 9144 images and 102 classes. We followed the standard protocol for evaluation with 10 folds and evaluate per class accuracy with 30 training samples per class, using a linear SVM used with the SLE descriptors. Applying our raw scattering network leads to an accuracy of $62.8\pm 0.7$, and the output features from $F_{1}$, $F_{2}$, and $F_{3}$ bring an absolute improvement of $13.7$, $17.3$, and $20.1$, respectively. The accuracy of the final SLE descriptor is thus $82.9\pm 0.4$, similar to that reported for the AlexNet final layer in and sparse coding with SIFT . However in both cases spatial variability is removed, either by Spatial Pyramid Pooling , or the cascade of large filters. By contrast, the concatenation of SLE descriptors are completely local. Similarly, the scattering network combined with ResNet-10 introduced in the next section, and followed by a linear SVM achieves 87.7 on Caltech-101, yet this descriptor is not local.

3 Interpreting SLE’s first layer

Finding structure in the kernel of the layers of depth less than $2$ is a complex task, and few empirical analyses exist that shed light on the structure of deeper layers. A scattering transform with scale $J$ can be interpreted as a CNN with depth $J$ , whose channels indexes correspond to different scattering frequency indexes, which is a structuration. This structure is consequently inherited by the first layer $F_{1}$ of our SLE $f$. We analyse $F_{1}$ and show that it explicitly builds invariance to local rotations, and also that the Fourier bases associated to rotations are a natural bases of our operator. It is a promising direction to understand the nature of the next two layers.

We first establish some mathematical notions linked to the rotation group that we use in our analysis. For the sake of clarity, we do not consider the roto-translation group. For a given input image $x$, let $r_{\theta}.x(u)\triangleq x(r_{-\theta}(u))$ be the image rotated by angle $\theta$, which corresponds to the linear action of rotation on images. Observe the scattering representation is covariant with the rotation in the following sense:

Additionally, in the case of the second order coefficients, $(\theta_{1},\theta_{2})$ is covariant with rotations, but $\theta_{2}-\theta_{1}$ is an invariant to rotation that corresponds to a relative rotation.

The unitary representation framework permits the building of a Fourier transform on a compact group, such as rotations. It is even possible to build a scattering transform on the roto-translation group . Fourier analysis permits the measurement of the smoothness of the operator and, in the case of a CNN operator, it is a natural basis.

We can now numerically analyse the nature of the operations performed along angle variables by the first layer $F_{1}$ of $f$, with output size $K=1024$. Let us define as $\{F^{0}_{1}S^{0}x,F^{1}_{1}S^{1}x,F^{2}_{1}S^{2}x\}$ the restrictions of $F_{1}$ to the order 0, 1, and 2 scattering coefficients respectively. Let $1\leq k\leq K$ be an index of a feature channel and $1\leq c\leq 3$ be the color index. In this case, $F^{0}_{1}S^{0}x$ is simply the weights associated to the smoothing $S_{0}x$. $F^{1}_{1}S^{1}x$ depends only on $(k,c,j_{1},\theta_{1})$, and $F^{2}_{1}$ depends on $(k,c,j_{1},j_{2},\theta_{1},\theta_{2})$. We would like to characterize the smoothness of these operators with respect to the variables $(\theta_{1},\theta_{2})$, because $Sx$ is covariant to rotations.

To illustrate that this is a natural basis we explicitly sparsify this operator in its frequency basis and verify that empirically the network accuracy is minimally changed. We do this by thresholding by $\epsilon$ the coefficients of the operators in the Fourier domain. Specifically we replace the operators $\hat{F}_{1}^{1}$, $\hat{F}_{1}^{2}$ by $1_{|\hat{F}_{1}^{1}|>\epsilon}\hat{F}_{1}^{1}$ and $1_{|\hat{F}_{1}^{2}|>\epsilon}\hat{F}_{1}^{2}$. We select an $\epsilon$ that sets $80\%$ of the coefficients to 0, which is illustrated in Figure 5. Without retraining our network performance degrades by only an absolute value of $2\%$ worse on Top 1 and Top 5 ILSVRC2012. We have thus shown that this basis permits a sparse approximation of the first layer, $F_{1}$. We now show evidence that this operator builds an explicit invariant to local rotations.

To aid our analysis we introduce the following quantities:

They correspond to the energy propagated by $F_{1}$ for a given frequency, and quantify the smoothness of our first layer operator w.r.t. the angular variables. Figure 6 shows variation of $\Omega_{1}\{F\}$ and $\Omega_{2}\{F\}$ as a function of the frequencies. For example, if $F_{1}^{1}$ and $F_{1}^{2}$ were convolutional along $\theta_{1}$ and $(\theta_{1},\theta_{2})$, these quantities would correspond to their respective singular values. One sees that the energy is concentrated in the low frequency domain, which indicates that $F_{1}$ builds explicitly an invariant to local rotations.

Cascading a Supervised Deep CNN Architecture

We demonstrate that cascading modern CNN architectures on top of the scattering network can produce high performance classification systems. We apply hybrid convolutional networks on the Imagenet ILSVRC 2012 dataset as well as the CIFAR-10 dataset and show that they can achieve performance comparable to modern end-to-end learned approaches. We then evaluate the hybrid networks in the setting of limited data by utilizing a subset of CIFAR-10 as well as the STL-10 dataset and show that we can obtain substantial improvement in performance over analogous end-to-end learned CNNs.

We showed in the previous section that a SLE followed by FC layers can produce results comparable to AlexNet on the ImageNet classification task. Here we consider cascading the scattering transform with a modern CNN architecture, such as ResNet . We take ResNet-18 as a reference and construct a similar architecture with only 10 layers on top of the scattering network. We utilize a scattering transform with $J=3$ such that the CNN is learned over a spatial dimension of $28\times 28$ and a channel dimension of 651 (3 color channels of 217 each). ResNet-18 typically has 4 residual stages of 2 blocks each which gradually decrease the spatial resolution . Since we utilize the scattering as a first stage we remove two blocks from our model. The network is described in Table V.

We use the same optimization and data augmentation procedure described in Section 4.2 but with decreases in the learning rate at 30, 60, and 80 epochs. We find that when both methods are trained with the same settings of optimization and data augmentation, and when the number of parameters is similar (12.8M versus 11.7 M) the scattering network combined with a ResNet can achieve analogous performance (11.4$\%$ Top 5 for our model versus 11.1$\%$), while utilizing fewer layers compared to a pure ResNet architecture. The accuracy is reported in Table III and compared to other modern CNNs.

This demonstrates both that the scattering networks does not lose discriminative power and that it can be used to replace early layers of standard CNNs. We also note that learned convolutions occur over a drastically reduced spatial resolution without resorting to pre-trained early layers, which can potentially lose discriminative information or become too task specific.

2 Deep Hybrid CNNs on CIFAR-10

We now consider the popular CIFAR-10 dataset consisting of color images composed of $5\times 10^{4}$ images for training, and $1\times 10^{4}$ images for testing divided into 10 classes. We use a hybrid CNN architecture with a ResNet built on top of the scattering transform.

For the scattering transform we used $J=2$ which means the output of the scattering stage will be $8\times 8$ spatially and 243 in the channel dimension. We follow the training procedure prescribed in utilizing SGD with momentum of 0.9, batch size of 128, weigh decay of $5\times 10^{-4}$, and modest data augmentation by using random cropping and flipping. The initial learning rate is 0.1, and we reduce it by a factor of 5 at epochs 60, 120 and 160. The models are trained for 200 epochs in total. We used the same optimization and data augmentation pipeline for training and evaluation in both case. We utilize batch normalization techniques at all layers which lead to a better conditioning of the optimization . Table IV reports the accuracy in the unsupervised and supervised settings and compares them to other approaches.

We compare to state-of-the-art approaches on CIFAR-10, all based on end-to-end learned CNNs. We use a similar hybrid architecture to the successful wide residual network (WRN) . Specifically we modify the WRN of 16 layers, which consists of 4 convolutional stages. With $k$ denoting the widening factor, after the scattering output we use a first stage of $32\times k$. We add intermediate $1\times 1$ convolutions to increase the effective depth, without substantially increasing the number of parameters. Finally we apply a dropout of 0.2 as specified in . Using a width of 32 we achieve an accuracy of $93.1\%$. This is superior to several benchmarks but performs worse than the original ResNet and the wide ResNet . We note that training procedures for learning directly from images, including data augmentation and optimization settings, have been heavily optimized for networks trained directly on natural images, while we use them largely out of the box.

3 Limited samples setting

A major application of a hybrid representation is in the setting of limited data. Here the learning algorithm is limited in the variations it can observe or learn from the data, such that introducing a geometric prior can substantially improve performance. We evaluate our algorithm on the limited sample setting using a subset of CIFAR-10 and the STL-10 dataset.

We take subsets of decreasing size of the CIFAR dataset and train both baseline CNNs and counterparts that utilize the scattering as a first stage. We perform experiments using subsets of 1000, 500, and 100 samples, which are split uniformly amongst the 10 classes.

We use as a baseline the Wide ResNet of depth 16 and width 8, which shows near state-of-the-art performance on the full CIFAR-10 task in the supervised setting. This network consists of 4 stages of progressively decreasing spatial resolution detailed in [59, Table 1]. We construct a comparable hybrid architecture that removes a single stage and all strides, as the scattering already down-sampled the spatial resolution. This architecture is described in Table VI. Unlike the baseline, referred from here-on as WRN 16-8, our architecture has 12 layers and equivalent width, while keeping the spatial resolution constant through all stages prior to the final average pooling. We also incorporate the numerical results obtained via a VGG of depth 16 for the sake of comparison.

We use the same training settings for our baseline, WRN 16-8, and our hybrid scattering and WRN-12. The settings are the same as those described for CIFAR-10 in the previous section, with the only difference being that we apply a multiplier to the learning rate schedule and to the maximum number of epochs. The multiplier is set to 10, 20, and 100 for the 1000, 500, and 100 sample cases, respectively. For example the default schedule of 60, 120, and 160 epochs becomes 600, 1200, and 1600 for the case of 1000 samples and a multiplier of 10. Finally in the case of 100 samples we use a batch size of 32 in lieu of 128.

Table VII corresponds to the averaged accuracy over 5 different subsets, with the corresponding standard error. In this small sample setting, a hybrid network outperforms the purely CNN based baselines, particularly when the sample size is smaller. This is not surprising as we incorporate a geometric prior in the representation.

3.2 STL-10

The STL-10 dataset consists of color images of size $96\times 96$, with only 5000 labeled images in the training set divided equally in 10 classes and 8000 images in the test set. The larger size of the images and the small number of available samples make this a challenging image classification task. The dataset also provides 100,000 unlabeled images for unsupervised learning. We do not utilize these images in our experiments, yet we find we are able to outperform all methods which learn unsupervised representations using these unlabeled images, obtaining very competitive results on the STL-10 dataset.

We apply a hybrid convolutional architecture, similar to the one applied in the small sample CIFAR task, adapted to the size of $96\times 96$. The architecture is described in Table VI and is similar to that used in the CIFAR small sample task. We use the same data augmentation as with the CIFAR datasets. We apply SGD with learning rate 0.1 and learning rate decay of 0.2 applied at epochs 1500, 2000, 3000, 4000. Training is run for 5000 epochs. We use at training and evaluation the predefined 10 folds of 1000 training images each, as given in . The averaged result is reported in Table VIII. Unlike other approaches, we do not use the 4000 remaining training images to perform hyper-parameter tuning on each fold, as this is not representative of small sample situations. Instead we train the same settings on each fold. The best reported result in the purely supervised case is a CNN whose hyper parameters have been automatically tuned using 4000 images for validation achieving 70.1$\%$ accuracy. The other competitive methods on this dataset utilize the unlabeled data to learn in an unsupervised manner before applying supervised methods. We also evaluate on the full training set of 5000 images obtaining an accuracy of $87.6\%$, which is quite higher than $81.3\%$ using unsupervised learning and the full training set. These techniques add several hyper parameters and require an additional engineering process. Applying a hybrid network is on the other hand straightforward and is very competitive with all the existing approaches without using any unsupervised learning. In addition to showing that hybrid networks perform well in the small sample regime, these results, along with our unsupervised CIFAR-10 result, suggest that completely unsupervised feature learning on image data may still not outperform supervised methods and pre-defined representations for downstream discriminative tasks. One possible explanation is that in the case of natural images, unsupervised learning of more complex variabilities than geometric ones (e.g the rototranslation group) might be ill-posed.

Unsupervised and Hybrid Unsupervised Learning with the Scattering Transform

This section describes the use of the Scattering Transform as an unsupervised representation and as part of hybrid unsupervised learning. First we evaluate the scattering as an unsupervised representation using the CIFAR-10 and ImageNet datasets, then we show that it can be used inside common unsupervised learning schemes by proposing a hybrid GAN combined with a Scattering Transform, which synthesizes Scattering Coefficients from random Gaussian noise on $32\times 32$ color images from ImageNet. Using the reconstruction proposed in Section 3.4 we show that we can generate images from this GAN model.

We first consider the CIFAR-10 dataset used in Section 5.2 and perform an experiment that allows us to evaluate the scattering transform as an unsupervised representation with a complex non-convolutional classifier. In a second experiment, we consider the linear classification task on ILSVRC 2012 often used to evaluate unsupervised representations .

For CIFAR-10, as in Section 5.2, we used $J=2$ which means the output of the scattering stage will be $8\times 8$ spatially and 243 in the channel dimension. This task has been commonly evaluated on CIFAR-10 with a non-linear classifier and we thus consider the use of a MLP. We follow the training procedure prescribed in utilizing SGD with momentum of 0.9, batch size of 128, weigh decay of $5\times 10^{-4}$, and modest data augmentation of the dataset by using random cropping and flipping. The initial learning rate is 0.1, and we reduce it by a factor of 5 at epochs 60, 120 and 160. The models are trained for 200 epochs in total. We used the same optimization and data augmentation pipeline for training and evaluation in both cases. We utilize batch normalization at all layers which leads to a better conditioning of the optimization . Table IV reports the accuracy in the unsupervised and supervised settings and compares them to other approaches. Combining the scattering transform with a NN classifier consisting of 3 hidden layers, with width $1.1\times 10^{4}$, we show that one can obtain a new state of the art classification for the case of unsupervised convolutional layers. More numerical comparisons with other unsupervised methods, such as random networks, can be found in . Scattering based approaches outperform all methods utilizing learned and not-learned unsupervised features, further demonstrating the discriminative power of the scattering network representation.

For the ILSVRC-2012 dataset we use a common evaluation based on training a linear classifier on top of the unsupervised representation . We used a standard training protocol with cross-entropy loss on top of a scattering transform produced with $J=4$. We apply standard data augmentation, optimizing with stochastic gradient descent with momentum $0.9$, weight decay set to $1e-7$, and learning rate drops at epochs 20, 40, and 60. The results are shown in Table IX and are compared with unsupervised and self-supervised baselines. Observe that a Scattering Transform improves significantly from a random baseline , and that it recognize a large number of images even when only considering the top result. The accuracy of a random baseline is still high, because the small support of the convolutional operators already incorporates some geometric structures in this type of pipeline. Modern learned unsupervised representations however can improve on this result.

It shows that being only 1-Lipschitz is not sufficient to be visually robust to such artifacts, when combined only with a linear classifier; using non-linear classifier, such as a CNN, designed to be robust to predefined noises could permit to tackle this issue.

2 Hybrid Unsupervised Learning with Scattering GAN

In this section we propose to construct a Generative Adversarial Network (GAN) in the space of scattering coefficients. This essentially constructs a hybrid generator and discriminator. The GAN is a state-of-the-art generative modeling framework. The use of the learned generator on top of a scattering transform can be well motivated if we consider the scattering transform as good a model of low level texture . Furthermore, as extensive data augmentation is often not required, it is possible to store scattering representations that have a smaller spatial resolution, permitting us to try rapidly a variety of architectures. We demonstrate in the following that a scattering representation can be used as the initialization of a generative model, similar to the classification case.

We follow the Deep Convolutional Generative Adversarial Network architectures proposed in in order to generate signals in the scattering space. We consider color images from the resized ImageNet dataset of size $32\times 32$ in the $YUV$ space that are processed by a Scattering Transform with $J=2$. The scattering coefficients were renormalized to lie between $-1$ and $1$. Their scattering representations are then fed to the generator and discriminators of our Scattering-DCGAN. In particular, the generator aims to synthesize scattering coefficients from a Gaussian noise with $d=100$. They are represented in Table X. Moreover we apply the recently proposed Wasserstein distance based objective .

We now describe our training procedure. We run the Adam optimizer for both the discriminator and generator during 600k iterations without observing significant instabilities during the optimization. The discriminator is trained during 5 successive iterations and the generator only 1, as done in , because we observed it leads to more realistic images. The generator takes as input a latent variable of 100 dimensions.

Section 3.4 shows that the scattering transform can be used to reconstruct images. We thus recover images generated from our model from the generated scattering coefficients, and they are shown in Figure 8. These images are qualitatively similar to other baselines, and it shows how one can use the scattering transform with more complex models. Generating coherent Scattering coefficients that leads to real images is challenging: the non-surjectivity of the scattering transform is due to physical constraints(e.g. interactions between different coefficients), yet we however did not incorporate this knowledge in our architectures.

Learning Scattering

Many theoretical arguments of deep learning rely on the universal approximation theorem . The flexibility of this deep learning frameworks raises the following question: can we approximate the first scattering layers by a deep network?

In order to explore this question, we consider a 5-layer convnet as a candidate to replace our scattering network on CIFAR10. Its architecture is described in Table XI, and it has the same output size as a scattering network. It has two downsampling steps, in order to mimic the behavior of a scattering network with $J=2$. We build a hybrid architecture, i.e. scattering followed by a Cascaded CNN, described in Table XI that leads to 91.4% on CIFAR10. Then we replace the scattering part by the CNN of Table XI, i.e. the Scattering Approximator. We train it, keeping the weights of the Cascaded CNN layers constant and equal to the optimal solution found with the scattering. Instead of minimizing a loss between the output of a scattering network and this network, we target the best input for the fixed convnet given the classification task.

This architecture can achieve $1\%$ accuracy below the original pipeline, which indicates it is possible to learn the Scattering representation. Using a shallower network seems to degrade the performances, but we did not investigate this question further. In any case, the learned network will not have any guarantee of stability properties present in the original scattering transform.

Conclusion

This work demonstrates a competitive approach for large scale visual tasks, based on scattering networks, in particular for ILSVRC2012. When compared with unsupervised representations on CIFAR-10 or small data regimes on CIFAR-10 and STL-10, we demonstrate state-of-the-art results. We build a supervised Shared Local Encoder (SLE) that permits the scattering networks to surpass other local encoding methods on ILSVRC2012. This network of just 3 learned layers permits a deteailed analysis of the performed operations. We additionally prove that it is possible to synthetize images from a GAN in the Scattering space.

Our work also suggests that pre-defined features are still of interest and can provide valuable insights into deep learning techniques and to allow them to be more interpretable. Combined with appropriate learning methods, they enable stronger theoretical guarantees, which are necessary to engineer better deep models and stable representations.

Acknowledgment

The authors would like to thank Mathieu Andreux, Tomás Angles, Joan Bruna, Carmine Cella, Bogdan Cirstea, Michael Eickenberg, Stéphane Mallat, Louis Thiry for helpful discussions and support. The authors would also like to thank Rafael Marini and Nikos Paragios for use of computing resources. We would like to thank Florent Perronnin for providing important details of their work. This work is funded by the ERC grant InvariantClass 320959, via a grant for PhD Students of the Conseil régional d’Ile-de-France (RDM-IdF), Internal Funds KU Leuven, FP7-MC-CIG 334380, DIGITEO 2013-0788D - SOPRANO, NSERC Discovery Grant RGPIN-2017-06936, an Amazon Research Award to Matthew Blaschko, and by the Research Foundation - Flanders (FWO) through project number G0A2716N. We thank also the CVN (CentraleSupelec) for providing financial support.

References