Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows

Introduction

In industrial manufacturing processes the quality of the products is constantly monitored and improved. Hence, small defects during fabrication need to be detected reliably. However, manufacturers do not know in advance which types of defects will occur and most of them appear so infrequently that no defective examples are available. Even if some defect types are known, new types can still occur any time due to unforeseeable events during manufacturing. Consequently, reliable defect detection cannot be done with supervised machine learning approaches. We propose a solution for semi-supervised defect detection where only positive examples and no defective examples are present during training. This is also known as anomaly detection.

In general, anomaly detection describes the problem of determining whether a data sample differs from a set of given normal data. There are various approaches for general anomaly detection on images, summarized in Section 2. Defect detection is a specific sub-problem where visually similar normal samples and only slightly different anomalous samples are present. While traditional anomaly detection methods are well-suited to data with high intra-class variance, they are not able to capture subtle differences. We tackle this problem by employing an accurate density estimator on image features extracted by a convolutional neural network. The feature distribution of normal samples is captured by utilizing the latent space of a normalizing flow . Unlike other generative models such as variational autoencoders or GANs , there exists a bijective mapping between feature space and latent space in which each vector is assigned to a likelihood. This enables DifferNet to calculate a likelihood for each image. From this likelihood we derive a scoring function to decide if an image contains an anomaly. Figure 1 visualizes the core idea behind our method. The most common samples are assigned to a high likelihood whereas uncommon images are assigned to a lower likelihood. Since defects are not present during training, they are mapped to a low likelihood. We further improve the descriptiveness of the feature extractor by using multi-scale inputs. To derive a meaningful scoring function, we include likelihoods of several transformations of the image. Thereby DifferNet gains flexibility and robustness to detect various types of defects. We show that our method considers even small changes in data while other approaches struggle to detect them. Moreover, due to the efficiency of the feature extractor and the proposed image transformations, our approach even outperforms the state of the art when trained with a low number of training samples. Besides defect detection, DifferNet’s architecture allows for localization of defects by having expressive gradients of input pixels regarding the scoring functions. A high magnitude of the gradient in an image region signals an area with anomalous features which helps to identify the defect.

Our work comprises the following contributions:

Detection of anomalies via the usage of likelihoods provided by a normalizing flow on multi-scale image features with multi-transform evaluation.

Anomaly localization without training labels, the necessity of any pixel-wise optimization and sub-image detection.

Applicability on small training sets. Even with a low number of training examples our approach achieves competitive results.

State-of-the-art detection performance on MVTec AD and Magnetic Tile Defects.

Code is available on GitHub https://github.com/marco-rudolph/differnet.

Related Work

Existing methods for anomaly detection can be roughly divided into approaches based on generative models and pretrained networks. The most relevant methods to our work are briefly presented in the following subsections. Note that we focus on works that deal with image anomaly detection rather than anomaly localization to keep the focus on our main problem.

There are several works which use the feature space of a pretrained network to detect anomalies. In most cases simple traditional machine learning approaches are used to obtain an anomaly score. Andrews et al. apply a One-Class-SVM on VGG features of the training images. Nazare et al. evaluated different normalization techniques to have a 1-Nearest-Neighbor-classifier on a PCA-reduced representation of the feature space. Localization of the anomalies is achieved by evaluating the method on many overlapping patches. However, this is very costly. Sabokrou et al. models the anomaly-free feature distribution as an unimodal Gaussian distribution. Therefore, they cannot capture multimodal distributions as opposed to our method.

These techniques only work for particular classes in defect detection. In contrast to our proposed DifferNet, existing approaches do not appear to be robust and powerful enough for defect detection. None of the above techniques take advantage of the flexibility of another neural network on top of the pretrained model. Another benefit of our method compared to these approaches is being able to compute gradients w.r.t. the inputs which can be utilized to compute anomaly maps.

1.2 Generative Models

Generative models, such as autoencoders and GANs , are able to generate samples from the manifold of the training data. Anomaly detection approaches using these models are based on the idea that the anomalies cannot be generated since they do not exist in the training set.

Autoencoder-based approaches try to detect anomalies by comparing the output of an autoencoder to its input. Thus, a high reconstruction error should indicate an anomalous region. Bergmann et al. proposes SSIM-loss to make the reconstruction error dependent on visual similarity. In many cases autoencoder-based methods fail because they generalize too strongly, i.e. anomalies can be reconstructed as good as normal samples. Gong et al. tackle the generalization problem by employing memory modules which can be seen as a discretized latent space. Zhai et al. connect regularized autoencoders with energy-based models to model the data distribution and classify samples with high energy as an anomaly.

GAN-based approaches assume that only positive samples can be generated. Schlegl et al. propose a two-stage training method: The GAN is learned first and an encoder is optimized as an inverse generator. Using the generator as decoder enables the calculation of a reconstruction loss alongside the difference in discriminator features of original and reconstructed image to obtain an anomaly score. Akcay et al. make use of adversarial training by letting an autoencoder directly act as generating part of the GAN. This enforces the property of the decoder to only generate normal-like samples which can be measured by the difference between the embedding of the original and the reconstructed data.

We argue that generative models are appropriate for a wide range of defect detection scenarios since they strongly depend on the anomaly type. For example, the size and structure of the defective area heavily influence the anomaly score. If the region of interest shows high frequency structures, they cannot be represented accurately. Often, other instance-specific structures influence the reconstruction error more than the anomaly. In contrast, we show that DifferNet handles significantly more and various defect types. Additionally, our method does not rely on a large number of training samples compared to generative models.

2 Normalizing Flows

Normalizing Flows (NF) are neural networks that are able to learn transformations between data distributions and well-defined densities. Their special property is that their mapping is bijective and they can be evaluated in both directions. First, they assign likelihoods to a given sample. Second, data is generated by sampling from the modeled distribution. The bijectivity is ensured by stacking layers of affine transforms which are fixed or autoregressive. A common class of such autoregressive flows is MADE (Germain et al. ) which makes use of the Bayesian chain rule to decompose the density. These models can learn distributions of large datasets containing of mostly small images. In contrast, we capture the distribution of a comparably small number of images at a high resolution. Autoregressive flows compute likelihoods fast, but are slow at sampling. Inverse autoregressive flows, proposed by Kingma et al. , show the exact opposite behavior. Real-NVP can be seen as a special inverse autoregressive flow which is simplified such that both forward and backward pass can be processed quickly. Similar to Ardizzone et al. proposed for Invertible Neural Networks , we integrate a parametrized clamping mechanism to the affine transformation of the Real-NVP-layers to obtain a more stable training process. Details to the so-called coupling layers and the clamping mechanism of Real-NVP are explained in Section 3.1.

The property of normalizing flows as an adequate estimator of probability densities to detect anomalies has not raised much attention yet, although some works present promising results using Real-NVP and MADE . However, none of the works deal with visual data.

Method

The normalizing flow acts as a bijector between feature space $Y$ and latent space $Z$ by using affine transformations. Likelihoods are computed from $Z$ according to the modelled distribution. We model $z\sim\mathcal{N}(0,\,I)$ in our latent space which gives us a well-defined density $p_{Z}(z)$ for $z$. Following from bijectivity, for every feature input there is a unique $z$ of the same dimension and vice versa.

with $\odot$ as the element-wise product. Using an exponential function before scaling preserves the affinity property by ensuring non-zero coefficients. The internal functions $s$ and $t$ can be any differentiable function, which in our case is implemented as a fully-connected network that regresses both components by splitting the output (see Figure 3). Similar to Ardizzone et al. , we apply soft-clamping to the values of $s$ to preserve model stability which is crucial in our case for better convergence. This is achieved by using the activation

as the last layer of $s$. This prevents large scaling components by restricting $s(y)$ to the interval $(-\alpha,\alpha)$.

1.2 Training

This is equivalent to maximizing the log-likelihood, which is more convenient here, since the terms simplify when inserting the density function of a standard normal distribution as $p_{Z}(z)$. We use the negative log-likelihood loss $\mathcal{L}(y)$ to obtain a minimization problem:

During Training, $\mathcal{L}(y)$ is optimized for features $y$ of different transformations of an input image for a fixed epoch length. Section 4.2 describes the training in more detail.

2 Scoring Function

We use the calculated likelihoods as a criterion to classify a sample as anomalous or normal. To get a robust anomaly score $\tau(x)$, we average the negative log-likelihoods using multiple transformations $T_{i}(x)\in\mathcal{T}$ of an image $x$:

As $\mathcal{T}$ we choose rotations and manipulations of brightness and contrast. An image is classified as anomalous if the anomaly score $\tau(x)$ is above the threshold value $\theta$. Thus, the decision can be expressed as

where $\mathcal{A}(x)=1$ indicates an anomaly. In Section 4 $\theta$ is varied to calculate the Receiver Operating Characteristic (ROC).

3 Localization

In contrast to several other approaches, ours is not optimized for localizing the defects on the image. Nevertheless, our method localizes areas where anomalous features occur. Our method allows for propagating the negative log-likelihood $\mathcal{L}$ back to the input image $x$. The gradient $\nabla x_{c}$ of each input channel $x_{c}$ is a value indicating how much the pixels influence the error which relates to an anomaly. For better visibility we blur these gradients with a Gaussian kernel $G$ and sum the absolute values over the channels $C$ according to

with $*$ as 2D convolution and $|\cdot|$ as the element-wise absolute value, which results in the gradient map $g_{x}$. Averaging the maps of multiple rotations of one single image – after rotating back the obtained maps – gives a robust localization.

Experiments

In this paper, we evaluate our approach on real-world defect detection problems. We use the challenging and newly proposed datasets MVTec AD and Magnetic Tile Defects (MTD) . The difficulty in these datasets lies in the similarity of anomalies and normal examples.

To the best of our knowledge, MVTec AD is the only publicly available multi-object and multi-defect anomaly dataset. It contains 5354 high-resolution color images of 10 object and 5 texture categories. The number of training samples per category ranges from 60 to 320, which is challenging for the estimation of the distribution of normal samples. Several defect types per category, such as little cracks, deformations, discolorizations and scratches are occurring in the test set. Some of which are shown in Figure 8. In total, the dataset includes 70 defect types. The anomalies differ in their size, shape and structure and thus cover several scenarios in industrial defect detection.

Magnetic Tile Defects comprises grayscale images of magnetic tiles under different illuminations with and without defects. Such tiles should provide a constant magnetic potential in engines. We split the 952 defect-free images randomly into a train and a test set, where the test set contains 20% of the data. All 392 defect images are used for testing. These show frayed or uneven areas, cracks, breaks and blowholes as anomalies, as shown in Figure 4. A lot of defect-free images contain variations that are similar to anomalies.

2 Implementation Details

For all experiments, we use the convolutional part of AlexNet as the feature extractor and apply global average pooling on each feature map at each scale. We tested more complex topologies, for instance ResNet and VGG , but did not observe better performance. We prefer the smaller AlexNet since it is sufficient for our purpose. The feature extractor is pretrained on ImageNET and remains fixed during training. We use features at $3$ scales with input image sizes of $448\times 448$, $224\times 224$ and $112\times 112$ pixels - resulting in $3\cdot 256=768$ features. The normalizing flow consists of 8 coupling blocks with fully connected networks as internal functions $s$ and $t$. These include 3 hidden dense layers with a size of 2048 neurons and ReLU activations. We set the mentioned clamping parameter $\alpha=3$. For training, we use the Adam Optimizer with the author-suggested $\beta$- parameters and a learning rate of $2\cdot 10^{-4}$. As transformations $\mathcal{T}$, random rotations, which are uniformly distributed in the interval $[0,2\pi]$, are applied. In addition, we manipulated the contrast and brightness of the magnetic tiles with an uniformly distributed random factor in the interval $[0.85,1.15]$. In each case of training and inference the same transformations are applied. We train our models for 192 epochs with a batch size of 96.

3 Detection

For reporting the performance of our method regarding the detection of anomalies, we follow and compute the Area Under Receiver Operator Characteristics (AUROC) depending on the scoring function which we obtained as described in Section 3. It measures the area under the true positive rate as a function of the false positive rate. The AUROC metric is not sensitive to any threshold or the percentage of anomalies in the test set. Besides other anomaly detection methods, we compare our method to the baselines one-class SVM (OCSVM) and the distance to the nearest neighbor (1-NN) after PCA reduction to 64 dimensions and z-score normalization . Note that both methods OCSVM and 1-NN are adapted to our setting: We used every technique of our pipeline (see Figure 2) but replaced the normalizing flow with them. We evaluate several transformations and used the mean respective score. Apart from these approaches and some state-of-the-art models, we compare our method with GeoTrans which cannot be assigned to generative and pretrained methods described in Section 2. GeoTrans computes an anomaly score based on the classification of conducted transformations. Table 1 shows the results for MVTec AD. Compared to other approaches, our method outperforms existing methods in almost every category, up to a large margin of 15%. In all of the 15 categories our method achieves an AUROC of at minimum $84\%$, which shows that our approach is not limited to a specific set of defects or features. The fact that 1-NN outperforms other competitors except us, demonstrates that our feature extraction and evaluation is well-suited for the problem.

We can observe similar characteristics on MTD, seen in Table 2. The ROC-Curve in Figure 5 shows that our method provides a much higher true positive rate for any false positive rate. DifferNet achieves a recall of about 50% without any false positive among 191 defect-free test images. The histogram of anomaly scores is visualized in Figure 6. There is a large subset of defective samples whose scores differ significantly from all scores of non-defective samples. The assignment of extremely high scores without any false positive is a characteristic of our method and can be similarly observed for other evaluated product categories.

4 Localization

Results of the localization procedure described in Section 3.3 are shown in Figure 8. The localizations are accurate for many types, sizes and shapes of anomalies; despite the average pooling of feature maps before being processed by the normalizing flow. Our architecture produces meaningful gradients which can be explained by the models architecture: First, AlexNet is relatively shallow such that noisy or vanishing gradients are prevented. Second, the bijectivity of the normalizing flow causes a direct relation between all image features $y$ and all values of $z$ with non-zero gradients. The gradients tend to appear speckled for larger anomalous regions. We conject the reason is that pixels, leading to features influencing the anomaly score, are usually not located evenly distributed in the corresponding region. However, our method enables the human to perceive the defective region and interpret which areas influenced the networks decision to what extent.

5 Ablation Studies

To quantify the effects of individual strategies used in our work, we performed an ablation study by comparing the performance on MVTec AD when modifying the strategies. In addition, the model’s behavior for different characteristics of the training set is analyzed. Preprocessing Pipeline and Evaluation. Table 3 compares the detection performance on MVTec AD for different configurations regarding multi-scaling, the usage of transformations in training and the number of used transformations $T_{i}$ for evaluation. Having one test transformation means that only the original image was used for evaluation. Note that we outperform existing methods even without the proposed transformations and multi-scale strategy. Since relevant features could appear at any scale, it is beneficial to include features at multiple scales which is shown by an AUROC improvement of $4.7\%$. Having transformed samples in training is crucial as it enables multi-transform evaluation and helps for generalization and data augmentation. The similar performances of configuration C and D reflect that applying transformations in training is only useful if they are performed in inference as well. The more of these transformations are then used, the more meaningful the score is, as the rising performance of configurations D to G shows. Number of Training Samples. We investigate the effect of the training set size on the detection performance as shown on Figure 7 and on the right of Table 1. The reported results are the average over three runs with different random subsets per training set size. It can be seen that our model and training procedure allows for a stable training even on small training sets. This can be explained by the usage of multiple transformations and the averaging of feature maps. Our model profits from this strategy which is a mixture of augmentation and compression. DifferNet requires only 16 training samples to outperform existing approaches that use the whole training set with 369 samples per class on average. For some classes, the subsets cannot represent the feature variation of normal samples. Multimodality. The feature distributions of the evaluated categories are unimodal. We also investigated the performance on multimodal distributions. Therefore, we also observed the detection performance when using all 15 categories of MVTec as training data. To capture this more complex distribution, we used 12 coupling blocks. The result is a mean AUROC of 90.2% which shows that our method is able to handle multimodal distributions well. The regressing sub-blocks inside the NF appear to capture the modes and switch between them depending on their input.

Conclusion

We presented DifferNet to detect defects in images by utilizing a normalizing-flow-based density estimation of image features at multiple scales. Likelihoods of several transformations of a single image are used to compute a robust anomaly score. Therefore, there is no need for a large amount of training samples. The design and scoring function is chosen such that image gradients can be exploited to localize defects. As shown, the method also scales to multimodal distributions which resembles real-world settings. In the future we plan to refine the concept in order to find anomalies in video data comparable to .

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122).

References