The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Motivation

The ability to compare data items is perhaps the most fundamental operation underlying all of computing. In many areas of computer science it does not pose much difficulty: one can use Hamming distance to compare binary patterns, edit distance to compare text files, Euclidean distance to compare vectors, etc. The unique challenge of computer vision is that even this seemingly simple task of comparing visual patterns remains a wide-open problem. Not only are visual patterns very high-dimensional and highly correlated, but, the very notion of visual similarity is often subjective, aiming to mimic human visual perception. For instance, in image compression, the goal is for the compressed image to be indistinguishable from the original by a human observer, irrespective of the fact that their pixel representations might be very different.

What we would really like is a “perceptual distance,” which measures how similar are two images in a way that coincides with human judgment. This problem has been a longstanding goal, and there have been numerous perceptually motivated distance metrics proposed, such as SSIM , MSSIM , FSIM , and HDR-VDP .

However, constructing a perceptual metric is challenging, because human judgments of similarity (1) depend on high-order image structure , (2) are context-dependent , and (3) may not actually constitute a distance metric . The crux of (2) is that there are many different “senses of similarity” that we can simultaneously hold in mind: is a red circle more similar to a red square or to a blue circle? Directly fitting a function to human judgments may be intractable due the the context-dependent and pairwise nature of the judgments (which compare the similarity between two images). Indeed, we show in this paper a negative result where this approach fails to generalize, even when trained on a large-scale dataset containing many distortion types.

Instead, might there be a way to learn a notion of perceptual similarity without directly training for it? The computer vision community has discovered that internal activations of deep convolutional networks, though trained on a high-level image classification task, are often surprisingly useful as a representational space for a much wider variety of tasks. For example, features from the VGG architecture have been used on tasks such as neural style transfer , image superresolution , and conditional image synthesis . These methods measure distance in VGG feature space as a “perceptual loss” for image regression problems .

But how well do these “perceptual losses” actually correspond to human visual perception? How do they compare to traditional perceptual image evaluation metrics? Does the network architecture matter? Does it have to be trained on the ImageNet classification task, or would other tasks work just as well? Do the networks need to be trained at all?

In this paper, we evaluate these questions on a new large-scale database of human judgments, and arrive at several surprising conclusions. We find that internal activations of networks trained for high-level classification tasks, even across network architectures and no further calibration, do indeed correspond to human perceptual judgments. In fact, they correspond far better than the commonly used metrics like SSIM and FSIM , which were not designed to handle situations where spatial ambiguities are a factor . Furthermore, the best performing self-supervised networks, including BiGANs , cross-channel prediction , and puzzle solving perform just as well at this task, even without the benefit of human-labeled training data. Even a simple unsupervised network initialization with stacked k-means beats the classic metrics by a large margin! This illustrates an emergent property shared across networks, even across architectures and training signals. Importantly, however, having some training signal appears crucial – a randomly initialized network achieves much lower performance.

Our study is based on a newly collected perceptual similarity dataset, using a large set of distortions and real algorithm outputs. It contains both traditional distortions, such as contrast and saturation adjustments, noise patterns, filtering, and spatial warping operations, and CNN-based algorithm outputs, such as autoencoding, denoising, and colorization, produced by a variety of architectures and losses. Our dataset is richer and more varied than previous datasets of this kind . We also collect judgments on outputs from real algorithms for the tasks of superresolution, frame interpolation, and image deblurring, which is especially important as these are the real-world use cases for a perceptual metric. We show that our data can be used to “calibrate” existing networks, by learning a simple linear scaling of layer activations, to better match low-level human judgments.

Our results are consistent with the hypothesis that perceptual similarity is not a special function all of its own, but rather a consequence of visual representations tuned to be predictive about important structure in the world. Representations that are effective at semantic prediction tasks are also representations in which Euclidean distance is highly predictive of perceptual similarity judgments.

We introduce a large-scale, highly varied, perceptual similarity dataset, containing 484k human judgments. Our dataset not only includes parameterized distortions, but also real algorithm outputs. We also collect judgments on a different perceptual test, just noticeable differences (JND).

We show that deep features, trained on supervised, self-supervised, and unsupervised objectives alike, model low-level perceptual similarity surprisingly well, outperforming previous, widely-used metrics.

We demonstrate that network architecture alone does not account for the performance: untrained nets achieve much lower performance.

With our data, we can improve performance by “calibrating” feature responses from a pre-trained network.

In order to evaluate existing similarity measures, a number of datasets have been proposed. Some of the most popular are the LIVE , TID2008 , CSIQ , and TID2013 datasets. These datasets are referred to Full-Reference Image Quality Assessment (FR-IQA) datasets and have served as the de-facto baselines for development and evaluation of similarity metrics. A related line of work is on No-Reference Image Quality Assessment (NR-IQA), such as AVA and LIVE In the Wild . These datasets investigate the “quality” of individual images by themselves, without a reference image. We collect a new dataset that is complementary to these: it contains a substantially larger number of distortions, including some from newer, deep network based outputs, as well as geometric distortions. Our dataset is focused on perceptual similarity, rather than quality assessment. Additionally, it is collected on patches as opposed to full images, in the wild, with a different experimental design (more details in Sec 2).

Prior work on deep networks and human judgments

Recently, advances in DNNs have motivated investigation of applications in the context of visual similarity and image quality assessment. Kim and Lee use a CNN to predict visual similarity by training on low-level differences. Concurrent work by Talebi and Milanfar train a deep network in the context of NR-IQA for image aesthetics. Gao et al. and Amirshahi et al. propose techniques involving leveraging internal activations of deep networks (VGG and AlexNet, respectively) along with additional multiscale post-processing. In this work, we conduct a more in-depth study across different architectures, training signals, on a new, large scale, highly-varied dataset.

Recently, Berardino et al. train networks on perceptual similarity, and importantly, assess the ability of deep networks to make predictions on a separate task – predicting most and least perceptually-noticeable directions of distortion. Similarly, we not only assess image patch similarity on parameterized distortions, but also test generalization to real algorithms, as well as generalization to a separate perceptual task – just noticeable differences.

Berkeley-Adobe Perceptual Patch Similarity (BAPPS) Dataset

To evaluate the performance of different perceptual metrics, we collect a large-scale highly diverse dataset of perceptual judgments using two approaches. Our main data collection employs a two alternative forced choice (2AFC) test, that asks which of two distortions is more similar to a reference. This is validated by a second experiment where we perform a just noticeable difference (JND) test, which asks whether two patches – one reference and one distorted – are the same or different. These judgments are collected over a wide space of distortions and real algorithm outputs.

We create a set of “traditional” distortions consisting of common operations performed on the input patches, listed in Table 2 (left). In general, we use photometric distortions, random noise, blurring, spatial shifts and corruptions, and compression artifacts. We show qualitative examples of our traditional distortions in Figure 2. The severity of each perturbation is parameterized - for example, for Gaussian blur, the kernel width determines the amount of corruption applied to the input image. We also compose pairs of distortions sequentially to increase the overall space of possible distortions. In total, we have 20 distortions and 308 sequentially composed distortions.

CNN-based distortions

To more closely simulate the space of artifacts that can arise from deep-learning based methods, we create a set of distortions created by neural networks. We simulate possible algorithm outputs by exploring a variety of tasks, architectures, and losses, as shown in Table 2 (right). Such tasks include autoencoding, denoising, colorization, and superresolution. All of these tasks can be achieved by applying the appropriate corruption to the input. In total, we generated 96 “denoising autoencoders” and use these as CNN-based distortion functions. We train each of these networks on the 1.3M ImageNet dataset for 1 epoch. The goal of each network is not to solve the task per se, but rather to explore common artifacts that plague the outputs of deep learning based methods.

Distorted image patches from real algorithms

The true test of an image assessment algorithm is on real problems and real algorithms. We gather perceptual judgments using such outputs. Data on real algorithms is more limited, as each application will have their own unique properties. For example, different colorization methods will not show much structural variation, but will be prone to effects such as color bleeding and color variation. On the other hand, superresolution will not have color ambiguity, but may see larger structural changes from algorithm to algorithm.

Superresolution

We evaluate results from the 2017 NTIRE workshop . We use 3 tracks from the workshop – $\times 2$, $\times 3$, $\times 4$ upsampling rates using “unknown” downsampling to create the input images. Each track had approximately 20 algorithm submissions. We also evaluate several additional methods, including bicubic upsampling, and four of the top performing deep superresolution methods . A common qualitative way of presenting superresolution results is zooming into specific patches and comparing differences. As such, we sample random $64\times 64$ triplets from random locations of images in the Div2K dataset – the ground truth high-resolution image, along with two algorithm outputs.

Frame interpolation

We sample patches from different frame interpolation algorithms, including three variants of flow-based interpolation , CNN-based interpolation , and phase-based interpolation on the Davis Middleburry dataset . Because artifacts arising from frame interpolation may occur at different scales, we randomly rescale the image before sampling a patch triplet.

Video deblurring

We sample from the video deblurring dataset , along with deblurring outputs from Photoshop Shake Reduction, Weighted Fourier Aggregation , and three variants of a deep video deblurring method .

Colorization

We sample patches using random scales on the colorization task, on images from the ImageNet dataset . The algorithms are from pix2pix , Larsson et al. , and variants from Zhang et al. .

2 Psychophysical Similarity Measurements

We randomly select an image patch $x$ and apply two distortions to produce patches $x_{0},x_{1}$. We then ask a human which is closer to the original patch $x$, and record response $h\in\{0,1\}$. On average, people spent approximately 3 seconds per judgment. Let $\mathcal{T}$ denote our dataset of patch triplets $(x,x_{0},x_{1},h)$.

A comparison between our dataset and previous datasets is shown in Table 1. Previous datasets have focused on collecting large numbers of human judgments for a few images and distortion types. For example, the largest dataset, TID2013 , has 500k judgments on 3000 distortions (from 25 input images with 24 distortions types, each sampled at 5 levels). We provide a complementary dataset that focuses instead on a large number of distortions types. In, addition, we collect judgments on a large number of $64\times 64$ patches rather than a small number of images. There are three reasons for this. First, the space of full images is extremely large, which makes it much harder to cover a reasonable portion of the domain with judgments (even $64\times 64$ color patches represent an intractable 12k-dimensional space). Second, by choosing a smaller patch size, we focus on lower-level aspects of similarity, to mitigate the effect of differing “respects of similarity” that may be influenced by high-level semantics . Finally, modern methods for image synthesis train deep networks with patch-based losses (implemented as convolutions) . Our dataset consists of over 161k patches, derived from the MIT-Adobe 5k dataset (5000 uncompressed images) for training, and the RAISE1k dataset for validation.

To enable large-scale collection, our data is collected “in-the-wild” on Amazon Mechanical Turk, as opposed to a controlled lab setting. Crump et al. show that AMT can be reliably used to replicate many psychophysics studies, despite the inability to control all environmental factors. We ask for 2 judgments per example in our “train” set and 5 judgments in our “val” sets. Asking for fewer judgments enables us to explore a larger set of image patches and distortions. We add sentinels which consist of pairs of patches with obvious deformations, e.g., a large amount of Gaussian noise vs a small amount of Gaussian noise. Approximately $~{}90\%$ of Turkers were able to correctly pass at least $93\%$ of the sentinels (14 of 15), indicating that they understood the task and were paying attention. We choose to use a larger number of distortions than prior datasets.

Just noticeable differences (JND)

A potential shortcoming of the 2AFC task is that it is “cognitively penetrable,” in the sense that participants can consciously choose which respects of similarity they will choose to focus on in completing the task , which introduces subjectivity into the judgments. To validate that the judgments actually reflected something objective and meaningful, we also collected user judgments of “just noticeable differences” (JNDs). We show a reference image, followed by a randomly distorted image, and ask a human if the images are the same or different. The two image patches are shown for 1 second each, with a 250 ms gap in between. Two images which look similar may be easily confused, and a good perceptual metric will be able to order pairs from most to least confusable. JND tests like this may be considered less subjective, since there is a single correct answer for each judgment, and participants are presumed to be aware of what correct behavior entails. We gather 3 JND observations for each of the 4.8k patches in our traditional and CNN-based validation sets. Each subject is shown 160 pairs, along with 40 sentinels (32 identical and 8 with large Gaussian noise distortion applied). We also provide a short training period of 10 pairs which contain 4 “same” pairs, 1 obviously different pair, and 5 “different” pairs generated by our distortions. We chose to do this in order to prime the users towards expecting approximately $40\%$ of the patch pairs to be identical. Indeed, $36.4\%$ of the pairs were marked “same” ($70.4\%$ of sentinels and $27.9\%$ of test pairs).

Deep Feature Spaces

We evaluate feature distances in different networks. For a given convolutional layer, we compute cosine distance (in the channel dimension) and average across spatial dimensions and layers of the network. We also discuss how to tune an existing network on our data.

We evaluate the SqueezeNet , AlexNet , and VGG architectures. We use 5 conv layers from the VGG network, which has become the de facto standard for image generation tasks . We also compare against the shallower AlexNet network, which may more closely match the architecture of the human visual cortex . We use the conv1-conv5 layers from . Finally, the SqueezeNet architecture was designed to be extremely lightweight ($2.8$ MB) in size, with similar classification performance to AlexNet. We use the first conv layer and some subsequent “fire” modules.

We additionally evaluate self-supervised methods, including puzzle-solving , cross-channel prediction , learning from video , and generative modeling . We use publicly available networks from these and other methods, which use variants of AlexNet .

Network activations to distance

Training on our data

We consider a few variants for training with our perceptual judgments: lin, tune, and scratch. For the lin configuration, we keep pre-trained network weights $\mathcal{F}$ fixed, and learn linear weights $w$ on top. This constitutes a “perceptual calibration” of a few parameters in an existing feature space. For example, for the VGG network, 1472 parameters are learned. For the tune configuration, we initialize from a pre-trained classification model, and allow all the weights for network $\mathcal{F}$ to be fine-tuned. Finally, for scratch, we initialize the network from random Gaussian weights and train it entirely on our judgments. Overall, we refer to these as variants of our proposed Learned Perceptual Image Patch Similarity (LPIPS) metric. We illustrate the training loss function in Figure 3 (right) and describe it further in the appendix.

Experiments

Results on our validation sets are shown in Figure 4. We first evaluate how well our metrics and networks work. All validation sets contain 5 pairwise judgments for each triplet. Because this is an inherently noisy process, we compute agreement of an algorithm with all of the judgments. For example, if there are 4 preferences for $x_{0}$ and 1 for $x_{1}$, an algorithm which predicts the more popular choice $x_{0}$ would receive $80\%$ credit. If a given example is scored with fraction $p$ humans in one direction and $1-p$ in the other, a human would achieve score $p^{2}+(1-p)^{2}$ on expectation.

Does the network have to be trained on classification?

In Figure 4, we show model performance across a variety of unsupervised and self-supervised tasks, shown in green – generative modeling with BiGANs , solving puzzles , cross-channel prediction , and segmenting foreground objects from video . These self-supervised tasks perform on par with classification networks. This indicates that tasks across a large spectrum can induce representations which transfer well to perceptual distances. Also, the performance of the stacked k-means method , shown in yellow, outperforms low-level metrics. Random networks, shown in orange, with weights drawn from a Gaussian, do not yield much improvement. This indicates that the combination of network structure, along with orienting filters in directions where data is more dense, can better correlate to perceptual judgments.

In Table 5, we explore how well our perceptual task correlates to semantic tasks on the PASCAL dataset , using results summarized in , including additional self-supervised methods . We compute the correlation coefficient between each task (perceptual or semantic) across different methods. The correlation from our 2AFC distortion preference task to classification and detection is .640 and .363, respectively. Interestingly, this is similar to the correlation between the classification and detection tasks (.429), even though both are considered “high-level” semantic tasks, and our perceptual task is “low-level.”

Do metrics correlate across different perceptual tasks?

We test if training for the 2AFC distortion preference test corresponds with another perceptual task, the JND test. We order patch pairs by ascending order by a given metric, and compute precision-recall on our CNN-based distortions – for a good metric, patches which are close together are more likely to be confused for being the same. We compute area under the curve, known as mAP . The 2AFC distortion preference test has high correlation to JND: $\rho=.928$ when averaging the results across distortion types. Figure 5 shows how different methods perform under each perceptual test. This indicates that 2AFC generalizes to another perceptual test and is giving us signal regarding human judgments.

Can we train a metric on traditional and CNN-based distortions?

In Figure 4, we show performance using our lin, scratch, and tune configurations, shown in purple, pink, and brown, respectively. When validating on the traditional and CNN-based distortions (Figure 4(a)), we see improvements. Allowing the network to tune all the way through (brown) achieves higher performance than simply learning linear weights (purple) or training from scratch (pink). The higher capacity network VGG also performs better than the lower capacity SqueezeNet and AlexNet architectures. These results verify that networks can indeed learn from perceptual judgments.

Does training on traditional and CNN-based distortions transfer to real-world scenarios?

We are more interested in how performance generalizes to real-world algorithms, shown in Figure 4(b). The SqueezeNet, AlexNet, and VGG architectures start at $64.0\%$, $65.0\%$, and $62.6\%$, respectively. Learning a linear classifier (purple) improves performance for all networks. Across the 3 networks and 4 real-algorithm tasks, 11 of the 12 scores improved, indicating that “calibrating” activations on a pre-existing representation using our data is a safe way to achieve a small boost in performance ($1.1\%$, $0.3\%$, and $1.5\%$, respectively). Training a network from scratch (pink) yields slightly lower performance for AlexNet, and slightly higher performance for VGG than linear calibration. However, these still outperform low-level metrics. This indicates that the distortions we have expressed do project onto our test-time tasks of judging real algorithms.

Interestingly, starting with a pre-trained network and tuning throughout lowers transfer performance. This is an interesting negative result, as training for a low-level perceptual task directly does not necessarily perform as well as transferring a representation trained for the high-level task.

Where do deep metrics and low-level metrics disagree?

In Figure 11, we show a qualitative comparison across our traditional distortions for a deep method, BiGANs , and a representation traditional perceptual method, SSIM . Pairs which BiGAN perceives to be far but SSIM to be close generally contain some blur. BiGAN tends to perceive correlated noise patterns to be a smaller distortion than SSIM.

Conclusions

Our results indicate that networks trained to solve challenging visual prediction and modeling tasks end up learning a representation of the world that correlates well with perceptual judgments. A similar story has recently emerged in the representation learning literature: networks trained on self-supervised and unsupervised objectives end up learning a representation that is also effective at semantic tasks . Interestingly, recent findings in neuroscience make much the same point: representations trained on computer vision tasks also end up being effective models of neural activity in macaque visual cortex . Moreover (and roughly speaking), the stronger the representation is at the computer vision task, the stronger it is as a model of cortical activity. Our paper makes a similar finding: the stronger a feature set is at classification and detection, the stronger it is as a model of perceptual similarity judgments, as suggested in Table 4. Together, these results suggest that a good feature is a good feature. Features that are good at semantic tasks are also good at self-supervised and unsupervised tasks, and also provide good models of both human perceptual behavior and macaque neural activity. This last point aligns with the “rational analysis” explanation of visual cognition , suggesting that the idiosyncrasies of biological perception arise as a consequence of a rational agent attempting to solve natural tasks. Further refining the degree to which this is true is an important question for future research.

This research was supported, in part, by grants from Berkeley Deep Drive, NSF IIS-1633310, and hardware donations by NVIDIA. We thank members of the Berkeley AI Research Lab and Adobe Research for helpful discussions. We thank Alan Bovik for his insightful comments. We also thank Radu Timofte, Zhaowen Wang, Michael Waechter, Simon Niklaus, and Sergio Guadarrama for help preparing data. RZ is partially supported by an Adobe Research Fellowship and much of this work was done while RZ was an intern at Adobe Research.

References

Appendix

We show full quantitative details in Appendix A. We also discuss training details in Appendix B. Finally, we show results on the TID2013 dataset in Appendix C.

Appendix A Quantitative Results

In Table 5, we show full quantitative results across all validation sets and considered metrics, including low-level metrics, along with random, unsupervised, self-supervised, supervised, and perceptually-learned networks.

In Figures 7, 8, 9, we plot performance in individual validation sets. Figure 7 shows our traditional and CNN-based distortions, and Figures 8, 9 show results on real algorithm applications individually.

If humans chose patches {$x_{1}$,$x_{0}$} with fraction {$p$,$1-p$}, the theoretical maximum for an oracle is $\max(p,1-p)$. However, human performance is lower. If an agent chooses them with probability {$q$,$1-q$}, the agent will agree with $qp+(1-q)(1-p)$ humans on expectation. With a human agent, $q=p$, the expected human score is $p^{2}+(1-p)^{2}$.

Linearly calibrating networks

Learning linear weights on top of the Alex model achieves state-of-the-art results on the real algorithms test set. The linear models have a learned linear layer on top of each channel, whereas the out-of-the-box versions weight each channel equally. In Figure 10(b), we show the learned weights for the Alex –frozen model. The conv1-5 layers contain 64, 192, 384, 256, and 256 channels, respectively, for a total of 1152 weights. For each layer, conv1-5, $79.7\%$, $71.4\%$, $56.8\%$, $46.5\%$, $27.7\%$, respectively, of the weights are zero. This means that a majority of the conv1 and conv2 units are ignored, and almost all of the conv5 units are used. Overall, about half of the units are ignored. Taking the cosine distance is equivalent to setting all weights to 1 (Figure 10(a)).

Data quantity for training models on distortions

The performance of the validation set on our distortions ($80.6\%$ and $81.4\%$ for Alex – tune and VGG – tune, respectively), is almost equal to human performance of $82.6\%$. This indicates that our training set size of 150k patch pairs and 300k judgments is nearly large enough to fully explore the traditional and CNN-based distortions which we defined. However, there is a small gap between the tune and scratch models ($0.4\%$ and $0.6\%$ for Alex and VGG, respectively).

Appendix B Model Training Details

We illustrate the loss function for training the network in Figure 3 (right) and describe it further in the supplementary material. Given two distances, $(d_{0},d_{1})$, we train a small network $\mathcal{G}$ on top to map to a score $\hat{h}\in(0,1)$. The architecture uses two 32-channel FC-ReLU layers, followed by a 1-channel FC layer and a sigmoid. Our final loss function is shown in Equation 2.

In preliminary experiments, we also tried a ranking loss, which attempts to force a constant margin between patch pairs $d(x,x_{0})$ and $d(x,x_{1})$. We found that using a learned network, rather than enforcing the same margin in all cases, worked better.

Here, we provide some additional details on model training for our networks trained on distortions. We train with 5 epochs at initial learning rate $10^{-4}$, 5 epochs with linear decay, and batch size 50. Each training patch pair is judged 2 times, and the judgments are grouped together. If, for example, the two judges are split, then the classification target ($h$ in Figure 3) will be set at 0.5. We enforce non-negative weightings on the linear layer $w$, since larger distances in a certain feature should not result in two patches becoming closer in the distance metric. This is done by projecting the weights into the constraint set at every iteration. In other words, we check for any negative weights, and force them to be 0. The project was implemented using PyTorch .

Appendix C TID2013 Dataset

Appendix D Changelog

v2

CVPR camera ready; moved TID results (Appendix C), SSIM vs BiGAN (Figure 11), and some training details into the Appendix to fit into 8 page limit; clarified that SSIM was not designed to handle geometric distortions and clarified that our dataset is a perceptual similarity dataset (as opposed to an IQA dataset); added linear weights for Squeeze-lin and VGG-lin architectures in Figure 10; miscellaneous small edits to text.