A Self-Supervised Descriptor for Image Copy Detection

Introduction

All online photo sharing platforms use content moderation to block or limit the propagation of images that are considered harmuful: terrorist propaganda, misinformation, harassment, pornography, etc. Some content moderation can be performed automatically, for unambiguous data like pornographic pictures, but this is much harder for complex data like memes or misinformation . In these cases, content is moderated manually. For of viral images, where copies of same image may be uploaded thousands of times, manual moderation of each copy is tedious and unnecessary. Instead, each image for which a manual moderation decision is taken can be recorded in a database, so that it can be re-identified later and handled automatically.

This paper is concerned with this basic task of re-identifcation. This is non trivial because copied images are often altered, for technical reasons (e.g. a user shares a mobile phone screenshot that captures additional content), or users may make adversarial edits to evade moderation.

Image re-identification is an image matching problem, with two additional challenges. The first is the enormous scale at which copy detection systems are deployed. At this scale, the only feasible approach is to represent images as short descriptor vectors, that can be searched efficiently with approximate nearest neighbor search methods . Copy detection systems typically proceed in 2 stages: a retrieval stage that produces a shortlist of candidate matches and a verification stage, often based on local descriptor matching that operates on the candidates. In this work, we are concerned with the first stage. Figure 1 shows the overall architecture of our Self Supervised Copy Detection (SSCD) approach.

The second challenge is that there is a hard match/non-match decision to take, and positive image pairs are rare. We wish to limit verification candidates using a threshold, which is a harder constraint than the typical image retrieval setting, where only the order of results matter.

SSCD uses differential entropy regularization to promote a uniform embedding distribution, which has three effects: (1) it makes distances from different embedding regions more comparable; (2) it avoids the embedding collapse described in , making full use of the embedding space; (3) it also improves ranking metrics that do not require consistent thresholds across queries.

Score normalization is important for ranking systems. An advanced score normalization relies on matching the query images with a set of background images. In this work, we show how this normalization can be incorporated in the image descriptor itself. We anticipate that this work will set a strong single-model baseline for image copy detection. We plan to release code and models for our method.

Section 2 discusses works related to this paper. Section 3 motivates the use of an entropy loss term in a simplified setting. Section 4 carefully describes SSCD. Section 5 presents results and ablations of our method. Section 6 points out a few observations about the copy detection task.

Related work

Content tracing on a user-generated photo sharing platform aims at re-identifying images when they circulate out and back into the platform. There are three broad families of tracing methods: metadata-based , watermarking and content-based. This work belongs to this last class.

Classical image datasets for content tracing, like Casia focus on image alterations like splicing, removal and copy-move transformations that alter only a small fraction of the image surface, so the re-identification is done reliably with simple interest-point based techniques. The challenge is to detect the tampered surface, which is typically approached with deep models inspired by image segmentation . A related line of research is image phylogeny: the objective is to identify the series of edits that were applied to an image between an initial and a final state . The Nimble/Media forensics series of competitions organized by NIST aim at benchmarking these tasks . In this work we focus on the identification itself, with strong transformations and near duplicates that need to be distinguished (see Figure 2).

Semantic and perceptual image comparison

Several definitions of near-duplicate image matching, form a continuum between pixel-wise copy and instance matching . The definition we use in this work is: images are considered copies iff they come from the same 2D image source. More relaxed definitions allow, for example, to match nearby frames in a video.

There is a large body of literature about solving instance matching i.e., recognizing images of the same 3D object with viewpoint/camera changes. In this work, we build on this literature because it addresses complex image matching, and to our knowledge, recent works and benchmarks for strict copy detection are rare .

Instance matching.

Classical instance matching relies on 3D matching tools, like interest points . CNN-based approaches use backbones from image classification, either pre-trained or trained end-to-end , with two adaptations: (1) the pooling layer that converts the last CNN activation map to a vector is a max-pooling , or more generally GeM pooling , a form of $L_{p}$ normalization where $p$ is adapted to the image resolution ; (2) careful normalization of the vectors. In addition to simple L2-normalization , “whitening” is often used to compare descriptors . An additional normalization technique contrasts the distances w.r.t. a background distribution of images . In this work, we apply these pooling and normalization techniques to copy detection.

Contrastive self-supervised learning.

A recent line of self-supervised learning research uses contrastive objectives that learn image representations that bring transformed images together. These methods either discriminate image features or the cluster assignments of these image features . These methods either rely on memory banks or large batch sizes . In particular, SimCLR uses matching transformed image copies as a surrogate task to learn a general image representation that transfer well to other tasks, such as image classification. A contrastive InfoNCE loss is used to map copies of the same source image nearby in the embedding space.

Differential entropy regularization.

Increasing the entropy of media descriptors forces them to spread over the representation space. Sablayrolles et al. observed that the entropy can be estimated locally with the Kozachenko-Leononenko differential entropy estimator , that can be incorporated directly into the loss to maximize descriptor entropy. The work of El-Nouby et al. is closest to our approach. It adds the entropy term to a contrastive loss at fine-tuning time to improve the accuracy for category and instance retrieval. Our approach is similar, applied to a self-supervised objective and image copy detection.

Motivation

In this section, we start from the SimCLR method, then perform a simple experiment where we combine it with the entropy loss from and witness how it impacts classification and copy detection tasks.

2 Entropy regularization

We use the differential entropy loss proposed in , based on the Kozachenko-Leononenko estimator. We adapt it to the repeated augmentation setting by only regularizing neighbors from different source images:

where $\hat{P}_{i}=P_{i}\cup\{i\}$. Since this entropy loss is a log of the distance to the nearest neighbor, its impact is very high for nearby vectors but dampens quickly when the descriptors are far apart. The effect is to “push” apart nearby vectors.

3 Experiment: SimCLR and entropy

For this experiment, we combine our contrastive loss with the entropy loss, using a weighting factor $\lambda$, similar to :

We then evaluate the impact of the combined loss on an image classifcation setting and a copy detection setting, see Section 5.1 for more details about the setup.

Figure 3 shows how varying entropy loss weight $\lambda$ impacts both tasks. As the entropy loss weight increases, ImageNet linear classification accuracy decreases: this loss term is not helpful for classification. Conversely, for copy detection the accuracy increases significantly.

Figure 4 shows the distribution of distances between matching images (positive pairs) and the nearest non-matching neighbors (negative pairs). Applying the entropy loss increases all distances and makes the negative distance distribution more narrow. The result is that there is a larger contrast between positive pairs and the mode of the negative distribution, i.e. they are more clearly separated.

Method

Having seen how the entropy loss improves copy detection accuracy, in this section we expand it into a robust image copy detection approach: SSCD. This entails adapting the architecture, the data augmentation, the pooling and adding a normalization stage, as shown in Figure 1.

SSCD uses a ResNet-50 convolutional trunk to extract image features. We standardize on this architecture because it is widely used, well optimized and still very competitive for image classification , but any CNN or transformer backbone could be used (see Section 5).

For classification, the last CNN activation map is converted to a vector by mean pooling. We use generalized mean (GeM) pooling instead, which was shown to improve the discriminative ability of descriptors. This is desirable for instance retrieval and our copy detection case alike. GeM introduces a parameter $p$, equivalent to average pooling when $p=1$ and max-pooling when $p\rightarrow\infty$. SSCD uses $p=3$, following common practice for image retrieval models .

While GeM pooling at inference time systematically improves accuracy, we observe that it is beneficial at training time only in combination with the differential entropy regularization, i.e. with a vanilla InfoNCE it is better to train with average pooling. We conjecture that GeM pooling may reduce the difficulty of the training task without the additional objective of maximally separating embedding points. We observe that learning the scalar $p$, as proposed in , fails for contrastive learning: the pooling parameter grows unbounded until training becomes numerically unstable.

Descriptor projection.

SimCLR uses a 2-layer MLP projection at training time. For inference, the MLP is discarded and CNN trunk features are used directly. The MLP is partly motivated to retain transformation-covariant features in the base network, which may be useful for downstream tasks, despite a training task that requires a transformation-invariant descriptor. Jing et al. also find that the MLP insulates the trunk model from an embedding collapse into a lower-dimensional space caused by the InfoNCE loss.

For SSCD, the training and inference tasks are the same, obviating the need for transformation-covariant features, and differential entropy regularization prevents the dimensional collapse. We replace the MLP with a simple linear projection to the target descriptor size, and retain this projection for inference.

2 Data Augmentation

Self-supervised contrastive objectives learn to match images across image transforms. These methods are sensitive to the augmentations seen at training time , since invariance to these transforms is the only supervisory signal.

Table 1 lists the SSCD augmentations used in our experiments. Note that since our main evaluation dataset (DISC2021) is built in part with data augmentation, there is a risk of overfitting to the augmentations of that dataset. This is mitigated by (1) DISC2021’s set of augmentations is not known precisely and (2) we present strong results trained using a simple blur augmentation. Our starting baseline is the default set of SimCLR augmentations.

Empirically, copy detection benefits from a stronger blur than is typically used for contrastive learning. We strengthen the blur augmentation compared to SimCLR. We suggest that invariance to blur confers a low-frequency bias, reducing the model’s sensitivity to high-frequency noise common to real world copies. We use this setting for most ablation steps, because it is easy to reproduce, and provides a good baseline setting for comparing methods. This augmentation was initially tuned on a proprietary dataset, and is unlikely to overfit to DISC2021.

Advanced augmentations.

We evaluate our method with additional augmentations, to demonstrate how SSCD extends as augmentations are added. Half of rotations rotate by multiples of 90 degrees and half are unconstrained. The text has a random font, text, opacity, font size, and color. We add emoji of random size. We apply JPEG compression with randomly sampled compression quality. These augmentations are somewhat inspired by DISC2021 but are still fairly generic for image copy detection problems.

Mixed images.

We use two augmentations that combine content from two images within a training batch. In a copy detection context, these augmentations model partial copies, where part of an image is included in a composite image. Mixup is a pixelwise weighted average of two images ($a$ and $b$) with parameter $\gamma\in$: $\gamma\cdot a+(1-\gamma)\cdot b$. CutMix moves rectangular regions from one image into another. See Appendix D for implementation details. Mixed images match multiple images in the batch, requiring changes to our losses, outlined below.

3 Loss Functions

SSCD uses a weighted combination of the contrastive InfoNCE and the entropy loss, as in Equation (4). However, we need to adapt both losses for the mixed-image augmentation case, where $P_{i}$ may contain multiple matching images.

We adapt the InfoNCE loss (see Section 3.1) to accommodate augmentations that mix features from multiple images. Given an image $i$ with full or partial matches $j\in P_{i}$, we modify the pairwise loss term from Equation (1) as:

where $\hat{P}_{i}=P_{i}\cup\{i\}$. We then combine these terms by taking a mean per image, so that each image contributes similarly to the overall loss, and average per-image losses. Note that this is equivalent to InfoNCE for non-mixed images.

Entropy loss.

Our formulation of the entropy loss in Equation (3) remains the same, with $\hat{P}_{i}$ updated to include multiple matching images.

Combination.

The losses are combined with entropy weight parameter $\lambda$:

Multi-GPU implementation.

The contrastive matching task benefits from a large batch size, since this provides stronger negatives. Losses are evaluated over the global batch, after aggregating image descriptors across GPUs. Descriptors from all GPUs are included in the negatives InfoNCE matches against, and we choose nearest neighbors for entropy regularization from the global batch. Batch normalization statistics are synchronized across GPUs to avoid leaking information within a batch. We use the LARS optimizer for stable training at large batch size.

4 Inference and retrieval

For inference, the loss terms are discarded. Features are extracted from the images using the convolutional trunk followed by GeM pooling, the linear projection head, and L2 normalization. Then we apply whitening to the descriptors. The whitening matrix is learned on the DISC2021 training set. The descriptors are compared with cosine similarity or equivalently with simple L2 distance.

5 Similarity normalization

We follow using similarity normalization as one of our evaluation settings. It uses a background dataset of images as a noise distribution, and produces high similarity scores only for queries whose reference similarity is greater than their similarity to nearest neighbors in the background dataset. Given a query image $q$ and a reference image $r$ with similarity $s(q,r)=\operatorname{sim}(z_{q},z_{r})$, the adjusted similarity is $\hat{s}_{0}(q,r)=s(q,r)-\beta s(q,b_{n})$ where $b_{n}$ is the $n$th nearest neighbor from the background dataset, and $\beta\geq 0$ is a weight.

Carrying around a bias term makes indexing of descriptors more complex. Therefore, we include the bias into the descriptors as an additional dimension:

Then we are back to $\hat{s}(q,r)=\operatorname{sim}(\hat{z}_{q},\hat{z}_{r})$. The descriptors are not normalized, i.e. the dot product similarity is not equivalent to L2 distance. If L2 distance is preferred for indexing, it is possible to convert the max dot product search task into L2 search using the approach from .

Similarity normalization consistently improves metrics. However it adds operational complexity, and may make it difficult to detect content similar to the background distribution. Therefore, we report results both with and without this normalization.

Experiments

In this section we evaluate SSCD for image copy detection. Despite its relative simplicity, it depends on various settings that we evaluate in an extensive ablation study.

Most evaluations are on the validation dataset of the Image Similarity Challenge, DISC2021 . DISC2021 contains both automated image transforms and manual edits. There are 1 million reference images and 50,000 query images, of which 10,000 are true copies. A disjoint 1 million image training set is used for model training and as background dataset for score normalization. The training set contains no copies or labels, but is representative of the image distribution of the dataset. The performance is evaluated with micro average precision ($\mu AP$) that measures the precision-recall tradeoff with a uniform distance threshold.

ImageNet.

For some experiments we train models on the ImageNet training set (ignoring the class labels). We use ImageNet linear classification to measure how our copy detection methods affect semantic representation learning.

Copydays

is a small copy detection dataset. Following common practice , we augment it with 10k distractors from YFCC100M , a setting known as CD10K, and evaluate the retrieval performance with mean average precision ($mAP$) on the “strong” subset of robustly transformed copies. In addition to this standard measure, we evaluate the $\mu AP$ on the overall dataset.

2 Training implementation

We use the training schedule and hyperparameters from SimCLR : batch size $N=4096$, resolution $224\times 224$, learning rate of $0.3\times N/256$, and a weight decay of $10^{-6}$. We train models for 100 epochs on either ImageNet or the DISC training set, using a a cosine learning rate schedule without restarts and with a linear ramp-up. We use the LARS optimizer for stable training at large batch size. We train at spatial resolution $224\times 224$.

We use a lower temperature than SimCLR, $\tau=0.05$ versus $0.1$, following an observation in that this setting yields better accuracy on the training task, while reducing accuracy of downstream classification tasks.

3 Evaluation protocol

We resize the small edge of an image to size 288 preserving aspect ratio for fully convolutional models. We use a larger inference size than seen at training to avoid train-test discrepancy . We use different preprocessing for the DINO ViT baseline, following their copy detection method. See Appendix D for details.

Descriptor postprocessing.

Image retrieval benefits from PCA whitening. SSCD descriptors are whitened, then L2 normalized. For baseline methods that use CNN trunk features, we L2 normalize both before and after whitening. SimCLR projection features often occupy a low-dimensional subspace, making whitening at full descriptor size unstable, and many representations perform better when whitened with low-variance dimensions excluded. For baseline methods, we try dimensionalities $\{d,\frac{3}{4}d,\frac{d}{2},\frac{d}{4},\ldots\}$ and choose the one that maximizes the final accuracy. For SSCD, we whiten at full descriptor size.

We use the FAISS library to apply embedding postprocessing and perform exhaustive k-nearest neighbor search. We train PCA on the DISC2021 training dataset, following standard protocol for this dataset.

4 Results

Table 2 reports DISC2021 results from the baseline methods published in and SSCD. Our evaluation protocol obtains somewhat stronger results for the Multigrain baseline (3rd row). The first observation is that SSCD improves the baseline accuracy by 2$\times$ to 5$\times$ before score normalization, demonstrating that copy detection benefits from specific architectural and training adaptations.

We present results on a few different SSCD models trained on ImageNet or DISC2021, using the three augmentation settings we propose. The intermediate model SimCLRCD has all of our proposed changes except the entropy loss. SSCDlarge model uses a larger descriptor size and a ResNeXt-101 trunk.

We evaluate SimCLR using both trunk and projected features, and find trunk features ($\mu AP$ $=13.1$) to outperform features from the projection head ($\mu AP$ $=9.4$) with and without score normalization. Further experiments (Appendix A) show the reverse when training with entropy loss: projected features have similar accuracy to trunk features, despite a much more compact representation.

The gain of SimCLRCD ($\mu AP$ $=39.8$ without score normalization) over SimCLR (13.1) is decomposed in Section 5.5. Introducing the entropy loss in SSCD contributes an additional 10% absolute of $\mu AP$, which is further increased by stronger augmentations (+6.2%) and training on a dataset with less domain shift (+4.7%). These findings are confirmed after score normalization.

Copydays results.

Table 3 reports results for baseline methods using publicly released models, but omit Multigrain settings that we were unable to reproduce. We used published preprocesing settings for baselines and whitening. Our DINO results outperform published results.

The first SSCD result is with all settings from our DISC2021 experiments, where we resize the short side of each image to 288 pixels. With no tuning on this dataset, our method outperforms published results. We also show results for SSCDlarge using a ResNeXt101 trunk and 1024 descriptor dimensions, at larger inference size. We report more results on CD10K in Appendix B.

In addition to state-of-the-art accuracy using the customary $mAP$ ranking metric, our method provides a significant improvement in the global $\mu AP$ metric, indicating better distance calibration. On high-resolution images that are common for image retrieval, we observe improved $mAP$ but degraded $\mu AP$. SSCD descriptors are more compact than baselines.

5 Ablations

Entropy weight.

Table 5 compares how varying entropy loss weight ($\lambda$) affects copy detection accuracy, using SimCLRCD as a baseline. Models for this experiment are trained using the strong blur augmentation setting.

As the entropy weight increases, we see a corresponding increase in global accuracy metrics. We also see a similar increase in per-query ranking metrics, such as recall at 1 and mean reciprocal rank (MRR). The increase in ranking metrics demonstrates that differential entropy regularization improves copy detection accuracy in general, beyond creating a more uniform notion of distance.

In contrast to metric learning contexts where entropy regularization has been used, copy detection benefits from higher $\lambda$ values. Our standard setting is $\lambda=30$, while reports reduced accuracy with $\lambda>1$, and uses values $<0.1$. At $\lambda>40$, training becomes unstable, and tends to minimize the entropy loss at the expense of the InfoNCE loss: embeddings are uniformly distributed, but meaningless because image copies are not near anymore.

Additional ablations.

We explore how batch size, training schedule, descriptor dimensions, and score normalization affect accuracy in Appendix A.

Discussion

We find, similar to , that SimCLR collapses to a subspace of approximately 256 dimensions when trained in 512 dimensions. Table 4 shows that SimCLR’s accuracy does not improve much when the descriptor size increases from 128 to 512 dimensions. SSCD’s entropy regularization resolves this collapse, and allows the model to use the full descriptor space.

Entropy regularization and whitening.

SSCD is much more accurate than baselines when compared without whitening or similarity normalization: 47.8 $\mu AP$ for $\lambda=30$ when trained on ImageNet, versus 26.8 for an equivalent $\lambda=0$ model. Both the entropy loss and post-training PCA whitening aim at creating a more uniform descriptor distribution. However PCA whitening can distort the descriptor space learned during training, particularly when many dimensions have trivial variance. Differential entropy regularization promotes an approximately uniform space, allowing the model to adapt to an approximately whitened descriptor during training, reducing the distortion whitening induces.

Uniform distribution as a perceptual prior.

For most experiments in this work we focus on the $\mu AP$ metric that requires a separation between matches and non-matches at a fixed threshold. However Table 5 shows that ranking metrics also improve with increased the entropy loss weight, i.e. better calibration across queries does not fully explain the benefit of entropy regularization.

Differential entropy regularization acts as a kind of prior, selecting for an embedding space that is uniformly distributed. We argue that, when applied to contrastive learning, this regularization is a perceptual prior, selecting for stronger copy detection representations. An ideal copy detection descriptor would map copies of the same image together, while keeping even semantically similar (same “class”) images far apart i.e. the descriptor distribution is uniform. This differs from the ideal properties of a representation for transfer learning to classification, where images depicting the same class should be nearby (a dense region) and well separated other classes (a sparse region between classes).

Visual results.

Figure 2 shows a few retrieval results, where SSCD outperforms the vanilla SimCLR. The two first examples demonstrate the impact of more appropriate data augmentation at training time: SSCD ignores text overlays and blur/color balance. The two last examples show that SimCLR falls back on low-level texture matching (grass) when SSCD correctly recovers the source image.

Limitations.

Our method is explicitly text-insensitive when training with text augmentation, and we find that it is somewhat text-insensitive even when trained without text augmentation. For this reason, SSCD is not precise when matching images composed entirely of text. Different photos of the same scene (e.g. of landmarks) may be identified as copies, even if the photos are distinct. Sometimes, images are combined to create a composite image or collage, where the copied content may occupy only a small region of the composite image. “Partial” copies of this kind are hard to detect with global descriptor models like SSCD, and local descriptor methods may be necessary in this case. Finally, matching at high precision often requires an additional verification step.

Ethical considerations.

We focus our investigation on the DISC2021 dataset, which is thoughtful in its approach to images of people, using only identifiable photos of paid actors who gave consent for their images to be used for research. Copy detection for content moderation is adversarial. There is a risk that publishing research for this problem will better inform actors aiming to evade detection. We believe that this is offset by the improvements that open research will bring.

This technology allows scaling manual moderation, which helps protect users form harmful content. However, it can also be used for e.g. political censorship. We still believe that advancing this technology is a net benefit.

Conclusion

We presented a method to train effective image copy detection models. We have demonstrated architecture and objective changes to adapt contrastive learning to copy detection. We show that the differential entropy regularization dramatically improves copy detection accuracy, promoting consistent separation of image descriptors.

Our method demonstrates strong results on DISC2021, significantly surpassing baselines, and transfers to Copydays, yielding state-of-the-art results. Our method is efficient because it relies on a standard trunk, uses smaller inference sizes than are typical for image retrieval, and produces a compact descriptor. Additionally, its calibrated distance metric limits candidates for verification. We believe that these results demonstrate a unique compatibility between uniform embedding distributions and the task of copy detection.

References

Appendix A Additional ablations

Table 6 shows how copy detection accuracy is affected by several hyper-parameters.

The descriptor dimension is a tradeoff between accuracy and the efficiency of the retrieval step. When constraining the descriptor to 256 dimensions for retrieval, we see highest accuracy for descriptors trained at that size.

Batch size.

The training objective learns to match pairs within the global batch (across all GPUs). A larger batch size makes the training task more challenging, improving the final accuracy. Large batch sizes require training with more machines, and incur synchronization overhead due in part to synchronized batch normalization.

Training schedule.

We compare accuracy as we vary the number of training epochs, and find no benefit to longer training schedules.

Variance between initializations.

We train using the same setting, initializing the model with five random seeds, and find a standard deviation of 0.2% $\mu AP$ and 0.1% $\mu AP_{SN}$.

Similarity normalization settings.

We show score normalized accuracy given several similarity normalization settings in Table 7. Several score normalization settings work similarly well. When using a single neighbor to normalize similarity, using the $2$nd nearest neighbor works best ($n=2$). When using an average similarity across multiple neighbors, averaging the first 2, 3 or 4 neighbors work similarly well. We find that $\beta=1$ is a good normalization weight. Our similarity normalized results use $n=1$, $n_{end}=3$, $\beta=1$, a setting that we found to work well across many descriptors.

Trunk and projected features.

We compare SSCD trunk and projected features in Table 8. Using the linear projection at inference time improves accuracy, despite a significantly more compact code.

Appendix B Full Copydays results

We provide additional Copydays results in Table 9, evaluating SSCD and SSCDlarge using preprocessing settings from prior published results. In each case, we evaluate our method with no tuning, e.g. we don’t adjust the GeM $p$ as proposed in .

We note that at $224^{2}$ inference size, ResNet50 has approximately $4\times$ the throughput as ResNeXt101 or ViT-B/16, and $20\times$ that of ViT-B/8.

Appendix C Embedding distribution

We plot principal values for SSCD ($\lambda=30$) compared to SimCLRCD ($\lambda=0$), and a uniform distribution in Figure 5. We see that the $\lambda=0$ model fails to make full use of the descriptor space, as observed in . With entropy regularization, all components have similar energy, spanning less than an order of magnitude (the maximum is $6.6\times$ the minimum).

Appendix D Implementation details

Mixup and Cutmix augmentations both combine content from two source images. The amount of content used from each image is determined by a mixing parameter $\gamma$, sampled from a $\beta$ distribution: $\gamma\sim\beta(\alpha,\alpha)$. We set $\alpha=2$ to reduce the prevalence of “trivial” mixed images that draw nearly all content from one of the inputs.

DINO baseline details.

We follow the copy detection method presented in for the DINO baseline. We use the concatenation of the CLS token and GeM pooled ($p=4$) patch token features as the descriptor.

Our DINO DISC evaluation uses the ViT-B/16 trunk. We resize inputs to $224\times 224$ without center cropping. This outperformed other preprocessing for this model, including our default aspect-ratio preserving resize, and resizing inputs to a larger fixed size ($288\times 288$). We suspect that ViT models may be less adaptable to rectangular inputs than fully convolutional networks.

Appendix E Visualizing matches

To view which parts of an image A match strongly to another image B, we experiment by keeping the activation map on A at full resolution by removing the GeM pooling operation. This results into one descriptor per activation map pixel, that can be compared with a global SSCD descriptor. We can thus build a spatial heatmap with the strongest activations. Figure 6 shows image pairs and the corresponding heatmaps. The areas on the left image that match with the image on the right are clearly identified.

Appendix F Retrieved matches

We compare the first result retrieved by SSCD and SimCLR on the DISC2021 dataset. Both models are trained on ImageNet and evaluated with whitening. We use trunk features for SimCLR, which are more accurate for this model. We do not use score normalization, since it has no effect on top-1 accuracy.

Table 10 shows quantitative results from this exercise. SSCD correctly identifies the copy as the first result $2\times$ as often as SimCLR. Correct SSCD matches are nearly a superset of SimCLR matches: very rarely does SimCLR have a correct first result that SSCD misses.

Figure 7 shows additional queries and retrieved results for examples that only SSCD correctly identifies. One pattern we observe is that SimCLR often matches images with similar types of distortion together. Images with text at an angle, or strong diagonal features, may be incorrectly matched with images with similar features. Images with a blurry, or grainy, quality are matched to other images with a similar quality. This is surprising given that SimCLR trains with a blur augmentation, albeit weaker, and should be somewhat blur invariant.