SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Introduction

A primary driver of recent success in machine learning has been the rise of self-supervised learning (SSL) scaled to ever larger models and unlabelled datasets . In particular, modern large datasets are often derived at global web-scale and are generally unfiltered, with the exception of NSFW filters. One such public dataset is LAION , a multi-modal dataset of 5 billion image/text pairs. Multi-modal models such as CLIP are trained for many epochs on these large datasets achieving impressive performance but at the cost of extremely long training durations.

The critical role of large datasets has led to increasing interest in scaling laws which enable us to predict how a model’s performance will change given more data and/or parameters, leading to the observation that test error generally scales as a power law with respect to data quantity . Power law scaling, however, is unsustainable as diminishing marginal returns are quickly hit such that ever increasing amounts of data are required to achieve ever diminishing improvements in performance. Notably, many of these models appear never to converge, as test performance continues to increase even after 10s of passes through these massive datasets . This result suggests that our best models are underfitting, likely as a result of spending an increasing fraction of learning time focusing on redundant data.

Improving data efficiency would therefore be quite impactful, either by enabling models to achieve the same performance much faster, or by enabling models to achieve better performance given the same computational budget. These observations have inspired recent work which suggests that by pruning training data according to an intelligent criterion, power law scaling with respect to data can be beaten and, given an optimal data ranking metric, exponential scaling might in principle be achieved . Recent explorations of this direction have shown promising results, with some works able to reduce data size by almost 5-fold with minimal performance loss .

However, optimal approaches to select data remain poorly understood. Such approaches might focus on one of several different classes of examples to be removed, roughly ordered by the complexity of their discovery:

Perceptual duplicates: We loosely define such data pairs to be perceptually identical to a typical human observer. The most straightforward version would be exact duplicates at the pixel or token level that could easily be found via exact duplicate detection in input space. However, such approaches might miss pairs of images with human imperceptible pixel level distortions. Most widely-used datasets have some exact duplicate filter already applied, though perceptual duplicates with slight pixel-level differences may pass through such filters.

Semantic duplicates: these are examples which contain largely identical information content, but remain perceptually distinct. For example, a pair of image views which are derived from the same image, but feature different margins, aspect ratios, color distributions, etc. could be considered semantic duplicates. A pair of sentences with the same structure but some words exchanged for synonyms would also be considered a semantic duplicate. Such pairs would rarely, if ever, be detected by exact duplicate filters as they would be far apart in pixel/token space.

Semantically redundant data: in contrast to semantic duplicates, semantically redundant data are not derived from the same underlying objects and would be clearly distinguishable to a human. However, the information contained in such examples may still contain substantial overlap. For example, consider the case of two different images of two different golden retrievers in two different parks. These images are neither perceptually nor semantically identical as the content of the images differs. However, the information contained in them is quite similar, leading us to think of such pairs as semantically redundant. Each additional semantically redundant data point will provide less and less new information, eventually converging to near-zero information gained from additional such data. Methods such as SSL Prototypes and memorization search for semantically non-redundant data subsets to train on.

Misleading data: these are data which rather than providing zero information (as in the previous categories) provide negative or harmful signal, in the sense that removing these data actually improves performance, rather than having a neutral effect. While such data are easy to conceive of in supervised learning (i.e. mislabeled examples), it is much less clear what such examples may be in the context of self-supervised learning.

In this work, we focus on the category of semantic duplicates: data which are semantically highly similar but which would be difficult to discover using simple deduplication approaches. These data points are challenging to identify because distance measures in input space are unlikely to uncover semantic duplicates. To overcome this limitation, we leverage pre-trained foundation models to compare data similarity in the learned embedding space rather than in input space. Comparing every data point to every other data point, however, is intractable, especially for web-scale datasets containing billions of examples. To make this computation possible, we use the clustering approach described in to segment the embedding space, allowing us to only search for duplicate pairs within a cluster. Using this approach, we make the following contributions:

We propose SemDeDup (Fig. 1, a), a simple, yet effective and computationally tractable way to identify semantic duplicates. Using this approach, we show that large web-scale datasets such as LAION contain large numbers of semantic duplicates, with 50% of examples containing at least one semantic duplicate.

Large fractions of semantic duplicates can be removed with little-to-no performance impact, greatly increasing training efficiency. We reduced the size of our LAION training set by 50% with minimal performance loss, and improved learning speed, achieving nearly the same performance 2x faster (Fig. 1, b), and moreover improved performance out-of-distribution.

We apply SemDeDup to C4, a large text corpus, beating prior SoTA deduplication while providing efficiency gains of 15%, sometimes even improving performance.

Overall, our results demonstrate a simple yet surprisingly effective approach to reduce the cost of training through the removal of semantic duplicates which is likely applicable to all web-derived datasets and may help to democratize the training of large-scale foundation models by improving data and compute efficiency.

Related Work

Much of the work in language and vision on deduplication has focused on the removal of exact duplicates. For example, removed duplicates between the YFCC15M dataset and the ImageNet validation set to prevent train-test leakage. The C4 text corpus - used for training T5 - has been deduplicated by discarding repeated occurrences of any three-sentence spans. showed that it’s possible to further deduplicate this dataset without loss of performance by computing approximate n-gram overlap between documents using the MinHash technique . also applied MinHash based deduplication to curate training data for the Gopher model and demonstrated that training on the deduplicated dataset can result in lower perplexity across various validation sets. found that deduplication prevents memorization in LLMs and thus mitigates privacy concerns. More recent works use forms of model-based feature extraction to improve the robustness of the similarity metric used for deduplication. created a supervised dataset for detecting duplicate news articles and trained models to predict those labels. In the domain of computer vision, improves on SSL techniques by removing near-duplicates in some high dimensional feature space they learn.

Beyond deduplication, a host of classical machine learning approaches seek to achieve data efficiency by finding coresets, defined as small subsets of the training data that can be used to train a machine learning algorithm to the same test accuracy achievable when training on the entire training data (see e.g. for reviews). However, many coreset algorithms are computationally prohibitive and therefore are difficult to scale to web-scale data. In contrast to many traditional coreset algorithms, we develop an exceedingly simple and tractable algorithm that achieves both computational and data efficiency at scale.

Recent approaches to achieve data efficiency in deep learning have operated in a supervised setting by defining and finding “hard” examples not easily learned by partially or fully trained (ensembles of) models . Perhaps the closest to our work is a recent effort to break beyond neural power law scaling by pruning unlabelled data, using the embedding space of a pre-trained foundation model . However, the largest dataset for which these works examined data pruning was ImageNet. In contrast, we move from relatively small, highly curated ImageNet scale to highly uncurated, web-scale data. Our analysis, at this new large and uncurated scale, reveals a possibly fundamental role for semantic deduplication as an important initial step in data-pruning for self-supervised learning that was not considered in prior data-pruning works.

SemDeDup

While identifying perceptual duplicates can be easily done in input space, identifying semantic duplicates is more difficult as they may be distant in either pixel or token space. To identify these pairs, we leverage the embedding space of a large pre-trained foundation model to provide a more semantically meaningful distance metric. To detect and remove semantically similar images, we use the following semantic de-duplication (SemDeDup) algorithm (Fig. 1, a). First, we embed each data point using a foundation model (CLIP for images and OPT for language). We then cluster the embeddings into $k$ clusters via k-means. Below, we choose $k=50,000$ clusters in CLIP image encoder embeddings and $k=11,000$ clusters in OPT-language model embeddings. Within each cluster, we compute all pairwise cosine similarities and set a threshold cosine similarity above which data pairs are considered semantic duplicates. Finally, from each group of semantic duplicates within a cluster, we keep the image with the lowest cosine similarity to the cluster centroid and remove the rest. We note that to determine duplicates, this method considers only the images and ignores the captions. A simplified pseudo code for SemDeDup is shown in Algorithm A7 in the appendix. We provide more details about the method in addition to experiments on choosing the value of $k$ in section 6.

Utilizing pre-trained foundation Models

Our method makes use of pre-trained foundation models to embed data examples. Considering that there are many of these ready-to-use pre-trained models available to the public, we can use embeddings from these models to guide curation of other datasets. Pre-trained models like Vision Transformers for vision tasks, OPT for natural language and CLIP for vision-language data have been used widely. In this work, we utilize pre-trained CLIP and OPT models for deduplication. In addition, in Section 6, we show that one can effectively use an on-the-shelf model pre-trained on one dataset to prune another dataset resulting in a considerable training cost saving.

Clustering to reduce computation

SemDeDup on LAION

If we consider pairs of data points to be semantic duplicates when their cosine similarity is at least $1-\epsilon$, then $\epsilon$ can be thought of as a deduplication dissimilarity threshold, with increasing $\epsilon$ reflecting an increasingly coarser notion of semantic equality. We expect that low thresholds of $\epsilon$ will find semantic duplicates, while higher thresholds will allow semantically redundant data pairs as well.

To evaluate SemDeDup’s ability to discover semantic redundancy in multi-modal data, we train CLIP models on the LAION dataset (Section 3). We first show that LAION contains extreme amounts of semantic redundancy (Section 4.2) and provide examples of the semantic duplicates discovered by SemDeDup (Section 4.3). Most critically, we demonstrate that removing the semantic duplicates discovered by SemDeDup has minimal to no impact on converged performance and increases learning speed (Section 4.4).

To train large-scale multi-modal models, we used the LAION dataset , an open multi-modal dataset containing up to 5 billion image-text pairs scraped from the web. LAION data were filtered using a pre-trained CLIP model to only retain image-text pairs with an embedding similarity greater than 0.28. Image-text pairs containing very short captions or small images were also removed. A simple de-duplication method based on the image url was also performed.

The majority of our experiments were performed on the LAION-440M filtered subset of LAION-2B introduced by . This dataset was filtered using a Complexity, Action, and Text (CAT) filtering according to three criteria: (1) high enough caption complexity; (2) the caption must contain an action; (3) any text present in the image cannot substantially overlap with the caption.

To ensure this CAT filtered LAION-440M subset did not impact our results, we also performed experiments on unfiltered data derived from LAION. Much of the original LAION-400M subset is no longer available due to broken urls, so we used a reduced version of the LAION-400M subset containing the 233 million data points we were able to collect, which we call LAION-233M.

CLIP training.

CLIP Evaluation

For CLIP evaluation we use zero-shot evaluation on 30 different datasets. Tables A4 and A5 in the Appendix list all the datasets we use for evaluation.

2 Extreme semantic redundancy at web-scale

How many semantically redundant pairs are there in LAION? Remarkably, we find that even tiny thresholds $\epsilon$ lead SemDeDup to remove large fractions of data in LAION440M (Fig. 3a), showing that LAION-440M contains large quantities of semantic duplicates. Surprisingly, $30\%$ of images in LAION-440M have a semantic duplicate at the highly stringent distance threshold of $\epsilon=0.00095$, while $50\%$ have a duplicate at the tight threshold of $\epsilon=0.03$ (Fig. 3c). Moreover, a histogram of pairwise cosine similarity in LAION-440M (Fig. 3d) reveals a high density of pairs at high cosine similarity, including a large contribution at $1$, reflecting highly similar semantic duplicates. These results demonstrate that LAION-440M contains large amounts of semantic redundancy.

3 What do semantic duplicates look like?

What leads to semantic duplicates? In Fig. 2, we show examples of semantic duplicates found at different thresholds $\epsilon$. At extremely low values of $\epsilon$ we find perceptual duplicates, and at slightly higher values of $\epsilon$, we find semantic duplicates, which are the same image but with distortions which evade exact de-duplication approaches such as different margins, crops, aspect ratios, and color filters, or slightly different peripheral details. Fig. A10, and A11 show examples of clusters that are semantically deduplicated at increasing levels of $\epsilon$, clearly indicating more semantic diversity in deduplicated clusters as $\epsilon$ increases.

Many semantic duplicates are of products which may have been displayed on multiple e-commerce websites, each with a slightly different style. As a result, semantic duplicates often contain different, but highly similar captions. While most clusters contained 20-40% duplicates, there are several remarkable outliers in redundancy in LAION-440M (Fig. A8), including one cluster containing $\approx 307,000$ copies of the European Union flag and another with $\approx 318,000$ copies of an icon of “Image not found."

At higher levels of $\epsilon$ in Fig. 2, and A9, we find fewer semantic duplicates, which are generally derived from the same source image, and more pairs which exhibit semantic redundancy instead, in which the same concept is present, but not derived from the same image source. For example, semantically redundant pairs may contain different images of similar objects or scenes.

4 Training on semantically deduplicated data improves efficiency

If SemDeDup is effective at finding semantic duplicates, we should be able to remove these duplicates with a minimal performance impact. To test this, we train CLIP models on subsets of LAION-440M deduplicated at different thresholds $\epsilon$, corresponding to smaller fractions of data as $\epsilon$ rises.

In Fig. 4 (a), we plot the top-1 zero-shot accuracy of our CLIP models on ImageNet-1k. Encouragingly, we found that SemDeDup can remove up to 37% of LAION440M with no performance drop, and 50% with minimal performance drop ($<0.5\%$). In contrast, randomly removing data results in much larger drops. In Fig. 4 (b), we show the average zero-shot performance across $24$ tasks, finding that on average, performance increased on de-duplicated data. See Table A4 for detailed performance on all $24$ tasks at $6$ deduplication thresholds as well as $1$ baseline and $4$ random controls. See also Fig. A4 for performance on $24$ individual tasks.

We also evaluated out-of-distribution robustness on $6$ datasets commonly used for this task: ImageNet-A, ImageNet-O , Imagenet-R , Imagenet-sketch , ImageNetV2 , and ObjectNet . We again found that SemDeDup increased average performance over baseline when removing 37% of the data, and matched performance when 50% was removed as shown in Fig. 5 (a). See Table A5 for detailed performance on $6$ OOD tasks at $6$ deduplication thresholds as well as $1$ baseline and $4$ random controls. We also note that SemDeDup outperforms random pruning on all individual out-of-distribution robustness datasets for all fractions of dataset kept. See Fig. A5 for performance on the $6$ individual tasks.

Fig 6 shows SemDeDup performance across $30$ combined zero-shot and OOD tasks when removing $37\%$ of the data, relative to a CLIP baseline trained on all the data. Remarkably, on about $20$ out of $30$ tasks, performance actually improves after removing pre-training data, whereas on all but about $3$ of the remaining tasks performance is not substantially reduced. Our observation that SemDeDup can improve performance in many cases is consistent with prior work which has found that removing duplicates may improve performance by discouraging memorization .

We emphasize that SemDeDup achieves these results on LAION-440M, an already highly curated dataset derived from LAION-2B which was found to have similar performance despite the almost five-fold reduction in data . However, to ensure that this curated subset did not bias our results, we also evaluated on LAION-233M, an uncurated subset of LAION-2B, finding qualitatively similar results (Fig. A6).

Because SemDeDup reduces the number of training points, it enables substantially faster training. In Fig. 5 (b), we plot the top-1 zero-shot accuracy on ImageNet-1k as a function of the number of iterations for different deduplication thresholds $\epsilon$. Notably, models trained on deduplicated data reach convergence in substantially fewer iterations.

Why do models trained on uncurated data exhibit slower learning? We posit that successive learning iterations involving semantic duplicates yield redundant information, thereby wasting valuable computation on data points that are highly similar to those the model has already seen. By removing these semantic duplicates, we increase the fraction of data points which provide a marginal information gain to the model, thereby increasing learning speed .

SemDeDup on Natural Language

We evaluate our trained language models on two independent validation sets: the validation text corpora used by OPT (referred to as "opt_valid") and a random sample of the instruction finetuning corpus used to train the OPT-IML family of models , composed of verbalized prompts corresponding to a wide range of NLP tasks and their solutions (referred to as "prompts_with_answers").

To perform SemDeDup, we pass documents through the open-sourced pre-trained 125M OPT model and save the last layer embedding for the last token in the document. We then apply the same method described in Section 3 with $\mathcal{K}=11000$ to cluster these embeddings. We compare to random pruning and the NearDup method described in . Note that the deduplication threshold values associated with different fractions of data remaining change compared to LAION-440M, as seen in Fig. A17.

2 Results on Language Modeling

In Fig. 7, we show the performance of SemDeDup versus random pruning. We observe that SemDeDup significantly outperforms random pruning as measured by perplexity on prompts_with_answers and average opt_valid performance. For a breakdown of performance on individual validation sets in opt_valid, see Fig. A20 where we observe that SemDeDup beats random pruning on every single validation set in opt_valid.

Training on less data for one epoch naturally causes performance to decrease. Thus, we also explore whether continuing to train on the same smaller pruned datasets for more epochs will match the performance of a baseline model trained on a larger dataset. In Fig. 8, we train on datasets pruned with SemDeDup, but perform the same number of total training steps as the baseline model on the larger dataset (which was trained for $1$ epoch). This causes the model to do multiple epochs over the pruned dataset. We observe that by training for multiple epochs over significantly pruned datasets we can reach the performance of a single-epoch run on the full dataset using 10-15% less compute. This is similar to the finding in Section 4.4. Notably, this efficiency gain is larger at higher pruning percentages, indicating that more aggressive pruning can yield more efficiency gains. This trend generally holds across the individual validation sets in opt_valid (see Fig. A21).

On the C4 validation set, we observe that SemDeDup still outperforms random pruning in Fig. A18. In Table A12 we compare SemDeDup to the NearDup baseline from . We observe that NearDup and SemDeDup have comparable performance as is expected, because with 4% pruning there is very little change to the underlying dataset.

3 What is being pruned in language data?

In Fig. A22 and Fig. A23 we choose specific clusters and show a random sample of documents retained in the cluster after performing SemDeDup for different values of $\epsilon$. In Fig. A22, we observe that at low values of $\epsilon$, we find semantic duplicates in the form of templated text, where typically few words (e.g. a geographic location or a name) is changed. This successfully evades exact-string deduplication methods but contains highly redundant information as seen in Fig. A22. In Fig. A23, we show an example of a cluster with semantically redundant duplicates — most examples in this cluster are advertisements about Nike shoes. These examples are not necessarily templated text or have exact string matches, but are highly redundant nonetheless. We see in Fig. A23 that at more aggressive pruning (i.e. higher $\epsilon$) these semantically redundant duplicates get pruned. We note that exact string duplicates (i.e.“perceptual duplicates for text") are rare since duplicate occurrences of any three-sentence spans were removed in C4 already.

Analysis of hyperparameter choices

Here we study the impact of changing the number of clusters $k$ in the k-means clustering step in SemDeDup described in section 3. In all our experiments in the main paper, we set $k$ = 50,000 for the LAION dataset and $k$ = 11,000 for the C4 dataset. To study the impact of the $k$ on the performance, we deduplicate LAION440M using different values for $k$ and train different CLIP models on the deduplicated data. We compare three values for $k$ (70,000, 50,000, and 10,000) when deduplicating LAION440M to 40% of its size. As we see in Table 1 the exact choice of $k$ has a very small impact on performance as measured by the zeroshot accuracy on ImageNet with a small improvement in the top1 accuracy as $k$ increases.

The key intuition is that the choice of $k$ implements a tradeoff in the probability of recovering all semantic duplicates of any data point, and the computational complexity of doing so. For example, assuming k-means finds equal cluster sizes, each data point will lie in a cluster of size $N/k$, and we are only searching for $\epsilon$-nearest neighbors (with cosine similarity > $1-\epsilon$) within each cluster. As $k$ decreases, cluster size $N/k$ increases, and the error probability of substantially many $\epsilon$ nearest neighbors of a data point lying outside it’s own cluster decreases, while the computational complexity of searching for all nearest neighbors within the cluster increases. As long as $k$ is small enough relative to the total dataset size $N$, so that $N/k$ is large enough to contain most nearest neighbors of each data point, the performance of SemDeDup should be robust to the choice of $k$.

2 Pre-trained models for extracting embeddings

As we describe in section 3, SemDeDup clusters the example embeddings extracted from a pre-trained foundation model and uses them for deduplication. To study the effect of the pre-training dataset of the foundation model on SemDeDup we deduplicate LAION440M using an OpenAI CLIP model pre-trained on a different dataset than LAION. We use the Open AI CLIP ViT-Base model pre-trained on a private dataset of 400 million image-caption pairs. We use the embeddings from this model to deduplicate LAION440M dataset to 40% of its size. As we see in Table 2, using Open AI CLIP model for extracting embeddings has a negligible impact on the performance.

3 Different strategies for choosing which semantic duplicates to keep

In section 3 and Algorithm A7, we describe the steps for deduplication with SemDeDup. From each group of duplicates (the circles in Figure 1), we keep the example with the lowest cosine similarity to the cluster centroid in the embedding space. This is the default setting for all experiments we run unless otherwise mentioned. In Table 3 we study the strategy we follow to choose the example to keep from each group of duplicates. We train three CLIP models on 40% of LAION440M deduplicated by SemDeDup for 32 epochs. We try three options for choosing the examples we keep 1) keeping examples with low similarity to centroids, 2) keeping random examples, and 3) keeping examples with high similarity to cluster centroids. We obverse that the difference between the three methods in zero-shot accuracy on ImageNet is negligible.

4 Training on deduplicated data for more iterations improves performance

Training on deduplicated data comes with the advantage that we train for fewer iterations under the match-epochs setting. For example, training on 50% of LAION440M for the same number of epochs as the baseline model (100% of the data) means that we train for only 50% of the number of training iterations. We find that we can achieve a good trade-off between performance and training speed when training on deduplicated data. We show that training on deduplicated LAION440M for more iterations improves the accuracy while still being below the number of iterations we train the baseline model for. In Table 4, we show results for different CLIP models, trained on 50% of LAION440M, for a different number of training iterations. We see that by continuing training the model until we reach 75% of the iterations relative to the baseline model, we outperform the baseline model on not only ImageNet, but also on average accuracy over 24 datasets, and on the $6$ out-of-distribution datasets.

5 Choosing the deduplication threshold ϵitalic-ϵ\epsilon

We tune the deduplication threshold $\epsilon$ for each dataset manually to get the desired deduplicated dataset size. To do that, we first run the clustering step of SemDeDup. Then we sample 10% of the clusters and tune $\epsilon$ on them. We found that using only 10% of clusters gives a good approximation of the final dataset size. We notice that the relationship between $\epsilon$ and the deduplicated dataset size is semi-linear for both LAION and C4 datasets (see Fig. 3, A1, and A17). When tuning $\epsilon$ we start with two values and run SemDeDup on 10% of the clusters (the time needed for this step is a few minutes. See the DeDup. Time column in Table A2 ). Then we linearly interpolate the two values of $\epsilon$ knowing their correspondence dataset size and the target dataset size to get a better value for $\epsilon$. In Fig. A1 we plot the duplicated dataset size as a function of $\epsilon$ for different values of the number of clusters $k$ used. We show that $k$ has a small impact on the value $\epsilon$ only when the duplicated dataset size is less than 50%.

Compute cost of running SemDeDup

We report in Table 5 the cost of running SemDeup on LAION440M in GPU hours. We see in the table that the overhead of deduplicating LAION440M doesn’t exceed 1% of the training cost in GPU hours. This results in substantial savings in the overall cost after deduplication. For example, training on 50% of the data saves 50% of the training cost while requiring only 1% of the training cost for deduplication. We also show in Table A2 the time needed for deduplicating LAION440M dataset using SemDeDup using 8 GPUs for clustering and 64 GPUs for CLIP training. Our implementation for SemDeDup parallelizes the operations across devices to speed up the deduplication. The table also shows how the time changes as we change the number of clusters.

However, we should note that the computational cost of SemDeDup can be amortized across the efficiency gains it can generate in training many downstream models by many other groups. For example, its typical use case would be to take a large web-scaled dataset, and semantically deduplicate it once, resulting in a much smaller foundation dataset that can be widely disseminated to the community. Then many different groups can train many different foundation models on this deduplicated foundation dataset, and all these groups will reap the training efficiency gains conferred by a less redundant smaller dataset. Thus the computational cost of finding the dataset can be amortized across the efficiency gains achieved on many downstream training runs, in direct analogy to how the computational cost of training a foundation model can be amortized across the computational efficiency gains with which it achieves high zero-shot or fine-tuning performance on many downstream applications.

Discussion

We introduced SemDeDup, a simple yet tractable and effective method which leverages pre-trained embeddings to remove semantic duplicates which are highly semantically similar but not identical. Removing semantic duplicates improves learning speed and out-of-distribution performance while providing efficiency gains of up to 50% on the largely uncurated LAION and 15% on the partially curated C4. SemDeDup demonstrates the importance of data quality and the potential of data curation to dramatically improve training efficiency.

While SemDeDup does an effective job of removing semantic duplicates and some semantically redundant data points, it is only one way to remove uninformative data points. In particular, this work does not capture many aspects of semantic redundancy, nor does it address removal of bad or misleading data, all of which can likely be exploited to make substantial further reductions to dataset size without sacrificing performance.

SemDeDup also requires access to a pre-trained embedding model relevant to the domain of interest, which may pose a problem for entirely novel domains unrelated to the wide array of publicly available pre-trained models. However, for most domains, pre-trained models are readily available, and many such models have been shown to generalize to related domains. We, therefore, expect that this limitation will only apply to a small fraction of the practical use cases for SemDeDup.

In LAION, we identified semantic duplicates based only on image data, but we ignored the caption information. Leveraging this information may lead to the identification of further semantic duplicates.

Our results on C4 showcase the potential of SemDeDup for NLP, but the gains were more modest due to the partially curated nature of C4 which has fewer duplicates than LAION. We also trained small models relative to the best models. It is possible that results may change with scale, though following , it is likely increasing scale would further improve the benefits of data curation.

Overall, the optimal data pruning policy for finding the smallest possible data subset under computational tractability and performance constraints remains, as ever, an extremely difficult open question. However, the remarkable efficacy of SemDeDup, especially given its underlying simplicity and scalability, suggests that the removal of semantic duplicates may well be an important prerequisite for any more sophisticated data pruning algorithm, especially when working with modern, large, highly uncurated, web-scale datasets.

Acknowledgements

We thank Mido Assran and Mansheej Paul for discussions. We also thank Mitchell Wortsman for support with OpenCLIP. We also thank Armen Aghajanyan for suggestions for handling training instability in our language model experiments.

References

Appendix A Additional Analysis

To further assess the impact of changing the value of $k$ we measure the intersection between datasets deduplicated by SemDeDup using different values for $k$. Let $D_{A}=\{a_{1},a_{2},...,a_{N}\}$ and $D_{B}=\{b_{1},b_{2},...,b_{N}\}$ be two datasets of the same size $N$. We define the percentage of intersection $I$ between $D_{A}$ and $D_{B}$ in equation 1 as the percentage of data points that appear in both datasets relative to the dataset size $N$. Note that $I(D_{A},D_{A})=100\%$. We find that deduplicating LAION440M dataset to 72% of its size using any value of $k$ values (10000, 25000, 50000, 70000) results in almost the same dataset with only 3% of the examples replaced when changing $k$. This is induced by the 97% percentage of intersection $I$ value between any pair of datasets deduplicated using two different values for $k$. We show in Fig. A2 the percentage of intersection ratio between different datasets when changing the number of clusters $k$ at different deduplication thresholds $\epsilon$. We also show in figure A1 that by using the same deduplication threshold value $\epsilon$ we get almost the same deduplicated dataset size for different values for $k$.

A.2 Estimating The Fraction of Duplicates Detected By SemDeDup

SemDeDup searches for duplicates within clusters. This results in reducing the floating point operations (FLOPs) required for deduplication by 5 order of magnitude for LAION440M dataset as described in section 3. Indeed, by searching for duplicates within clusters, we ignore duplicates across different clusters if they exist. Here we try to estimate the efficiency of SemDeDup in detecting all the duplicates in the dataset. Let $D_{\epsilon}$ represent the total number of duplicates in the dataset at a specific value of deduplication threshold $\epsilon$, and $D^{s}_{\epsilon}$ represent the total number of duplicates detected by SemDeDup. We define the deduplication efficiency $\eta_{\epsilon}$ (eq. 2) as the fraction of duplicates detected by SemDeDup from the total number of duplicates in the datasets at a specific value of $\epsilon$. For example, a deduplication efficiency of 100% corresponds to detecting all the duplicates in a dataset. As computing the exact value of $D_{\epsilon}$ is computationally expensive, we approximate its value by the number of duplicates between the cluster items and its 20 nearest neighbor clusters and donate this approximated value by $D^{{}^{\prime}}_{\epsilon}$. We sampled part (2000 clusters) of the LAION440M dataset randomly and compute the value of the deduplication efficiency $\eta$ in eq. 2 for different values of $\epsilon$ and k-means clusters $k$. As we see in Table A1, for $k$=50,000, SemDeDup can effectively detect more than 94% of the duplicates when keeping 63% of LAION440M dataset and 89% of the duplicates when keeping 40%.

Appendix B CLIP Zeroshot Evaluation

In this section, we show the result of zeroshot evaluation for CLIP. We note that the models trained on dataset deduplicated using SemDeDup outperform the baseline model in many tasks. In Table A4 we list the top1 zeroshot accuracy on 24 tasks and in Table A5 we show the top1 zeroshot accuracy on 6 datasets for out-of-distribution robustness evaluation. Our complete evaluation set has $30$ different datasets in total. When using only 63% of LAION-440M, SemDeDup outperforms the baseline model in 19 out of the 30 tasks. Fig. (A4) and Fig. (A5) show the performance of different models as a function of training dataset size.

Appendix C LAION-233M De-duplication

To support our results on LAION-440M, we also de-duplicate a much smaller dataset of 233 million images. We call this dataset LAION-233M. Usually, CLIP needs to be trained on more than 400 million images as introduced in , so de-duplicating LAION-233M is more challenging in this respect. We train a baseline model on the 233 million images and two models on 55% of the data, one on a random subset and the other on deduplicated subset using SemDeDup. We trained all the models using the same hyperparameters we used for training on LAION-440M. We show ImageNet top1 zeroshot accuracy for these models in Fig. A6. The baseline model achieved 64.62% accuracy, while the SemDeDup model achieved 63.61% outperforming the model trained on the random subset (61.3% accuracy).

Appendix D Visualizing Examples Before and After De-duplication

To visually show which images are removed by SemDeDup from LAION440M dataset, we visualize some images from a random cluster before and after deduplication. To do that, we choose a cluster randomly and sort its examples by the cosine similarity to the centroid. By doing that, we can show similar images next to each other in a sequence. Then we visualize a sequence of images before de-duplication. After that, we run SemDeDup, remove duplicates, and sort the remaining examples again. Finally, we visualize the sequence of images from the same indices we visualize before de-duplication. Figures (A10 and A11) show that after applying SemDeDup with different values for the de-duplication threshold $\epsilon$, we keep the unique images.

Appendix E Perplexity Values for SemDeDup on Language Modeling

Appendix F Qualitative Examples of SemDeDup on C4

Appendix G K-means Clustering Details

We use the $faiss$ library for clustering. $faiss$ is a library for efficient clustering on millions of vectors with GPU support. We use Spherical k-means as we found it better for clustering on ImageNet. Spherical k-means normalizes the cluster centroids after every iteration to have a unit length. This requires the data to also be normalized before clustering. In all our experiments, we run 100 clustering iterations for LAION440M and 20 iterations for C4. We found that centroids do not move after this number of iterations.