MixMatch: A Holistic Approach to Semi-Supervised Learning

Introduction

Much of the recent success in training large, deep neural networks is thanks in part to the existence of large labeled datasets. Yet, collecting labeled data is expensive for many learning tasks because it necessarily involves expert knowledge. This is perhaps best illustrated by medical tasks where measurements call for expensive machinery and labels are the fruit of a time-consuming analysis that draws from multiple human experts. Furthermore, data labels may contain private information. In comparison, in many tasks it is much easier or cheaper to obtain unlabeled data.

Semi-supervised learning (SSL) seeks to largely alleviate the need for labeled data by allowing a model to leverage unlabeled data. Many recent approaches for semi-supervised learning add a loss term which is computed on unlabeled data and encourages the model to generalize better to unseen data. In much recent work, this loss term falls into one of three classes (discussed further in Section 2): entropy minimization —which encourages the model to output confident predictions on unlabeled data; consistency regularization—which encourages the model to produce the same output distribution when its inputs are perturbed; and generic regularization—which encourages the model to generalize well and avoid overfitting the training data.

In this paper, we introduce $\operatorname{MixMatch}$, an SSL algorithm which introduces a single loss that gracefully unifies these dominant approaches to semi-supervised learning. Unlike previous methods, $\operatorname{MixMatch}$ targets all the properties at once which we find leads to the following benefits:

Experimentally, we show that $\operatorname{MixMatch}$ obtains state-of-the-art results on all standard image benchmarks (section 4.2), and reducing the error rate on CIFAR-10 by a factor of 4;

We further show in an ablation study that $\operatorname{MixMatch}$ is greater than the sum of its parts;

We demonstrate in section 4.3 that $\operatorname{MixMatch}$ is useful for differentially private learning, enabling students in the PATE framework to obtain new state-of-the-art results that simultaneously strengthen both privacy guarantees and accuracy.

In short, $\operatorname{MixMatch}$ introduces a unified loss term for unlabeled data that seamlessly reduces entropy while maintaining consistency and remaining compatible with traditional regularization techniques.

Related Work

To set the stage for $\operatorname{MixMatch}$, we first introduce existing methods for SSL. We focus mainly on those which are currently state-of-the-art and that $\operatorname{MixMatch}$ builds on; there is a wide literature on SSL techniques that we do not discuss here (e.g., “transductive” models , graph-based methods , generative modeling , etc.). More comprehensive overviews are provided in . In the following, we will refer to a generic model ${\rm p}_{\rm{model}}(y\mid x;\theta)$ which produces a distribution over class labels $y$ for an input $x$ with parameters $\theta$.

A common regularization technique in supervised learning is data augmentation, which applies input transformations assumed to leave class semantics unaffected. For example, in image classification, it is common to elastically deform or add noise to an input image, which can dramatically change the pixel content of an image without altering its label . Roughly speaking, this can artificially expand the size of a training set by generating a near-infinite stream of new, modified data. Consistency regularization applies data augmentation to semi-supervised learning by leveraging the idea that a classifier should output the same class distribution for an unlabeled example even after it has been augmented. More formally, consistency regularization enforces that an unlabeled example $x$ should be classified the same as $\operatorname{Augment}(x)$, an augmentation of itself.

In the simplest case, for unlabeled points $x$, prior work adds the loss term

Note that $\operatorname{Augment}(x)$ is a stochastic transformation, so the two terms in eq. 1 are not identical. “Mean Teacher” replaces one of the terms in eq. 1 with the output of the model using an exponential moving average of model parameter values. This provides a more stable target and was found empirically to significantly improve results. A drawback to these approaches is that they use domain-specific data augmentation strategies. “Virtual Adversarial Training” (VAT) addresses this by instead computing an additive perturbation to apply to the input which maximally changes the output class distribution. MixMatch utilizes a form of consistency regularization through the use of standard data augmentation for images (random horizontal flips and crops).

2 Entropy Minimization

A common underlying assumption in many semi-supervised learning methods is that the classifier’s decision boundary should not pass through high-density regions of the marginal data distribution. One way to enforce this is to require that the classifier output low-entropy predictions on unlabeled data. This is done explicitly in with a loss term which minimizes the entropy of ${\rm p}_{\rm{model}}(y\mid x;\theta)$ for unlabeled data $x$. This form of entropy minimization was combined with VAT in to obtain stronger results. “Pseudo-Label” does entropy minimization implicitly by constructing hard (1-hot) labels from high-confidence predictions on unlabeled data and using these as training targets in a standard cross-entropy loss. MixMatch also implicitly achieves entropy minimization through the use of a “sharpening” function on the target distribution for unlabeled data, described in section 3.2.

3 Traditional Regularization

Regularization refers to the general approach of imposing a constraint on a model to make it harder to memorize the training data and therefore hopefully make it generalize better to unseen data . We use weight decay which penalizes the $L_{2}$ norm of the model parameters . We also use $\operatorname{MixUp}$ in $\operatorname{MixMatch}$ to encourage convex behavior “between” examples. We utilize $\operatorname{MixUp}$ as both as a regularizer (applied to labeled datapoints) and a semi-supervised learning method (applied to unlabeled datapoints). $\operatorname{MixUp}$ has been previously applied to semi-supervised learning; in particular, the concurrent work of uses a subset of the methodology used in MixMatch. We clarify the differences in our ablation study (section 4.2.3).

MixMatch

In this section, we introduce $\operatorname{MixMatch}$, our proposed semi-supervised learning method. $\operatorname{MixMatch}$ is a “holistic” approach which incorporates ideas and components from the dominant paradigms for SSL discussed in section 2. Given a batch $\mathcal{X}$ of labeled examples with one-hot targets (representing one of $L$ possible labels) and an equally-sized batch $\mathcal{U}$ of unlabeled examples, $\operatorname{MixMatch}$ produces a processed batch of augmented labeled examples $\mathcal{X}^{\prime}$ and a batch of augmented unlabeled examples with “guessed” labels $\mathcal{U}^{\prime}$. $\mathcal{U}^{\prime}$ and $\mathcal{X}^{\prime}$ are then used in computing separate labeled and unlabeled loss terms. More formally, the combined loss $\mathcal{L}$ for semi-supervised learning is defined as

where $\operatorname{H}(p,q)$ is the cross-entropy between distributions $p$ and $q$, and $T$, $K$, $\alpha$, and $\lambda_{\mathcal{U}}$ are hyperparameters described below. The full $\operatorname{MixMatch}$ algorithm is provided in algorithm 1, and a diagram of the label guessing process is shown in fig. 1. Next, we describe each part of $\operatorname{MixMatch}$.

As is typical in many SSL methods, we use data augmentation both on labeled and unlabeled data. For each $x_{b}$ in the batch of labeled data $\mathcal{X}$, we generate a transformed version $\hat{x}_{b}=\operatorname{Augment}(x_{b})$ (algorithm 1, line 3). For each $u_{b}$ in the batch of unlabeled data $\mathcal{U}$, we generate $K$ augmentations $\hat{u}_{b,k}=\operatorname{Augment}(u_{b}),k\in(1,\ldots,K)$ (algorithm 1, line 5). We use these individual augmentations to generate a “guessed label” $q_{b}$ for each $u_{b}$, through a process we describe in the following subsection.

2 Label Guessing

For each unlabeled example in $\mathcal{U}$, $\operatorname{MixMatch}$ produces a “guess” for the example’s label using the model’s predictions. This guess is later used in the unsupervised loss term. To do so, we compute the average of the model’s predicted class distributions across all the $K$ augmentations of $u_{b}$ by

in algorithm 1, line 7. Using data augmentation to obtain an artificial target for an unlabeled example is common in consistency regularization methods .

In generating a label guess, we perform one additional step inspired by the success of entropy minimization in semi-supervised learning (discussed in section 2.2). Given the average prediction over augmentations $\bar{q}_{b}$, we apply a sharpening function to reduce the entropy of the label distribution. In practice, for the sharpening function, we use the common approach of adjusting the “temperature” of this categorical distribution , which is defined as the operation

where $p$ is some input categorical distribution (specifically in $\operatorname{MixMatch}$, $p$ is the average class prediction over augmentations $\bar{q}_{b}$, as shown in algorithm 1, line 8) and $T$ is a hyperparameter. As $T\rightarrow 0$, the output of $\operatorname{Sharpen}(p,T)$ will approach a Dirac (“one-hot”) distribution. Since we will later use $q_{b}=\operatorname{Sharpen}(\bar{q}_{b},T)$ as a target for the model’s prediction for an augmentation of $u_{b}$, lowering the temperature encourages the model to produce lower-entropy predictions.

3 MixUp

We use $\operatorname{MixUp}$ for semi-supervised learning, and unlike past work for SSL we mix both labeled examples and unlabeled examples with label guesses (generated as described in section 3.2). To be compatible with our separate loss terms, we define a slightly modified version of $\operatorname{MixUp}$. For a pair of two examples with their corresponding labels probabilities $(x_{1},p_{1}),(x_{2},p_{2})$ we compute $(x^{\prime},p^{\prime})$ by

where $\alpha$ is a hyperparameter. Vanilla $\operatorname{MixUp}$ omits eq. 9 (i.e. it sets $\lambda^{\prime}=\lambda$). Given that labeled and unlabeled examples are concatenated in the same batch, we need to preserve the order of the batch to compute individual loss components appropriately. This is achieved by eq. 9 which ensures that $x^{\prime}$ is closer to $x_{1}$ than to $x_{2}$. To apply $\operatorname{MixUp}$, we first collect all augmented labeled examples with their labels and all unlabeled examples with their guessed labels into

(algorithm 1, lines 10–11). Then, we combine these collections and shuffle the result to form $\mathcal{W}$ which will serve as a data source for $\operatorname{MixUp}$ (algorithm 1, line 12). For each the $i^{th}$ example-label pair in $\hat{\mathcal{X}}$, we compute $\operatorname{MixUp}(\hat{\mathcal{X}}_{i},\mathcal{W}_{i})$ and add the result to the collection $\mathcal{X}^{\prime}$ (algorithm 1, line 13). We compute $\mathcal{U}^{\prime}_{i}=\operatorname{MixUp}(\hat{\mathcal{U}}_{i},\mathcal{W}_{i+|\hat{\mathcal{X}}|})$ for $i\in(1,\ldots,|\hat{\mathcal{U}}|)$, intentionally using the remainder of $\mathcal{W}$ that was not used in the construction of $\mathcal{X}^{\prime}$ (algorithm 1, line 14). To summarize, $\operatorname{MixMatch}$ transforms $\mathcal{X}$ into $\mathcal{X}^{\prime}$, a collection of labeled examples which have had data augmentation and $\operatorname{MixUp}$ (potentially mixed with an unlabeled example) applied. Similarly, $\mathcal{U}$ is transformed into $\mathcal{U}^{\prime}$, a collection of multiple augmentations of each unlabeled example with corresponding label guesses.

4 Loss Function

Given our processed batches $\mathcal{X}^{\prime}$ and $\mathcal{U}^{\prime}$, we use the standard semi-supervised loss shown in eqs. 3, 4 and 5. Equation 5 combines the typical cross-entropy loss between labels and model predictions from $\mathcal{X}^{\prime}$ with the squared $L_{2}$ loss on predictions and guessed labels from $\mathcal{U}^{\prime}$. We use this $L_{2}$ loss in eq. 4 (the multiclass Brier score ) because, unlike the cross-entropy, it is bounded and less sensitive to incorrect predictions. For this reason, it is often used as the unlabeled data loss in SSL as well as a measure of predictive uncertainty . We do not propagate gradients through computing the guessed labels, as is standard

5 Hyperparameters

Since $\operatorname{MixMatch}$ combines multiple mechanisms for leveraging unlabeled data, it introduces various hyperparameters – specifically, the sharpening temperature $T$, number of unlabeled augmentations $K$, $\alpha$ parameter for $\operatorname{Beta}$ in $\operatorname{MixUp}$, and the unsupervised loss weight $\lambda_{\mathcal{U}}$. In practice, semi-supervised learning methods with many hyperparameters can be problematic because cross-validation is difficult with small validation sets . However, we find in practice that most of $\operatorname{MixMatch}$’s hyperparameters can be fixed and do not need to be tuned on a per-experiment or per-dataset basis. Specifically, for all experiments we set $T=0.5$ and $K=2$. Further, we only change $\alpha$ and $\lambda_{\mathcal{U}}$ on a per-dataset basis; we found that $\alpha=0.75$ and $\lambda_{\mathcal{U}}=100$ are good starting points for tuning. In all experiments, we linearly ramp up $\lambda_{\mathcal{U}}$ to its maximum value over the first $16{,}000$ steps of training as is common practice .

Experiments

We test the effectiveness of $\operatorname{MixMatch}$ on standard SSL benchmarks (section 4.2). Our ablation study teases apart the contribution of each of $\operatorname{MixMatch}$’s components (section 4.2.3). As an additional application, we consider privacy-preserving learning in section 4.3.

Unless otherwise noted, in all experiments we use the “Wide ResNet-28” model from . Our implementation of the model and training procedure closely matches that of (including using 5000 examples to select the hyperparameters), except for the following differences: First, instead of decaying the learning rate, we evaluate models using an exponential moving average of their parameters with a decay rate of $0.999$. Second, we apply a weight decay of $0.0004$ at each update for the Wide ResNet-28 model. Finally, we checkpoint every $2^{16}$ training samples and report the median error rate of the last 20 checkpoints. This simplifies the analysis at a potential cost to accuracy by, for example, averaging checkpoints or choosing the checkpoint with the lowest validation error.

2 Semi-Supervised Learning

First, we evaluate the effectiveness of $\operatorname{MixMatch}$ on four standard benchmark datasets: CIFAR-10 and CIFAR-100 , SVHN , and STL-10 . Standard practice for evaluating semi-supervised learning on the first three datasets is to treat most of the dataset as unlabeled and use a small portion as labeled data. STL-10 is a dataset specifically designed for SSL, with 5,000 labeled images and 100,000 unlabeled images which are drawn from a slightly different distribution than the labeled data.

As baselines, we consider the four methods considered in ($\Pi$-Model , Mean Teacher , Virtual Adversarial Training , and Pseudo-Label ) which are described in section 2. We also use $\operatorname{MixUp}$ on its own as a baseline. $\operatorname{MixUp}$ is designed as a regularizer for supervised learning, so we modify it for SSL by applying it both to augmented labeled examples and augmented unlabeled examples with their corresponding predictions. In accordance with standard usage of $\operatorname{MixUp}$, we use a cross-entropy loss between the $\operatorname{MixUp}$-generated guess label and the model’s prediction. As advocated by , we reimplemented each of these methods in the same codebase and applied them to the same model (described in section 4.1) to ensure a fair comparison. We re-tuned the hyperparameters for each baseline method, which generally resulted in a marginal accuracy improvement compared to those in , thereby providing a more competitive experimental setting for testing out $\operatorname{MixMatch}$.

2.2 Results

For CIFAR-10, we evaluate the accuracy of each method with a varying number of labeled examples from $250$ to $4000$ (as is standard practice). The results can be seen in fig. 2. We used $\lambda_{\mathcal{U}}=75$ for CIFAR-10. We created 5 splits for each number of labeled points, each with a different random seed. Each model was trained on each split and the error rates were reported by the mean and variance across splits. We find that $\operatorname{MixMatch}$ outperforms all other methods by a significant margin, for example reaching an error rate of $6.24\%$ with $4000$ labels. For reference, on the same model, fully supervised training on all $50000$ samples achieves an error rate of $4.17\%$. Furthermore, $\operatorname{MixMatch}$ obtains an error rate of $11.08\%$ with only $250$ labels. For comparison, at $250$ labels the next-best-performing method (VAT ) achieves an error rate of $36.03$, over $4.5\times$ higher than $\operatorname{MixMatch}$ considering that $4.17\%$ is the error limit obtained on our model with fully supervised learning. In addition, at $4000$ labels the next-best-performing method (Mean Teacher ) obtains an error rate of $10.36\%$, which suggests that $\operatorname{MixMatch}$ can achieve similar performance with only $1/16$ as many labels. We believe that the most interesting comparisons are with very few labeled data points since it reveals the method’s sample efficiency which is central to SSL.

Some prior work has also considered the use of a larger, $26$ million-parameter model. Our base model, as used in , has only $1.5$ million parameters which confounds comparison with these results. For a more reasonable comparison to these results, we measure the effect of increasing the width of our base ResNet model and evaluate $\operatorname{MixMatch}$’s performance on a 28-layer Wide Resnet model which has $135$ filters per layer, resulting in $26$ million parameters. We also evaluate $\operatorname{MixMatch}$ on this larger model on CIFAR-100 with $10000$ labels, to compare to the corresponding result from . The results are shown in table 2. In general, $\operatorname{MixMatch}$ matches or outperforms the best results from , though we note that the comparison still remains problematic due to the fact that the model from also uses more sophisticated “shake-shake” regularization . For this model, we used a weight decay of $0.0008$. We used $\lambda_{\mathcal{U}}=75$ for CIFAR-10 and $\lambda_{\mathcal{U}}=150$ for CIFAR-100.

As with CIFAR-10, we evaluate the performance of each SSL method on SVHN with a varying number of labels from $250$ to $4000$. As is standard practice, we first consider the setting where the $73257$-example training set is split into labeled and unlabeled data. The results are shown in fig. 3. We used $\lambda_{\mathcal{U}}=250$. Here again the models were evaluated on 5 splits for each number of labeled points, each with a different random seed. We found $\operatorname{MixMatch}$’s performance to be relatively constant (and better than all other methods) across all amounts of labeled data. Surprisingly, after additional tuning we were able to obtain extremely good performance from Mean Teacher , though its error rate was consistently slightly higher than $\operatorname{MixMatch}$’s.

Note that SVHN has two training sets: train and extra. In fully-supervised learning, both sets are concatenated to form the full training set ($604388$ samples). In SSL, for historical reasons the extra set was left aside and only train was used ($73257$ samples). We argue that leveraging both train and extra for the unlabeled data is more interesting since it exhibits a higher ratio of unlabeled samples over labeled ones. We report error rates for both SVHN and SVHN+Extra in table 3. For SVHN+Extra we used $\alpha=0.25,\lambda_{\mathcal{U}}=250$ and a lower weight decay of $0.000002$ due to the larger amount of available data. We found that on both training sets, $\operatorname{MixMatch}$ nearly matches the fully-supervised performance on the same training set almost immediately – for example, $\operatorname{MixMatch}$ achieves an error rate of $2.22\%$ with only 250 labels on SVHN+Extra compared to the fully-supervised performance of $1.71\%$. Interestingly, on SVHN+Extra $\operatorname{MixMatch}$ outperformed fully supervised training on SVHN without extra ($2.59\%$ error) for every labeled data amount considered. To emphasize the importance of this, consider the following scenario: You have $73257$ examples from SVHN with $250$ examples labeled and are given a choice: You can either obtain $8\times$ more unlabeled data and use $\operatorname{MixMatch}$ or obtain $293\times$ more labeled data and use fully-supervised learning. Our results suggest that obtaining additional unlabeled data and using $\operatorname{MixMatch}$ is more effective, which conveniently is likely much cheaper than obtaining $293\times$ more labels.

STL-10 contains $5000$ training examples aimed at being used with $10$ predefined folds (we use the first 5 only) with $1000$ examples each. However, some prior work trains on all $5000$ examples. We thus compare in both experimental settings. With $1000$ examples $\operatorname{MixMatch}$ surpasses both the state-of-the-art for $1000$ examples as well as the state-of-the-art using all $5000$ labeled examples. Note that none of the baselines in table 2 use the same experimental setup (i.e. model), so it is difficult to directly compare the results; however, because $\operatorname{MixMatch}$ obtains the lowest error by a factor of two, we take this to be a vote in confidence of our method. We used $\lambda_{\mathcal{U}}=50$.

2.3 Ablation Study

Since $\operatorname{MixMatch}$ combines various semi-supervised learning mechanisms, it has a good deal in common with existing methods in the literature. As a result, we study the effect of removing or adding components in order to provide additional insight into what makes $\operatorname{MixMatch}$ performant. Specifically, we measure the effect of

using the mean class distribution over $K$ augmentations or using the class distribution for a single augmentation (i.e. setting $K=1$)

removing temperature sharpening (i.e. setting $T=1$)

using an exponential moving average (EMA) of model parameters when producing guessed labels, as is done by Mean Teacher

performing $\operatorname{MixUp}$ between labeled examples only, unlabeled examples only, and without mixing across labeled and unlabeled examples

using Interpolation Consistency Training , which can be seen as a special case of this ablation study where only unlabeled mixup is used, no sharpening is applied and EMA parameters are used for label guessing.

We carried out the ablation on CIFAR-10 with $250$ and $4000$ labels; the results are shown in table 4. We find that each component contributes to $\operatorname{MixMatch}$’s performance, with the most dramatic differences in the $250$-label setting. Despite Mean Teacher’s effectiveness on SVHN (fig. 3), we found that using a similar EMA of parameter values hurt $\operatorname{MixMatch}$’s performance slightly.

3 Privacy-Preserving Learning and Generalization

Learning with privacy allows us to measure our approach’s ability to generalize. Indeed, protecting the privacy of training data amounts to proving that the model does not overfit: a learning algorithm is said to be differentially private (the most widely accepted technical definition of privacy) if adding, modifying, or removing any of its training samples is guaranteed not to result in a statistically significant difference in the model parameters learned . For this reason, learning with differential privacy is, in practice, a form of regularization . Each training data access constitutes a potential privacy leakage, encoded as the pair of the input and its label. Hence, approaches for deep learning from private training data, such as DP-SGD and PATE , benefit from accessing as few labeled private training points as possible when computing updates to the model parameters. Semi-supervised learning is a natural fit for this setting.

We use the PATE framework for learning with privacy. A student is trained in a semi-supervised way from public unlabeled data, part of which is labeled by an ensemble of teachers with access to private labeled training data. The fewer labels a student requires to reach a fixed accuracy, the stronger is the privacy guarantee it provides. Teachers use a noisy voting mechanism to respond to label queries from the student, and they may choose not to provide a label when they cannot reach a sufficiently strong consensus. For this reason, if $\operatorname{MixMatch}$ improves the performance of PATE, it would also illustrate $\operatorname{MixMatch}$’s improved generalization from few canonical exemplars of each class.

We compare the accuracy-privacy trade-off achieved by $\operatorname{MixMatch}$ to a VAT baseline on SVHN. VAT achieved the previous state-of-the-art of $91.6\%$ test accuracy for a privacy loss of $\varepsilon=4.96$ . Because $\operatorname{MixMatch}$ performs well with few labeled points, it is able to achieve $95.21\pm 0.17\%$ test accuracy for a much smaller privacy loss of $\varepsilon=0.97$. Because $e^{\varepsilon}$ is used to measure the degree of privacy, the improvement is approximately $e^{4}\approx 55\times$, a significant improvement. A privacy loss $\varepsilon$ below 1 corresponds to a much stronger privacy guarantee. Note that in the private training setting the student model only uses 10,000 total examples.

Conclusion

We introduced $\operatorname{MixMatch}$, a semi-supervised learning method which combines ideas and components from the current dominant paradigms for SSL. Through extensive experiments on semi-supervised and privacy-preserving learning, we found that $\operatorname{MixMatch}$ exhibited significantly improved performance compared to other methods in all settings we studied, often by a factor of two or more reduction in error rate. In future work, we are interested in incorporating additional ideas from the semi-supervised learning literature into hybrid methods and continuing to explore which components result in effective algorithms. Separately, most modern work on semi-supervised learning algorithms is evaluated on image benchmarks; we are interested in exploring the effectiveness of $\operatorname{MixMatch}$ in other domains.

We would like to thank Balaji Lakshminarayanan for his helpful theoretical insights.

References

Appendix A Notation and definitions

Appendix B Tabular results

Training the same model with supervised learning on the entire $50000$-example training set achieved an error rate of $4.13\%$.

B.2 SVHN

Training the same model with supervised learning on the entire $73257$-example training set achieved an error rate of $2.59\%$.

B.3 SVHN+Extra

Training the same model with supervised learning on the entire $604388$-example training set achieved an error rate of $1.71\%$.

Appendix C 13-layer ConvNet results

Early work on semi-supervised learning used a 13-layer convolutional network architecture . In table 8 we present results on a similar architecture. We caution against comparing these numbers directly to previous work as we use a different implementation and training process .