LEEP: A New Measure to Evaluate Transferability of Learned Representations

Introduction

Transferability estimation (Eaton et al., 2008; Ammar et al., 2014; Sinapov et al., 2015) is the problem of quantitatively estimating how easy it is to transfer knowledge learned from one classification task to another. Specifically, given a source task, represented by a labeled data set or a pre-trained model, and a target task, represented by a labeled data set, transferability estimation aims to develop a measure (or a score) that can tell us, ideally without training on the target task, how effectively transfer learning algorithms can transfer knowledge from the source task to the target task.

Answering this question is important, since good estimations of transferability can help understand the relationships between tasks (Tran et al., 2019), select groups of highly transferable tasks for joint training (Zamir et al., 2018), or choose good source models for a given target task (Achille et al., 2019; Bao et al., 2019; Bhattacharjee et al., 2019). Previous approaches to transferability estimation often require running a transfer learning algorithm that involves expensive parameter optimization (Zamir et al., 2018; Achille et al., 2019), do not have a simple interpretation (Bao et al., 2019), or make strong assumptions about the data sets that limit their applicability (Zamir et al., 2018; Tran et al., 2019).

We propose a novel measure called the Log Expected Empirical Prediction (LEEP) for transferability estimation of deep networks that overcomes all the shortcomings above. In contrast to previous approaches, LEEP scores are obtained without training on the target task, thus avoiding the expensive parameter optimization step. Additionally, they have a simple interpretation and can be applied in general settings to a wide range of modern deep networks.

In particular, LEEP scores are obtained from a source model and a target data set by making a single forward pass of the model through the target data. This is a simpler process than previous methods, such as Taskonomy (Zamir et al., 2018) and Task2Vec (Achille et al., 2019), where one has to re-train at least part of the source model on the target data set. Furthermore, LEEP has a simple interpretation: it is the average log-likelihood of the expected empirical predictor, a simple classifier that makes prediction based on the expected empirical conditional distribution between source and target labels. Finally, LEEP does not make any assumption on the source and target input samples, except that they have the same size. This is more general and applicable than previous work (Zamir et al., 2018; Tran et al., 2019) where source and target data sets were assumed to share the same input samples.

Contributions. We formally define LEEP and rigorously analyze it, both theoretically and empirically. We show two theoretical properties of the measure: (1) LEEP is upper bounded by the average log-likelihood of the optimal model, obtained by re-training the head classifier while freezing the feature extractor; (2) LEEP is related to the negative conditional entropy measure proposed by Tran et al. (2019).

We conduct extensive experiments to evaluate our LEEP measure in several scenarios. We show that the measure is useful for predicting the performance of two commonly used transfer learning algorithms – head classifier re-training (Donahue et al., 2014; Razavian et al., 2014) and model fine-tuning (Agrawal et al., 2014; Girshick et al., 2014) – not only for large target data sets, but also for small or imbalanced target data sets that are difficult to use for re-training. We also show that LEEP can predict the convergence speed of the fine-tuning method for transfer learning.

We further demonstrate that LEEP can predict the performance of a recently developed meta-transfer learning method, the Conditional Neural Adaptive Processes (Requeima et al., 2019). Meta-transfer learning (Wei et al., 2018b; Sun et al., 2019; Requeima et al., 2019) is a framework for learning to transfer using several meta-training tasks. Importantly, to our knowledge, our work is the first to develop a transferability measure for meta-transfer learning.

We empirically compare our method with the very recent negative conditional entropy measure (Tran et al., 2019) and H scores (Bao et al., 2019). Our comparisons show that LEEP better correlates with the actual transfer accuracy than these methods. Finally, we demonstrate the effectiveness of LEEP for the source model selection problem in comparison with the negative conditional entropy and H scores.

Log Expected Empirical Prediction

The transfer learning setting above is very common in computer vision for leveraging expensive pre-trained deep learning models. For instance, a popular transfer learning method takes a model, pre-trained on ImageNet, and re-trains its classifier (the head) on the target data set while freezing the feature extractor. The result is a new model for the target task that uses the representation learned on the source with a head classifier learned on the target (Donahue et al., 2014; Razavian et al., 2014; Zeiler & Fergus, 2014; Oquab et al., 2014; Whatmough et al., 2019). We call this the head re-training method. Another popular transfer learning method, called the fine-tuning method, fine-tunes the feature extractor while re-training the new head classifier on the target data set to get a model for the target task (Agrawal et al., 2014; Girshick et al., 2014; Chatfield et al., 2014; Dhillon et al., 2020).

In this work, we study the transferability estimation problem, which aims to develop a measure (or a score) that can tell us, without training on the target data set, how effectively these transfer learning algorithms can transfer knowledge learned in the source model $\theta$ to the target task, using the target data set $\mathcal{D}$. We emphasize that although transferability estimation can tell us the effectiveness of transfer learning, it is not a solution for transfer learning in itself. Instead, transfer learning is achieved by e.g., the head re-training or fine-tuning methods above. In our experiments in Sec. 5, we will use these transfer learning methods to test our transferability measure.

We now describe our proposed measure, LEEP, that requires no expensive training on the target task and offers an a priori estimate of how well a model will transfer to the target task. The measure can be efficiently computed from the source model $\theta$ and the target data set $\mathcal{D}$. The computation involves the following three steps.

Step 1: Compute dummy label distributions of the inputs in the target data set $\mathcal{D}$. We apply $\theta$ to each input $x_{i}$ in $\mathcal{D}$ to get the predicted distribution over $\mathcal{Z}$, the label set of the source task. We denote this predicted distribution by $\theta(x_{i})$, which is a categorical distribution over $\mathcal{Z}$. Note that $\theta(x_{i})$ is a dummy distribution over labels of the source task, since these labels may not be meaningful for the example $x_{i}$. For instance, if $\theta$ is a model pre-trained on ImageNet and $\mathcal{D}$ is the CIFAR data set, then $\theta(x_{i})$ is a distribution over ImageNet labels, which may not be semantically related to the true label of $x_{i}$ in the CIFAR data set.

Step 2: Compute the empirical conditional distribution $\hat{P}(y|z)$ of the target label $y$ given the source label $z$. We next compute the empirical conditional distribution $\hat{P}(y|z)$ for all $(y,z)\in\mathcal{Y}\times\mathcal{Z}$. To this end, we first compute the empirical joint distribution $\hat{P}(y,z)$ for all label pairs $(y,z)\in\mathcal{Y}\times\mathcal{Z}$:

where the summation $\sum_{i:y_{i}=y}$ means we sum over all indices $i\in\{1,2,\ldots,n\}$ that satisfy $y_{i}=y$. In the above equation, $\theta(x_{i})_{z}$ is the probability of the label $z$ according to the categorical distribution $\theta(x_{i})$.

From this empirical joint distribution, we can compute the empirical marginal distribution $\hat{P}(z)$:

and then the empirical conditional distribution $\hat{P}(y|z)$:

Step 3: Compute LEEP using $\theta(x)$ and $\hat{P}(y|z)$. For any input $x\in\mathcal{X}$, consider a classifier that predicts a label $y$ of $x$ by first randomly drawing a dummy label $z$ from $\theta(x)$ and then randomly drawing $y$ from $\hat{P}(y|z)$. Equivalently, this classifier can predict $y$ by directly drawing a label from the distribution $p(y|x;\theta,\mathcal{D})=\sum_{z\in\mathcal{Z}}\hat{P}(y|z)~{}\theta(x)_{z}$. We call this classifier the Expected Empirical Predictor (EEP).

For the target data set $\mathcal{D}$, we define LEEP as the average log-likelihood of the EEP classifier given the data $\mathcal{D}$. Formally, our LEEP measure of transferability is defined as:

Intuitively, this measure tells us how well the EEP performs on $\mathcal{D}$. We argue that since the EEP is constructed mainly from the source model $\theta$ with a minimal use of $\mathcal{D}$ (i.e., $\mathcal{D}$ is only used to compute the simple empirical conditional distribution), it can serve as an indicator of how “close” $\theta$ and $\mathcal{D}$ are, and hence a measure of transferability.

From its definition, the LEEP measure is always negative and larger values (i.e., smaller absolute values) indicate better transferability. When the target task contains more classes, LEEP scores tend to be smaller. The measure is also efficient to compute since its computational bottleneck (step 1 above) requires only a single forward pass through the target data set $\mathcal{D}$.

Theoretical Properties of LEEP

Assume $\theta=(w,h)$ where $w$ is a feature extractor that maps an input $x\in\mathcal{X}$ to a representation (or embedding), $r=w(x)$, and $h$ is a classifier (or head) that takes the representation $r$ as input and returns a probability distribution over $\mathcal{Z}$. Next, assume we fix $w$ and re-train the classifier by maximum likelihood, using the target data set $\mathcal{D}$, to obtain the new classifier $k^{*}$. That is,

where $l(w,k)=\frac{1}{n}\sum_{i=1}^{n}\log p(y_{i}|x_{i};w,k)$ is the average log-likelihood of $(w,k)$ on the target data set $\mathcal{D}$, and $\mathcal{K}$ is the space of classifiers from which we would like to select $k$. We note that $\mathcal{K}$ contains classifiers that map $w(x)$ to labels in $\mathcal{Y}$. If we assume $\mathcal{K}$ contains the EEP, we can easily show the following property of the LEEP measure (see proof in Appendix A):

LEEP is a lower bound of the optimal average log-likelihood. Formally, $T(\theta,\mathcal{D})\leq l(w,k^{*})$.

The assumption that $\mathcal{K}$ contains the EEP can be easily satisfied if we select $\mathcal{K}=\bar{\mathcal{K}}\cup\{k_{\text{EEP}}\}$, where the classifier $k_{\text{EEP}}$ is the EEP and $\bar{\mathcal{K}}$ is a space of classifiers that we can easily optimize over (e.g., the parameter space of a linear classifier). We can then solve Eq. (3) by the following two-stage process. First, we solve $\bar{k}=\arg\max_{k\in\bar{\mathcal{K}}}~{}l(w,k)$ using, for instance, stochastic gradient descent (SGD) (Robbins & Monro, 1951; Bottou, 1991). Then, we solve $k^{*}=\arg\max_{k\in\{\bar{k},k_{\text{EEP}}\}}l(w,k)$ by simple comparison. In our experiments, we skip this second step and choose $k^{*}=\bar{k}$, as usually done in practice. The results reported in Sec. 5.1 show that LEEP correlates well with the test accuracy of $\bar{k}$, indicating that the measure can be used to estimate the performance of the transferred model in practice.

We note that in the setting of Tran et al. (2019), the source data set is given, instead of the source model $\theta$. Hence, NCE is a more natural measure of transferability in their case. In contrast, LEEP is more natural for our setting, since we are only given the source model $\theta$. In return, by assuming a source model, LEEP is not restricted to their setting, where the two tasks are defined over the exact same input instances.

Related Work

Our work is related to several research areas in machine learning and computer vision, including transfer learning (Weiss et al., 2016; Yang et al., 2020), meta-transfer learning (Wei et al., 2018b; Sun et al., 2019; Requeima et al., 2019), task space modeling (Zamir et al., 2018; Achille et al., 2019), and domain adaptation (Sun et al., 2016; Azizzadenesheli et al., 2019). We discuss below previous work that is closely related to ours.

Transfer learning. Our paper addresses the problem of predicting the performance of transfer learning algorithms between two classification tasks, without actually executing these algorithms. This problem is also called transferability estimation between classification tasks (Bao et al., 2019; Tran et al., 2019). Early theoretical work in transfer and multi-task learning studied the relatedness between tasks and proposed several types of distances between tasks. These distances include the $\mathcal{F}$-relatedness (Ben-David & Schuller, 2003), $\mathcal{A}$-distance (Kifer et al., 2004; Ben-David et al., 2007), and discrepancy distance (Mansour et al., 2009). Although useful for theoretical analysis, these approaches are unsuited for measuring transferability in practice because they cannot be computed easily and are symmetric. Transferability measures should be non-symmetric since transferring from one task to another (e.g., from a hard task to an easy one) is different from transferring in the reverse direction (e.g., from the easy task to the hard one).

More recently, Azizzadenesheli et al. (2019) studied domain adaptation under label shift. The authors considered the case where only the marginal label distribution changes between domains. This assumption is more restrictive than the setting in our paper as we allow both the input distribution and the label distribution to change arbitrarily. Task similarity was also considered for Gaussian process based transfer learning between regression tasks (Cao et al., 2010; Wei et al., 2018a).

The most related work to ours are transferability measures recently proposed by Tran et al. (2019) and Bao et al. (2019). Tran et al. (2019) developed the negative conditional entropy measure between the source and target label sets, under the assumption that the source and target data sets share the same input examples. Our paper removes this requirement and allows the input data to come from arbitrarily different distributions. Bao et al. (2019) developed a transferability measure based on H scores, which are derived from information-theoretic principles. Although, like us, their transferability measure can be applied to general settings, their measure is hard to interpret since it involves solving a Hirschfeld-Gebelein-Rényi maximum correlation problem (Hirschfeld, 1935; Gebelein, 1941; Rényi, 1959). LEEP scores, on the other hand, have a simple interpretation related to the EEP and are easy to implement and compute.

Meta-transfer learning. Meta-transfer learning is a framework for learning to transfer from a source task to a target task (Wei et al., 2018b; Sun et al., 2019; Requeima et al., 2019). Similar to our transfer learning setting, Sun et al. (2019) and Requeima et al. (2019) also adapted a pre-trained model on the source task to the target task. These meta-learning methods learn the adaptation from several meta-training tasks, which consist of additional target data sets and target test sets from different domains that are intended to mimic the transfer learning scenario, where one wants to transfer knowledge to unseen tasks. Because of the additional data sets, the transfer learning mechanism in meta-learning departs from that of regular transfer learning. Nevertheless, we show that LEEP scores can also predict the performance of meta-transfer learning algorithms, such as the conditional neural adaptive processes (CNAPs) recently proposed by Requeima et al. (2019). To our knowledge, we are the first to develop a transferability measure that can be applied to meta-transfer learning.

Task space representation. Our paper is related to task space representation (Edwards & Storkey, 2017; Zamir et al., 2018; Achille et al., 2019; Jomaa et al., 2019) in the sense that transferability may be estimated from a distance between the representations (or embeddings) of tasks. For instance, Task2Vec (Achille et al., 2019) tried to map tasks (or data sets) to vectors in a vector space. Transferability between tasks was then estimated using a non-symmetric distance between the corresponding vectors. This method requires training a large reference network (called the probe network), adapting it to the target data set, and computing the Fisher information matrix to obtain a task embedding. Our method, on the other hand, is much simpler and computationally more efficient. Additionally, we require only the source model and a small target data set, which are both usually available in practice.

Edwards & Storkey (2017) extended the variational auto-encoder (Kingma & Welling, 2014) to compute statistics of data sets that are useful in a range of applications, including clustering data sets or classifying unseen classes. These statistics, however, were not shown to be useful for transferability estimation. Another related work, Taskonomy (Zamir et al., 2018), created a taxonomy of tasks that revealed task structures useful for reducing the number of labeled training data. Taskonomy involves several steps, one of which requires computing the task affinity matrix by re-training networks on target data sets. LEEP scores can be regarded as an efficient approximation of the task affinity, as we avoid network re-training.

Experiments

We evaluate the ability of LEEP to predict the performance of transfer and meta-transfer learning algorithms, prior to applying these algorithms in practice. We further show that LEEP is useful even in the small or imbalanced data settings, where training on the target task could be hard. We compare LEEP with the state of the art NCE transferability measure of Tran et al. (2019) and H score of Bao et al. (2019). Finally, we demonstrate the use of LEEP for source model selection. Our experiments are implemented in Gluon/MXNet (Chen et al., 2015; Guo et al., 2019).

We show that LEEP scores effectively predict the accuracies of models transferred from source to target tasks. We consider two different source models, each one representing a different source task: ResNet18 (He et al., 2016), which is pre-trained on ImageNet (Russakovsky et al., 2015), and ResNet20 (He et al., 2016), which is pre-trained on CIFAR10 (Krizhevsky, 2009). For each model, we construct 200 different target tasks from the CIFAR100 data set (Krizhevsky, 2009) as follows. The label set of each target task is constructed by randomly drawing a subset of the 100 classes, with the subset’s size ranging from 2 to 100. The target data set then consists of all training examples of the selected classes in the CIFAR100 data set. We use the test examples of these selected classes as the target test set to compute the accuracy of the transferred model.

We experiment with two commonly used transfer learning methods to train a transferred model to a target task:

Re-train head. This method keeps the feature extractor layers of the source model fixed and then trains a new head classifier using the target data set from scratch. We re-train the new classifier by running SGD on the cross entropy loss.

Fine-tune. This method replaces the head classifier of the source model with a new head and then fine-tunes the entire model – the feature extractor and the new head – using the target data set. Fine-tuning is performed again by running SGD on the cross entropy loss.

To clarify, we let the feature extractor be the portion of the source model’s network up to and including the penultimate layer. The head classifier is the network’ last fully connected layer. For each target task, models transferred using these two methods were evaluated on the target test set to obtain the test accuracies. We then compare these accuracies with our LEEP scores evaluated on these target tasks.

In all tests, we ran SGD for 100 epochs with learning rate 0.01 and batch size 10 since they were sufficient to obtain good transferred models. We found that varying the number of epochs does not significantly change the results of our experiments, although in principle, fine-tuning the whole network until convergence could decouple its dependence on the source task.

Fig. 1 shows the LEEP scores and the corresponding test accuracies for transferred models on 200 target tasks. Following the correlation analysis by Nguyen et al. (2019) and Tran et al. (2019), we compute the Pearson correlation coefficients and the $p$ values between LEEP scores and test accuracies, which are shown in Table 1 (see the first two rows of each transfer method). From Fig. 1 and Table 1, LEEP scores clearly correlate with test accuracies, with correlation coefficients higher than 0.94 and $p<0.001$ in all cases. Evidently, LEEP scores are a reliable indicator of the performance of transferred models.

2 LEEP vs. Transfer Accuracy in Small Data Regime

Transfer learning is often performed in cases where target data sets are small. We next evaluate LEEP in such small data settings. We focus on scenarios where the size of the target data set might be insufficient to train a deep model from scratch, but reasonable for transfer learning to be effective. Particularly, we repeat the experiments of Sec. 5.1 with the additional restriction that target data sets contain exactly five random classes and 50 random examples per class. Thus, the target data sets contain only 10% of the training examples per class, compared to the full CIFAR100 data set.

We further add experiments where target data sets are constructed from the FashionMNIST data set (Xiao et al., 2017). In this case, the target data sets are restricted to contain four random classes and 30 random examples per class (i.e., 0.5% of the training examples per class compared to the full data set). Since FashionMNIST contains grayscale images, we duplicate the grayscale channel three times to be consistent with the source model’s input domain. Because the target data set size is small, we re-train the head classifier by running SGD for only 50 epochs.

Table 1 (see the 3${}^{\text{rd}}$ to 6${}^{\text{th}}$ rows of the first two algorithms) shows the correlation coefficients between LEEP scores and test accuracies in these settings. The results indicate a reasonably positive correlations with coefficients greater than 0.5 in most cases, with the exception of fine-tuning from the CIFAR10 pre-trained model. We note that, in general, the number of examples per class in the target data set would affect the effectiveness of LEEP. For instance, if the number of examples is too small, the empirical conditional distribution, and consequently LEEP, would become unreliable.

To factor out the noise when evaluating LEEP scores on small target data sets, we also consider partitioning the scores’ range into five equal bins and averaging the test accuracies of tasks in each bin. This allows us to compare the average test accuracies of tasks belonging in five transferability levels. Fig. 2(a) shows a typical result for this case, where higher transferability levels generally imply better accuracies for both transfer methods. Full results are given in Fig. 6 in Appendix B. These results testify that LEEP scores can predict accuracies of transferred models, even in small data regimes.

We also evaluate LEEP in the noisy, small data setting, where we repeat the experiment between ImageNet and CIFAR100 but randomly flip the labels in the target data set to a wrong one with probability 0.15. We report the correlation coefficients for this case in Table 1 (see the 7${}^{\text{th}}$ rows of the first two algorithms). The results also show positive correlations between LEEP scores and test accuracies, although the correlations are weaker than those with correct labels.

3 LEEP vs. F1 Score on Imbalanced Data

Imbalanced target data sets are commonly encountered in practice (Al-Stouhi & Reddy, 2016; Wang et al., 2017). We now evaluate LEEP in this setting. Specifically, we also repeat the experiments of Sec. 5.1 and 5.2 where, this time, we restrict target data sets to imbalanced binary classification sets. The two classes of each target data set are drawn randomly. The size of the smaller class is chosen uniformly from 30 to 60, while the other class is five times larger. Because the target data sets are small with only two classes, we only need to re-train the head or fine-tune for 20 epochs.

Table 1 (see the last four rows of the first two algorithms) shows the correlation coefficients between LEEP scores and test F1 scores in these settings. The results also indicate a reasonably positive correlations with coefficients greater than 0.5 in all cases. Fig. 2(b) further shows the average test F1 scores in five transferability levels predicted from LEEP for a typical case, where higher transferability levels imply better F1 scores for both transfer methods. Full results are given in Fig. 7 in Appendix B. These results again confirm that LEEP scores can predict the performance of transferred models, even for imbalanced target data sets.

4 LEEP vs. Accuracy of Meta-Transferred Models

Meta-transfer learning is a framework for learning to adapt from a source task to a target task (Wei et al., 2018b; Sun et al., 2019; Requeima et al., 2019). Next, we show that LEEP can also predict the test accuracies of CNAPs (Requeima et al., 2019), a recently proposed meta-transfer learning method. CNAPs adapt a pre-trained model on the source task to a target task by adding scale and shift parameters to each channel of its convolutional layers. The additional parameters are outputs of adaptation networks, which are trained by meta-learning from several training tasks. When given a target data set, the adaptation networks return the appropriate scale and shift parameters that can augment the source model to make predictions on the target test set.

In our experiment, we follow the original training procedure proposed for CNAPs by Requeima et al. (2019), where the source model is a ResNet18 pre-trained on ImageNet, and the adaptation networks are trained using the Meta-data set (Triantafillou et al., 2020). We also test CNAPs on 200 random target tasks drawn from CIFAR100 as follows. Each target data set contains five random labels and 50 random examples per class, drawn from the test set of CIFAR100. The remaining 50 test examples of the selected classes are used as the target test set. This testing procedure is consistent with the one used by Requeima et al. (2019), except that in our experiments, we fix the number of classes.

The last row of Table 1 shows the correlation coefficient between LEEP scores and test accuracies of CNAPs. The coefficient is 0.591 with $p<0.001$, indicating that LEEP scores are a good measure of meta-transferability. Similar to Sec. 5.2 and 5.3, Fig. 3 provides the average test accuracies of tasks in five LEEP score transferability levels. Fig. 3 clearly shows that higher transferability levels correspond to better test accuracies for CNAPs.

5 LEEP vs. Convergence of Fine-tuned Models

For each target task, let us consider a reference model: one that is trained from scratch using only the target data set. When it is easier to transfer from a source task to the target task, we often expect the fine-tuned models to converge more quickly and even exceed the performance of this reference model. The experiments in this section show that LEEP scores indeed predict this behavior.

We use the same small data settings defined in Sec. 5.2 and train a reference model for each target task. Reference models are trained using SGD with 100 epochs for target tasks from CIFAR100 and with 40 epochs for those from FashionMNIST. When fine-tuning a model, we track the difference between the fine-tuned model’s test accuracy and the final test accuracy of the corresponding reference model (i.e., we always compare against the fully trained reference model). Similar to Sec. 5.2, we also consider different transferability levels according to LEEP scores, and average the accuracy differences of all tasks within the same transferability level.

Fig. 4 plots the average accuracy difference curves of five different transferability levels when transferring to CIFAR100 target tasks. Results for FashionMNIST target tasks are similar and given in Fig. 8 in Appendix B. From Fig. 4, on average, fine-tuned models on tasks in higher transferability levels have better convergence speeds (i.e., their curves reach zero faster). Furthermore, these models can outperform the reference models by larger margins (i.e., their curves reach higher values). In all cases, the fine-tuned models match the performance of the reference models using far fewer training epochs. These results confirm the advantage of transfer learning in small data settings, especially between highly transferable tasks, which in turn, can be efficiently predicted using our LEEP scores.

6 Comparison of LEEP, NCE, and H scores

We compare the LEEP measure with the NCE measure proposed by Tran et al. (2019) and the H score proposed by Bao et al. (2019). Particularly, for all the experimental settings in Sec. 5.1, 5.2, 5.3, and 5.4 above, we compute the NCE score for each transfer from a source model to a target task using the method described in Sec. 3. The computation is efficient since it also requires a single forward pass through the target data set to get the dummy labels. After obtaining the NCE scores, we evaluate their Pearson correlation coefficients and $p$ values with test accuracies (or F1 for imbalanced data) of the three transfer (or meta-transfer) learning algorithms. Similarly, we also compute the correlation coefficients and $p$ values of the H scores between the features returned by the source model and the target labels. The features are obtained by applying the source model to the target inputs.

Table 1 shows the correlation coefficients of NCE and H scores in comparison with those of LEEP scores. When compared with NCE, LEEP scores have equal or better correlations with transfer performance in all except for two cases of the fine-tuning method (see the second and third rows of this method in Table 1). Even in these two cases, LEEP scores are only slightly worse than NCE scores. These comparisons confirm that LEEP scores are better than NCE scores for our transfer settings, with up to 30% improvement in terms of correlation coefficients.

When compared with H scores, LEEP scores give better correlations in 16/23 cases. We note that the performance of H scores is not consistent in all experiments. Particularly, it completely fails to capture the transferability in 11 cases (those marked with asterisks). Even when it successfully captures transferability, it performs worse than LEEP on large target data sets. By comparison, our LEEP scores capture the transferability in all cases.

7 LEEP for Source Model Selection

We evaluate LEEP for the source model selection problem, where we need to select the best source model from 9 candidate models and transfer it to CIFAR100. The candidate models are pre-trained on ImageNet and include ResNet18, ResNet34, ResNet50 (He et al., 2016), MobileNet1.0, MobileNet0.75, MobileNet0.5, MobileNet0.25 (Howard et al., 2017), DarkNet53 (Redmon & Farhadi, 2018), and SENet154 (Hu et al., 2018). Our target data set is the full CIFAR100 training set, while the target test set is the full CIFAR100 test set. We compare our LEEP measure to the NCE (Tran et al., 2019) and H score (Bao et al., 2019) baselines. We also consider ImageNet top-1 accuracy as an additional baseline since previous work (Kornblith et al., 2019) has shown that ImageNet accuracy can predict the performance of transferred models.

Fig. 5 shows the results of this experiment for the head re-training method. From the figure, LEEP scores can predict well the test accuracy of models whose head classifiers are re-trained. In comparison, the other baselines all perform worse than LEEP. For example, NCE fails to predict the performance of MobileNet1.0, while H score and ImageNet accuracy fail on the SENet154.

We also give results for the fine-tuning method in Fig. 9 in Appendix B. Generally, all the considered measures do not predict well the test accuracy of fine-tuned models, especially for ResNet18 and ResNet34 source models. One possible explanation is that the performance of fine-tuned models is sensitive to the architecture and the size of the source networks. However, the transferability measures considered in this section do not take these factors into account.

Discussions

We proposed LEEP, a novel transferability measure that can efficiently estimate the performance of transfer and meta-transfer learning algorithms before actually executing them. We show both theoretically and empirically that LEEP is an effective transferability measure that is useful in several scenarios. Below are more discussions about our work.

Source model assumption. Our work assumes a source model pre-trained on the source task. If, instead, we are given the source data set, we can first train a source model from the data and then use this model to compute the LEEP scores. The LEEP scores would depend on the architectural choice and thus the performance of the source model. This is expected and has been pointed out in previous work (Kornblith et al., 2019; Tran et al., 2019).

Rationale for LEEP and possible extensions. The design of LEEP was aimed at being algorithm independence, i.e., it should work for many different transfer learning algorithms. In fact, the performance of transfer learning algorithms depends partially on the relationship between the source model and the target data set, and thus LEEP tries to quantify such relationship to achieve this independence. We note that LEEP was not intended to be a transfer learning algorithm although the EEP can be viewed as a transfer learning algorithm that was designed for simplicity and speed instead of accuracy. Transfer learning should be performed by a proper algorithm, e.g., by re-training the head classifier or fine-tuning the model parameters.

Although it might seem surprising that LEEP works well in our experiments without explicitly using the feature distributions, we note that LEEP in fact takes into account the feature distributions indirectly. Specifically, given a pre-trained source network, the output label distribution is a linear transformation of the features (i.e., output of the penultimate layer) followed by the Softmax. Thus, the dummy source label distribution indirectly contains information about the input features.

We can extend LEEP to include the learned features by transforming the feature vector into a probability distribution directly using a Softmax, and then computing LEEP with this new dummy distribution. In this case, the support of the dummy distribution has the same length as that of the feature vector and would have a different interpretation than ours. This would be a direction for future work.

Effects of heterogeneous source and target tasks. LEEP can be applied when the source and target tasks have different label semantics or when their input spaces are heterogeneous. In these cases, the source model would be more uncertain about the inputs from the target data set, leading to a more uniform dummy label distribution. By definition, LEEP scores will be smaller and thus indicating a harder transfer for these cases.

Applications of LEEP. Our experiments, reported in Sec. 5, showed that LEEP is applicable in diverse scenarios. In general, LEEP scores can be used to efficiently select highly transferable pairs of source and target tasks, yielding high transfer accuracy and good convergence speeds. This ability can support source model selection in transfer/meta-transfer learning scenarios, where we need to select a source model among several others in order to optimize for best performance on a target task.

Aside from transfer and meta-transfer learning, our LEEP scores are potentially useful for continual learning (Zenke et al., 2017; Swaroop et al., 2019), multi-task learning (Misra et al., 2016; Standley et al., 2019), and feature selection (Yosinski et al., 2014). For instance, LEEP scores can be used to estimate the hardness of task sequences, thereby helping to analyze properties of continual learning algorithms (Nguyen et al., 2019). For multi-task learning, LEEP scores can be used for selecting groups of tasks for joint training (Standley et al., 2019). Finally, LEEP scores could be useful for hyperparameter transfer learning and Bayesian optimization (Perrone et al., 2018). These are promising research directions that we leave to future work.

Appendix A Proofs

This proof is straight-forward because $l(w,k^{*})$ is the maximal average log-likelihood over $k\in\mathcal{K}$, $T(\theta,\mathcal{D})$ is the average log-likelihood of the EEP, and the EEP is in $\mathcal{K}$. Thus, $T(\theta,\mathcal{D})\leq l(w,k^{*})$.

A.2 Proof of Property 2

Let $Z=(z_{1},z_{2},\ldots,z_{n})$ be the dummy labels of $(x_{1},x_{2},\ldots,x_{n})$ obtained when computing the NCE, and let $Y=(y_{1},y_{2},\ldots,y_{n})$ be the true label set. We have:

According to the proof of Theorem 1 of Tran et al. (2019), we have:

Appendix B Full Experimental Results

Fig. 6 shows the results for all experimental settings with small balanced target data sets.

Fig. 7 shows the results for all experimental settings with small imbalanced target data sets.

Fig. 8 shows the results for all experimental settings with the convergence speed of fine-tuned models. For a clearer comparison, we only consider two LEEP transferability levels for target tasks constructed from FashionMNIST.

Fig. 9 shows the results for all experimental settings in the source model selection problem.

References