Interventional Few-Shot Learning

Introduction

Few-Shot Learning (FSL) — the task of training a model using very few samples — is nothing short of a panacea for any scenario that requires fast model adaptation to new tasks , such as minimizing the need for expensive trials in reinforcement learning and saving computation resource for light-weight neural networks . Although we knew that, more than a decade ago, the crux of FSL is to imitate the human ability of transferring prior knowledge to new tasks , not until the recent advances in pre-training techniques, had we yet reached a consensus on “what & how to transfer”: a powerful neural network $\Omega$ pre-trained on a large dataset $\mathcal{D}$. In fact, the prior knowledge learned from pre-training prospers today’s deep learning era, e.g., $\mathcal{D}$ = ImageNet, $\Omega$ = ResNet in visual recognition ; $\mathcal{D}$ = Wikipedia, $\Omega$ = BERT in natural language processing .

In the context of pre-trained knowledge, we denote the original FSL training set as support set $\mathcal{S}$ and the test set as query set $\mathcal{Q}$, where the classes in $(\mathcal{S},\mathcal{Q})$ are unseen (or new) in $\mathcal{D}$. Then, we can use $\Omega$ as a backbone (fixed or partially trainable) for extracting sample representations $\mathbf{x}$, and thus FSL can be achieved simply by fine-tuning the target model on $\mathcal{S}$ and test it on $\mathcal{Q}$ . However, the fine-tuning only exploits the $\mathcal{D}$’s knowledge on “what to transfer”, but neglects “how to transfer”. Fortunately, the latter can be addressed by applying a post-pre-training and pre-fine-tuning strategy: meta-learning . Different from fine-tuning whose goal is the “model” trained on $\mathcal{S}$ and tested on $\mathcal{Q}$, meta-learning aims to learn the “meta-model” — a learning behavior — trained on many learning episodes $\{(\mathcal{S}_{i},\mathcal{Q}_{i})\}$ sampled from $\mathcal{D}$ and tested on the target task $(\mathcal{S},\mathcal{Q})$. In particular, the behavior can be parametrized by $\phi$ using model parameter generator or initialization . After meta-learning, we denote $\Omega_{\phi}$ as the new model starting point for the subsequent fine-tuning on target task $(\mathcal{S},\mathcal{Q})$. Figure 1 illustrates the relationships among the above discussed FSL paradigms.

It is arguably a common sense that the stronger the pre-trained $\Omega$ is, the better the downstream model will be. However, we surprisingly find that this may not be always the case in FSL. As shown in Figure 2(a), we can see a paradox: though stronger $\Omega$ improves the performance on average, it indeed degrades that of samples in $\mathcal{Q}$ dissimilar to $\mathcal{S}$. To illustrate the “dissimilar”, we show a 5-shot learning example in Figure 2(b), where the prior knowledge on “green grass” and “yellow grass” is misleading. For example, the “Lion” samples in $\mathcal{Q}$ have “yellow grass”, hence they are misclassified as “Dog” whose $\mathcal{S}$ has major “yellow grass”. If we use stronger $\Omega$, the seen old knowledge (“grass” & “color”) will be more robust than the unseen new knowledge (“Lion” & “Dog”), and thus the old becomes even more misleading. We believe that such a paradox reveals an unknown systematic deficiency in FSL, which has been however hidden for years by our gold-standard “fair” accuracy, averaged over all the random $(\mathcal{S},\mathcal{Q})$ test trials, regardless of the similarity between $\mathcal{S}$ and $\mathcal{Q}$ (cf. Figure 2(a)). Though Figure 2 only illustrates the fine-tune FSL paradigm, the deficiency is expected in the meta-learning paradigm, as fine-tune is also used in each meta-train episode (Figure 1). We will analyze them thoroughly in Section 5.

In this paper, we first point out that the cause of the deficiency: pre-training can do evil in FSL, and then propose a novel FSL paradigm: Interventional Few-Shot Learning (IFSL), to counter the evil. Our theory is based on the assumption of the causalities among the pre-trained knowledge, few-shot samples, and class labels. Specifically, our contributions are summarized as follows.

We begin with a Structural Causal Model (SCM) assumption in Section 2.2, which shows that the pre-trained knowledge is essentially a confounder that causes spurious correlations between the sample features and class labels in support set. As an intuitive example in Figure 2(b), even though the “grass” feature is not the cause of the “Lion” label, the prior knowledge on “grass” still confounds the classifier to learn a correlation between them.

In Section 2.3, we illustrate a causal justification of why the proposed IFSL fundamentally works better: it is essentially a causal approximation to many-shot learning. This motivates us to develop three effective implementations of IFSL using the backdoor adjustment in Section 3.

Thanks to the causal intervention, IFSL is naturally orthogonal to the downstream fine-tuning and meta-learning based FSL methods . In Section 5.2, IFSL improves all baselines by a considerable margin, achieving new 1-/5-shot state-of-the-arts: 73.51%/83.21% on miniImageNet , 83.07%/88.69% on tieredImageNet , and 50.71%/64.43% on cross-domain CUB .

We further diagnose the detailed performances of FSL methods across different similarities between $\mathcal{S}$ and $\mathcal{Q}$. We find that IFSL outperforms all baselines in every inch.

Problem Formulations

We are interested in a prototypical FSL: train a $K$-way classifier on an $N$-shot support set $\mathcal{S}$, where $N$ is a small number of training samples per class (e.g., $N$=1 or 5); then test the classifier on a query set $\mathcal{Q}$. As illustrated in Figure 1, we have the following two paradigms to train the classifier $P(y|\mathbf{x};\theta)$, predicting the class $y\in\{1,...,K\}$ of a sample $\mathbf{x}$:

Fine-Tuning. We consider the prior knowledge as the sample feature representation $\mathbf{x}$, encoded by the pre-trained network $\Omega$ on dataset $\mathcal{D}$. In particular, we refer $\mathbf{x}$ to the output of the frozen sub-part of $\Omega$ and the rest trainable sub-part of $\Omega$ (if any) can be absorbed into $\theta$. We train the classifier $P(y|\mathbf{x};\theta)$ on the support set $\mathcal{S}$, and then evaluate it on the query set $\mathcal{Q}$ in a standard supervised way.

2 Structural Causal Model

From the above discussion, we can see that $(\phi,\theta)$ in meta-learning and $\theta$ in fine-tuning are both dependent on the pre-training. Such “dependency” can be formalized with a Structural Causal Model (SCM) proposed in Figure 3(a), where the nodes denote the abstract data variables and the directed edges denote the (functional) causality, e.g., $X\rightarrow Y$ denotes that $X$ is the cause and $Y$ is the effect. Now we introduce the graph and the rationale behind its construction at a high-level. Please see Section 3 for the detailed functional implementations.

$\boldsymbol{D\to X}$. We denote $X$ as the feature representation and $D$ as the pre-trained knowledge, e.g., the dataset $\mathcal{D}$ and its induced model $\Omega$. This link assumes that the feature $X$ is extracted by using $\Omega$.

$\boldsymbol{D\rightarrow C\leftarrow X}$. We denote $C$ as the transformed representation of $X$ in the low-dimensional manifold, whose base is inherited from $D$. This assumption can be rationalized as follows. 1) $D\rightarrow C$: a set of data points are usually embedded in a low-dimensional manifold. This finding can be dated back to the long history of dimensionality reduction . Nowadays, there are theoretical and empirical evidences showing that disentangled semantic manifolds emerge during training deep networks. 2) $X\rightarrow C$: features can be represented using (or projected onto) the manifold base linearly or non-linearly . In particular, as later discussed in Section 3, we explicitly implement the base as feature dimensions (Figure 3(b)) and class-specific mean features (Figure 3(c)).

$\boldsymbol{X\rightarrow Y\leftarrow C}$. We denote $Y$ as the classification effect (e.g., logits), which is determined by $X$ via two ways: 1) the direct $X\rightarrow Y$ and 2) the mediation $X\rightarrow C\rightarrow Y$. In particular, the first way can be removed if $X$ can be fully represented by $C$ (e.g., feature-wise adjustment in Section 3). The second way is inevitable even if the classifier does not take $C$ as an explicit input, because any $X$ can be inherently represented by $C$. To illustrate, suppose that $X$ is a linear combination of two base vectors plus a noise residual: $\mathbf{x}=c_{1}\mathbf{b}_{1}+c_{2}\mathbf{b}_{2}+\mathbf{e}$, any classifier $f(\mathbf{x})$ = $f(c_{1}\mathbf{b}_{1}+c_{2}\mathbf{b}_{2}+\mathbf{e})$ will implicitly exploit the $C$ representation in terms of $\mathbf{b}_{1}$ and $\mathbf{b}_{2}$. In fact, this assumption also fundamentally validates unsupervised representation learning . To see this, if $C\not\rightarrow Y$ in Figure 3(a), uncovering the latent knowledge representation from $P(Y|X)$ would be impossible, because the only path left that transfers knowledge from $D$ to $Y$: $D\rightarrow X\rightarrow Y$, is cut off by conditioning on $X$: $D\not\rightarrow X\rightarrow Y$.

An ideal FSL model should capture the true causality between $X$ and $Y$ to generalize to unseen samples. For example, as illustrated in Figure 2(b), we expect that the “Lion” prediction is caused by the “lion” feature per se, but not the background “grass”. However, from the SCM in Figure 3(a), the conventional correlation $P(Y|X)$ fails to do so, because the increased likelihood of $Y$ given $X$ is not only due to “X causes Y” via $X\rightarrow Y$ and $X\rightarrow C\rightarrow Y$, but also the spurious correlation via 1) $D\rightarrow X$, e.g., the “grass” knowledge generates the “grass” feature, and 2) $D\rightarrow C\rightarrow Y$, e.g., the “grass” knowledge generates the “grass” semantic, which provides useful context for “Lion” label. Therefore, to pursue the true causality between $X$ and $Y$, we need to use the causal intervention $P(Y|do(X))$ instead of the likelihood $P(Y|X)$ for the FSL objective.

3 Causal Intervention via Backdoor Adjustment

By now, an astute reader may notice that the causal graph in Figure 3(a) is also valid for Many-Shot Learning (MSL), i.e., conventional learning based on pre-training. Compared to FSL, the $P(Y|X)$ estimation of MSL is much more robust. For example, on miniImageNet, a 5-way-550-shot fine-tuned classifier can achieve 95% accuracy, while a 5-way-5-shot one only obtains 79%. We used to blame FSL for insufficient data by the law of large numbers in point estimation . However, it does not answer why MSL converges to the true causal effects as the number of samples increases infinitely. In other words, why $P(Y|do(X))\approx P(Y|X)$ in MSL while $P(Y|do(X))\not\approx P(Y|X)$ in FSL?

In this paper, we propose to use the backdoor adjustment to achieve $P(Y|do(X))$ without the need for many-shot, which certainly undermines the definition of FSL. The backdoor adjustment assumes that we can observe and stratify the confounder, i.e., $D=\{d\}$, where each $d$ is a stratification of the pre-trained knowledge. Formally, as shown in Appendix 2, the backdoor adjustment for the graph in Figure 3(a) is:

where $g$ is a function defined later. However, it is not trivial to instantiate $d$, especially when $D$ is a 3rd-party delivered pre-trained network where the dataset is unobserved . Next, we will offer three practical implementations of Eq. (1) for Interventional FSL.

Interventional Few-Shot Learning

Our implementation idea is inspired from the two inherent properties of any pre-trained DNN. First, each feature dimension carries a semantic meaning, e.g., every channel in convolutional neural network is well-known to encode visual concepts . So, each feature dimension represents a piece of knowledge. Second, most prevailing pre-trained models use a classification task as the objective, such as the 1,000-way classifier of ResNet and the token predictor of BERT . Therefore, the classifier can be considered as the distilled knowledge, which has been already widely adopted in literature . Next, we will detail the proposed Interventional FSL (IFSL) by providing three different implementationsWe assume that the combinations of the feature dimensions or classes are linear, otherwise the adjustment requires prohibitive $\mathcal{O}(2^{n})$ sampling. We will relax this assumption in future work. for $g(\mathbf{x},d)$, $P(Y|X,D,C)$, and $P(D)$ in Eq. (1). In particular, the exact forms of $P(Y|\cdot)$ across different classifiers are given in Appendix 5.

Feature-wise Adjustment. Suppose that $\mathcal{F}$ is the index set of the feature dimensions of $\mathbf{x}$, e.g., from the last-layer of the pre-trained network $\Omega$. We divide $\mathcal{F}$ into $n$ equal-size disjoint subsets, e.g., the output feature dimension of ResNet-10 is 512, if $n=8$, the $i$-th set will be a feature dimension index set of size 512/8 = 64, i.e., $\mathcal{F}_{i}=\{64(i-1)+1,...,64i\}$. The stratum set of pre-trained knowledge is defined as $D:=\{d_{1},\ldots,d_{n}\}$, where each $d_{i}=\mathcal{F}_{i}$.

(i) $g(\mathbf{x},d_{i})\coloneqq\{k|k\in\mathcal{F}_{i}\cap\mathcal{I}_{t}\}$, where $\mathcal{I}_{t}$ is an index set whose corresponding absolute values in $\mathbf{x}$ are larger than the threshold $t$. The reason is simple: if a feature dimension is inactive in $\mathbf{x}$, its corresponding adjustment can be omitted. We set $t$=1e-3 in this paper.

(ii) $P(Y|X,D,C)=P(Y|[\mathbf{x}]_{c})$, where $c=g(\mathbf{x},d_{i})$ is implemented as the index set defined above, $[\mathbf{x}]_{c}=\{x_{k}\}_{k\in c}$ is a feature selector which selects the dimensions of $\mathbf{x}$ according to the index set $c$. The classifier takes the adjusted feature $[\mathbf{x}]_{c}$ as input. Note that $d$ is already absorbed in $c$, so $[\mathbf{x}]_{c}$ is essentially a function of $(X,D,C)$.

(iii) $P(d_{i})=1/n$, where we assume a uniform prior for the adjusted features.

(iv) The overall feature-wise adjustment is:

It is worth noting that the feature-wise adjustment is always applicable, as we can always have the feature representation $\mathbf{x}$ from the pre-trained network. Interestingly, our feature-wise adjustment sheds some light on the theoretical justifications for the multi-head trick in transformers . We will explore this in future work.

Class-wise Adjustment. Suppose that there are $m$ pre-training classes, denoted as $\mathcal{A}=\{a_{1},\ldots a_{m}\}$. In class-wise adjustment, each stratum of pre-trained knowledge is defined as a pre-training class, i.e., $D:=\{d_{1},\ldots,d_{m}\}$ and each $d_{i}=a_{i}$.

(i) $g(\mathbf{x},d_{i})\coloneqq P(a_{i}|\mathbf{x})\mathbf{\bar{x}}_{i}$, where $P(a_{i}|\mathbf{x})$ is the pre-trained classifier’s probability output that $\mathbf{x}$ belongs to class $a_{i}$, and $\mathbf{\bar{x}}_{i}$ is the mean feature of pre-training samples from class $a_{i}$. Note that unlike feature-wise adjustment where $c$ is an index set, here $c=g(\mathbf{x},d_{i})$ is implemented as a real vector.

(ii) $P(Y|X,D,C)=P(Y|\mathbf{x}\oplus g(\mathbf{x},d_{i}))$, where $\oplus$ denotes vector concatenation.

(iii) $P(d_{i})=1/m$, where we assume a uniform prior of each class.

(iv) The overall class-wise adjustment is:

where we adopt the Normalized Weighted Geometric Mean (NWGM) approximation to move the outer sum $\sum P$ into the inner $P(\sum)$. This greatly reduces the network forward-pass consumption as $m$ is usually large in pre-training dataset. Please refer to Appendix 3 for the detailed derivation.

Combined Adjustment. We can combine feature-wise and class-wise adjustment to make the stratification in backdoor adjustment much more fine-grained. Our combination is simple: applying feature-wise adjustment after class-wise adjustment. Thus, we have:

Related Work

Few-Shot Learning. FSL has a wide spectrum of methods, including fine-tuning , optimizing model initialization , generating model parameters , learning a feature space for a better separation of sample categories , feature transfer , and transductive learning that additionally uses query set data . Thanks to them, the classification accuracy has been drastically increased . However, accuracy as a single number cannot explain the paradoxical phenomenon in Figure 2. Our work offers an answer from a causal standpoint by showing that pre-training is a confounder. We not only further improve the accuracy of various FSL methods, but also explain the reason behind the improvements. In fact, the perspective offered by our work can benefit all the tasks that involve pre-training—any downstream task can be seen as FSL compared to the large-scale pre-training data.

Negative Transfer. The above phenomenon is also known as the negative transfer, where learning in source domain contributes negatively to the performance in target domain . Many research works have being focused on when and how to conduct this transfer learning . Yosinski et al. split ImageNet according to man-made objects and natural objects as a test bed for feature transferability. They resemble the $\mathcal{S}\not\sim\mathcal{Q}$ settings used in Figure 2(a). Other work also revealed that using deeper backbone might lead to degraded performance when the domain gap between training and test is large . Some similar findings are reported in the few-shot setting and NLP tasks . Unfortunately, they didn’t provide a theoretical explanation why it happens.

Causal Inference. Our work aims to deal with the pre-training confounder in FSL based on causal inference . Causal inference was recently introduced to machine learning and has been applied to various fields in computer vision. proposes a retrospective for image captioning and other applications include image classification , imitation learning , long-tailed recognition and semantic segmentation . We are the first to approach FSL from a causal perspective. We would like to highlight that data-augmentation based FSL can also be considered as approximated intervention. These methods learn to generate additional support samples with image deformation or generative models . This can be view as physical interventions on the image features. Regarding the causal relation between image $X$ and label $Y$, some works adopted anti-causal learning , i.e., $Y\to X$, where the assumption is that labels $Y$ are disentangled enough to be treated as Independent Mechanism (IM) , which generates observed images $X$ through $Y\to X$. However, our work targets at the more general case where labels can be entangled (e.g.“lion” and “dog” share the semantic “soft fur”) and the IM assumption may not hold. Therefore, we use causal prediction $X\to Y$ as it is essentially a reasoning process, where the IM is captured by $D$, which is engineered to be disentangled through CNN (e.g., the conv-operations are applied independently). In this way, $D$ generates visual features through $D\to X$ and emulates human’s naming process through $D\to Y$ (e.g., “fur”, “four-legged”$\to$ “meerkat”). In fact, the causal direction $X\to Y$ (NOT anti-causal $Y\to X$) has been empirically justified in complex CV tasks .

Experiments

Datasets. We conducted experiments on benchmark datasets in FSL literature: 1) miniImageNet containing 600 images per class over 100 classes. We followed the split proposed in : 64/16/20 classes for train/val/test. 2) tieredImageNet is much larger compared to miniImageNet with 608 classes and each class around 1,300 samples. These classes were grouped into 34 higher-level concepts and then partitioned into 20/6/8 disjoint sets for train/val/test to achieve larger domain difference between training and testing. 3) Caltech-UCSD Birds-200-2011 (CUB) for cross-domain evaluation. It contains 200 classes and each class has around 60 samples. The models used for CUB test were trained on the miniImageNet. Training and evaluation settings on miniImageNet and tieredImageNet are included in Appendix 5.

Implementation Details. We pre-trained the 10-layer ResNet (ResNet-10) and the WideResNet (WRN-28-10) as our backbones. Our proposed IFSL supports both fine-tuning and meta-learning. For fine-tuning, we applied average pooling on the last residual block and used the pooled features to train classifiers. For meta-learning, we deployed 5 representative methods that cover a large spectrum of meta-learning based FSL: 1) model initialization: MAML , 2) weight generator: LEO , transductive learning: SIB , 4) metric learning: MatchingNet (MN) , and 5) feature transfer: MTL . For both fine-tuning and meta-learning, our IFSL aims to the learn classifier $P(Y|do(X))$ instead of the conventional $P(Y|X)$. Detailed implementations are given in Appendix 5.

Evaluation Metrics. Our evaluation is based on the following metrics: 1) Conventional accuracy (Acc) is the average classification accuracy commonly used in FSL . 2) Hardness-specific Acc. For each query, we define a hardness that measures its semantic dissimilarity to the support set, and accuracy is then computed at different levels of query hardness. Specifically, query hardness is computed by $h=\log\left((1-s)/s\right)$ and $s=exp{\langle\mathbf{r}^{+},\mathbf{p}_{c=gt}^{+}\rangle}/{\sum\nolimits_{c}exp\langle\mathbf{r}^{+},\mathbf{p}_{c}^{+}\rangle}$, where $\langle\cdot\rangle$ is the cosine similarity, $(\cdot)^{+}$ represents the ReLU activation function, $\mathbf{r}$ denotes the $\Omega$ prediction logits of query, $\mathbf{p}_{c}$ denotes the average prediction logits of class $c$ in the support set and $gt$ is the ground-truth of query. Using Hardness-specific Acc is similar to evaluating the hardness of FSL tasks , while ours is query-sample-specific and hence is more fine-grained. Later, we will show its effectiveness to unveil the spurious effects in FSL. 3) Feature localization accuracy (CAM-Acc) quantifies if a model “pays attention” to the actual object when making prediction. It is defined as the percentage of pixels inside the object bounding box by using Grad-CAM score larger than $0.9$. Compared to Acc that shows if the prediction is correct, CAM-Acc reveals whether the prediction is based on the correct visual cues.

2 Results and Analysis

Conventional Acc. 1) From Table 5.1, we observe that IFSL consistently improves fine-tuning and meta-learning in all settings, which suggests that IFSL is agnostic to methods, datasets, and backbones. 2) In particular, the improvements are typically larger on 1-shot than 5-shot. For example, in fine-tuning, the average performance gain is 1.15% on 5-shot and 3.58% on 1-shot. The results support our analysis in Section 2.3 that FSL models are more prone to bias in lower-shot settings. 3) Regarding the average improvements on fine-tuning vs. meta-learning (e.g.$k$-NN and MN), we observe that IFSL improves more on fine-tuning in most cases. We conjecture that this is because meta-learning is an implicit form of intervention, where randomly sampled meta-training episodes effectively stratify the pre-trained knowledge. This suggests that meta-learning is fundamentally superior over fine-tuning due to increased robustness against confounders. We will investigate this potential theory in future work. 4) Additionally we see that the improvements on miniImageNet are usually larger than that on tieredImageNet. A possible reason is the much larger training set for tieredImageNet: it substantially increases the breadth of the pre-trained knowledge and the resulting models explain query samples much better. 5) According to Table 5.1 and Table 5.1, it is clear that our $k$-NN+IFSL outperforms IdeMe-Net using the same pre-trained ResNet-10. This shows that using data augmentation — a method of physical data intervention as in IdeMe-Net is inferior to our causal intervention in IFSL. 6) Overall, our IFSL achieves the new state-of-the-art on both datasets. Note that IFSL is flexible to be plugged into different baselines.

Hardness-specific Acc. 1) Figure 6() shows the plot of Hardness-specific Acc of fine-tuning. We notice that when query becomes harder, ResNet-10 (blue curves) becomes superior to WRN-28-10 (red curves). This tendency is consistent with Figure 2(a) illustrating the effect of the confounding bias caused by pre-training. 2) Intriguingly, in Figure 6(), we notice that this tendency is reversed for meta-learning, i.e., deeper backbone always performs better. The improved performance of deeper backbone on hard queries suggests that meta-learning should have some functions to remove the confounding bias. This evidence will inspire us to provide a causal view of meta-learning in future work. 3) Overall, Figure 5 shows that using IFSL futher improves fine-tuning and meta-learning consistently across all hardness, validating the effectiveness of the proposed causal intervention.

CAM-Acc & Visualization. In Figure 6, we compare +IFSL to baseline linear classifier on the left and to baseline MAML on the right, and summarize CAM-Acc results in the upper-right table. From the visualization, we see that using IFSL let the model pay more attention to the objects. However, notice that all models failed in the categories colored as red. A possible reason behind the failures is the extremely small size of the object — models have to resort to context for prediction. From the numbers, we can see our improvements for 1-shot are larger than that for 5-shot, consistent with our findings using other evaluation metrics. These results suggest that IFSL helps models use the correct visual semantics for prediction by removing the confounding bias.

Cross-Domain Generalization Ability. In Table 5.1, we show the testing results on CUB using the models trained on the miniImageNet. The setting is challenging due to the big domain gap between the two datasets. We chose linear classifier as it outperforms cosine and $k$-NN in cross-domain setting and compared with transductive method — SIB. The results clearly show that IFSL works well in this setting and brings consistent improvements, with the average 1.94% of Acc. In addition, we can see that applying IFSL brings larger improvements to the inductive linear classifier than to the transductive SIB. It is possibly because transductive methods involve unlabeled query data and performs better than inductive methods with the additional information. Nonetheless we observe that IFSL can further improve SIB in cross-domain (Table 5.1) and single-domain (Table 5.1) generalization.

Conclusions

We presented a novel casual framework: Interventional Few-Shot Learning (IFSL), to address an overlooked deficiency in recent FSL methods: the pre-training is a confounder hurting the performance. Specifically, we proposed a structural causal model of the causalities in the process of FSL and then developed three practical implementations based on the backdoor adjustment. To better illustrate the deficiency, we diagnosed the classification accuracy comprehensively across query hardness, and showed that IFSL improves all the baselines across all the hardness. It is worth highlighting that the contribution of IFSL is not only about improving the performance of FSL, but also offering a causal explanation why IFSL works well: it is a causal approximation to many-shot learning. We believe that IFSL may shed light on exploring the new boundary of FSL, even though FSL is well-known to be ill-posed due to insufficient data. To upgrade IFSL, we will seek other observational intervention algorithms for better performance, and devise counterfactual reasoning for more general few-shot settings such as domain transfer.

Acknowledgements

The authors would like to thank all the anonymous reviewers for their constructive comments and suggestions. This research is partly supported by the Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University (NTU), Singapore; the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 and Tier 2 grant; and Alibaba Innovative Research (AIR) programme. We also want to thank Alibaba City Brain Group for the donations of GPUs.

Broader Impact

The proposed method aims to improve the Few-Shot Learning task. Advancements in FSL helps the deployment of machine learning models in areas where labelled data is difficult or expensive to obtain and it is closely related to social well-beings: few-shot drug discovery or medical imaging analysis in medical applications, cold-start item recommendation in e-commerce, few-shot reinforcement learning for industrial robots, etc.. Our method is based on causal inference and the analysis is rooted on causation rather than correlation. The marriage between causality and machine learning can produce more robust, transparent and explainable models, broadening the applicability of ML models and promoting fairness in artificial intelligence.

References

A.1 Instrumental Variable

In this section, we will show that in our causal graph for many-shot learning, the sampling ID $I$ is essentially an instrumental variable for $X\rightarrow Y$ that achieves $P(Y|I)\approx P(Y|do(X))$. Before introducing instrumental variable, we first formally define d-separation , which gives a criterion to study the dependencies between nodes (data variables) in any structural causal model.

d-separation. A set of nodes $Z$ blocks a path $p$ if and only if 1) $p$ contains a chain $A\rightarrow B\rightarrow C$ or a fork $A\leftarrow B\rightarrow C$ and the middle node $B$ is in $Z$; 2) $p$ contains a collider $A\rightarrow B\leftarrow C$ such that the middle node $B$ and its descendants are not in $Z$. If conditioning on $Z$ blocks every path between $X$ and $Y$, we say $X$ and $Y$ are d-separated conditional on $Z$, i.e., $X$ and $Y$ are independent given $Z$ (X\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y|Z).

Instrumental Variable. For a structual causal model $\mathcal{G}$, a variable Z is an instrumental variable (IV) to $X\rightarrow Y$ by satisfying the graphical criteria : 1) (Z\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y)_{\mathcal{G}_{\overline{X}}}~{}; 2) (Z\not\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}X)_{\mathcal{G}}~{}, where $\mathcal{G}_{\overline{X}}$ is the manipulated graph where all incoming arrows to node $X$ are deleted. For the SCM of many-shot learning in Figure 4(a), it is easy to see that $I$ satisfies both criteria and therefore it is an IV for $X\rightarrow Y$. However, in the few-shot SCM in Figure 4(b), the paths $I\leftarrow X\leftarrow D\rightarrow C\rightarrow Y$ and $I\leftarrow X\rightarrow C\rightarrow Y$ are not blocked in $\mathcal{G}_{\overline{X}}$, which means the first criterion is not met (I\not\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Y)_{\mathcal{G}_{\overline{X}}}~{} and $I$ is not an instrumental variable in the few-shot learning case.

Instrumental variable can help find the true causal effect even in the presence of confounder. This is due to the collider junction that makes the IV and confounder independent (I\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}D in Figure 4(a)). To see this, we will first consider a simplified case of Figure 4(a) where each causal link represents a linear relationship and we aim to find the true causal effect from $X\rightarrow Y$ through linear regression. Without loss of generality, let $I,X,Y$ take the value of real number. Denote $r_{IX},r_{XY},$ and $r_{IY}$ as the slope of regression line between $I$ and $X$, $X$ and $Y$, $I$ and $Y$ respectively. Notice that $r_{XY}$ is spurious as it is contaminated by the backdoor path $X\leftarrow D\rightarrow C\rightarrow Y$. However, since the path $I\rightarrow X\leftarrow D\rightarrow C\rightarrow Y$ is blocked due to collider at $X$, $r_{IY}$ is free from confounding bias. Therefore $r_{IY}/r_{IX}$ gives the true causal effect from $X\rightarrow Y$. Similarly, in the classification case of many-shot learning, a classifier is trained to maximize the conditional probability on the IV $P(Y|I)$. As the ID-sample matching $I\rightarrow X$ is deterministic, the classifier eventually learns to predict based on the true causal relationship $X\rightarrow Y$. Yet in the complex case of image classification, it is unreasonable to assume linear relationships between variables. In the nonlinear case, it is shown in that observations on IV provide a bound for the true causal effect. This means that learning based on $P(Y|I)$ provides an approximation to the true causal effect, i.e.$P(Y|I)\approx P(Y|do(X))$.

A.2 Derivation of Backdoor Adjustment for the Proposed Causal Graph

We will show the derivation of the backdoor adjustment for the causal graph in Figure 3(a) using the three rules of do-calculus .

For a causal directed acyclic graph $\mathcal{G}$, let $X,Y,Z$ and $W$ be arbitrary disjoint sets of nodes. We use $\mathcal{G}_{\overline{X}}$ to denote the manipulated graph where all incoming arrows to node $X$ are deleted. Similarly $\mathcal{G}_{\underline{X}}$ represents the graph where outgoing arrows from node $X$ are deleted. We use lower case $x,y,z$ and $w$ for specific values taken by each set of nodes: $X=x,Y=y,Z=z$ and $W=w$. For any interventional distribution compatible with $\mathcal{G}$, we have the following three rules:

Rule 1 Insertion/deletion of observations:

where $Z(W)$ is the set of nodes in $Z$ that are not ancestors of any $W$-node in $\mathcal{G}_{\overline{X}}$.

In our causal graph, the desired interventional distribution $P(Y|do(X=\mathbf{x}))$ can be derived by:

where Eq. (A8) and Eq. (A11) follow the law of total probability; Eq. (A9) uses Rule 3 given D\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}X in $\mathcal{G}_{\overline{X}}$; Eq. (A10) uses Rule 2 to change the intervention term to observation as (Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}X|D) in $\mathcal{G}_{\underline{X}}$; Eq. (A12) is because in our causal graph, $C$ takes a deterministic value given by function $g(\mathbf{x},d)$. This reduces summation over all values of $C$ in Eq. (A11) to a single probability measure in Eq. (A12).

A.3 Derivation of NWGM Approximation

where $D$ represents the set of stratifications, $f_{y}$ is the classifier logit for class $y$, $\mathbf{c}=g(\mathbf{x},d)$ is the feature concatenated to $\mathbf{x}$ in Eq. (3) and (4) and $P(d)$ is the prior for each stratificaction.

It is shown in that Eq. (A13) can be approximated by the Normalized Weighted Geometric Mean (NWGM) as:

where Eq. (A14) follows , Eq. (A15) follows the definition of NWGM, Eq. (A16) is because $exp(a)^{b}=exp(ab)$.

For the $k$-NN classifier, our implementation calculates class centroids using the mean feature of the $K$ support sets and then uses the nearest centroid for prediction ($1$-NN). Specifically, let $\mathbf{x}$ be a feature vector and $\mathbf{x^{\prime}}$ be the $i$th class centroid, $i\in\{1,\ldots,K\}$. The logit for class $i$ is given by $f_{i}(\mathbf{x})=-\left\lVert\mathbf{x}-\mathbf{x^{\prime}}\right\rVert^{2}$. It is shown in that $k$-NN classifier that uses squared Euclidean distance to generate logits is equivalent to a linear classifier with a particular parameterization. Therefore, our analysis on linear classifier follows for $k$-NN.

A.4 Algorithms for Fine-tuning and Meta-Learning with IFSL

In this section, we will briefly revisit the settings of fine-tuning and meta-learning and introduce how to integrate IFSL into them.

In fine-tuning, the goal is to train a classifier $\theta$ conditioned on the current support set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n_{s}}$, where $\mathbf{x}_{i}$ is the feature generated by $\Omega$ for $i$th sample, $y_{i}$ is the ground-truth label for $i$th sample and $n_{s}$ is the support set size. This is achieved by first predicting the support label $\hat{y}$ using the classifier $P(y|\mathbf{x};\theta)$. Then with the predicted label $\hat{y}$ and ground-truth label $y$, one can calculate a loss $\mathcal{L}(\hat{y},y)$ (usually cross-entropy loss) to update the classifier parameter, e.g.through stochastic gradient descent. Adding IFSL to fine-tuning is simple: 1) Pick an adjustment strategy introduced in Section 3. Each implementation defines the set of pre-trained knowledge stratifications $D$, function form of $g(X,D)$, function form of $P(Y|X,D,C)$ and the prior $P(D)$; 2) The classifier prediction is now based on $P(Y|do(X);\theta)$. The process of fine-tuning with IFSL is summarized in Algorithm 1. Note that for the non-parametric $k$-NN classifier, the fine-tuning process is not applicable. When adding IFSL to $k$-NN, each sample is represented by the adjusted feature instead of original feature $\mathbf{x}$. Please refer to the classifier inputs in Eq. (2), (3) and (4) for the exact form of adjusted feature.

In meta-learning, the goal is to learn the additional “learning behavior” parameterized by $\phi$ using training episodes $\{(\mathcal{S}_{i},\mathcal{Q}_{i})\}$ sampled from training dataset $\mathcal{D}$. The classifier in meta-learning makes predictions by additionally conditioning on the learning behavior, written as $P_{\phi}(y|\mathbf{x};\theta)$. Within each episode, $\theta$ is first fine-tuned on the support set $\mathcal{S}_{i}$. Then the fine-tuned model is tested on the query set $\mathcal{Q}_{i}$ to obtain the loss $\mathcal{L}_{\phi}(\mathcal{S}_{i},\mathcal{Q}_{i})$ (e.g.using cross-entropy loss). Finally the loss is used to update $\phi$ using an optimizer. It is also easy to integrate IFSL into meta-learning by only changing the classifier from $P_{\phi}(y|\mathbf{x};\theta)$ to $P_{\phi}(y|do(\mathbf{x});\theta)$. The flow of meta-learning with IFSL is presented in Algorithm 2. Firstly notice that the initialization of $\theta$ in each task may depend on $\phi$ or $\mathcal{S}_{i}$. For example, in MAML $\phi$ essentially defines an initialization of model parameters, and in LEO the initial classifier parameter is generated conditioned on $\phi$ and $\mathcal{S}_{i}$. Secondly, although the fine-tuning of $\theta$ largely follows Algorithm 1, some meta-learning methods additionally utilize meta-knowledge $\phi$. For example, in SIB the gradients for updating $\theta$ are predicted by $\phi$ using unlabelled query features instead of calculated from $\mathcal{L}(\hat{y},y)$ as in Algorithm 1.

A.5 Implementation Details

Prior to fine-tuning or meta-learning, we pre-trained a deep neural network (DNN) as feature extractor on the train split of a dataset. We use ResNet-10 or WRN-28-10 as feature extractor backbone. This section will present the architecture and exact training procedure for our backbones.

Network Architecture. The architecture of our ResNet-10 and WRN-28-10 backbone is shown in Figure A1. Specifically, each convolutional layer is described as “$n\times n$ conv, $p$”, where $n$ is the kernel size and $p$ is the number of output channels. Convolutional layers with “$/2$” have a stride of 2 and are used to perform downsampling. The solid curved lines represent identity shortcuts, and the dotted lines are projection shortcuts implemented by $1\times 1$ convolutions. The batch normalization and ReLU layers are omitted in Figure A1 to highlight the key structure of the two backbones.

Pre-training Procedure. The networks are trained from scratch with stochastic gradient descent in a fully-supervised manner, i.e., minimizing cross-entropy loss on the train split of a dataset. Specifically the training is conducted on 90 epochs with early stopping using validation accuracy. We used batch size of 256 and image size of $84\times 84$. For data augmentation, a random patch is sampled from an image, resized to $84\times 84$ and randomly flipped along horizontal axis before used for training. The initial learning rate is set to 0.1 and it is scaled down by factor of 10 every 30 epochs.

A.5.2 Fine-Tuning

We consider linear, cosine and $k$-NN classifier for our fine-tuning experiments. In a $K$-way FSL problem, the detailed implementations for the classifier function $f(\mathbf{x})$ are:

$k$-NN. Our implementation of $k$-NN is similar to . For each of the $K$ classes, we first calculated the average support set feature (centroid) denoted as $\mathbf{x}_{i},i\in\{1,\ldots,K\}$. The classifier output for class $i$ is then given by $f_{i}(\mathbf{x})=-\left\lVert\mathbf{x}-\mathbf{x}_{i}\right\rVert^{2}$. Notice that the prediction given by this classifier will be the nearest centroid.

We froze the backbone and used the average pooling layer output of $\Omega$ to learn the classifier. The output logits from classifier functions are normalized using softmax to generate probability output $P(y|\mathbf{x})$. For linear and cosine classifier, we followed and trained the classifier for 100 iteration with a batch size of 4. For fine-tuning baseline, we set the learning rate as $1\times 10^{-2}$ and weight decay as $1\times 10^{-3}$. For IFSL, we set the learning rate as $5\times 10^{-3}$ and weight decay as $1\times 10^{-3}$. $k$-NN classifier is non-parametric and can be initialized directly from support set.

A.5.3 Meta-Learning

MAML. MAML aims to learn an initialization of network parameters such that it can be fine-tuned within a few steps to solve a variety of few-shot classification tasks. When using pre-trained network with MAML, it has been shown that learning initialization of the backbone can lead to unsatisfactory performance . Therefore in our experiment, we froze the backbone and appended a 2-layer MLP with ReLU activation in between the hidden layers and a linear classifier after the average pooling layer of $\Omega$. The hidden dimension of the layers in MLP is the same as output dimension of $\Omega$ (512 for ResNet-10 and 640 for WRN-28-10). The initialization of MLP and the linear classifier is meta-learnt using MAML. For hyper-parameters, we set the inner loop learning rate $\alpha=0.01$, the outer loop learning rate $\beta=0.01$ and the number of adaptation steps as $20$. For IFSL, we adopted the same hyper-parameter setting and set $n$=8 for feature-wise and combined adjustment. Implementation-wise, we adopted the released codehttps://github.com/wyharveychen/CloserLookFewShot from and performed experiments on MAML without using first-order approximation. Following the implementation in , the model was trained on 10,000 randomly sampled tasks with model selection using validation accuracy. We used 2,000 randomly sampled tasks for validation and testing.

MTL. MTL learns scaling and shifting parameters at each convolutional layer of the backbone. We used the MTL implementation released by the authorhttps://github.com/yaoyao-liu/meta-transfer-learning which adopts linear classifier. We integrated our ResNet-10 and WRN-28-10 backbones into the released code. The learning rate for scaling and shifting weights $\phi_{SS}$ and initial classifier parameters was set to $1\times 10^{-4}$ uniformly. We set the inner loop learning rate for classifier as $1\times 10^{-2}$ and the inner loop update step as 100. For IFSL, we adopted the same hyper-parameter setting and set $n$=8 for feature-wise and combined adjustment. We trained the MTL model on 10,000 randomly sampled tasks with model selection using validation accuracy and used 2,000 randomly sampled tasks for validation and testing. We used 3 RTX 2080 Ti for MTL experiments on WRN-28-10 backbone.

LEO. LEO learns to generate classifier parameters conditioned on support set and the generated parameters are further fine-tuned within each FSL task. Our experiments were conducted on the released code of LEOhttps://github.com/deepmind/leo using linear classifier. Following author’s implementation, we saved the center cropped features from our pre-trained backbones and used the saved features to train LEO. For baseline, we used the hyper-parameter settings released by the author. For IFSL, we set $n$=8 for feature-wise and combined adjustment and halved the outer loop learning rate compared to baseline. The model was trained up to 100,000 randomly sampled tasks from training split with early stopping using validation accuracy. We used 2,000 randomly sampled tasks for validation and testing.

Matching Net. Matching Net (MN) is a metric-based method that learns a distance kernel function for $k$-NN. We used the Matching Net implementation in . The implementation follows the setup in and uses LSTM-based fully conditional embedding. We set the learning rate as 0.01 uniformly. For IFSL, we used $n$=16 for feature-wise and combined adjustment. The model was trained using 10,000 randomly sampled tasks with model selection using validation accuracy. We used 2,000 randomly sampled tasks for validation and testing.

SIB. SIB initializes classifier from support set and generates gradients conditioned on unlabelled query set features to update classifier parameters. We followed the SIB implementation released by the authorhttps://github.com/hushell/sib_meta_learn which uses cosine classifier. In the transductive setting, the query set size is set to 15. In the inductive setting, we used only 1 query sample randomly selected from the $K$ classes in each episode. In terms of hyper-parameter settings, we took 3 synthetic gradient steps ($K=3$) for all our experiments. For baseline, the learning rate for SIB network and classifier was set to $1\times 10^{-3}$ following author’s implementation. For IFSL, we set the learning rate to $5\times 10^{-4}$ and used $n$=4 for feature-wise and combined adjustment. In both transductive and inductive settings, we meta-trained SIB using 50,000 randomly sampled tasks with model selection using validation accuracy. We used 2,000 randomly sampled tasks for validation and testing.

A.6 Additional Results

In this section, we include additional results on 1) Conventional Acc in Table A.6.1 supplementary to Table 1; 2) Hardness-Specific Acc in Figure A2 for miniImageNet and Figure A3 for tieredImageNet, supplementary to Figure 5; 3) CAM-Acc in Table A.6.3 supplementary to Figure 6; 4) Cross-Domain Evaluation in Table A.6.4 supplementary to Table 3.

A.6.2 Hardness-Specific Acc

A.6.3 CAM-Acc

A.6.4 Cross-Domain Evaluation