Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models

Introduction

Supervised learning techniques train networks with full access to all data, which can result in a lack of flexibility when extending them to acquire knowledge from new tasks. Continual Learning (CL) has emerged as a solution, enabling ongoing model training on sequentially arriving data while retaining the learned information . Conventional CL settings consider either newly introduced classes or domain distribution shifts, referred to as class incremental and domain incremental learning . However, with only one type of increment considered, these existing works limit their applicability in complex real-world scenarios.

Consider a more challenging Domain-Class Incremental Learning (DCIL) setting, where both the domain data distribution and classes to be classified can keep varying among all tasks, as illustrated in Fig. 1(a). Vanilla image encoder-based techniques are infeasible under such circumstances due to their non-scalable classification head design. Recently, the advent of contrastively trained Vision-Language Models (VLMs), such as CLIP , has made it possible to address this demanding but practical problem. VLMs are trained on web-scale image-text pairs and hold a powerful zero-shot generalization ability to identify nearly infinite classes, making them capable of confronting this severe task variation scenario .

However, the use of vision-language models introduces new challenges to incremental training. Conventional continual learning schemes aim to prevent models from forgetting previously learned knowledge, which is termed as backward forgetting . Existing works have explored the potential of the regularization mechanism, rehearsal buffer, and architecture design to mitigate backward forgetting, achieving promising results . Nevertheless, when these approaches are applied to vision-language models, a different form of catastrophic forgetting emerges: models tend to forget the knowledge learned during the pre-training phase, thus compromising their powerful zero-shot generalization capacity. This problem is termed as forward forgetting because it occurs when VLMs perform “forward” prediction on the unknown distributed data. Fig. 1(a) illustrates the two types of forgetting.

Recent work ZSCL made an attempt to address the forward forgetting issue on CLIP. They introduced a large-scale reference dataset to perform knowledge distillation and incorporated a weight ensemble scheme . However, this approach requires intensive computation and external data, which could be infeasible in real-world scenarios. Meanwhile, existing VLM-based parameter-efficient continual learning methods , mostly utilizing prompt tuning mechanisms, fail to retain the pre-trained knowledge and cause zero-shot ability degradation, as shown in Fig. 1(b). We attribute this issue to information interference: newly introduced task-specific parameters can disturb the pre-trained knowledge. Illustrations of these methods are shown in Fig. 1(c).

To alleviate the forward forgetting problem of VLMs with a computationally and parameter-efficient approach, we introduce the Distribution-aware Interference-free Knowledge Integration (DIKI) framework. Specifically, we inject task-specific information into frozen VLM for each task, storing learned knowledge efficiently. (1) To maintain the pre-trained knowledge in VLMs, our knowledge integration mechanism is designed to resolve the information interference issue prevalent in existing methods. By employing our fully residual design and zero-initialization strategy, we can inject new knowledge while keeping the pre-trained knowledge untouched, introducing minimal noise to the pre-trained model compared to prompt tuning. (2) With this advantage, we further introduce a distribution-aware integration calibration mechanism, explicitly identifying the unseen distributed samples and controlling the implanted information for them, thereby enhancing the model generalization capabilities.

Our contributions are summarized in threefold:

We introduce the parameter-efficient DIKI to retain pre-trained knowledge in VLMs under the DCIL setting. It resolves the information interference issue, mitigating the need for heavy computation and external data.

To alleviate the forward forgetting, DIKI implants new knowledge in a fully residual manner, leaving pre-trained knowledge undisturbed. With this residual property, a distribution-aware integration calibration is incorporated to further boost performance on unseen tasks.

Comprehensive experiments demonstrate that we achieve state-of-the-art performance with only 0.86% trained parameters and significantly less training time compared to the previous methods.

Related Works

Continual learning. Existing continual learning algorithms can be broadly classified into three categories . Regularization-based methods introduce an extra regularization term in the loss function, consolidating previous knowledge when learning on new data. In contrast, architecture-based methods dedicate different model parameters to each task, storing task knowledge with specific expanded network components. With memory replay technique, rehearsal-based methods retrain current step model with stored exemplars in raw format or generated pseudo-samples with a generative model, which has been questioned for its rationality by recent work . While achieving promising results, these solutions only consider one type of increment, either domain shift or new classes, along the continual training process, resulting in limited applicability in real-world scenarios. Instead, we investigate the forgetting problem under a domain-class incremental learning setting to adapt to a broader variety of situations.

Parameter-efficient fine-tuning. Fully fine-tuning a large pre-trained model is computationally expensive and requires a large-scale dataset . Alternatively, parameter-efficient fine-tuning approaches only introduce a small set of parameters to rapidly adapt a pre-trained model to downstream tasks, such as LoRA , prompt tuning and adapters . Due to their simple and portable design, prompt tuning techniques have attracted many applications in a variety of areas . However, existing prompt learning-based methods typically prepend the learnable parameters to the original input tokens, where lies the information interference issue and eventually causes pre-trained knowledge loss during the training process.

Vision-language models. Trivial visual-only models extract features from images and then utilize a fixed head to derive final predictions, constraining their flexibility across tasks . Vision-Language Models (VLMs) present a solution by leveraging the interaction between image and text descriptions . Trained on web-scale image-text pair datasets, V-L models can identify nearly infinite classes and can be easily transferred to unseen domains, holding a strong zero-shot ability. However, most previous VLMs continual learning methods have not considered the zero-shot performance drop during the training process, which can cause a significant model degradation towards unseen data distributions.

Preliminaries

Continual learning protocol. Continual learning aims to sequentially learn different tasks without forgetting previously learned knowledge. Considering $N$ sequentially arrived tasks $\left[\mathcal{T}^{1},\mathcal{T}^{2},\cdots,\mathcal{T}^{N}\right]$, each task $\mathcal{T}^{i}$ contains a dataset $D^{i}=\{x^{i}_{j},y^{i}_{j}\}_{j=1}^{N^{i}}$, where $x^{i}_{j}$ is an image and $y^{i}_{j}$ is corresponding one-hot label inside current dataset, and $N_{i}$ is the number of image samples. Additionally, a class name set $C^{i}=\{c^{i}_{j}\}_{j=1}^{N_{c}^{i}}$ is included, linking the label index to a category name used by the VLMs.

Vision-language models. Towards the challenging DCIL setting, training a vanilla image encoder-based model, such as ResNets and ViTs , is not practical for incrementally learning intensely shifted domains and classes. Hence, pre-trained vision-language models are introduced for their robust zero-shot transfer capabilities. CLIP consists of an image encoder $f$ and a text encoder $g$, which are trained to generate closely aligned feature representations for paired image-text samples. At inference time, $f$ first encodes the input image $x$ into a feature vector $f(x)$. Concurrently, potential class names are embedded into a template, like “a photo of {$c$}”, and then encoded by $g$ to form text embeddings $\{t_{j}\}_{j=1}^{N_{c}}$. The model predictions are determined by the largest similarity scores between image embedding and all text embeddings, formulated as $s_{j}=\Braket{f(x),t_{j}}$, where $\Braket{\cdot,\cdot}$ denotes the cosine similarity.

Task-specific prompt learning. Following the success of , a series of works begin to explore the potential of parameter-efficient fine-tuning in continual learning. A common practice is learning and storing a set of lightweight prompts for each task, forming a “prompt pool” during the continual learning phase, formulated as:

where $N$ is the task number, $l$ and $d$ are the prompt length and the feature embedding dimension.

The prompt selection process mentioned above is implemented by query-key matching. During the continual training stage, average feature representations $\mathbf{I}=\{I^{i}\}_{i=1}^{N}$ for each task are learned by maximizing cosine similarity or by applying clustering algorithm . When a test sample $\bm{x}$ comes, a key lookup regime is performed:

With the most relevant key $I_{s}$, corresponding prompts $P_{s}$ are selected and attached to the frozen model, performing inference process.

Methodology

Is prepending the best choice? Despite methods that prepend prompt to input tokens are widely used for their simplicity in implementation, we identified that they are suffering from issues in two folds.

Firstly, concatenating the prompts with input tokens causes them to interact during the attention process, and influences the pre-trained knowledge extraction, which will be discussed below. When the test samples are drawn from the distribution where the model learned the prompts, the adapted model can preserve relatively satisfactory results. However, once encountering samples with a distribution shift, this interference could result in model degradation and a loss of its vital zero-shot generalization ability, causing forward forgetting issues.

Besides, simply prepending prompts inevitably increases the token length across all transformer blocks, which is not desirable in many scenarios with token length constraints. In addition, its scalability is limited: a long prompt context can distract the text encoder from informative class names, resulting in poor text embedding representation.

The existence of the above issues indicates that prompt tuning-based methods do not satisfy the “residual property”: we expect learned parameters should be a residual path paralleled to the frozen backbone, supplementing novel knowledge without affecting the crucial pre-trained knowledge. Therefore, we propose a Interference-free Knowledge Integration (IKI) scheme to inject newly learned knowledge into a pre-trained VLM with introducing minimal noise to it.

To address this issue, we compute attention outputs for self-attention within input tokens and cross-attention between prompts and input tokens separately, as shown in Fig. 2 (b). In other words, we only train a residual attention branch, leaving the existing attention score untouched. With newly introduced keys $K_{r}$ and values $V_{r}$, the output of our residual attention branch can be formulated as:

Ideally, a desirable residual block should not affect the original branch before being trained on downstream datasets, i.e. at initialization time. Widely used protocols initialize prompts with uniform or normal distribution, which injects random noise into the pre-trained VLMs even when no knowledge has been learned. Specifically, we enforce residual attention addition to be an identity function by zero-initialize the parameters $V_{r}$:

Note that we only constrain values $V_{r}^{\text{init}}$ to be zero at the beginning, while keeping $K_{r}$ random initialized. That’s because initializing both $K_{r}$ and $V_{r}$ to zero matrix will prevent $K_{r}$ from updating by gradient flow, and make $V_{r}$ degenerate to vectors with same values. We prove this in the supplementary materials.

Since zero-initialization is more like a choice rather than a technique, some studies have adopted it across various tasks. However, these works leverage it to ensure a stable and progressive training regime, a concern that is not present in DCIL scenarios. We argue that zero-initialization is essential for our residual attention design to inject new knowledge into the pre-trained VLMs with minimal noise introduced, which is demonstrated in Sec. 5.2.

2 Distribution-aware Integration Calibration

Observations. At inference time, the query-key matching mechanism described in Eq. 3 is performed to retrieve appropriate learned prompts for the current test sample. This approach is tailored for conventional continual learning settings, which only considers the backward forgetting mentioned in Sec. 1. However, when confronted with data from unseen domains, this trivial matching design is enforced to assign a relatively similar task for test samples, despite there’s a significant distribution gap between them.

Benefiting from the residual design of our proposed IKI, we can introduce less noise in such mismatch scenarios compared with previous methods. Nonetheless, when the discrepancy between training and testing distribution increases, it’s inevitable to cause model degradation to some extent and hurt the zero-shot ability that VLMs learned during the pre-train phase.

ZSCL tackles this problem via distillation. They build a reference dataset with 100k images from ImageNet to distill pre-trained knowledge from the original CLIP to the current model at every training step, explicitly performing rehearsal to avoid forgetting. This approach could be effective, but it relies on large-scale storage and high computation resources as shown in Tab. 5, making it impractical under real-world circumstances.

One intuitive solution to this issue is controlling to what extent knowledge is implanted into the model. However, previous prepending-based prompt tuning techniques have only two choices: either appending learned prompts or leaving the original CLIP model untouched. Thanks to the graceful residual property from our IKI, we obtain the ability to control this paralleled branch.

DIKI: calibrate the integration with distribution. To determine the likelihood that a test sample belongs to a learned task, we maintain a feature distribution instead of a single key vector for every task. Here we simply apply multivariate Gaussian distribution and find it works well. Formally, we build a $\mathcal{N}^{i}(\bm{\mu}^{i},\bm{\Sigma}^{i})$ for task $i$ during training stage:

where $f(\bm{x}^{i}_{j})$ is the image feature extracted by frozen encoder. With these estimated distributions, the possibility of a test sample being drawn from each $\mathcal{N}^{i}$ can be calculated. Here we compute the logarithm of the probability density as a scoring function for input $\bm{x}$ on each learned task:

where $\varphi$ is the probability density function.

Intuitively, a sample with a higher score $S^{i}$ is more likely to be drawn from task $i$, and parameters $K_{r}^{i},V_{r}^{i}$ should be introduced for model prediction. Besides, we should also take into account that income sample $\bm{x}$ might come from some new distributions, which is suggested if all $S^{i}$ are low. Thus we utilize the maximum score $\hat{S}=\max_{i\in[1,N]}S^{i}$ to weight the residual attention output:

where $\mathcal{M}$ is a mapping function that scales the score $\hat{S}$ to the range $$. Here we find a simple Sigmoid function$\sigma(x)=\frac{1}{1+e^{-x}}$ works well here. We also conduct experiments in Sec. 5.2 to demonstrate the rationality and correctness of the calibration technique on IKI outputs.

Empowered by this distribution-aware integration calibration mechanism, the pre-trained zero-shot ability of VLMs can be retained better by assign lower weight to unfamiliar images, further resolving the forward forgetting issue.

Experiments

Benchmarks. To demonstrate the effectiveness of DIKI under the domain-class incremental learning setting, we conduct experiments on the recently proposed MTIL benchmark. MTIL consists of 11 diverse datasets: Aircraft , Caltech101 , CIFAR100 , DTD , EuroSAT , Flowers , Food , MNIST , OxfordPet , StanfordCars , and SUN397 . It’s a very challenging benchmark with total of 1201 classes and severe data distribution shift across different tasks, which is infeasible for vanilla image encoder-based methods. Thus, vision-language models are necessarily included. The Order-I in original paper is applied. We also introduce the modified MTIL-FS benchmark for few-shot setting evaluation, in which only 16 samples per class of each dataset are used for training to simulate the data deficient scenario. More details can be found in the supplementary materials.

Evaluation metrics. To evaluate both backward and forward forgetting issues mentioned in Sec. 1, we adopt Transfer, Avg. and Last metrics from . Last score is the model performance after all continual training, representing the degree of backward forgetting and being widely used in conventional continual learning. For forward forgetting issues, i.e. the loss of zero-shot ability, we evaluate model average accuracy on task $i+1,i+2,...,N$ after its training on task $i$, denoted by Transfer. Lastly, Avg. is the average accuracy across all time steps. Detailed formulations can be found in the supplementary materials.

Comparison methods. We compare our DIKI against both full-parameter fine-tuning and parameter-efficient fine-tuning methods. For full fine-tuning, we choose ZSCL, ZSCL* , LwF , iCaRL , LwF-VR , and WiSE-FT following . For parameter efficient ones, L2P , DualPrompt , and S-Prompts are selected for the similar task-specific parameter training procedure to our DIKI. Note that original L2P and DualPrompt are designed for ViT , we reproduce them on CLIP for fair comparisons. More reproduction details can be found in the supplementary materials.

Implementation details. We adopt CLIP ViT-B/16 as our vision-language model for fair comparisons. In the training process, we optimize the cross entropy loss between model prediction and ground truth. SGD optimizer with cosine learning rate scheduler is applied for all experiments, and the learning rate and batch size are set to 5 and 128, separately. Models are trained with 10 epochs on each task. For trainable parameters $K_{r}$ and $V_{r}$, we set both the length $l$ and training layer depth to 8 as discussed in the supplementary materials. To avoid floating point arithmetic precision problems, a small number $10^{-7}$ is added to diagonal elements of covariance matrix $\Sigma^{i}$ with minor influence on final accuracy. All experiments are conducted on one NVIDIA 3090 GPU.

Tab. 1 contains the Transfer, Avg. and Last scores among all methods on MTIL benchmark. “Extra data” includes memory buffers and reference datasets which used in distillation , and “# Param.” is the number of trainable parameters. “Zero-shot” results are simply derived from leveraging the original CLIP weight on each task and perform as a comparison reference for Transfer metric. Note that Transfer scores can be higher than zero-shot results, because knowledge learned from current task $i$ may contain some task-invariant information which can boost the performance of future tasks $i+1,i+2,...,N$. “Upper Bound” is calculated by applying full parameter fine-tuning technique on each separate dataset, as a guide for Last score.

As indicated by the bold values, our DIKI outperforms the previous state-of-the-art method across all three metrics with only 0.86% trainable parameters, while alleviating the requirement for any external data. Thanks to the task-specific parameter training technique, we can memorize previous tasks’ knowledge without rehearsal buffers and parameter ensemble, maintaining a high Last score with low computational complexity. Moreover, compared with task-specific prompt tuning methods (L2P, DualPrompt, and S-Prompts), we achieve significant improvement on Transfer metric, which shows that our DIKI mechanism can effectively inject new information to the frozen backbone without interfering with pre-trained knowledge.

We also conduct experiments on the 16-shot MTIL-FS benchmark. Abbreviated results are shown in Tab. 3 and the full table can be found in the supplementary materials. Since we only update a small amount of parameters, we gain more improvement over ZSCL compared to full parameter training. In addition, with minimal noise introduced, our fully residual IKI design demonstrates enhanced competitiveness when training data is deficient, compared to other interruptive prompt tuning methods.

2 Analysis

Ablation study. We ablate our proposed modules of DIKI on MTIL benchmark in Tab. 3. Firstly we consider Transfer score (i.e. zero-shot ability): from the first two rows, it can be seen that the zero-initialization mechanism is ineffective to prompt tuning methods, because they can still disturb the pre-trained knowledge by softmax function inside the attention calculation. However, with our residual attention design, the effect of zero-initialization is activated. They can work together to avoid introducing irrelevant information to the frozen backbone. Thanks to the fully residual property, distribution-aware calibration can be exploited to further boost performance by identifying unseen distributions.

Considering the Last metric, our interference-free mechanism stores more task knowledge because of its clear information injection process, thus enhancing the last state accuracy. However since our zero-initialization and distribution-aware calibration are designed to improve the retention of pre-trained knowledge, the addition of them does not result in an increase on Last score.

Effect of zero-initialized residual attention. To demonstrate the effect of our zero-initialization paradigm, we conduct experiments for different distributed initialization strategies on the MTIL benchmark, as shown in Fig. 4. Following previous common practice , we choose uniform distribution with different bounds to initialize our trainable $K_{r}$ and $V_{r}$ in Eq. 6. Results show that with different initialization values, the model can achieve constant final performance after being trained on all tasks (Last score keeps invariant). However, as the initialization bound increases, model’s zero-shot ability degenerates due to the noise introduced by random initialization (Transfer score is decreasing).

Effect of distribution aware calibration. To demonstrate our calibration technique, we conduct experiments with manually set calibration weights. Specifically, we train the model exclusively on the first task of MTIL (Aircraft dataset) and test it on all tasks, including trained and unseen datasets. Here we replace $\mathcal{M}(\hat{S})$ in Eq. 10 with fixed values, as shown in Fig. 4. When the weight is set to 1.0, which means full use of newly learned knowledge, the trained task accuracy is maximized while the vital zero-shot ability is interfered with. Conversely, as weight decreases, the zero-shot capability returns, while trained task accuracy decreases due to the reduced incorporation of new knowledge.

Our distribution-aware attention calibration tailors appropriate weights for different inference samples by the distribution modeling, allocating higher/lower weights to samples from learned/unseen domains. It alleviates the need to select a “balance point” which compromises overall performance.

Effect of IKI on CIL. To validate the universality of the proposed IKI, we evaluate it on the conventional Class Incremental Learning (CIL) task. Specifically, IKI is integrated into existing prompt-based CIL methods, serving as a replacement for their original prepending mechanisms. Experiments are conducted on the 10-split CIFAR-100 dataset following the common protocol , as shown in Tab. 5. IKI explicitly formulates a knowledge injection process, thus boosting the average accuracy by achieving superior performance on each task. For the forgetting metric, result of L2P remains comparable due to the absence of shared information across tasks. Conversely, for methods with shared prompts (DualPrompt and CODA-P ), our non-interference attention mechanism facilitates the knowledge shareability and alleviates the forgetting problem.

Training cost analysis. We compare the computational requirement of our DIKI and previous state-of-the-art method ZSCL in Tab. 5. Benefiting from our parameter efficient framework, the training process of DIKI only lasts 2.3 hours on a single GPU, while ZSCL requires 4 GPUs, nearly half a day for training, and extra 100k images to perform distillation. With a much faster model adaptation speed, our method can be more effective and adoptable in tackling real-world continual learning problems.

Qualitative visualization results. We implement Grad-CAM on the attention maps of the CLIP visual encoder, following the practice used in , as depicted in Fig. 5. Specifically, we load the model which is only trained on the first dataset Aircraft of MTIL benchmark, and test it on several subsequent unseen datasets. We observe that the vanilla prompting way (employed by L2P, Dualprompt, and S-Prompts) interferes with pre-trained knowledge and undermines the zero-shot ability. However, with utilizing our DIKI, the generalization ability acquired during pre-training is preserved.

Conclusions

This study introduced Distribution-aware Interference-free Knowledge Integration (DIKI) mechanism for domain-class incremental learning. DIKI preserves the pre-trained knowledge of VLMs while effectively implanting new task information, without heavy computation and external data. DIKI infuses new knowledge into a frozen backbone in a fully residual manner, effectively mitigating the forward forgetting issue. A distribution-aware integration calibration technique is also integrated, which controls the information injection for data from unseen distributions. Experiments show that DIKI surpasses the previous SOTA method with only 0.86% trainable parameters.

Limitations and future work. Our DIKI follows a task-specific tuning paradigm, where the training on different tasks is independent. Although some recent CIL research works have verified the effect of sharing knowledge across tasks , we find these solutions are impractical within the DCIL context. Experiments are conducted in the supplementary materials. We attribute this to the significant domain gap among DCIL datasets, which hinders the shareability of knowledge from different tasks. Future works could explore suitable knowledge-sharing strategies tailored to the DCIL problems.

References