InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

Introduction

Large language models (LLMs) largely promote the development of artificial general intelligence (AGI) systems with their impressive capabilities in open-world language tasks, and their model scale and performance are still increasing at a fast pace. Vision large language models (VLLMs) , which leverage LLMs, have also achieved significant breakthroughs, enabling sophisticated vision-language dialogues and interactions. However, the progress of vision and vision-language foundation models, which are also crucial for VLLMs, has lagged behind the rapid growth of LLMs.

To bridge vision models with LLMs, existing VLLMs commonly employ lightweight “glue” layers, such as QFormer or linear projection , to align features of vision and language models. Such alignment contains several limitations: (1) Disparity in parameter scales. The large LLMs now boosts up to 1000 billion parameters, while the widely-used vision encoders of VLLMs are still around one billion. This gap may lead to the under-use of LLM’s capacity. (2) Inconsistent representation. Vision models, trained on pure-vision data or aligned with the BERT series , often exhibit representation inconsistencies with LLMs. (3) Inefficient connection. The “glue” layers are usually lightweight and randomly initialized, which may not capture the rich cross-modal interactions and dependencies that are crucial for multi-modal understanding and generation.

These limitations reveal a large gap in both parameter scale and feature representation ability between the vision encoder and the LLM. To bridge this gap, our inspiration lies in elevating the vision encoder to align with the parameter scale of the LLM and subsequently harmonizing their representations. However, the training of such large-scale models necessitates a vast amount of image-text data obtained from the Internet. The significant heterogeneity and quality variations within this data pose considerable challenges to the training process. To enhance the efficacy of the training, generative supervision is considered as a complementary approach to contrastive learning, as depicted in Figure 1. This strategy aims to provide additional guidance to the model during training. Yet, the suitability of low-quality data for generative training remains a concern. Besides, how to effectively represent the users’ commands and align the representations between the vision encoder and LLM is another open question.

To address these issues, we formulate the InternVL, a large-scale vision-language foundation model, which aligns the representation of the scaled-up vision encoder with the LLM and achieves state-of-the-art performance on various visual and vision-language tasks. As shown in Figure 1 (c), InternVL has three key designs: (1) Parameter-balanced vision and language components: It includes a vision encoder scaled up to 6 billion parameters and an LLM middleware with 8 billion parameters, where the middleware functions as a substantial “glue” layer to reorganize visual features based on user commands. Unlike prior vision-only (Figure 1 (a)) or dual-tower (Figure 1 (b)) structures, our vision encoder and middleware offer flexible combinations for both contrastive and generative tasks. (2) Consistent representations: To maintain the consistency of representations between the vision encoder and LLM, we employ a pre-trained multilingual LLaMA , to initialize the middleware and align the vision encoder with it. (3) Progressive image-text alignment: We leverage image-text data from diverse sources, ensuring training stability through a progressive alignment strategy. This strategy initiates contrastive learning on large-scale noisy image-text data and subsequently transitions to generative learning on fine-grained data. This approach ensures a consistent enhancement of model performance and task scope.

These designs endow our model with several advantages: (1) Versatile. It functions as a standalone vision encoder for perception tasks, or collaborates with the language middleware for vision-language tasks and multi-modal dialogue systems. The language middleware bridges the gap between the vision encoder and the LLM decoder. (2) Strong. By leveraging the training strategy, large-scale parameters, and web-scale data, our model has a powerful representation that helps to achieve state-of-the-art results on various vision and vision-language tasks, as shown in Figure 2. (3) LLM-friendly. Due to the aligned feature space with LLMs, our model can smoothly integrate with existing LLMs, such as LLaMA series , Vicuna , and InternLM . These features distinguish our model from the previous approaches and establish a leading vision-language foundation model for various applications.

In summary, our contribution has three folds:

(1) We present a large-scale vision-language foundation model—InternVL, which aligns the large-scale vision encoder with LLMs for the first time. The model demonstrates strong performance on a wide range of generic visual-linguistic tasks, including visual perception tasks, vision-language tasks, and multi-modal dialogue.

(2) We introduce a progressive image-text alignment strategy for the efficient training of large-scale vision-language foundation models. This strategy maximizes the utilization of web-scale noisy image-text data for contrastive learning and fine-grained, high-quality data for generative learning.

(3) We extensively compare the proposed model with the current state-of-the-art vision foundation models and VLLMs. The results indicate that InternVL achieves leading performance on a broad range of generic visual-linguistic tasks, including image classification (ImageNet), semantic segmentation (ADE20K), video classification (Kinetics), image-text retrieval (Flickr30K & COCO), video-text retrieval (MSR-VTT), and image captioning (COCO & Flickr30K & NoCaps). Meanwhile, it is also effective for multi-modal dialogue (MME & POPE & Tiny LVLM).

Related Work

The past decade has witnessed significant development in foundation models within the field of computer vision. Starting with the pioneering AlexNet , a variety of convolutional neural networks (CNNs) have emerged, continuously refreshing the ImageNet benchmark . In particular, the introduction of residual connections effectively addressed the problem of vanishing gradients. This breakthrough led to an era of “big & deep” neural networks, signifying that, with adequate training and data, larger and deeper models can achieve better performance. In other words, scaling up matters.

In recent years, ViT has opened up new possibilities for network architectures in the computer vision field. ViT and its variants have significantly increased their capacity and excelled in various important visual tasks. In the LLM era, these vision foundation models often connect with LLMs through some lightweight “glue” layers . However, a gap exists as these models primarily derive from visual-only datasets like ImageNet or JFT , or are aligned with the BERT series using image-text pairs, lacking direct alignment with LLMs. Additionally, the prevalent vision models employed to connect with LLMs are still limited to around 1 billion parameters , which also constrains the performance of VLLMs.

2 Large Language Models

Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought exclusive to humans . The emergence of GPT-3 brought a significant leap in capabilities, particularly in few-shot and zero-shot learning, highlighting the immense potential of LLMs. This promise was further realized with the advancements of ChatGPT and GPT-4 . The progress in the field has been further accelerated by the emergence of open-source LLMs, including the LLaMA series , Vicuna , InternLM , MOSS , ChatGLM , Qwen , Baichuan , and Falcon , among others . However, in real scenarios, interactions are not limited to natural language. The vision modality can bring additional information, which means more possibilities. Therefore, exploring how to utilize the excellent capabilities of LLMs for multi-modal interactions is poised to become the next research trend.

3 Vision Large Language Models

Recent advancements have seen the creation of vision large language models (VLLMs) , which aim to enhance language models with the capability to process and interpret visual information. Flamingo uses the visual and language inputs as prompts and shows remarkable few-shot performance for visual question answering. Subsequently, GPT-4 , LLaVA series and MiniGPT-4 have brought in visual instruction tuning, to improve the instruction-following ability of VLLMs. Concurrently, models such as VisionLLM , KOSMOS-2 , and Qwen-VL et al. have improved VLLMs with visual grounding capabilities, facilitating tasks such as region description and localization. Many API-based methods have also attempted to integrate vision APIs with LLMs for solving vision-centric tasks. Additionally, PaLM-E and EmbodiedGPT represent advanced efforts in adapting VLLMs for embodied applications, significantly expanding their potential applications. These works showcase that VLLMs have achieved significant breakthroughs. However, the progress of vision and vision-language foundation models, equally essential for VLLMs, has not kept pace.

Proposed Method

As depicted in Figure 3, unlike traditional vision-only backbones and dual-encoder models , the proposed InternVL is designed with a vision encoder InternViT-6B and a language middleware QLLaMA. Specifically, InternViT-6B is a vision transformer with 6 billion parameters, customized to achieve a favorable trade-off between performance and efficiency. QLLaMA is a language middleware with 8 billion parameters, initialized with a multilingual-enhanced LLaMA . It could provide robust multilingual representation for image-text contrastive learning, or serve as a bridge to connect the vision encoder and the off-the-shelf LLM decoder.

To align the two large-scale components with substantial gaps in modalities and structures, we introduce a progressive alignment training strategy. The training strategy is conducted progressively, beginning with contrastive learning on large-scale noisy data, and gradually moving towards generative learning on exquisite and high-quality data. In this way, we ensure the effective organization and full utilization of web-scale image-text data from a variety of sources. Then, equipped with the aligned vision encoder and language middleware, our model functions like a Swiss Army knife. It boasts a flexible composition that can be adapted for a wide array of generic visual-linguistic tasks. These tasks range from visual perception and image/video-text retrieval to image captioning, visual question answering, and multi-modal dialogue, among others.

2 Model Design

Large-Scale Vision Encoder: InternViT-6B. We implement the vision encoder of InternVL with vanilla vision transformer (ViT) . To match the scale of LLMs, we scale up the vision encoder to 6 billion parameters, resulting in the InternViT-6B model. To obtain a good trade-off between accuracy, speed, and stability, we conduct a hyperparameter search for InternViT-6B. We vary the model depth within {32, 48, 64, 80}, the head dimension within {64, 128}, and the MLP ratio within {4, 8}. The model width and the head number are calculated based on the given model scale and other hyperparameters.

We employ contrastive learning on a 100M subset of the LAION-en dataset to measure the accuracy, speed, and stability of InternViT-6B variants with different configurations. We report the following findings: (1) Speed. For different model settings, when computation is not saturated, the models with smaller depths exhibit faster speed per image. However, as the GPU computation is fully utilized, the speed difference becomes negligible; (2) Accuracy. With the same number of parameters, the depth, head dimension, and MLP ratio have little impact on the performance. Based on these findings, we identified the most stable configuration for our final model, as shown in Table 1.

Language Middleware: QLLaMA. The language middleware QLLaMA is proposed to align visual and linguistic features. As shown in Figure 3, QLLaMA is developed based on the pre-trained multilingual LLaMA , and newly added 96 learnable queries and cross-attention layers (1 billion parameters) that are randomly initialized. This manner allows QLLaMA to smoothly integrate visual elements into the language model, thereby enhancing the coherence and effectiveness of the combined features.

Compared to recently popular approaches that use lightweight “glue” layers, such as QFormer and linear layers to connect vision encoder and LLMs, our method has three advantages: (1) By initializing with the pre-trained weights of , QLLaMA can transform image tokens generated by InternViT-6B into the representation that is aligned with the LLMs; (2) QLLaMA has 8 billion parameters for vision-language alignment, which are 42 times larger than the QFormer. Therefore, even with a frozen LLM decoder, InternVL can achieve promising performance on multi-modal dialogue tasks. (3) It can also be applied to contrastive learning, providing a powerful text representation for image-text alignment tasks, such as zero-shot image classification and image-text retrieval.

“Swiss Army Knife” Model: InternVL. By flexibly combining the vision encoder and the language middleware, InternVL can support various vision or vision-language tasks.

(2) For contrastive tasks, as shown in Figure 4 (a) (b), we introduce two inference modes: InternVL-C and InternVL-G, using the vision encoder or the combination of InternViT and QLLaMA to encode visual features. Specifically, we apply attention pooling to the visual features of InternViT or the query features of QLLaMA, to calculate the global visual feature $I_{f}$. Besides, we encode text as $T_{f}$ by extracting the feature from the [EOS] token of QLLaMA. By computing similarity scores between $I_{f}$ and $T_{f}$, we support various contrastive tasks such as image-text retrieval.

(3) For generative tasks, unlike QFormer , QLLaMA inherently has promising image captioning abilities thanks to its scaled-up parameters. The queries of QLLaMA reorganize the visual representations from InternViT-6B and play as the prefix texts for QLLaMA. The subsequent text tokens are generated one by one sequentially.

(4) For multi-modal dialogue, we introduce InternVL-Chat, leveraging InternVL as the visual component to connect with LLMs. For this purpose, we have two distinct configurations. One option is to employ the InternViT-6B independently, as shown in Figure 4 (c). The alternative is to employ the complete InternVL model concurrently, as illustrated in Figure 4 (d).

3 Alignment Strategy

As shown in Figure 3, the training of InternVL consists of three progressive stages, including vision-language contrastive training, vision-language generative training, and supervised fine-tuning. These stages effectively leverage public data from diverse sources, ranging from noisy image-text pairs on the web to high-quality caption, VQA, and multi-modal dialogue datasets.

Vision-Language Contrastive Training. In the first stage, we conduct contrastive learning to align InternViT-6B with a multilingual LLaMA-7B on web-scale, noisy image-text pairs. The data are all publicly available and comprise multilingual content, including LAION-en , LAION-multi , LAION-COCO , COYO , Wukong , etc. We use the combination of these datasets and filter out some extremely low-quality data to train our model. As summarized in Table 2, the original dataset contains 6.03 billion image-text pairs, and 4.98 billion remains after cleaning. More details about data preparation will be provided in the supplementary materials.

During training, we adopt the LLaMA-7B to encode the text as $T_{f}$, and use InternViT-6B to extract the visual feature $I_{f}$. Following the objective function of CLIP , we minimize a symmetric cross-entropy loss on the similarity scores of image-text pairs in a batch. This stage allows InternVL to excel on contrastive tasks like zero-shot image classification and image-text retrieval, and the vision encoder of this stage can also perform well on visual perception tasks like semantic segmentation.

Vision-Language Generative Training. In the second stage of training, we connect InternViT-6B with QLLaMA and adopt a generative training strategy. Specifically, QLLaMA inherits the weights of LLaMA-7B in the first stage. We keep both InternViT-6B and QLLaMA frozen and only train the newly added learnable queries and cross-attention layers with filtered, high-quality data. Table 2 summarizes the datasets for the second stage. It can be seen that we further filtered out data with low-quality captions, reducing it from 4.98 billion in the first stage to 1.03 billion.

Following the loss function of BLIP-2 , the loss in this stage is computed as the sum of three components: image-text contrastive (ITC) loss, image-text matching (ITM) loss, and image-grounded text generation (ITG) loss. This enables the queries to extract powerful visual representations, and further align feature space with LLMs, attributable to the effective training objectives and the utilization of our large-scale, LLM-initialized QLLaMA.

Supervised Fine-tuning. To demonstrate the benefits of InternVL in creating multi-modal dialogue systems, we connect it with an off-the-shelf LLM decoder (e.g., Vicuna or InternLM ) through an MLP layer, and conduct supervised fine-tuning (SFT). As detailed in Table 3, we collect a wide range of high-quality instruction data, totaling approximately 4 million samples. For non-dialogue datasets, we follow the method described in for conversion. Owing to the similar feature space of QLLaMA and LLMs, we can achieve robust performance even when freezing the LLM decoder, choosing to train just the MLP layer or both the MLP layer and QLLaMA. This approach not only expedites the SFT process but also maintains the original language capabilities of the LLMs.

Experiments

Stage 1. In this stage, the image encoder InternViT-6B is randomly initialized , and the text encoder LLaMA-7B is initialized with the pre-trained weights from . All parameters are fully trainable.

Stage 2. In this stage, InternViT-6B and QLLaMA inherit their weights from the first stage, while the new learnable queries and cross-attention layers in QLLaMA are randomly initialized. Benefiting from the powerful representations learned in the first stage, we keep both InternViT-6B and QLLaMA frozen and only train the new parameters.

Stage 3. At this stage, we have two different configurations. One is to use InternViT-6B separately, as shown in Figure 4 (c). The other is to use the entire InternVL model simultaneously, as shown in Figure 4 (d). More details will be provided in the supplementary materials.

2 Visual Perception Benchmarks

First of all, we validate the visual perception capabilities of InternViT-6B, the most core component of InternVL.

Transfer to Image Classification. We evaluate the quality of visual representation produced by InternViT-6B using the ImageNet-1K dataset. Following common practices , we adopt the linear probing evaluation, i.e. training a linear classifier while keeping the backbone frozen. In addition to the ImageNet-1K validation set, we also report performance metrics on several ImageNet variants , to benchmark the domain generalization capability. As shown in Table 4, InternViT-6B achieves a very significant improvement over previous state-of-the-art methods on linear probing. To our knowledge, this represents the currently best linear evaluation results without the JFT dataset .

Transfer to Semantic Segmentation. To investigate the pixel-level perceptual capacity of InternViT-6B, we conduct extensive experiments of semantic segmentation on the ADE20K dataset. Following ViT-22B , we begin with few-shot learning experiments, i.e. fine-tuning the backbone with a linear head on a limited dataset. As indicated in Table 5(a), InternViT-6B consistently outperforms ViT-22B across five experiments with varying proportions of training data. Additionally, Table 5(b) presents our further verification in three distinct settings, including linear probing, head tuning , and full-parameter tuning. Notably, in the case of linear probing, InternViT-6B attains 47.2 mIoU, a substantial +12.6 mIoU improvement over ViT-22B. These results underscore the strong out-of-the-box pixel-level perceptual capacity of our InternViT-6B.

3 Vision-Language Benchmarks

In this section, we evaluate the inherent capabilities of InternVL on various vision-language tasks.

Zero-Shot Image Classification. We conduct thorough validation of the zero-shot image classification capability of InternVL-C. As depicted in Table 6(a), InternVL-C attains leading performance on various ImageNet variants and ObjectNet . Compared to EVA-02-CLIP-E+ , it exhibits stronger robustness to distribution shift, manifesting in a more consistent accuracy across ImageNet variants. Additionally, as shown in Table 6(b), our model showcases robust multilingual capabilities, outperforming competing models on the multilingual ImageNet-1K benchmark.

Zero-Shot Video Classification. Following previous methods , we report the top-1 accuracy and the mean of top-1 and top-5 accuracy on Kinetics-400/600/700 . As shown in Table 8, when sampling only a single center frame in each video, our method achieves an average accuracy of 76.1%, 75.5%, and 67.5% on the three datasets, surpassing EVA-02-CLIP-E+ by +6.3, +6.2, and +4.1 points, respectively. Additionally, when uniformly sampling 8 frames in each video, we obtain at least 3.3 points of improvement compared to the single-frame setting, outperforming ViCLIP trained using web-scale video data. In summary, InternVL-C exhibits remarkable generalization capabilities in video classification.

Zero-Shot Image-Text Retrieval. InternVL exhibits a powerful multilingual image-text retrieval capability. In Table 7, we evaluate these capabilities in English using the Flickr30K and COCO datasets, as well as in Chinese using the Flickr30K-CN and COCO-CN . Additionally, we leverage the XTD dataset to evaluate the multilingual image-text retrieval capability across 8 languages (see supplementary materials). In summary, InternVL-C achieves state-of-the-art performance across most retrieval metrics, and with the second stage of pre-training, InternVL-G further enhances zero-shot image-text retrieval performance. These improvements in retrieval tasks suggest a more effective alignment between visual and linguistic features, through additional image encoding using the language middleware–QLLaMA.

Zero-Shot Image Captioning. Benefiting from vision-language generative training on a vast collection of high-quality image-text pairs, our QLLaMA possesses promising capability in zero-shot image captioning. As shown in Table 10, QLLaMA surpasses other models in zero-shot performance on the COCO Karpathy test set . It also achieves comparable results to current state-of-the-art models on both the Flickr30K Karpathy test and the NoCaps val set . When InternVL is linked with an LLM (e.g., Vicuna-7B/13B ) and subjected to SFT, a notable enhancement in zero-shot performance is observed for both Flickr30K and NoCaps, as shown in Table 9.

4 Multi-Modal Dialogue Benchmarks

Beyond the traditional multi-modal tasks, the emergence of ChatGPT has led to a growing focus on evaluating the performance of multi-modal models in real usage scenarios, specifically within the realm of multi-modal dialogue. We conducted testing of InternVL-Chat models on two prominent multi-modal dialogue benchmarks, including MME and POPE . MME is a comprehensive benchmark that includes 14 sub-tasks focusing on the model’s perception and cognition capabilities. POPE is a popular dataset used to evaluate object hallucination. As shown in Table 9, it clearly demonstrates that our models exhibit superior performance compared with previous methods, under the condition of fair trainable parameter counts.

5 Ablation Study

Hyperparameters of InternViT-6B. As discussed in Section 3.2, we explored variations in model depth {32, 48, 64, 80}, head dimension {64, 128}, and MLP ratio {4, 8}, resulting in 16 distinct models. In selecting the optimal model, we initially narrowed down our focus to 6 models, chosen based on their throughput, as listed in Table 11. These models underwent further evaluation using contrastive learning on a 100M subset of LAION-en over 10K iterations. For the experimental setup, the primary difference was the use of a randomly initialized text encoder from CLIP-L , in order to speed up the training. For the sake of accuracy, inference speed, and training stability, we ultimately chose variant 3 as the final InternViT-6B.

Consistency of Feature Representation. In this study, we validate the consistency of the feature representation of InternVL with off-the-shelf LLMs. We adopt a minimalist setting, i.e. conducting a single-stage SFT using only the LLaVA-Mix-665K dataset. Moreover, only the MLP layers are trainable, thereby confirming the inherent alignment level among features from various vision foundation models and LLMs. The results are shown in Table 12. We observed that compared to EVA-E , our InternViT-6B achieves better performance under this simple setup. Additionally, it is noteworthy that performance across all three tasks saw significant improvement when using QLLaMA as the “glue layer”. These significant improvements clearly delineate that the feature representation of InternVL is more consistent with the off-the-shelf LLM.

Conclusion

In this paper, we present InternVL, a large-scale vision-language foundation model that scales up the vision foundation model to 6 billion parameters and is aligned for generic visual-linguistic tasks. Specifically, we design a large-scale vision foundation model InternViT-6B, progressively align it with an LLM-initialized language middleware QLLaMA, and leverage web-scale image-text data from various sources for efficient training. It bridges the gap between vision foundation models and LLMs, and demonstrates proficiency in a wide range of generic visual-linguistic tasks, such as image/video classification, image/video-text retrieval, image captioning, visual question answering, and multi-modal dialogue. We hope this work could contribute to the development of the VLLM community.

Acknowledgement

We thank Shenglong Zhang, Beitong Zhou, Xinyue Zhang, Dongxing Shi, Weigao Sun, Xingcheng Zhang, and Zhifeng Yue for their contributions to the optimization of the training framework. We thank Zhenhang Huang for his assistance in data preparation.

Appendix A Supplementary Materials

Zero-Shot Image Classification on 20 Datasets. In this section, we expand our examination to showcase the effectiveness and robustness of InternVL in 20 different zero-shot image classification benchmarks. As indicated in Table 16, InternVL registers an average performance of 78.1% across all 20 benchmarks. This performance notably exceeds that of the previously leading method, EVA-02-CLIP-E+ , by a margin of 1.0 points. This underscores that, beyond ImageNet and its variants, InternVL possesses robust generalization capabilities across a variety of different domains in zero-shot image classification.

Zero-Shot Image-Text Retrieval on XTD. Table 13 reports the results of InternVL on the multilingual image-text retrieval dataset XTD , spanning eight languages. As can be seen, InternVL-C achieves an average recall@10 score of 95.1% across these languages. The second stage model, InternVL-G, further improves retrieval performance. It attains the highest scores in each individual language and establishes a new record for average performance at 96.6%.

Zero-Shot Video Retrieval. In Table 14, we present our results of zero-shot video-text retrieval on the MSR-VTT dataset using our InternVL models, i.e. InternVL-C and InternVL-G. In the 1-frame setting, we select a single central frame from each video. In the 8-frame setting, we uniformly extract 8 frames from each video, treat them as independent images for encoding, and then average the embeddings. The results showcase consistent improvement across various metrics such as R@1, R@5, R@10, and the average score. Importantly, both models exhibit promising outcomes in single-frame and multi-frame configurations, with InternVL-G achieving slightly higher performance than InternVL-C, especially in the multi-frame setting. These results underscore the effectiveness of QLLaMA in harmonizing visual and linguistic features.

Fine-tuned Image-Text Retrieval. In Table 15, we report the fine-tuned image-text retrieval results of InternVL, on both the English and Chinese versions of the Flickr30K dataset . The specific hyperparameters for fine-tuning are shown in Table 21. As can be seen, our models obtain competitive performance, with InternVL-G-FT marginally surpassing InternVL-C-FT in both datasets. Notably, in the highly challenging Flickr30K-CN, both models show a promising ability to handle cross-lingual retrieval tasks. These results demonstrate the effectiveness of our language middleware, especially in the retrieval tasks.

Tiny LVLM. Tiny LVLM is an ability-level benchmark for evaluating the performance of multimodal dialogue models. It provides a systematic assessment of five categories of multimodal capabilities, including visual perception, visual knowledge acquisition, visual reasoning, visual commonsense, and object hallucination. We report our results on Tiny LVLM in Table 17.

A.2 More Ablation Studies

Compatibility with Other LLM. In this experiment, we test the compatibility of InternVL with LLMs other than Vicuna . The experimental setup used here is the same as in Table 9 of the main paper. As shown in Table 18, InternLM-7B achieves slightly better performance than Vicuna-7B . This indicates that our InternVL exhibits promising compatibility with various LLMs.

Efficiency Analysis. In this study, we analyze the computational efficiency of InternVL in encoding image-text pairs. The entire encoding process consists of two parts: image encoding and text encoding. The analysis covered two models (InternVL-C and InternVL-G) and their performance across three different image sizes (224, 336, and 448). The results are shown in Table 19.

From these results, we find that: (1) As the image size increases, the encoding time also significantly increases, leading directly to a decrease in frame rate; (2) InternVL-G slightly increased the encoding time due to the introduction of QLLaMA for secondary image encoding, but it still maintains a reasonable frame rate across all image sizes; (3) Even though we scale up the text encoder, the additional cost of text encoding is not significant, as the main time expenditure lies in image encoding. In summary, when choosing between InternVL-C and InternVL-G, one should weigh the trade-off between computational efficiency and potential performance improvements based on specific requirements. Additionally, these results were measured using PyTorch with Flash Attention and bf16 precision, and there is still considerable room for optimization, such as using model quantization and TensorRT.

A.3 Detailed Training Settings

Settings of Stage 1. As shown in Table 20, in this stage, the image encoder InternViT-6B is randomly initialized using the BEiT’s initialization method , and the text encoder LLaMA-7B is initialized with the pre-trained weights from , a multilingual LLaMA-7B. All parameters are fully trainable. We employ the AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.95$, weight decay at 0.1, and a cosine learning rate schedule starting at 1e-3 and 1e-4 for the image and text encoders, respectively. We adopt a uniform drop path rate of 0.2. The training involves a total batch size of 164K across 640 A100 GPUs, extending over 175K iterations to process about 28.7 billion samples. To enhance efficiency, we initially train at a 196$\times$196 resolution, masking 50% of image tokens , and later switch to 224$\times$224 resolution without masking for the final 0.5 billion samples.

Settings of Stage 2. In this stage, InternViT-6B and QLLaMA inherit their weights from the first stage, while the learnable queries and cross-attention layers in QLLaMA are randomly initialized. Benefiting from the powerful encoding capabilities learned in the first stage, we keep both InternViT-6B and QLLaMA frozen and only train the newly added parameters. The input images are processed at a resolution of 224$\times$224. For optimization, the AdamW optimizer is employed with $\beta_{1}=0.9$, $\beta_{2}=0.98$, weight decay set at 0.05, and a total batch size of 20K. The training extends over 80K steps across 160 A100 GPUs, inclusive of 2K warm-up steps, and is governed by a cosine learning rate schedule with a peak learning rate of 5e-5. More detailed training settings are listed in Table 20.

Settings of Stage 3. At this stage, we have two different configurations. One is to use InternViT-6B separately, as shown in Figure 4 (c). The other is to use the entire InternVL model simultaneously, as shown in Figure 4 (d).

(1) InternVL-Chat (w/o QLLaMA): For this setup, we follow the training recipes of LLaVA-1.5 . We use the same hyperparameters and datasets for supervised fine-tuning, i.e. we first train the MLP layers with the LGS-558K dataset, and then train the LLM with the LLaVA-Mix-665K dataset, both for one epoch.

(2) InternVL-Chat (w/ QLLaMA): For this more advanced setup, we also conducted the training in two steps. We first train the MLP layers with our custom SFT dataset and then fine-tune the LLM with it. Due to the expansion of the dataset, we increased the batch size to 512.

Settings of Retrieval Fine-tuning. In this experiment, all parameters of InternVL are set to be trainable. We conduct separate fine-tuning on the Flickr30K and Flickr30K-CN . Following common practice , a 364$\times$364 resolution is adopted for fine-tuning. To avoid over-fitting, we apply a layer-wise learning rate decay of 0.9 to both InternViT-6B and QLLaMA, along with a drop path rate of 0.3 for InternViT-6B. The AdamW optimizer is utilized, with a total batch size of 1024, for fine-tuning the InternVL model across 10 epochs. For more detailed training settings, please refer to Table 21.

Settings of ImageNet Linear Probing. We follow the common practices of linear probing in previous methods . Specifically, we employ an additional BatchNorm to normalize the pre-trained backbone features during training. Besides, we concatenate the average-pooled patch token features with the class token. The linear head is trained using the SGD optimizer for 10 epochs on ImageNet-1K , with a total batch size of 1024, a peak learning rate of 0.2, 1 epoch warm-up, and no weight decay. Data augmentation involves random-resized-crop and flip. For more training details, please see Table 22.

Settings of ADE20K Semantic Segmentation. In Table 23, we have listed the hyperparameters for three different configurations in ADE20K semantic segmentation, including linear probing, head tuning, and full-parameter tuning.

A.4 Data Preparation for Pre-training

Training Data for Stage 1 & Stage 2. During the first and second stages, we employed a vast collection of image-text pair data (see Figure 5 (a)), such as LAION-en , LAION-multi , LAION-COCO , COYO , Wukong , among others . A detailed introduction to these datasets is provided in Table 24.

Training Data Cleaning for Stage 1 & Stage 2. To fully utilize web-scale image-text data, we adopted different data filtering strategies in stage 1 and stage 2.

(1) Stage 1: In the first stage, we applied only minor data filtering, thus retaining the vast majority of the data. We considered six factors: CLIP similarity, watermark probability, unsafe probability, aesthetic score, image resolution, and caption length, to remove extreme data points and avoid disrupting training stability. Additionally, we removed data that was duplicated with ImageNet-1K/22K , Flickr30K , and COCO to ensure the reliability of our zero-shot evaluations. Due to download failures and the use of our data filtering pipeline, the total amount of data retained in the first stage was 4.98 billion.

(2) Stage 2: In the second stage, we implemented a more stringent data filtering strategy. With generative supervision included, we deleted most of the low-quality data based on the captions, mainly considering the length, completeness, readability, and whether they were gibberish or boilerplate (like menus, error messages, or duplicate text), contained offensive language, placeholder text, or source code. We retained only 1.03 billion entries.

Testing Datasets for Image Classification. We conducted extensive validation on image classification tasks (see Figure 5 (b)), including the linear probing performance of InternViT-6B and the zero-shot performance of InternVL-C. These datasets used are listed in Table 24.

Testing Datasets for Video Classification. As shown in Figure 5 (c), to evaluate the capabilities of video classification, we utilize the following Kinetics datasets: Kinetics 400 , Kinetics 600 , and Kinetics 700 .

Testing Datasets for Image-Text Retrieval. We use five datasets (see Figure 5 (d)) to evaluate InternVL’s zero-shot, multilingual image-text retrieval capabilities. A detailed introduction to these datasets is provided in Table 25.

Testing Dataset for Video-Text Retrieval. As shown in Figure 5 (e), we use the MSR-VTT dataset to evaluate our InternVL in zero-shot video-text retrieval.

Testing Dataset for Image Captioning. As illustrated in Figure 5 (f), we use three image captioning datasets to test our InternVL model. A detailed introduction to these datasets is provided in Table 26.

Testing Dataset for Semantic Segmentation. We use the ADE20K dataset to study the pixel-level perceptual capacity of InternViT-6B, as shown in Figure 5 (g). A detailed introduction to this dataset is provided in Table 26.

A.5 Data Preparation for SFT

Training Data for SFT. In this stage, we collect a wide range of high-quality instruction data. For non-dialogue datasets, we follow the method described in for conversion. A detailed introduction is provided in Table 27.

Testing Datasets for SFT. We validate the effectiveness of our supervised fine-tuned InternVL-Chat models on three tasks, including image captioning, visual question answering, and multi-modal dialogue. There datasets are listed in Table 28. For most of these datasets, we employ the same response formatting prompt as for LLaVA-1.5 .

References