MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Introduction

General-purpose vision-language pre-trained models (VLMs) have made significant advancements (Li et al., 2022; 2023d; 2023g; Zhu et al., 2023; Li et al., 2023b). Recent VLMs mostly augment a large language model (LLM) with a visual encoder and exhibit impressive zero-shot capacities in various visual tasks. However, unlike LLMs that can extract rich background knowledge and task information from the prompt with in-context learning (ICL), most VLMs still struggle to understand complex multi-modal prompts that include multiple images. Previous studies (Li et al., 2023d; b) primarily focus on handling the user queries with a single image rather than multi-modal prompts with interleaved multiple images and text. Although some VLMs like Flamingo (Alayrac et al., 2022) and Kosmos-1 (Huang et al., 2023b) can handle user queries with multiple images, their pre-training data can not provide more sophisticated multi-modal prompts than interleaved image and text crawled from the web (Awadalla et al., 2023). Hence, there is a gap between the prompts used in pre-training these VLMs and the user queries in real-world scenarios, which always contain multiple images and more sophisticated text. Specifically, these VLMs may suffer from the following three limitations, which makes VLMs less effective in downstream vision-language tasks.

Hard to Understand Text-to-Image Reference: Previous studies rarely attempt to address the issue of text-to-image reference in the multi-modal prompts. However, there are often intricate referential relationships between the text and images in user queries, with different words mentioning different images. For example, the user may ask a specific question about multiple images(Fig. 1.c and Fig. 1.f) or use multiple images as exemplars to ask the question only about a specific image(Fig. 1.d). However, the training data used in previous studies (Li et al., 2023d; Alayrac et al., 2022; Huang et al., 2023a) are crawled from the web and may lack explicit text-to-image references. VLMs thus might fail to handle user queries involving intricate text-to-image references.

Hard to Understand the Relationships between Multiple Images: There are often spatial, temporal, and logical relationships between multiple images, and correctly understanding them allows the model to handle user queries better. However, the pre-training data used by previous VLMs (Alayrac et al., 2022) are collected from the internet, lacking close connections among images, especially when these images are far apart on the same webpage. It hampers the ability of VLMs to understand the intricate relationships among the images and further limits their reasoning ability.

Hard to Learn from In-Context Multi-Modal Demonstrations: Previous studies have shown that pretrained LLMs can benefit from few in-context demonstrations (Brown et al., 2020; Dong et al., 2023). However, the ICL ability of current VLMs is rather limited, specifically: 1) VLMs like BLIP-2 (Li et al., 2023d), LLaVA (Li et al., 2023b) only support multi-modal prompts with a single image, hampering their abilities to use multiple multi-modal demonstrations to enhance their performance during the inference; 2)Although VLMs such as Flamingo (Alayrac et al., 2022) support multi-image inputs during pretraining and emerge ICL abilities, their context schemes fall to provide text-image references and closely related images. It inhibits them from offering sophisticated enough prompts to the VLMs, thereby limiting the effectiveness of their ICL ability. Besides, the lack of further supervised instruction tuning hinders their effectiveness across downstream tasks.

In this paper, to address the aforementioned limitations 1) We present MMICL, a new approach to allow VLMs to efficiently deal with multi-modal inputs, including relationships among multiple images and text-to-image references. 2) We propose a novel context scheme in which incorporating an extra image declaration section, along with the inclusion of image proxy tokens, enhances the ICL ability of the VLM. 3) We construct a multi-modal in-context learning dataset in accordance with the proposed scheme. The dataset is adapted from a range of existing datasets and can be used to provide support for the training of more capable VLMs.

Our experiments show that MMICL achieves new state-of-the-art performance on various of vision-language benchmarks including MME (Fu et al., 2023) and MMBench (Liu et al., 2023c) Results of MMICL are submitted on August 28th, 2023.. Comprehensive examinations of the three limitations we aim to address reveal that MMICL exhibits exceptional ability in understanding text-to-image references (13-points improvement on the vision-language compositionality benchmark, Winoground (Thrush et al., 2022a)) and intricate relationships among images(12-points improvement on the multi-image reasoning benchmark, RAVEN (Huang et al., 2023a)). Moreover, MMICL demonstrates impressive multi-model ICL performance across various tasks. We also observe that MMICL efficiently mitigates the language bias, which often causes VLMs to ignore visual contents when facing extensive textual contexts, leading to hallucinations.

MMICL

Most VLMs utilize Visual-Prompt Generators (VPG) (e.g., Resampler (Alayrac et al., 2022), Q-former (Li et al., 2023d)) to extract visual embeddings from the image features encoded by vision backbones and use visual embeddings to help LLMs understand visual inputs. The model architecture shown in Fig. 2.a belongs to VLMs that focus on prompts with a single image, such as Blip-2 (Li et al., 2023d), which always places the image at the top of the entire input and can not handle the inputs with multiple images In Fig. 2.b, VLMs with few-shot ability, such as Flamingo (Alayrac et al., 2022), encode images into image embeddings with a fixed number of visual tokens and use cross-attentions in LLM to mixture the visual and text content. Different from previous work, MMICL shown in Fig. 2.c treats image and text representations equally and establishes the reference between image and text via image declaration. It enables users to have the flexibility to input multiple images and text in any desired order, with no restrictions on the quantity or placement of images in contexts. As shown in Fig. 4, each given image is encoded by a vision encoder (e.g., ViT (Radford et al., 2021)) to get the image representation. Then, we use the Q-former as the VPG to extract the visual embedding. We utilize a fully connected layer as the projection layer to convert each visual embedding to the same dimension as the text embedding of the LLM. Finally, we combine the visual embeddings of multiple images with text embeddings in an interleaved style and feed them into the LLM. We set the weights for mapping query and value vectors in the attention layer of LLM as learnable to better adapt to multi-modal prompts with multiple images. More details are presented in Appendix D.

2 The Design of Context Scheme of MMICL

In this section, we outline the design of the Context Scheme for MMICL. The proposed scheme is devised to proficiently transform the interleaved image-text data into the training context for MMICL.

Users may use textual descriptions to refer to particular images in their queries. Such reference can provide information about the visual content mentioned in the text to the VLM, allowing it to learn alignment between two modalities. To precisely link text and image, we form image declaration templates for each image in mixed inputs, as shown in Fig. 3.a. Firstly, we allocate a unique image proxy ([IMG$j$]) to reference the visual embedding of image $j$, which provides a unique identifier for VLMs to index and distinguish between visual and text embeddings. Then, we utilize natural language prompts to establish references between text and image. Incorporating the explicit text-to-image reference in the image declaration assists the model in correlating the text with the appropriate image. Meanwhile, the image declaration, maintained as textual content, can also preserve the flexibility to appear at any position within the prompt. Each instance $\mathbf{I}_{i}$ follows the structure, where the $\mathbf{X}_{i}$ symbolizes the set of image decorations that can be placed anywhere within the instance $\mathbf{I}_{i}$. $\mathbf{q}_{i}$ and $\mathbf{a}_{i}$ denote the question with instruction and corresponding answer, respectively.

2.2 Multi-model Data with Interconnected Images

To incorporate abundant multi-image information within the context schema of MMICL, we generate interconnected multi-image data that includes spatial, logical, and temporal relationships. It aids MMICLin understanding the intricate relationships among images in user queries. Specifically, we derive frames from videos to build multi-image data. The frames extracted from video inherently sustain close temporal and spatial relations, which infuse spatial and temporal correlation information among images into the context scheme. Besides, we build multi-image data from images depicting multiple object interactions. We detect the objects within the image and generate bounding boxes for each object. We acquire multiple sub-images of different objects by cropping the image according to bounding boxes. We then replace the textual references to these objects with their corresponding cropped images, thus forming interleaved multi-modal data with logical and causal interconnected images, as delineated in Fig. 3.b. Each instance $\mathbf{I}_{i}$ comprises a question-answer text pair along with $K$ images, where the $\mathbf{x}_{i,k}\in\mathbf{X}_{i}$ represents the image declaration for the $k$-th image.

2.3 Unified Multi-modal In-Context Format for Different Tasks

We propose a design for producing multi-modal in-context learning data for different tasks to enrich the context scheme of MMICL. It aims to improve the instruction-aware ability of VLM and expand its abilities for proficient multi-modal in-context learning. Specifically, we start by crafting diverse instructions for each task and generate different templates for the task utilizing these instructions. We then fill in the randomly selected template with the original task to assemble data equipped with instructions as Appendix F. Moreover, we convert the data into a multi-modal in-context format by constructing few-shot exemplars generated by sampling instances from the data. These exemplars are combined with the input instance to produce the multi-modal in-context data. In this way, we can transform all tasks into a unified multi-modal in-context format, as illustrated in Fig. 3.c. This method facilitates amassing an extensive amount of high-quality data from different tasks, enriching the context schema of MMICL with an abundant diversity of multi-modal in-context data teeming with diverse instructions. Ultimately, this improves the model’s ability to follow instructions and multi-modal in-context learning ability. Each instance $\mathbf{I}_{i}$ comprises N exemplars.

Each exemplar $\mathbf{P}_{j}=(\mathbf{X}_{j},\mathbf{q}_{j},\mathbf{a}_{j})$, $\mathbf{X}_{j}$ denotes the image declaration of the $j$-th exemplar. $\mathbf{q}_{j}$ and $\mathbf{a}_{j}$ denote the question and answer for the $j$-th exemplar, respectively.

3 Multimodality In-Context Learning (MIC) Dataset Construction

To help VLMs understand the complex prompts, we construct MIC dataset by gathering data from public data resources and converting them based on the context scheme. It has three key aspects: 1) image declaration, 2) multi-modal data with closely related images, and 3) multi-modal in-context data for different tasks. Training set of MIC comes from 16 datasets across 8 categories, while the test set comes from 18 datasets across 10 categories. Additional details can be found in Appendix B and Appendix C.

Firstly, we create an image declaration per instance in all datasets using Algorithm 1 to generate datasets with explicit text-to-image reference. We then have annotators scrutinize every dataset’s samples and provide task instructions. This practice aids in gaining a comprehensive understanding of the task and helps craft high-quality templates. Next, we employ ChatGPT We use the gpt-3.5-turbo version of ChatGPT. to rewrite the instructions to describe the key characteristics of each task accurately. After ChatGPT generates the instructions, we undergo a manual review to guarantee the high quality of the instructions. We select ten suitable templates matching as candidates, then merge the original dataset’s input into a randomly chosen template. We assemble demonstrations for each instance from the dataset by selecting a small amount of data and arranging them sequentially. These demonstrations are integrated with the input instance to generate multi-modal contextual dataExcept for the video datasets, vcr dataset, and LLaVa dataset. More detail can be found in Appendix B. We construct multi-image data by extracting eight frames per video from MSRVTT (Xu et al., 2016) and MSRVTTQA (Xu et al., 2016) datasets. We also crop images from the VCR (Zellers et al., 2019) dataset using object bounding boxes to produce intertwined multi-modal data with closely related images. We convert all data into a vision-language Q&A format to create high-quality multi-modal training data and accumulate 5.8M samples in MIC dataset. Due to resource constraints, we use approximately 10% of MIC with the sampling strategy described in Appendix E to finetune MMICL. It is anticipated that a larger model trained on all of our data would yield a more promising result.

4 Training Paradigm

Stage \@slowromancapi@: Pretraining. This stage aims to assist the model in aligning the image and text embeddings. During this stage, both the vision encoder and the LLM remain frozen. The VPG (i.e., Q-Former) and projection layer are trained to learn visual embeddings that can be interpreted by the LLM.

Stage \@slowromancapii@: Multi-Modal In-Context Tuning. In this stage, we aim to address the aforementioned limitations and take our model a step further by extending it to multi-modal in-context learning. Specifically, we aim to make the model understand the intricate referential relationships between the text and images and the complex relationships among multiple images and ultimately acquire a proficient multi-modal in-context learning ability. Therefore, we perform multi-modal In-Context Tuning on MIC dataset. During the stage \@slowromancapii@, we freeze the image encoder, Q-former, and LLM while jointly training the projection layer and query and value vectors.

Experiment

Evaluation Setup. We aim to develop general-purpose VLMs that can generally adapt to diverse, challenging multi-modal prompts. Therefore, we evaluate our models in several vision-language benchmarks, including tasks that involve images and videos. The metrics used in these benchmarks and further details are shown in Appendix L.

Models and Baselines. We provide two versions of MMICL: (1) MMICL (FLAN-T5) which uses BLIP-2 (Li et al., 2023d) as the backbone and (2) MMICL (Instruct-FLAN-T5) which uses InstructBLIP (Dai et al., 2023) as the backbone. We also adopt XL and XXL of FLANT5 (Chung et al., 2022) model for both versions. We compare MMICL with following strong baselines: Flamingo (Alayrac et al., 2022), KOSMOS-1 (Huang et al., 2023a), BLIP-2-FLAN-T5, InstructBLIP-FLAN-T5, Shikra (Chen et al., 2023), Otter (Li et al., 2023a), Ying-VLM (Li et al., 2023e). The details of MMICL and baselines are shown in Appendix G, and Appendix M.

2 General Performance Evaluations

We evaluate the general performance of MMICL on both MME (Fu et al., 2023) and MMBench (Liu et al., 2023c) benchmarksAll the reported performance for the baseline methods is from the leaderboard of MME (Fu et al., 2023) and MMBench (Liu et al., 2023c). We report the result of MMICL with FLANT5-XXL backbone.. MME evaluates VLMs with 14 sub-tasks that encompass cognition and perception abilities. Results in Table 1 show that MMICL can achieve the best average scores compared with current VLMs on cognition and perception tasks. MMICL also demonstrates outstanding performance and significantly surpasses other VLMs on the MMBench benchmark, which thoroughly evaluates the diverse skills of VLMs. The detailed results are presented in Table 21. See Appendix H and I for MMICL’s evaluation detail and comparisons with other VLMs.

3 Performance Prob

The Winoground (Thrush et al., 2022b) proposes a task of correctly matching two given images and captions, as depicted in the left of Fig. 5. The challenge lies in the fact that both captions consist of the exact same words, albeit in a different order. VLMs must compare both images and texts to discern their subtle differences and capture the implicit reference between them. Therefore, we select the Winoground to evaluate whether VLMs understand the text-to-image reference. Results in Table 2 demonstrate that MMICL captures the referential relationship between image and text, surpassing previous baselines.

3.2 Understanding Complex Image-to-Image Relationship

RAVEN (Zhang et al., 2019; Huang et al., 2023a) test is widely used to evaluate the nonverbal reasoning ability of VLMs. It requires visual and logical skills to understand the relationships among images.

We conduct zero-shot experiments on the Raven test to evaluate VLM’s ability to understand image-to-image relationships. Each instance has $3$ or $8$ images as inputs and $6$ candidate images with a unique answer, and the goal is to predict the right image as shown in the right of Fig. 5. The result in Table 3 shows that MMICL achieves 12 points improvement compared to KOSMOS-1. It indicates that MMICL is able to capture the complex image-to-image relationships and conduct nonverbal visual reasoning tasks.

4 Learning from In-Context Multi-Modal Demonstrations

As shown in Table 4, we evaluate the multi-modal in-context learning ability of MMICL across various vision-language tasks. MMICL outperforms other VLMs on both the held-in and held-out datasets and achieves the state-of-art few-shot performance. For example, few-shot evaluation (4-shot) of MMICL on the VizWiz benchmark outperforms the baseline Flamingo-9B (Alayrac et al., 2022) and KOSMOS-1 (Huang et al., 2023b) by $15.38$ and $14.98$ points, respectively. Since VizWiz has never been exposed in the training data, this superior suggests the ability of MMICL to generalize to new tasks with a few exemplars. The few-shot performance of Flickr30K decreases with examples given because the captions examples may provide noise for the VLM to finish the task(i.e., in-context exemplars generally do not provide hinds for models to perform image captioning tasks).

5 Hallucination and Language Bias of VLMS

Current VLMs have significant visual hallucinations (Li et al., 2023f), preventing VLMs from benefiting from multi-modal ICL. Especially when dealing with complex prompts with multiple images (e.g., multi-modal chain of thoughts (Zhang et al., 2023b)), VLMs often overlook visual content when facing extensive text. This language bias reduces their efficiency in answering questions that require both images and text. ScienceQA-IMG (Lu et al., 2022) is a challenging task that requires a model to use both modalities to answer the question. We manually split the dataset into two groups: questions needing images to answer and those not. Extensive experiments in Table 5 demonstrate that MMICL effectively mitigates language bias as it performs equally well in both groups. On the other hand, other VLMs suffer from language bias and exhibit vastly different performances in the two groups. Specifically, MMICL achieves a significant improvement compared to other VLMs with a similar model structure (e.g., Instructblip and Ying-VLM) in reducing language bias. Comparison with Otter shows that the lack of understanding in text-to-image reference and multiple-image relationships can result in significant language bias for Otter, even with the multimodal instruction in-context tuning. ShrikaWe use 0708 version of Shikra, which performs better for multi-choice questions to ensure fair competition. mitigates the language bias by including spatial coordinate inputs and achieves the lowest performance gap except for MMICL. We also examined object hallucination in MMICLin Appendix K, which shows impressive performance.

6 Ablation Study on Training Paradigm

We conduct an ablation study on various tasks to evaluate the effect of multi-modal in-context tuning. Table 6 displays a significant enhancement of MMICL‘s performance due to the multi-modal in-context tuning. Significant improvements can be observed across all types and sizes of models, especially for tasks that involve multiple images. Specifically, MMICL (Stage \@slowromancapi@ + Stage \@slowromancapii@) gained 15.75 and 21.05 points improvement in IconQA-img and Bongard-HOI respectively, compared to the Stage \@slowromancapi@ only model. This indicates that with the help of Stage \@slowromancapii@, MMICL can handle complex multi-modal prompts and accomplish challenging tasks with multiple images. Result in Appendix J also confirms this point with the outstanding performance of MMICL across various video datasets.

Related Work

Vision-Language Pretraining: Recent VLMs (Zhu et al., 2023; Liu et al., 2023b; Li et al., 2022; Alayrac et al., 2022; Dai et al., 2023) have been proven effective for aligning visual inputs and frozen LLMs to obtain cross-modal generalization ability. However, previous works overlooked multi-image VLMs, mainly focusing on handling single-image prompts. Tsimpoukelli et al. (2021) supports multi-image inputs using self-attention for images but performs poorly in downstream tasks. Although Flamingo (Alayrac et al., 2022) supports Few-Shot Learning in VLMs and uses cross-attention to capture text-image relationships, it still suffers from exact reference to specific images.

Multi-Modal Instruction Tuning: Instruction tuning (Kung & Peng, 2023; Wei et al., 2022) achieves great success in cross-task generalization for LLMs. However, multi-modal instruction tuning still requires further exploration. Multiinstruct (Xu et al., 2023) introduces instruction tuning to enhance the performance of VLMs in instruction-following ability. Due to the architectural design, Multiinstruct still struggles with complex contexts containing multiple images. Otter (Li et al., 2023a) fine-tunes Openflamingo (Awadalla et al., 2023) to augment its instruction comprehension capabilities. However, Otter’s dataset lacks text-to-image references and interconnected image-to-image data. This limitation hinders its capability to handle complex contexts that involve visual-textual relationships.

Conclusion

In this paper, we highlight the limitations of VLMs handling the complex multi-modal prompts with multiple images, which makes VLMs less effective in downstream vision-language tasks. We introduce MMICL to address the aforementioned limitations and take our model a step further by extending it to multi-modal in-context learning. This breakthrough enables VLMs to better understand complex multi-modal prompts. Furthermore, MMICL sets a new state-of-the-art performance on the general VLM benchmarks and complex multi-modal reasoning benchmarks.

References

Appendix A Related Work

Our work is inspired by recent vision-language pre-training works (Zhu et al., 2023; Liu et al., 2023b; Li et al., 2022; 2023d), which have been proven effective for aligning visual inputs and frozen LLMs to obtain cross-modal generalization ability.

BLIP-2 (Li et al., 2023d) bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model.

InstructBLIP (Dai et al., 2023) performs vision-language instruction tuning based on the pre-trained BLIP-2 models with converted multi-modal datasets and the LLaVA (Liu et al., 2023b) dataset generated by GPT-4.

MiniGPT-4 (Zhu et al., 2023)aligns a CLIP visual encoder with a frozen Vincuna (Chiang et al., 2023) with an artificially collected dialog dataset

Shikra (Chen et al., 2023), a VLM which can handle spatial coordinate inputs and outputs in natural language. It makes Shikra excel at referential dialogue and general vision-language tasks, resulting in outstanding performance.

However, there is still less work focusing on VLMs with multi-image inputs.

Flamingo (Tsimpoukelli et al., 2021) achieves multi-visual inputs based on self-attention for images but performs poorly in downstream tasks. Flamingo supports Few-Shot Learning (FSL) in VLM via ICL by leveraging its robust capability to handle multi-visual inputs and uses cross-attention instead of self-attention to get better performance. However, it still suffers from the unableness to explicitly point images, so they introduce a hacky cross-attention mask.

Kosmos-1 (Huang et al., 2023a), is trained from scratch on billion-scale multi-modal corpora, including interleaved text-image web page data, image-text caption, and language-only instruction tuning data. It can multi-modal Few-Shot Learning and Chain-of-Thought processes, thereby achieving formidable performance.

Otter (Li et al., 2023a), an open-source implementation of flamingo and trained with multi-modal instruction in-context tuning data.

Najdenkoska et al. (2023) uses meta-learning objective to train an adapter that aggregates multiple image features so the original VLM and adapter become a better few-shot learner.

A.2 In-Context Learning

It has been well-explored to enable ICL in pre-trained language models (PLM). MetaICL (Min et al., 2021) proposes a meta-training framework for few-shot learning to tune a PLM to do in-context learning on a large set of training tasks. LM-BFF (Gao et al., 2020) studies few-shot fine-tuning of PLMs. However, ICL in VLM is still less explored. Recent works in VLM mainly focus on zero-shot evaluation with single image input.

Appendix B Multi-modal ICL Data

We construct two training datasets, text-image interleaved data and in-context learning data, for the text-image relationship challenge and image-image relationship challenge, respectively. In this section, we will cover the data resources.

Appendix C Data Resource

The data resource used in constructing the MIC dataset is displayed in Fig. 6. Our training dataset comes from 8 task categories and 16 datasets.

Image Captioning aims to produce descriptions of the given images according to different needs. Our training dataset includes MS COCO (Lin et al., 2014), DiffusionDB (Wang et al., 2022b), and Flickr 30K (Young et al., 2014).

Knowledgeable Visual Question Answering (KVQA) requires the model to make use of commonsense knowledge outside the input image to answer questions. Our training dataset includes OK-VQA (Marino et al., 2019).

Image Question Answering (IQA) requires the model to answer the questions based on the image correctly. Our training dataset includes VQAv2 (Goyal et al., 2017), ST-VQA (Biten et al., 2019), Text-VQA (Singh et al., 2019), WikiART (Saleh & Elgammal, 2015) and RefCOCO (Yu et al., 2016).

Video Question Answering (VideoQA) requires the model to answer questions based on the video correctly. We extract eight frames per video as visual inputs for Video QA tasks. Our training dataset includes MSRVTTQA (Xu et al., 2016).

Video Captioning requires the model to give the caption based on the video. We extract eight frames per video as visual inputs for Video Captioning tasks. Our training dataset includes MSRVTT (Xu et al., 2016).

Visual Reasoning requires the model to correctly perform image reasoning and answer questions. Our training dataset includes GQA (Hudson & Manning, 2019), VCR (Zellers et al., 2019), and NLVR2 (Suhr et al., 2018).

Image Classification involves classifying an image based on a given set of candidate labels. Our training dataset includes MiniImage (Russakovsky et al., 2015).

Visual Dialog requires the model to hold a meaningful dialog about visual content with humans in natural, conversational language. Our training dataset includes LLAVA-Instruct-150K (Liu et al., 2023b).

Our testing dataset comes from 10 task categories and 18 datasets.

Image Captioning includes the Nocaps (Agrawal et al., 2019) dataset.

Knowledgeable Visual Question Answering (KVQA) includes the ScienceQA (Lu et al., 2022) and A-OKVQA (Schwenk et al., 2022) datasets.

Image Question Answering (IQA) includes the VizWiz (Bigham et al., 2010) dataset.

Visual Reasoning includes the Winoground (Thrush et al., 2022b), VSR (Liu et al., 2022) and IconQA (Lu et al., 2021) dataset. Winoground proposes a task of matching two given images and two captions correctly. The challenge of this task is that both captions contain a completely identical set of words, only in a different order. VSR describes the spatial relation of two individual objects in the image, and a VLM needs to judge whether the caption correctly describes the image (True) or not (False). The IconQA dataset has two sub-datasets: image question answering with multiple text choice and image question answering with multiple image choice.

Web Page Question Answering (Web QA) includes the Websrc (Chen et al., 2021a; Huang et al., 2023a) datasets. The model must answer questions based on the web image and the optional extracted texts. We sampled 2000 instances from Websrc for the evaluation. To align with KOSMOS-1 (Huang et al., 2023a), we only use the web image as input.

Video Question Answering (VideoQA) includes the iVQA (Yang et al., 2021), MVSD (Chen & Dolan, 2011), and NextQA (Xiao et al., 2021) dateset. The NextQA dataset has two sub-datasets: video question answering with multiple choice and open-domain video question answering.

Few-shot Image Classification includes the HatefulMemes (Kiela et al., 2020) and Bonard-HOI (Jiang et al., 2022) dataset. HatefulMemes requires the model to determine if a meme is hateful based on the image and explanation provided. Bonard-HOI is the benchmark for evaluating the model’s ability in Few-Shot Visual Reasoning for Human-Object Interactions. It provides few-shot examples with challenging negatives, where positive and negative images only differ in action labels. The model is then asked whether the final image is positive or negative. We sampled 2000 instances from Bonard-HOI for the evaluation.

Nonverbal Reasoning includes the Raven IQ test (Huang et al., 2023a). Each instance in the Raven IQ test has 3 or 8 images as inputs and six candidate images with a unique correct completion, and the goal is to predict the next image from the candidates.

Visual Dialog includes the visual dialog dataset (Das et al., 2017). We use the question of the final dialogue as the question for instance and take all preceding dialogues as the context to perform open-domain image question answering.

OOD Generalization includes the Minecraft dataset that we construct using Minecraft (Cipollone et al., 2014) game which requires the VLM to identify whether an animal (i.e., cow, llama, chicken, donkey, and so on) is present in a picture.

More detailed task descriptions and statistics about the datasets are shown in Table 8.

Appendix D Model structure

As shown in Fig. 7 MMICL treats the image and language representations equally and combines them into interleaved image-text representations, similar to the original input. Each given image is encoded by a vision encoder (e.g., ViT (Radford et al., 2021; Fang et al., 2023)) to get the vision representation of the image. Then, we use the Q-former as the VPG to extract the visual embedding. We utilize a fully connected layer as the projection layer to convert each visual embedding to the same dimension as the text embedding of the LLMs. This alignment helps the LLM to understand the images. Our approach treats the visual and text embedding equally, enabling a flexible combination of visual and textual content. Finally, we combine the visual embeddings of multiple images with text embeddings in an interleaved style and then feed them into the LLM. We set the weights for mapping query and value vectors in the attention layer of LLM as learnable to better adapt to the multi-modal context with multiple images. During the pre-training, we freeze the image encoder, Q-former, and the backbone LLM while jointly training the language projection and the query and value vectors of the LLM.

Appendix E Data Balance

Previous studies have shown that the data balance of training data could significantly influence the model performance (Dai et al., 2023). Mixing the training data of each dataset uniformly could cause the model to overfit smaller datasets and underfit larger datasets, causing poor performance. In order to alleviate this problem, we employ a sampling strategy to sample datasets with probabilities proportional to the square root of the number of training samples following Dai et al. (2023). Formally, given $D$ datasets with $N_{\cdot}$ training samples $\{N_{1},N_{2},\dots,N_{D}\}$, the probability $p_{d}$ of data samples being selected from a dataset during training is as follows.

Appendix F Instruction template for Data construction

As Sec. 2.2.3, the constructions of MICrequire carefully designed templates. The instruction templates for each task are presented in this section. The templates for tasks MSCOCO, Flick30k, Nocaps, and Diffusiondb are presented in Table 9. The templates for tasks MiniImagenet are presented in Table 10. The templates for tasks VQAv2, S-VQA, WikiART and RefCOCO are presented in Table 11. The templates for task OKVQA are presented in Table 13. The templates for task MSRVTT are presented in Table 14. The templates for tasks MSRVTTQA and MSVD are presented in Table 15

Appendix G Experiment Details

Following Chung et al. (2022), we use FLANT5-XL and FLANT5-XXL (Chung et al., 2022) as the backbone LLMs. In Stage \@slowromancapi@, we set the vision encoder and language model to be frozen and utilize the COCO captioning data and LAION-400M data (Schuhmann et al., 2021) to perform feature alignment training on the Q-former. We keep the other part of the VLM frozen and jointly train the Q-former and projection layer. To benefit from BLIP-2’s significant visual representation extraction ability, we integrate its powerful vision encoder to initialize the Q-former and projection layer. We use BLIP-2 and InstructBlip as the backbone for MMICL, so Stage \@slowromancapi@ is skipped.. In Stage \@slowromancapii@, we train the model for three epochs with a lower learning rate of $1e-5$. The weights of mapping query and value vectors in the attention layer of LLMs are learnable in this stage to better adapt to the multi-modal prompts with multiple images. In this stage, we freeze the visual encoder, Q-former, and the backbone LLM and jointly train the projection layer, the query vectors, and the value vectors of the LLM.

All experiments are conducted with 6 NVIDIA A40 GPUs with the zero2-offload (Rajbhandari et al., 2020) of Deepspeed (Rasley et al., 2020) with the trainer of huggingface transformers (Wolf et al., 2020). The batch size is 10 and 4 for MMICL (FLAN-T5-XL) and MMICL (FLAN-T5-XXL), respectively. The largest MMICL (FLAN-T5-XXL) requires about two days for the Stage \@slowromancapii@.

Appendix H MME Benchmark

MME comprehensively evaluates VLMs with 14 sub-tasks that encompass perception and cognition abilities. Other than OCR, perception ability includes the recognition of coarse-grained and fine-grained objects. The former identifies the existence, count, position, and color of objects. The latter recognizes movie posters, celebrities, scenes, landmarks, and artworks. The cognition includes commonsense reasoning, numerical calculation, text translation, and code reasoning.

MME evaluates a wide range of multi-modal abilities. The compared baselines include LLaVA (Liu et al., 2023b), MiniGPT-4 (Zhu et al., 2023), MultiModal-GPT (Gong et al., 2023), VisualGPM-6B (Du et al., 2021), VPGTrans (Zhang et al., 2023a) , LaVIN (Luo et al., 2023), mPLUG-Owl (Ye et al., 2023), LLaMA-Adapter-V2 (Gao et al., 2023), InstructBLIP (Dai et al., 2023), Otter (Li et al., 2023a), BLIP-2 (Li et al., 2023d), LRV-Instruction (Liu et al., 2023a), Cheetor (Li et al., 2023c), GIT2 (Wang et al., 2022a), Lynx (Zeng et al., 2023), BLIVA (Hu et al., 2023). We also provide more detail evaluation results for MMICL at Table 17, Table 18, Table 19, and Table 20. Results show that MMICL can achieve the best average scores in comparisons with current VLMs.

Appendix I MMBench benchmark

MMBench (Liu et al., 2023c) is a thoughtfully designed benchmark that thoroughly evaluates the diverse skills of vision-language models. The results of all different VLMs from the test set are presented in Table 21.

Appendix J Understanding Multiple Images in the Multi-Modal Prompt

Videos contain more temporal information compared to static images. We test MMICL across different video-languages tasks to evaluate whether the MMICL is able to support the multiple images in the complex prompts. The result is present in Table 22. Our model, MMICL, achieved significant improvement of $10.86$, $4.53$, and $2.45$ points for MSVD-QA (Chen & Dolan, 2011), NExT-QA (Xiao et al., 2021), and iVQA (Yang et al., 2021) respectively, when compared to the strongest baselines. It is important to note that our training dataset did not include any videos. This indicates that MMICL effectively enhances the model’s ability to understand temporal information in videos.

Appendix K Object Hallucination Evaluation

We test the following VLMs on the POPE benchmark to evaluate their object hallucination performance: MMICL, Shikra (Chen et al., 2023), InstructBLIP (Dai et al., 2023), MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023b), MM-GPT (Gong et al., 2023) and mPLUG-Owl (Ye et al., 2023). The result is present in the Table LABEL:table:POPE.

Appendix L Details for Evaluation

In this Section. we provide details for evaluation in our experiments as Sec. 3.

We provide evaluation metrics as Table 24

L.2 VQA Tools

We use the same VQA Tools as the original VQA paper (Agrawal et al., 2016) and use it in all metrics using the VQA accuracy.

Appendix M Baselines

We primarily compare MMICL with recently proposed powerful multi-modal approaches, including:

(1) Flamingo (Alayrac et al., 2022) where a VLM is trained on large-scale multi-modal- web corpora containing arbitrarily interleaved text and images;

(2) KOSMOS-1 (Huang et al., 2023a) which is trained from scratch on web-scale multi-modal corpora;

(3) BLIP-2-FLAN-T5 (Li et al., 2023d) where an instruction-tuned Flan-T5 (Chung et al., 2022) is connected with a powerful visual encoder to perform a series of multi-modal tasks;

(4) InstructBLIP-FLAN-T5 (Dai et al., 2023), a recently proposed instruction tuning enhanced multi-modal agents with FLAN-T5 with converted multi-modal datasets and the LLaVA (Liu et al., 2023b) dataset generated by GPT-4 (OpenAI, 2023);

(5) Shikra (Chen et al., 2023), a VLM that can handle spatial coordinate inputs and outputs in natural language without the need for extra vocabularies or external plugin models. All inputs and outputs of Shikra are in natural language form.

(6) Otter (Li et al., 2023a), an open-source implementation of flamingo (Alayrac et al., 2022). By utilizing multi-modal instruction in-context tuning data, Otter fine-tunes Openflamingo to augment its instruction comprehension capabilities while maintaining its ability to learn in context;

(7) Ying-VLM (Li et al., 2023e), a VLM model trained on Multi-Modal multilingual instruction tuning dataset, showcasing its potential to answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese.

Appendix N OOD Generalization to Unseen Domain

In an unseen challenging domain with limited exemplars, analyzing regular patterns, reasoning, and learning new knowledge (OOD Generalization to unseen domain) is a great way to test multi-modal ICL ability.

We construct a task using Minecraft (Cipollone et al., 2014), which requires the VLM to identify whether an animal (i.e., cow, llama, chicken, donkey, and so on) is present in case (d) of Fig. 1.

We collect 550 cases and transfer the task to a vision-to-text question-answering task to evaluate the performance of OOD generalization of MMICL. The results are shown in Table 25. Results demonstrate that MMICL is able to generalize to the Minecraft domain even if the images are extremely different compared to the images used by training at Stage I and II as Sec. 2.4.