Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Introduction

Vision language pre-training aims to learn vision language alignments from a large number of image-text pairs. A pre-trained Vision Language Model (VLM) fine-tuned with a small amount of labeled data has shown the state-of-the-art performances in many Vision Language (V+L) tasks such as visual question answering and image-text retrieval.

Existing methods learning vision language alignments fall into two approaches as shown in Figure 1 (a, b). Most of them detect objects in the image and align the text with fine-grained (object-centric) features. They either utilize pre-trained object detectors (Tan & Bansal, 2019; Lu et al., 2019; Li et al., 2019, 2020a; Chen et al., 2020; Li et al., 2020b; Gan et al., 2020) or conduct object detection on-the-fly in the pre-training process (Su et al., 2020; Xu et al., 2021a). The other methods do not rely on object detection and only learn alignments between the texts and coarse-grained (overall) features of the image (Huang et al., 2020, 2021; Kim et al., 2021; Li et al., 2021a).

Both the fine-grained and coarse-grained approaches have drawbacks. Object detection identifies all possible objects in the image, and some of them might not be relevant to the text. Object-centric features cannot easily represent relations among multiple objects, e.g. “ man crossing the street”. Moreover, it is challenging to pre-define the categories of objects suitable for downstream tasks. On the other hand, the coarse-grained approaches cannot effectively learn fine-grained alignments between vision and language, e.g. object-level, which has shown to be critical for some downstream tasks such as visual reasoning, visual grounding, and image captioning.

Ideally, we want a VLM to learn multi-grained alignments between vision and language in pre-training, which are not restricted to object-level or image-level, and leverage the learned alignments to downstream V+L tasks. Unfortunately, existing methods cannot satisfactorily handle multi-grained alignments between vision and language.

In this paper, we propose performing multi-grained vision language pre-training by aligning text descriptions with the corresponding visual concepts in images. Taking Figure 1 as an example, we have the following data for training: 1) the image caption describing the whole image; 2) region annotations such as “man wearing backpack” each of which has been related to a region in the image, while previous approaches roughly align the region descriptions with the whole image; 3) object labels such as “backpack” which are utilized by previous methods to train object detectors. We re-formulate the data, so that an image may have multiple bounding boxes, and a text We take the object labels as text descriptions of objects. is directly associated with the visual concept in each box. The ‘visual concept’ (Krishna et al., 2017; Zhang et al., 2021; Changpinyo et al., 2021) may be an object, a region, or the image itself, as the example in Figure 1 (c). By doing so, our approach learns unlimited visual concepts associated with diverse text descriptions, which are also not restricted to object-level or image-level.

Our multi-grained model, denoted as X-VLM, consists of an image encoder that produces representations of visual concepts (including the image itself) in an image, a text encoder, and a cross-modal encoder that conducts cross-attention between the vision features and language features to learn vision language alignments. The key to learning multi-grained alignments is to optimize X-VLM by: 1) locating visual concepts in the image given associated texts by a combination of box regression loss and intersection over union loss; 2) in the meantime aligning the texts with the visual concepts, e.g. by a contrastive loss, a matching loss, and a masked language modeling loss, where the alignments are in multi-granularity, as illustrated in Figure 1 (c). In fine-tuning and inference, X-VLM can leverage the learned multi-grained alignments to perform the downstream V+L tasks without bounding box annotations in the input images.

We demonstrate the effectiveness of our approach on various downstream tasks. On image-text retrieval, X-VLM learning multi-grained vision language alignments outperforms VinVL (Zhang et al., 2021) which is based on object-centric features, achieving an absolute gain of 4.65% in terms of R@1 score on MSCOCO. X-VLM also outperforms ALIGN (Jia et al., 2021), ALBEF (Li et al., 2021a), and METER (Dou et al., 2021) by a large margin even though they are pre-trained on more data or have more parameters. On visual reasoning tasks, X-VLM achieves absolute improvements of 0.79% on VQA and 1.06% on NLVR2 compared to VinVL (Zhang et al., 2021), with a much faster inference speed. X-VLM also outperforms SimVLMbase (Wang et al., 2021) pre-trained with 1.8B in-house data, especially on NLVR2 by 2.4%. On visual grounding (RefCOCO+), X-VLM achieves absolute improvements of $4.5\%$ compared to UNITER (Chen et al., 2020) and $1.1\%$ compared to MDETR (Kamath et al., 2021) which is specialized for grounding tasks. X-VLM also has comparable performance with SimVLMbase in the image caption generation task.

The contributions of this work are as follows:

We propose performing multi-grained vision language pre-training to handle the alignments between texts and visual concepts.

We propose to optimize the model (X-VLM) by locating visual concepts in the image given the associated texts and in the meantime aligning the texts with the visual concepts, where the alignments are in multi-granularity.

We empirically verify that our approach effectively leverages the learned multi-grained alignments in fine-tuning. X-VLM consistently outperforms existing state-of-the-art methods on many downstream V+L tasks.

Related Work

The existing work on vision language pre-training typically falls into two categories: fine-grained and coarse-grained.

Most existing methods belong to the fine-grained approach, which relies on object detection (Tan & Bansal, 2019; Lu et al., 2019; Li et al., 2019, 2020a; Chen et al., 2020; Li et al., 2020b; Gan et al., 2020; Li et al., 2021b). An object detector first identifies all regions that probably contain an object, then conducts object classification on each region. An image is then represented by dozens of object-centric features of the identified regions. Object detectors, such as Faster R-CNN (Ren et al., 2015), Bottom-Up and Top-Down Attention (BUTD) (Anderson et al., 2018), are trained on image annotations of common objects, e.g. COCO (Lin et al., 2014) (110K images) and Visual Genome (Krishna et al., 2017) (100K), and can be utilized. VinVL (Zhang et al., 2021) has, for example, achieved SoTA performances on many V+L tasks by utilizing a powerful object detector pre-trained with a large collection of image annotations (2.5M images). The challenge with the approach is that object-centric features cannot represent relations among multiple objects in multiple regions. Furthermore, it is not easy to define the categories of objects in advance that are useful for downstream V+L tasks.

The coarse-grained approach builds VLMs by extracting and encoding overall image features with convolutional network (Jiang et al., 2020; Huang et al., 2020, 2021) or vision transformer (Kim et al., 2021; Li et al., 2021a). The performances are usually not as good as the fine-grained approach. Though object-centric features are only related to certain objects, learning fine-grained alignments, e.g. object-level, has shown to be critical for some downstream tasks such as visual reasoning and visual grounding. To cope with the problem, SOHO (Huang et al., 2021) employs online clustering on image features to obtain more comprehensive representations, ViLT (Kim et al., 2021) uses a more advanced vision transformer, i.e. Swin-Transformer (Liu et al., 2021b) for image encoding, and ALBEF (Li et al., 2021a) exploits contrastive learning and momentum distillation in learning of image-text alignments. However, the improvements still cannot close the gap with the fine-grained approach.

Recently, there emerge some methods managing to learn both object-level and image-level alignments. However, these approaches still rely on object detectors and thus suffer from the aforementioned problems. For example, VL-BERT (Su et al., 2020) incorporates Faster R-CNN into pre-training. E2E-VLP (Xu et al., 2021a) adds an end-to-end object detection module (i.e. DETR (Carion et al., 2020)). Uni-EDEN (Li et al., 2022) uses Faster R-CNN as the vision backbone. KD-VLP (Liu et al., 2021a) relies on external object detectors to perform object knowledge distillation. In contrast, X-VLM does not rely on object detection. Besides, X-VLM learns multi-grained vision language alignments, which are not restricted to object-level or image-level. Also, unlike Uni-EDEN, which aligns objects to language by object classification and aligns images to language by caption generation, X-VLM learns visual concepts in different granularities in a unified way. We will show the effectiveness of X-VLM in the experiments.

Method

We re-formulate the widely used pre-training datasets (see Section 4.1) so that an image may have multiple bounding boxes, and each of them is associated with a text that describes an object or a region, denoted as $(I,T,\{(V^{j},T^{j})\}^{N})$. Note that some images do not have associated texts, i.e., $T$ is NaN, and some images do not have bounding boxes, i.e., $N=0$. Here, $V^{j}$ is an object or region in the bounding box ${\boldsymbol{b}}^{j}=(cx,cy,w,h)$ represented by the normalized center coordinates, width, and height of the box. When the image itself represents a visual concept, ${\boldsymbol{b}}=(0.5,0.5,1,1)$. Figure 2 illustrates the architecture and pre-training objectives of X-VLM.

2 Vision Encoding

The image encoder efficiently produces multi-grained visual concept representations in an image. The encoder is based on vision transformer (Dosovitskiy et al., 2020). It first splits an image into non-overlapping patches and linearly embeds all patches. Then, these patches are passed into the transformer layers, yielding $\{{\boldsymbol{v}}_{1},...,{\boldsymbol{v}}_{N^{I}}\}$. For an image of resolution of 224x224 and patch size of 32x32, we have $N^{I}=49$.

3 Cross-Modal Modeling

As shown in Figure 2, we optimize X-VLM by locating visual concepts in the image given the corresponding texts and in the meantime aligning the texts and visual concepts, where the alignments are in multi-granularity.

Bounding Box Prediction We let the model predict the bounding box ${\boldsymbol{b}}^{j}$ of visual concept $V^{j}$ given the image representation and the text representation, where ${\boldsymbol{b}}^{j}=(cx,cy,w,h)$. By locating different visual concepts in the same image, we expect that the model better learns fine-grained vision language alignments. The bounding box is predicted by:

Meanwhile, we align texts and visual concepts by three objectives which are widely used in vision language pre-training (Chen et al., 2020; Radford et al., 2021; Li et al., 2021a). We extend the objectives to incorporate multi-grained visual concepts in the images.

Contrastive Learning We predict (visual concept, text) pairs, denoted $(V,T)$, from in-batch negatives. Note that visual concepts include objects, regions, and images. Similar to Radford et al. (2021), we randomly sample a mini-batch of $N$ pairs, and calculate the in-batch vision-to-text similarity and text-to-vision similarity.

Similarly, the text-to-vision similarity is:

where ${\boldsymbol{y}}^{\textrm{match}}$ is a 2-dimensional one-hot vector representing the ground-truth label.

Masked Language Modeling We predict the masked words in the text based on the visual concept. We randomly mask out the input tokens with a probability of 25%, and the replacements are 10% random tokens, 10% unchanged, and 80% [MASK]. We use the cross-modal encoder’s outputs, and append a linear layer followed by softmax for prediction. Let $\hat{T}$ denote a masked text, and ${\boldsymbol{p}}^{j}(V,\hat{T})$ denote the predicted probability of the masked token $t_{j}$. We minimize the cross-entropy loss:

where ${\boldsymbol{y}}^{j}$ is a one-hot distribution in which the ground-truth token $t_{j}$ has the probability of one.

Finally, the pre-training objective of X-VLM is defined as:

Experiment

We compare X-VLM with existing approaches at two settings, as listed in Table 1. We refer to them as the 4M setting and 16M setting respectively. Following UNITER (Chen et al., 2020) and other existing work, we prepare our pre-training data using two in-domain datasets, COCO (Lin et al., 2014) and Visual Genome (VG) (Krishna et al., 2017), and two out-of-domain datasets, SBU Captions (Ordonez et al., 2011) and Conceptual Captions (CC) (Sharma et al., 2018).

In the 4M setting, we utilize image annotations only from COCO and VG, which contain 2.5M object annotations and 3.7M region annotations. Note that BUTD, the most widely used object detector, is trained on the same set of object annotations. The existing methods of only learning image-text alignments also utilize the region annotations of VG under the assumption that region descriptions can describe the whole images. In contrast, we take the object labels as text descriptions of objects, and re-formulate the image annotations so that an image has multiple boxes and each box is associated with a text. The text describes the visual concept in the box, which can be an object, a region, or the image itself.

In the 16M setting, we exploit a much noisier Conceptual 12M dataset (CC-12M) (Changpinyo et al., 2021) following ALBEF (Li et al., 2021a). We additionally exploit Objects365 (Shao et al., 2019) and OpenImages (Kuznetsova et al., 2018) following VinVL (Zhang et al., 2021).

Since most downstream V+L tasks are built on top of COCO and VG, we exclude all images that also appear in the validation and test sets of downstream tasks to avoid information leak. We also exclude all co-occurring Flickr30K (Plummer et al., 2015) images via URL matching, because COCO and VG are from Flickr, and there are some overlaps.

2 Implementation Details

The image encoder of X-VLM is vision transformer (Dosovitskiy et al., 2020), which is initialized with Swin Transformerbase (Liu et al., 2021b). The text encoder and the cross-modal encoder consist of six transformer layers respectively. The text encoder is initialized using the first six layers of BERTbase (Devlin et al., 2019), and the cross-modal encoder is initialized using the last six layers. In total, X-VLM has 215.6M parameters for pre-training.

X-VLM takes images of resolution of $224\times 224$ as input. For text input, we set the maximum number of tokens to 30. During fine-tuning, we increase the image resolution to $384\times 384$ and interpolate the positional embeddings of image patches following Dosovitskiy et al. (2020).

We apply mixed precision for pre-training. In the 4M setting, we train the model for 200K steps on 8 NVIDIA A100 GPUs and the batch size is set to 1024, which tasks $\sim 3.5$ days. In the 16M setting, we train the model on 24 GPUs with a batch size of 3072. We sample the data by making half of the images in a batch containing bounding box annotations. We use the AdamW (Loshchilov & Hutter, 2019) optimizer with a weight decay of 0.02. The learning rate is warmed-up to $1e^{-4}$ from $1e^{-5}$ in the first 2500 steps and decayed to $1e^{-5}$ following a linear schedule.

3 Downstream Tasks

We adapt X-VLM to five downstream V+L tasks. We follow the settings in the previous work on fine-tuning (see Appendix A.2). Note that we have cleaned the pre-training datasets to avoid data leaks since downstream V+L tasks have overlaps in images with COCO and Visual Genome.

Visual Question Answering (Goyal et al., 2017) It requires the model to predict an answer given an image and a question. Following the previous work (Cho et al., 2021; Li et al., 2021a), we use a six-layer Transformer decoder to generate answers based on the outputs of the cross-modal encoder. We fine-tune the model for 10 epochs. During inference, we constrain the decoder to only generate from the 3,129 candidate answers to make a fair comparison with existing methods.

Natural Language for Visual Reasoning (NLVR2 (Suhr et al., 2019)) The task lets the model determine whether a text describes the relations between two images. Following ALBEF, we extend the cross-modal encoder to enable reasoning over two images, and perform an additional pre-training step for one epoch using the 4M images: given two images and a text, the model assigns the text to either the first image, the second image, or none of them. Then, we fine-tune the model for 10 epochs.

Image Captioning The task requires a model to generate textual descriptions of input images. We evaluate X-VLM on the COCO Captioning dataset (Chen et al., 2015). We report BLEU-4 and CIDEr scores on the Karparthy test split. To apply X-VLM for captioning, we do not need to add a decoder. Instead, we simply adapt X-VLM to a multi-modal decoder. Specifically, we train X-VLM with language modeling loss for one epoch on 4M data. Then, we fine-tune it on the COCO Captioning dataset. Additionally, following VinVL, we also report the results after applying CIDEr optimization (Rennie et al., 2017) for the second stage of fine-tuning, which are denoted with +.

4 Results on Image-Text Retrieval

Table 2 compares X-VLM with SoTA approaches on MSCOCO and Flickr30K, which are based on either object-centric features (i.e. UNITER and VinVL) or overall image features (i.e. ALIGN, METER, and ALBEF). ALIGN (Jia et al., 2021) is a dual-encoder model similar to CLIP (Radford et al., 2021) specially for image-text retrieval tasks, which is trained on in-house 1.8B image-text pairs. Other VLMs, including our approach, for more general purposes, have a cross-modal encoder and thus use the output of the cross-modal encoder for ranking.

Even though existing approaches either have more parameters or more training data, X-VLM under the 4M setting outperforms all the previous methods by a large margin, achieving new SoTA results. Specifically, X-VLM(4M) which learns multi-grained vision language alignments outperforms VinVL which is based on object-centric features. In contrast, ALBEF which learns only image-text alignments outperforms VinVL only when increasing the training data to 14M. Compared to METER-Swin (Dou et al., 2021) which also uses Swin Transformer as the image encoder, X-VLM has better performance. Furthermore, even though X-VLM(4M) has already achieved very high performance on the image-text retrieval tasks, we still obtain improvements on R@1 when increasing the training instances to 16M. Additionally, Appendix A.3 shows that when increasing the training data to 16M, X-VLM obtains substantial improvements on zero-shot image-text retrieval. Moreover, X-VLM also outperforms ALIGN on zero-shot MSCOCO by a large margin.

Additionally, METER provides an empirical study of VLMs and shows that the vision backbone (or parameter initialization) is important for the model performance. From Swin Transformer to CLIP-ViT, METER improves significantly on both retrieval and VQA (Table 2 and 3). We also have some preliminary observations and leave detailed studies of different backbones of X-VLM for future work.

5 Results on Visual Reasoning

Table 3 shows experimental results on visual reasoning (VQA and NLVR2). First, though ALBEF(14M) outperforms VinVL on image-text retrieval, the coarse-grained approaches such as SOHO, METER-Swin, and ALBEF, all have worse performances than VinVL in visual reasoning tasks, except that METER-CLIP and SimVLM outperform VinVL on VQA. Besides, VinVL also substantially outperforms previous methods that rely on object detectors to learn both object-level and image-level alignments, such as E2E-VLP and KD-VLP.

Nevertheless, X-VLM(4M) with moderate model size and pre-trained on fewer instances outperforms VinVL. Specifically, X-VLM(4M) achieves absolute improvements of $1.52\%$ on VQA and $0.86\%$ on NLVR2 (average on metrics) over VinVL. Meanwhile, as reported in Li et al. (2021a), X-VLM, which encodes images without an object detection process, enjoys $\sim 10$ times faster inference speed than VinVL. The results indicate that our approach of X-VLM is both effective and efficient. X-VLM also outperforms SimVLMbase which is pre-trained on in-house 1.8B data, especially on NLVR2.

6 Results on Visual Grounding

Table 3 reports the performance of X-VLM on RefCOCO+. X-VLM(4M) achieves absolute improvements of $4.5\%$ compared to UNITER. As aforementioned, previous approaches formulate grounding as a ranking task by relying on the region proposals provided by object detectors. In contrast, X-VLM is able to directly predict the target boxes, which is much simpler and more efficient. Furthermore, X-VLM for general V+L purposes outperforms MDETR (Kamath et al., 2021) specialized for visual grounding tasks. X-VLM(4M) using the same set of image annotations achieves absolute improvements of $1.1\%$ (average on metrics), compared to MDETR.

We also evaluate X-VLM in the weakly-supervised setting, proposed by ALBEF. X-VLM(4M) obtains 68.46/76.53/57.09 for vald/testAd/testBd respectively, achieving an absolute improvement of $10.5\%$ (average on metrics) compared to ALBEF(14M). When increasing pre-training images to 16M, X-VLM obtains 77.26/84.11/67.13.

Figure 3 provides a few examples of images from the test set of RefCOCO+. For the supervised setting, we show the bounding boxes predicted by X-VLM given the text descriptions. For the weakly-supervised setting, following ALBEF, we provide the Grad-CAM visualization, which uses the cross-attention maps in the fourth layer of the cross-modal encoder. The visualization examples show that X-VLM has a strong ability of cross-modal understanding. It successfully predicts the correct regions in images, even though the text descriptions only differ in a single word. Furthermore, X-VLM can align each word in the text to the corresponding image region. We provide more examples in Appendix A.4, showing X-VLM’s superior performance in vision language alignment.

7 Results on Image Captioning

We show that X-VLM, usually considered as an “encoder-only” model, has comparable performance with SoTA generative methods on image caption generation, as indicated in Table 3. Specifically, X-VLM pre-trained on 16M instances performs similarly to SimVLM which uses not only 1.8B in-house image-text pairs but also a large-scale text corpus. Besides, we observe that CIDEr optimization largely boosts the CIDEr scores. X-VLM in moderate model size also has comparable performance to VinVLlarge.

8 Ablation Study

We also conduct an in-depth ablation study to investigate the role of different components in the X-VLM, as shown in Table 4. All compared model variants are trained on 4M images for 80K steps with a batch size of 3072 to ensure a fair comparison. We use Recall@1 as an evaluation measure in the retrieval tasks and Meta-Sum as a general measure. We report RefCOCO+ evaluation results in the weakly-supervised setting.

First, we evaluate the effectiveness of visual concepts in different granularities, i.e. w/o object and w/o region. The results show that training without either of them hurts the performance, demonstrating the necessity of learning multi-grained alignments. Besides, we can observe that w/o region makes the performance drop more drastically than w/o object. Furthermore, the ablation study shows that bounding box prediction is a critical component of X-VLM, as w/o bbox loss leads to the lowest Meta-Sum. We also report the results of ‘w/o all’ where all the above components are ablated. Though in the 4M setting, only 210K images have dense annotations, X-VLM can leverage the data to learn multi-grained vision language alignment and substantially improve the performances in the downstream V+L tasks (Meta-Sum from 580.6 to 605.2).

Conclusion and Discussion

In this paper, we have proposed X-VLM, a strong and efficient approach to perform multi-grained vision language pre-training. Training of the model is driven by locating visual concepts in the image given the associated texts and aligning texts with relevant visual concepts, where the alignments are in multi-granularity. We have pre-trained X-VLM with 4M and 16M images, which are of moderate size. Also, X-VLM only consists of 216M parameters. These choices are made because we want to make our experiments as “green” (Schwartz et al., 2020; Xu et al., 2021b) as possible and be accessible to a larger group of people. Experiments on downstream V+L tasks, including image-text retrieval, visual reasoning, visual grounding, and image caption generation have shown that X-VLM outperforms the existing methods which could be larger and/or pre-trained on more data. As suggested by the comparison between X-VLM(4M) and X-VLM(16M), adding more pre-training datasets will probably lead to further performance improvements. As for applications, X-VLM has shown better performance in understanding fine-grained vision language alignments. For example, it can generate image captions probably having more object details, which makes it a better choice to help people with disability in vision to understand images. On the other hand, X-VLM in moderate model size is also easier to deploy.

Acknowledgements

We would like to thank Wangchunshu Zhou, Wenguan Huang, and Xiu-jun Li at ByteDance for their generous assistance in data collection and insightful comments in technical discussions.

References

Appendix A Appendix

Table 5 gives statistics of object and region annotations of each dataset. Only the Visual Genome dataset contains region annotations. Besides, the OpenImages dataset offers some relationship annotations, indicating pairs of objects in particular relations (e.g. ”woman playing guitar”, ”beer on table”), object properties (e.g. ”table is wooden”), and human actions (e.g. ”woman is jumping”), which can also be viewed as region annotations.

Note that we filtered out some samples because of: 1) invalid annotations (e.g. negative values for bounding boxes or boxes being outside of the images); 2) boxes being too small ($<$ 1%); 3) highly overlapped textual descriptions of regions ($>$ 75%), etc. After pre-processing, we keep: for example, COCO objects 446,873 (from 859,999), VG objects 2,043,927 (from 3,802,349), VG regions 3,699,598 (from 5,402,953).

A.2 Implementation Details of Downstream Tasks

We follow the settings in existing methods for fine-tuning. We describe how we implement fine-tuning on the downstream V+L tasks, and we also provide our fine-tuning scripts for more details. Note that we have cleaned our pre-training datasets to avoid data leaks since downstream V+L tasks have overlaps in images with COCO and Visual Genome.

Visual Question Answering (VQA (Goyal et al., 2017)) Following existing methods (Tan & Bansal, 2019; Chen et al., 2020; Li et al., 2021a), we use both train and validation sets for training, and include additional question-answer pairs from Visual Genome. The VQA model contains a 6-layer transformer-based decoder to generate answers based on the outputs of the cross-modal encoder following previous work (Cho et al., 2021; Li et al., 2021a). The decoder is initialized using the pre-trained weights from the cross-modal encoder. Then, the model is fine-tuned by optimizing the auto-regressive loss for 10 epochs. During inference, we constrain the decoder to only generate from the 3,129 candidate answers There is a NULL answer. Thus, the actual number of candidate answers is 3,128. to make a fair comparison with existing methods.

Natural Language for Visual Reasoning (NLVR2 (Suhr et al., 2019)) Since the task asks the model to distinguish whether a text describes a pair of images, we follow ALBEF to extend the cross-modal encoder to enable reasoning over two images. We also perform an additional pre-training step for 1 epoch using the 4M images: given a pair of images and a text, the model needs to assign the text to either the first image, the second image, or none of them. Then, we fine-tune the model for 10 epochs.

Image Captioning The task requires a model to generate textual descriptions of input images. We evaluate X-VLM on the COCO Captioning dataset (Chen et al., 2015). We report BLEU-4 and CIDEr scores on the Karparthy test split. To apply X-VLM for captioning, we do not need to add a decoder. Instead, we simply adapt X-VLM to a multi-modal decoder. Specifically, we train X-VLM with language modeling loss for one epoch on 4M data. Then, we fine-tune it on the COCO Captioning dataset with naive cross-entropy loss for five epochs. Additionally, following VinVL, we also report the results after applying CIDEr optimization (Rennie et al., 2017) for the second stage of fine-tuning which takes another five epochs.

A.3 Zero-Shot Image-Text Retrieval Results

Table 6 shows zero-shot image-text retrieval results and compares X-VLM with the dual encoder SoTAs (CLIP and ALIGN) which are pre-trained using only the retrieval objective. We can observe that though X-VLM is pre-trained using the combination of different objectives, it still has very competitive results on zero-shot retrieval tasks.

A.4 Case Study

Figure 4 and 5 provide visualizations of some images from the test set of RefCOCO+. We show the bounding boxes predicted by X-VLM given the text descriptions. For the weakly-supervised setting, we provide the Grad-CAM visualization which uses the cross-attention maps in the fourth layer of the cross-modal encoder. We can observe that in both settings X-VLM can predict correct regions even though the textual descriptions only differ in a single word. X-VLM can also align each word in the text to the corresponding image region, showing X-VLM’s superior ability of multi-grained vision language alignments.