Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

Introduction

In recent years, the rapid development of Large Language Models (LLMs) (OpenAI, 2023; Bai et al., 2023a; Cai et al., 2024) has spurred increasing efforts to extend their multimodal capabilities (Chen et al., 2023; Liu et al., 2023e). As shown in Fig. 1 (a), most existing Multimodal Large Language Models (MLLMs) adopt a modular architecture, where visual encoding and language decoding are processed separately. This is typically achieved by combining a pre-trained visual encoder like CLIP-ViT (Radford et al., 2021) with an LLM (Liu et al., 2023e; Chen et al., 2024c; Li et al., 2023a). Recent research has also started exploring monolithic MLLMs (Bavishi et al., 2023; Diao et al., 2024; Chen et al., 2024b), as shown in Fig. 1 (b), which integrate visual perception and multimodal understanding directly into a single LLM. Due to their simplicity and unity, monolithic MLLMs can be more easily deployed using existing LLM inference libraries (LMDeployContributors, 2023) and show superior efficiency than modular MLLMs (Diao et al., 2024; Chen et al., 2024b).

Despite recent progress, training a monolithic MLLM with promising performance still remains challenging. In particular, monolithic MLLMs struggle to replicate the successful usage of pre-trained visual encoders in modular MLLMs for visual perception. Therefore, researchers often rely on additional pre-training to compensate for the shortcomings in visual perception in monolithic MLLMs (ChameleonTeam, 2024; Diao et al., 2024). A straightforward approach is the native pre-training (ChameleonTeam, 2024), which pre-trains a monolithic MLLM from scratch using a mixture of text-only and multimodal data. However, it requires prohibitively high training costs and often suffers from challenges of unstable optimization (ChameleonTeam, 2024). Another common solution is the continuous pre-training (Diao et al., 2024), which extends the pre-trained LLM to multimodality via additional visual pre-training. Benefiting from the knowledge in the pre-trained LLM, the cost of continuous pre-training becomes much more affordable. Nevertheless, due to the catastrophic forgetting issue (Zhai et al., 2023), the pre-trained language knowledge is inevitably undermined during continuous pre-training, thereby weakening the multimodal capabilities.

In this paper, we aim to address the forgetting issue of continuous pre-training through delta tuning (Ding et al., 2022). In particular, we argue that such issue arises from the shared architecture for joint vision and language modeling, where optimizations for vision can negatively impact language capabilities. Therefore, it is a natural thought to introduce an independent visual parameter set into the pre-trained LLM, which allows formulating visual pre-training with partial parameter tuning. This can help retain the language knowledge by freezing the entire LLM during continuous pre-training, while improving visual learning. This principle is also aligned with previous endeavors in modular MLLMs, e.g., QwenVL (Bai et al., 2023b) and InternVL-1.5 (Chen et al., 2024c), where the visual parameters are placed outside the LLM.

Based on the above principle, we propose a novel monolithic MLLM, namely Mono-InternVL. As shown in Fig. 2, the visual parameters in Mono-InternVL are designed as a set of expert networks that are seamlessly integrated into the LLM via the mixture-of-experts mechanism. To better capture the visual knowledge, these experts are initialized from the Feed-Forward Networks (FFNs) in the pre-trained LLM. Based on this architecture, we present an innovative visual pre-training method called Endogenous Visual Pre-training (EViP). Specifically, EViP is formulated as a progressive learning process of three stages: 1) concept learning to grasp basic visual concepts, 2) semantic learning to capture high-level semantics, e.g., world knowledge, and 3) alignment learning to align knowledge with downstream tasks. Benefiting from the architecture and the pre-training strategy, the visual scalability of Mono-InternVL is fully unleashed, where the downstream performance consistently improves as the scale of the pre-training data increases. After visual pre-training, Mono-InternVL accommodates complex multimodal tasks via supervised instruction tuning.

To validate our method, we develop Mono-InternVL using the pre-trained LLM InternLM2-1.8B (Cai et al., 2024), and conduct extensive experiments on 16 multimodal benchmarks. Experimental results not only demonstrate the significant performance improvements of Mono-InternVL against previous monolithic MLLMs, but also validate its superior efficiency compared to existing modular MLLMs. For instance, Mono-InternVL with 1.8 billion activated parameters can outperform existing monolithic MLLMs with 7 billion parameters by a significant margin, e.g., +15.5% over EVE on average. Compared to the state-of-the-art modular MLLM InternVL-1.5 (Chen et al., 2024c), Mono-InternVL shows superior performance on 6 multimodal benchmarks while reducing first token latency by 67%. In conclusion, our contributions can be summarized in threefold:

We present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts architecture. This architecture can effectively extend the pre-trained LLM to a monolithic MLLM while retaining the pre-trained knowledge.

We propose a novel visual pre-training approach for Mono-InternVL called endogenous visual pre-training (EViP). EViP adopts a progressive learning strategy to encourage visual experts of Mono-InternVL to continuously grasp visual knowledge from noisy data to high-quality data.

Mono-InternVL demonstrates for the first time that the leading performance of MLLM no longer depends on the pre-trained visual encoder, thereby opening new avenues for designing future MLLMs. In particular, Mono-InternVL achieves the state-of-the-art (SoTA) results compared to existing MLLMs on 6 multimodal benchmarks.

Related Work

Modular multimodal large language models. Recent progress in large language models (LLMs) has catalyzed the integration of vision and language modalities, giving rise to multimodal large language models (MLLMs). Both commercial entities, such as GPT-4V (Yang et al., 2023) and Gemini series (Team et al., 2023), and other open-source initiatives, e.g. BLIP series (Li et al., 2022; 2023a; Dai et al., 2023), LLaVA series (Liu et al., 2023e; d; 2024), InternVL (Chen et al., 2023; 2024c), have pursued this integration, often linking LLMs (Touvron et al., 2023a; b; Team, 2023; Cai et al., 2024) with large vision models (LVMs) (Radford et al., 2021; Dosovitskiy et al., 2021; Dehghani et al., 2023; Chen et al., 2023) via intermediate layers. Specifically, there are lightweight MLLMs (i.e., $\leq$ 4B parameters), such as MobileVLM-V2 (Chu et al., 2024), Mini-Gemini (Li et al., 2024), MM1 (McKinzie et al., 2024), DeepSeek-VL (Lu et al., 2024), PaliGemma (Beyer et al., 2024) and MiniCPM-V (Yao et al., 2024). However, such encoder-based vision-language models (modular MLLMs) encounter challenges like limitations in input processing due to pre-trained visual encoders, deployment inefficiencies, and complexities in balancing the capacities of LLMs and LVMs, as also pointed out by (Diao et al., 2024).

Monolithic multimodal large language models. The issues related to modular MLLMs have steered research toward encoder-free architectures, also known as monolithic MLLMs, which can be summarized into two categories. The first category obtains continuous visual tokens through a lightweight structure before feeding into MLLMs. For instance, Fuyu-8B (Bavishi et al., 2023) processes images directly through a simple linear projection, adeptly handling high-resolution images without using a dedicated visual encoder. EVE-7B (Diao et al., 2024) prioritizes vision-language pre-alignment from an LLM-centric perspective and enhances image recognition through visual representation supervision. SOLO (Chen et al., 2024b) introduces an open-source training recipe for developing monolithic MLLMs. In contrast, the second category introduces VQ tokenizer-based models to generate discrete visual tokens to support image generation tasks, with representative works such as Chameleon (ChameleonTeam, 2024), Show-o (Xie et al., 2024), Transfusion (Zhou et al., 2024), and Emu3 (Wang et al., 2024b).

Multimodal mixture-of-experts. VLMo (Bao et al., 2022) and BEiT-3 (Wang et al., 2022) employ a pool of modality experts to replace the feed-forward network in standard Transformer to capture modality-specific information by switching to different modality experts, and use the shared self-attention across modalities to align visual and linguistic information. VL-MoE (Shen et al., 2023) introduces mixture-of-experts (MoE) (Yuksel et al., 2012) based on the above works for better training and deploying. MoMa (Lin et al., 2024) also uses multimodal mixture-of-experts for pre-training of MLLMs (ChameleonTeam, 2024) and cooperates with sparse components, e.g. MoE and mixture-of-depths (MoD) (Raposo et al., 2024) to improve the efficiency of pre-training from scratch with trillions of mixed-modal tokens. Inspired by the above literature, we introduce multimodal mixture-of-experts (i.e., a visual expert and a language expert) for pre-training monolithic MLLMs and use a novel progressive learning strategy, named endogenous visual pre-training (EViP), to address the unique challenges faced by monolithic MLLMs.

Mono-InternVL

As shown in Fig. 2, we first outline the architecture of Mono-InternVL, which consists of tokenizers and a multimodal mixture-of-experts structure.

As shown in Fig. 2, $\mathcal{F}_{\text{llm}}$ is designed as a multimodal mixture-of-experts structure. In particular, we adopt the static routing strategy that assigns visual and textual experts to the corresponding tokens. Therefore, the l-th LLM layer can be defined by

Here, $\text{MHA}(\cdot)$ and $\text{RMSNorm}(\cdot)$ denote the multi-head attention (Vaswani et al., 2017) and the layer normalization (Zhang & Sennrich, 2019), respectively. $\text{MMoE}(\cdot)$ is the proposed multimodal mixture-of-experts, formulated by

As defined in Eq. 4 and 5, the MMoE structure has two distinct advantages over the existing monolithic MLLM. Firstly, the visual learning of Mono-InternVL can greatly benefit from the pre-trained language knowledge, while the language ability can still be preserved by freezing $\text{FFN}_{t}$. Secondly, the MMoE structure significantly enhances the model’s capacity for vision-and-language modeling, while the additional inference cost is almost negligible due to the MoE mechanism.

2 Endogenous Visual Pre-training

Endogenous Visual Pre-training (EViP) aims to maximize the benefits of Mono-InternVL from visual experts through pre-training on massive noisy and synthetic data. Unlike existing methods (Diao et al., 2024; ChameleonTeam, 2024), we formulate EViP from the perspective of delta tuning (Ding et al., 2022), in which most of the LLM parameters are frozen to preserve its pre-trained knowledge. Therefore, the objective of EViP can be defined by

where $\mathcal{L}(\cdot)$ and $\hat{y}$ denote the auto-regressive loss and the ground-truth, respectively. As shown in Fig. 3, $\Delta\theta$ denotes parameters of patch embedding and visual experts in the concept and semantic learning, i.e., $\theta_{v}$, while in the alignment learning stage $\Delta\theta$ also includes the parameters of multi-head attentions. Based on Eq. 6, EViP is designed as a progressive learning process. As shown in Fig. 3 and Tab. 1, EViP consists of three sub-stages, namely concept learning (S1.1), semantic learning (S1.2) and alignment learning (S1.3). For different sub-stages, we use carefully partitioned data to achieve coarse-to-fine visual learning.

Concept learning. Concept learning aims to encourage the model to learn fundamental visual concepts, such as object categories or basic shapes. Therefore, we first pre-train Mono-InternVL with about 922 million noisy samples, which are sampled from Laion-2B (Schuhmann et al., 2022a) and Coyo-700M (Byeon et al., 2022). In this sub-stage, Mono-InternVL employs a simple prompt to perform generative learning, i.e., “provide a one-sentence caption for the image”. Meanwhile, we constrain the maximum number of image patches of the visual tokenizer to 1,280 for training efficiency. To ensure that the foundational language capabilities are preserved while enabling visual specialization, the entire LLM is kept frozen during concept learning, and only the patch embedding and visual experts are optimized.

Semantic learning. After concept learning, Mono-InternVL is able to understand basic concepts in the image, but organizing this information to produce reasonable descriptions remains challenging. To achieve a higher-level visual understanding, we utilize the pre-trained InternVL-8B (Chen et al., 2024c) to produce short captions for 258 million images. Compared to the original noisy captions, synthetic captions typically depict complex visual knowledge, such as relationship and world knowledge, etc., while containing less noisy information unrelated to the image, e.g., time of shooting, and the photographer. In this sub-stage, we adopt the same optimization strategy as concept learning, except that the maximum number of image patches is increased to 1,792.

Alignment learning. To meet the visual requirements of downstream tasks, we further perform alignment learning on Mono-InternVL. As shown in Tab. 1, our alignment data is sampled from the pre-training data of InternVL-1.5 (Chen et al., 2024c), including 143 million samples of image captioning, detection and optical character recognition (OCR). In particular, captioning data, detection data and OCR data account for about 53.9%, 5.2% and 40.9% of the total, respectively. In this sub-stage, we utilize the task-specific prompts from InternVL-1.5 for the generative learning, and increase the maximum number of image patches to 3,328. Compared to previous sub-stages, the multi-head attention layers are additionally optimized to achieve better vision-language alignment.

3 Instruction Tuning

In this stage, we follow InternVL (Chen et al., 2024c) to use around 5 million bilingual instructions for supervised learning, covering various tasks such as visual question answering, multimodal dialogue, mathematics, knowledge, etc. In addition to this, we further include additional instruction data for video understanding and handwritten text recognition. In this stage, the entire models are optimized and the maximum number of image patches is increased to 6,400 to accommodate high-resolution images. Details of instruction data can be found in Appendix §A.1.

Experiments

We evaluate Mono-InternVL and existing MLLMs on 16 comprehensive multimodal benchmarks. Specifically, general MLLM benchmarks encompass MMBench-EN test (Liu et al., 2023f), MMVet (Yu et al., 2023b), MMMU val (Yue et al., 2023), MME (Fu et al., 2023), MathVista testmini (Lu et al., 2023), SEED Image (Ge et al., 2024), OCRBench (Liu et al., 2023g), HallusionBench (Guan et al., 2023), and CCBench dev (Liu et al., 2023f). Visual question answering benchmarks include TextVQA val (Singh et al., 2019), SQA test (Lu et al., 2022a), GQA test-dev (Hudson & Manning, 2019), DocVQA test (Clark & Gardner, 2018), AI2D test (Kembhavi et al., 2016), ChartQA test (Masry et al., 2022), and InfographicVQA test (Mathew et al., 2022). The evaluation metrics follow the existing methods (Chen et al., 2024c; Diao et al., 2024). Part of the results of Chameleon and EVE are evaluated with VLMEvalKit (Duan et al., 2024) or from the OpenCompass leaderboard (Contributors, 2023).

2 Implementation Details

Mono-InternVL is implemented based on InternLM2-1.8B (Cai et al., 2024) with newly added visual tokenizer and visual experts. The visual experts are initialized from pre-trained MLPs in InternLM2-1.8B to leverage existing learned representations for improved visual feature extraction, which accounts for 1.2 billion parameters. We adopt a similar dynamic high-resolution strategy from InternVL-1.5 (Chen et al., 2024c) to align an optimal resolution for input image, which is then patchfied to visual tokens. The remaining configurations are kept identical to InternLM2-1.8B. The endogenous visual pre-training and instruction tuning take approximately 16 days (646k iterations) and 1 day (14k iterations) on 256 NVIDIA A100 GPUs, respectively. More detailed training configurations are given in Appendix §A.2.

3 Quantitative Experiments

Comparison with existing MLLMs. In Tab. 2 and 3, we compare Mono-InternVL and existing MLLMs on 16 multimodal benchmarks. From Tab. 2, the first observation is that most modular MLLMs outperform existing monolithic MLLMs by significant margins. For example, the average performance of InternVL-1.5-2B (Chen et al., 2024c) on 9 MLLM benchmarks greatly exceeds the SoTA monolithic MLLM, i.e., + 15.5% over EVE-7B (HD) (Diao et al., 2024). These results strongly suggest the challenges in existing monolithic MLLMs. In contrast, Mono-InternVL-2B with a slightly smaller model size can even outperform the SoTA modular MLLM, i.e., + 0.8% against InternVL-1.5-2B on average. Notably, Mono-InternVL-2B demonstrates distinct advantages on MathVista and OCRBench, suggesting its seamless text recognition and reasoning capabilities. Moreover, superior bilingual ability of Mono-InternVL-2B is also validated on CCBench, which contains a large amount of questions related to Chinese culture. Compared to existing monolithic MLLMs, performance gains of Mono-InternVL are more distinct, e.g., +15.4% over EVE-7B (HD) (Diao et al., 2024) on MMVet and +7.9% over Emu3 (Wang et al., 2024b) on TextVQA, while using a much smaller parameter scale. Similar advantages of Mono-InternVL can also been witnessed in Tab. 3, e.g., +2.1% on TextVQA. Nevertheless, we also observe that Mono-InternVL is still inferior to InternVL-1.5 on high-resolution benchmarks, e.g., -12.4% on InfoVQA. This is because specific optimizations for high-resolution encoding are not the focus of the paper, e.g., positional embedding and high-resolution training data, which we plan to explore in future research. Overall, these comparisons significantly validate the architecture and training strategy of Mono-InternVL.

In Tab. 4, we further compare the pre-training performance of Mono-InternVL and existing MLLMs. From this table, we can observe that with concept and semantic learning, Mono-InternVL-S1.2 already exceeds existing modular MLLMs, e.g., +13.8 CIDEr over MM1 (McKinzie et al., 2024) on COCO Captions, demonstrating that Mono-InternVL-S1.2 is effective in capturing fundamental multimodal relationships. It is worth noting that pre-training in Mono-InternVL-S1.2 only consumes about 0.9B image-text pairs, but the cost in MM1 and Flamingo is much more expensive, e.g., more than 2B data. Compared to monolithic MLLMs, Mono-InternVL also demonstrates superior performance. For instance, even though Chameleon has a much larger model size, it is still inferior to Mono-InternVL-S1.3 by -2.6 CIDEr on Flickr30k (Young et al., 2014). These results further confirm the effectiveness of EViP for Mono-InternVL.

Ablation studies. To validate the design of Mono-InternVL, we conduct extensive ablation studies in Tab. 5 and Fig. 4. Specifically, Tab. 5 compares different strategies for visual pre-training. The first row is the common strategy used in existing monolithic MLLMs, i.e., full tuning of the LLM, which yields the worst downstream performance in the table. After employing visual experts (the second row), such a full-tuning strategy becomes more effective, e.g., +1.6% on GQA. These comparisons well validate that the shared architecture for joint vision and language modeling is sub-optimal in monolithic MLLMs. Besides, we also observe that the delta tuning strategy greatly benefits the visual pre-training, providing +18.8% and 16.1% gains on SQA-I and AI2D, respectively. Compared to full tuning, delta tuning can effectively preserve the knowledge of the pre-trained LLM, which is also crucial for maintaining the language understanding capabilities required for effective multimodal interactions.

Fig. 4 further demonstrates the relationship between downstream performance and pre-training data size. From it we can observe that performance of Mono-InternVL will gradually reach an upper bound in the concept learning. Through additional semantic learning and alignment learning, capabilities of Mono-InternVL consistently boost as the data size increases. It is important to note that that the alignment learning plays a significant role for VQA and MME, which can provide sufficient task-related knowledge, e.g., OCR knowledge. These results not only demonstrate the data scalability of Mono-InternVL, but also confirm the advantages of coarse-to-fine learning in EViP.

Comparison of inference efficiency. In Tab. 6, we compare the inference speed of Mono-InternVL and InternVL-1.5 using the popular deployment library LMDeploy (LMDeployContributors, 2023). From this table, we can find that due to the elimination of visual encoder, Mono-InternVL demonstrates superior efficiency under different number of input tokens. In particular, the first-token time is greatly reduced in Mono-InternVL, e.g., up to -67% against InternVL-1.5. Benefiting from this, the overall throughput is correspondingly increased by around 31%. These results greatly validate the efficiency of Mono-InternVL. We note that this is only an initial attempt, and using Turbomind backend or further optimization techniques may yield better performance.

4 Qualitative Experiments

To gain in-depth insights into Mono-InternVL, we visualize its attention maps of different layers in Fig. 5. From this figure, we can draw two noteworthy findings. Firstly, despite the global connectivity of the Transformer architecture, locality still exists in the visual encoding of shallow layers. As shown in Fig. 5, in the first layer, visual tokens only interact with their nearby content, yielding patterns that closely resemble those produced by convolutional neural networks (He et al., 2016). Secondly, modalities are barely interactive at shallow layers but gradually fused as the layers deepen. As illustrated in Fig. 5, the attention weights between visual and textual tokens are extremely small in the first layer and become larger in deeper layers. We believe these examples will provide useful hints for the design of monolithic MLLMs. More examples are given in Appendix §A.4.

Conclusion

In this paper, we propose Mono-InternVL, a monolithic multimodal large language model (MLLM) that integrates visual encoding and textual decoding into a single LLM. In Mono-InternVL, a set of visual experts is embedded into the pre-trained LLM via a mixture-of-experts mechanism. By freezing the LLM, Mono-InternVL ensures that visual capabilities are optimized without compromising the pre-trained language knowledge. Based on this structure, an innovative Endogenous Visual Pre-training (EViP) is introduced to realize coarse-to-fine visual learning. Extensive experiments demonstrate the effectiveness and efficiency of Mono-InternVL compared to existing MLLMs. Our work greatly pushes the boundaries of monolithic MLLMs, providing new possibilities for the development of MLLMs.

References

Appendix A Appendix

The datasets used in the instruction fine-tuning stage are listed in Tab. 7.

A.2 More Training Details

Hyper-parameters used in the training stages are listed in Tab. 8.

A.3 More Ablation Studies

In Fig. 6, we provide the full results of Fig. 4 with all the downstream tasks.

In Tab. 9, we examine the effects of freezing and unfreezing attention layers in alignment learning. We observe that unfreezing attention results in consistent improvements across all metrics, suggesting that it is crucial to optimize the multi-head attentions in this sub-stage for better vision-language alignment.

A.4 Visualizations

Prompt: What is the content in this picture? Response: The image is a graphical representation, primarily informational in nature. It features two individuals depicted in line art style. The background of the image is lightcolored. At the top of the image, there’s text that reads “THE NOBEL PRIZE IN PHYSICS 2024.” This indicates that the award being discussed in this image pertains to physics and is scheduled for the year 2024. Below this header, there are two names mentioned: John J. Hopfield and Geoffrey E. Hinton. Further down, there is a quote attributed to both named individuals: “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” This quote suggests that both John J. Hopfield and Geoffrey E. Hinton made significant contributions to the field of artificial intelligence and neural networks, which may have involved pioneering discoveries and innovations that were crucial for developing machine learning technologies. Finally, at the bottom, there’s another text stating: “THE ROYAL SWEDISH ACADEMY OF SCIENCES.” This indicates that John J. Hopfield and Geoffrey E. Hinton have been recognized by the Royal Swedish Academy of Sciences. Overall, this image appears to serve as a tribute or recognition of the achievements of these scientists in the field of physics, particularly relating to artificial intelligence and neural networks. Prompt: Describe this image. Response: This image depicts a bustling urban scene at night in an animation style, with a large electronic billboard prominently displaying the words ”YUNIKA VISION” in bright white letters against a black background. The billboard is surrounded by several other tall buildings, including those of well-known companies such as HITACHI, TOSHIBA, SHARP, SONY, PANASONIC, and NEC. In the foreground, there are multiple train tracks that intersect near what appears to be an elevated railway or subway system. Several trains are visible, indicating active transportation amidst the busy cityscape. The skyline is dotted with various other commercial and residential structures, contributing to the sense of a thriving metropolis. Trees can also be seen around the area, providing some greenery amidst the urban environment. Overall, the image captures the dynamic energy of a modern city at night, with its blend of technology, commerce, and human activity. Visual grounding

Prompt: Detect the $<$ref$>$the American Shorthair$<$/ref$>$ in the image with its bounding box. Response: $<$ref$>$ the American Shorthair $<$/ref$>$ $<$box$>$ [] $<$/box$>$ Prompt: Please describe the detected object. Response: The object in the image is a cat. The cat has gray fur with black stripes and is wearing a pink sweater. It appears to be sitting on a windowsill, looking up attentively towards the camera. Prompt: Detect the $<$ref$>$the tallest building$<$/ref$>$ in the image with its bounding box. Response: $<$ref$>$ the tallest building $<$/ref$>$ $<$box$>$ [] $<$/box$>$ Prompt: Please infer where this place is. Response: This image depicts a stunning view of the skyline of Shanghai, China. The prominent features in the skyline include several high-rise buildings with unique structures such as the Oriental Pearl Tower and the Shanghai World Financial Center. There is also a distinctive arch bridge spanning across a river, which appears to be part of Shanghai’s famous Huangpu River waterfront area known for its historic sites and modern development. VQA

Prompt: Describe the image. Response: The image features a large adult dog lying on its side, likely the mother of several puppies. The adult dog has brown and white fur with distinct markings on its face. She is surrounded by several small puppies, all of which are nursing from her body or resting on a light blue towel that covers part of the floor. The setting appears to be indoors, possibly in a home environment given the tiled flooring and the soft, comfortable-looking towel. The atmosphere seems calm and nurturing as the dogs interact closely together. Prompt: Please tell me the exact number of golden retrievers in the picture. Response: There are a total of six golden retrievers in the picture. Chinese OCR

Prompt: 识别图中文字。 Response: 图中文字如下： “你知道吗。晚上和早上的风 开始冷了，树叶也开始落了 下次见面就该穿长袖了 对了。好久不见。” Code generation

Prompt: Convert the formula in the figure to latex format. Response: Certainly! Here is the formula from the figure in LaTeX format: Document understanding

Prompt: What is the age of subject B.L.? Response: B.L. is 5 years old at the time of study on the effect of milk protein dilution. Prompt: Whose initial weight is the largest? What is it? Response: The largest initial weight in the table is for P.K. with 61.5 pounds (lb.). Prompt: How many children participated in the experiment in total? Response: There are a total of 14 children who participated in the experiment. Math