Language Models are General-Purpose Interfaces

Introduction: Design Principles

The large-scale language model serves as a general-purpose interface not only for language tasks, but also for vision, and multimodal tasks. Language models have open-ended output space, which generalizes to a wide range of tasks. As long as we can describe the predictions via natural language, the downstream task can fit in with language-model-based task layer. It is natural that transforming various predictions to free-text sequences . For example, we can transform the target labels, and answers to texts for classification, and question answering, respectively. In addition, with the help of the universal task layer, the prediction process can go beyond single turn, i.e., a multi-turn dialogue interface can be built upon language models by conditioning on history context. Such unification of various tasks is important to general-purposed AI, which unifies representations, transformations, and expressions into a shared module.

GPT-3 has shown that the intriguing properties emerge from causal language model pretraining. Because of the favorable sample efficiency and inductive bias of causal language modeling (i.e., all tokens make predictions and produce supervision signals) compared with other counterparts (such as masked language modeling), it is effective to give models the desired properties via causal language modeling. The capabilities of zero- and few-shot learning are critical to be a general-purpose task layer. Zero-shot generalization indicates that language models have learned an enormous amount of world knowledge and patterns by reading large-scale text corpora. The memorized information can serve as reusable background knowledge and basic skills for a wide range of end tasks. Moreover, in-context learning enables us to easily adapt either pretrained or finetuned models to new scenarios. For example, we can use task instructions to repurpose the model, and use demonstrations of some examples to conduct few-shot learning.

Although causal language models are good at zero- and few-shot generalization, BERT and T5 show that having bidirectional encoders pretrained by masked language modeling achieves much better finetuning performance. Once the whole input is given, non-causal modeling is quite rational for encoding data. Because all the context can access each other, while causal modeling can only make use of history tokens one by one. The advantage of finetuning is helpful for the data-rich setting where there are many annotated data available. In addition, non-causal encoder pretrained by the masked language modeling objective achieves competitive performance on cross-lingual transfer , which makes it effective to adapt models to the multilingual setting.

Semi-causal language modeling plays the role of linking together non-causal encoders and the causal language model. It is a meta task in the sense of universal interface pretraining of pretrained encoders. Specifically, non-causal encoders learn to represent various input data, and a causal language model serves as a universal task layer. Non-causal encoders dock with a causal language model, so that we can benefit from both modeling methods described as above. In comparison with previous encoder-decoder pretraining (such as prefix language modeling, and T5; Raffel et al. 82), our task non-causally encodes random spans of the whole sequence, while generating the rest via causal language modeling. Moreover, in terms of architecture, we directly feed the outputs of bidirectional encoders into the causal decoder, rather than relying on cross attention . Besides, multiple bidirectional encoders can be mounted to the causal language model, but the encoder-decoder architecture usually has only one encoder.

Cognition is usually categorized into two levels : System 1 (i.e., intuitive, and unconscious) and System 2 (i.e., sequential, conscious, planning, and reasoning). In the proposed framework, the modules can be regarded as an implementation of these two levels, respectively. To be specific, non-causal encoders pretrained by masked data modeling, such as BERT and BEiT , are used as a perception layer to encode various input modalities. The encoding modules can be viewed as System 1. After we obtain the input representations, we feed them to the causal language model, which has shown promising performance on commonsense reasoning and planning . The universal task layer is designed to play a role of System 2 in our method.

The universal task layer based on causal language modeling enables users to interact with pretrained non-causal encoders using natural language. First, language can be used as a programming language for the underlying pretrained or finetuned models, which is compiled by the universal interface. For example, we can write text-based instructions and explanations to repurpose and guide the model behaviors. Second, the universal interface enables the models to present the results using free texts, making predictions directly understandable and explainable. Third, the proposed framework natively supports multi-turn conversational interactions. In each turn, we can feed the encoded input to the interface layer and then generate response results in a semi-causal manner.

MetaLM: Meta Language Model

Guided by the design principles in Section 1, we present Meta Language Model (MetaLM), a semi-causal language model that plays the role of a general-purpose interface and supports interactions with various foundation models. An overview of our framework is shown in Figure 2. Specifically, a collection of pretrained encoders, that perceive diverse modalities, dock with a language model. The language model is regarded as a universal task layer (i.e., general-purpose interface), which unifies various tasks as free-text generation.

In order to pretrain MetaLM, we propose a semi-causal language modeling task to jointly learn the modules. MetaLM subsumes the advantages and capabilities from both worlds. From the language model, MetaLM inherits the capabilities of in-context learning, multi-turn interaction, and open-ended generation. Moreover, the underlying foundation models are conducive to finetuning because of bidirectional modeling .

Input representations of MetaLM are grouped into two categories. The first type is contextualized representations obtained by the underlying encoders and then projected by a connector layer. For example, as shown in in Figure 2, the image patches and $x_{7},x_{8}$ are encoded by the bidirectional vision-language encoder. The second category is token embeddings of texts, such as $x_{5}$, and $x_{6}$ in Figure 2. The representations of these two categories are summed with positional embeddings before feeding into the general-purpose interface.

2 Model Architecture

As shown in Figure 3, we summarize the model architectures of three language model variants and the proposed semi-causal language model. First, causal language model (such as GPT; Brown et al. 14) is a left-to-right Transformer decoder. Second, prefix language model uses the encoder-decoder architecture with cross-attention connections to complete the sequence. Third, non-causal language model is a bidirectional encoder, which is usually pretrained by masked language modeling . Forth, the proposed semi-causal language model has a unidirectional Transformer decoder, and multiple bidirectional encoders that dock with the decoder. In other words, our model processes the whole session from left to right, while having some spans pre-encoded by non-causal encoders.

We use Transformer to build the models. Given an input sequence, we first pack their vector representations together. Then we feed the vectors into a multi-layer Transformer, which encodes the input to contextualized representations. In each Transformer block, there is a multi-head self-attention layer and a feed-forward network layer that are used to aggregate the hidden states of the previous layer. Moreover, attention masks are used to control the context access. We use a triangular matrix as the attention mask for the universal task layer, so that it processes the input from left to right. For the bidirectional encoder, we allow all the tokens to access each other. After obtaining the output vectors of the universal task layer, we use a softmax classifier to predict over the vocabulary. The weight matrix is shared with the input token embeddings.

As shown in Figure 2, there is a connector layer between the universal task layer and various bidirectional encoders. The connectors project vector representations of bidirectional encoders before feeding them into the general-purpose interface. Moreover, the connectors are used to match the output dimensions of foundation models with the universal task layer. We empirically find that both linear projection and feed-forward network work well in our experiments.

3 Proposed Objective: Semi-Causal Language Modeling

In order to pretrain MetaLM, we introduce the semi-causal language modeling objective. As shown in Figure 2, our pretraining task autoregressively generates the tokens of a sequence, while some spans are represented by bidirectional encoders.

Given an input sequence $\bm{x}=x_{1},x_{2},...,x_{n}$, we assume there are $k$ non-causal spans denoted as $\{\bm{x}_{s_{1}}^{e_{1}},...,\bm{x}_{s_{k}}^{e_{k}}\}$, where $\bm{x}_{s_{i}}^{e_{i}}=x_{s_{i}},...,x_{e_{i}-1}$. For each non-causal span $\bm{x}_{s_{i}}^{e_{i}}$, we use a bidirectional encoder to obtain its vector representations $\bm{h}(\bm{x}_{s_{i}}^{e_{i}})$. The choose of bidirectional encoders is dependent on the modality of the non-causal span.

Then the semi-causal language modeling objective is formulated as:

where $e_{0}=1$, $s_{(k+1)}=n$, and $\{\bm{h}(\bm{x}_{s_{j}}^{e_{j}})\}_{j<i}=\{\bm{h}(\bm{x}_{s_{1}}^{e_{1}}),\cdots,\bm{h}(\bm{x}_{s_{(i-1)}}^{e_{(i-1)}})\}$. Notice that the next token of each non-causal span is generated at the last position of the span. Typically the number of non-causal spans and their positions are randomly sampled. The spans do not have overlaps with each other.

By leveraging the proposed objective, we jointly pretrain the general-purpose interface and the underlying foundational models, and seamlessly connect them together. We pretrain MetaLM for both the language-only (Section 3) and vision-language (Section 4) settings.

4 Capabilities on Downstream Tasks

MetaLM can adapt to a new task by conditioning on natural language instructions or several input-output pairs (i.e., demonstrations), without updating any parameter. We first describe the usage of $k$-shot learning. For each demonstration input, we conduct bidirectional encoding. Then we feed the encoded vectors and the label into the general-purpose interface. By conditioning on the given demonstrations, MetaLM predicts the target output of unseen examples. For zero-shot generalization, there is only the test input, typically with prompts used to describe the task. We feed the example with the task instruction into bidirectional encoders. The target output is generated by the universal task layer.

Finetuning is especially helpful when many annotated examples of the downstream task are available. We unify various tasks to the open-ended generation format, i.e., targets are transformed to free texts. During finetuning, MetaLM learns to generate the target output, conditioning on the bidirectionally encoded input. Compared with causal language models, MetaLM inherits the excellent finetuning capability of bidirectional encoders.

A typical usage is that we first finetune the model on a large amount of data, and then use in-context learning to customize the finetuned model. So we can easily transfer the knowledge of labeled data to new tasks. As we subsume the advantages of both causal and non-causal modeling, MetaLM unlocks the combinations of the capabilities, i.e., good finetuning performance of non-causal modeling, and in-context learning of causal modeling.

MetaLM supports multi-turn interactions between users and pretrained models. For each turn, non-causal modules encode user inputs, which accepts multimodal contents by using the corresponding pretrained encoders. The output responses are generated by the general-purpose interface. By conditioning on the history conversations, MetaLM naturally works as a conversational interface. Moreover, the conversation can include multiple modalities instead of plain texts.

Experiments on Language-Only Tasks

We first conduct experiments on language-only datasets to demonstrate the versatility and effectiveness of MetaLM. Here the non-causal encoder is a pretrained language foundation model that docks with the universal task layer. The intriguing capabilities emerge through pretraining, which enables the general-purpose interface to transfer across tasks and scenarios.

We elaborate on language-only evaluation settings in Table 1. We demonstrate the capabilities of MetaLM, including multitask finetuning (Section 3.3), single-task finetuning (Section 3.4), instruction tuning (Section 3.5), and in-context learning (Section 3.6). The capabilities are task-agnostic and broadly applicable to understanding, generation, and interaction, which facilitates skill adaptation and communication with users. Moreover, the evaluation settings of multitask finetuning and instruction tuning are seamlessly built upon the capability combination of finetuning and in-context learning. In addition, because the tasks are unified in the free-text format, we can handle diverse downstream tasks using the same interface.

Figure 4 illustrates how to apply our model to different scenarios. Generally, the input examples and instructions are fed to the non-causal language encoder, and the target outputs are produced from the universal task layer. Moreover, the predictions are generated in a generative manner, which is open-ended.

2 Pretraining Setup

We use sinusoidal position embeddings for the language model. The number of layers is $L=24$, each layer consists of $A=32$ attention heads and the hidden dimension is $H=2048$. The number of parameters is about 1.3B. For the non-causal part, we use encoder-only Transformers, where $A=16$, $H=1024$, $L=24$. We utilize the learnable position embedding and relative position bias for the non-causal model. The number of parameters is about 366M. We use DeepNorm for Transformers. The connector module is a linear projection layer in our implementation.

The maximum input lengths for non-causal and semi-causal models are 512 and 2048, respectively. We randomly sample random spans whose lengths are between 64 and 128, and feed them to the non-causal part. The total length of non-causal spans is 25% of the original sequence length. The spans do not cross document boundaries. We pretrain the semi-causal language model from scratch. The non-causal module is initialized from a pretrained bidirectional encoder, using the replaced token detection task . During pretraining, we freeze all parameters of the non-causal encoder except the last two layers. We pretrain MetaLM for 300k steps with a batch size of 1024 and use Adam for optimization. We disable dropout of the semi-causal model and set the dropout rate of the non-causal model to 0.1. We use a learning rate of 6e-4 with warm-up. Please refer to Appendix A.1 for more pretraining details.

We pretrain the model on Pile , which is a massive English text dataset constructed from diverse data sources and targeted at training large-scale language models. We exclude data splits of GitHub, arXiv, and PubMed Central. Please refer to Appendix B.1 for detailed descriptions about Pile. The pretraining data is tokenized by SentencePiece . We construct the input in the “full-sentence” format , i.e., each input sequence is packed with full sentences sampled contiguously from one or more documents. We additionally introduce three special tokens for input construction: indicates the start of a sequence, indicates the end of a paragraph and indicates the end of a document.

3 Multitask Finetuning

We first evaluate MetaLM under the multitask finetuning setting. To be specific, we unify a wide range of tasks in an open-ended generation manner, so that they can be processed by the universal task layer without any task-specific architecture. Figure 4(a) shows an example of how MetaLM handles multitask finetuning. During finetuning, we randomly sample training examples and feed the inputs into the bidirectional language encoder. The finetuning objective is to maximize the likelihood of the correct labels generated from the interface.

We conduct experiments on a mixture of 34 NLP datasets (refer to Appendix B.2 for more details) grouped into ten task clusters, including both language understanding tasks and generation tasks:

Natural Language Inference: ANLI (R1-R3), CB, MNLI, QNLI, RTE, SNLI, WNLI

Sentiment Classification: IMDB, SST-2, Sentiment140, Yelp

Paraphrase Detection: QQP, MRPC, Paws Wiki

Coreference Resolution: DPR, Winogrande, WSC

Commonsense Reasoning: HellaSwag, PiQA, COPA

Reading Comprehension: DROP, SQuADv1, SQuADv2, OBQA, BoolQ

MetaLM is finetuned on a mixture of all the mentioned datasets. We limit the maximum number of training examples in each dataset to 30k. We follow the prompts used in . If the dataset is a multi-choice task, all possible options are provided in the template. For instance, the input format of an example from a sentiment classification dataset is “ Would the following phrase be considered positive or negative? [text] OPTIONS: Positive Negative TARGET:”. The model determines the sentiment by generating Positive or Negative.

We finetune MetaLM for 20k steps with a batch size of 256. The total length of input and answer tokens is restricted to 2048. Following , we pack multiple training examples into one sequence to make computation batch-friendly. The learning rate is set to 1e-4. For more details, please refer to Appendix A.2.

For multi-choice tasks, we report the exact match score without decoding constraints. For SQuAD, DROP, and closed-book QA datasets, we report the F1 score with greedy decoding. When evaluating on the struct2text and summarization clusters, we use beam search with a beam size of 4 and a length penalty of $\alpha=0.6$. We report ROUGE scores for the above two clusters.

3.2 Results

Table 2 compares the multitask finetuning results of MetaLM and GPT. The GPT baseline follows the same configuration and training corpus for a fair comparison. Each result represents the average score of all datasets of one task cluster. The full results of all task clusters are reported in Appendix C. We also illustrate the score differences between MetaLM and GPT for all datasets in Figure 5.

We observe that MetaLM consistently surpasses GPT by a large margin on almost all the task clusters. The results indicate that our method inherits the performant finetuning ability from the non-causal encoder. Particularly, MetaLM performs much better than GPT on NLU tasks. It partially confirms that non-causal modeling is conducive to finetuning . For more challenging tasks, such as natural language inference, and reading comprehension, the improvement of MetaLM is very prominent (14.1% and 9.6%). Furthermore, we find that finetuning of GPT brings relatively small gains on commonsense reasoning tasks, whose results are comparable to zero-shot generalization. By contrast, finetuning of MetaLM obtains decent gains over zero-shot numbers. With regard to language generation, MetaLM consistently outperforms GPT except on struct-to-text datasets. For closed-book question answering and text summarization, MetaLM achieves better performance than GPT too, benefiting from the non-causal modeling of input text.

4 Single-Task Finetuning

We explore the finetuning capability of MetaLM under data-rich settings. We design a new finetuning paradigm for MetaLM. For each downstream task, we only update the parameters of the non-causal encoder while keeping the language model frozen. We demonstrate that the proposed strategy achieves excellent performance, and preserves the general-purpose interface’s capabilities of in-context learning and open-endedness.

We conduct single-task finetuning on the natural language inference dataset MNLI . We use the template “ Premise:[*] Hypothesis:[*] Label:”. The task is to determine whether a hypothesis is true, false or undetermined given a premise. The corresponding labels are “entailment”, “contradiction” and “neutral”, respectively. During finetuning, we freeze the general-purpose interface and only update the non-causal encoder and the connector. In contrast, all parameters are updated for the GPT baseline. We finetune both MetaLM and GPT for three epochs with a learning rate of 5e-5 and a batch size of 32.

4.2 Results

Table 3 reports single-task finetuning accuracy. MNLI-m and -mm represent the matched and the mismatched validation sets respectively. Each score is the average of three runs with different random seeds. Compared with GPT, MetaLM improves the accuracy of MNLI by 3.4 absolute points, despite updating much fewer parameters. In addition to Section 3.3, the results show that bidirectional encoders benefit finetuning performance . Furthermore, we also present three strong baselines derived from finetuning bidirectional language encoders, including BERT , RoBERTa and ELECTRA . All these three models are in large size. Results show that MetaLM achieves comparable or better performance than the bidirectional encoders.

5 Instruction-Tuned Zero-Shot Generalization

We investigate instruction tuning for MetaLM, which finetunes the model on a variety of tasks with instructions. After finetuning, we evaluate the performance of instruction following and zero-shot generalization for the models. Because our goal is to investigate the zero-shot generalization on held-out tasks. Therefore, when evaluating on a specific dataset, all datasets in the same category (i.e., task cluster) are not seen during the training stage. For example, if we evaluate on the classification dataset SST-2, the entire cluster of sentiment analysis is excluded during instruction tuning.

We follow the evaluation pipeline proposed in FLAN . We conduct instruction tuning with MetaLM and GPT on the same dataset mixture described in Section 3.3 except for the summarization cluster. For each dataset, we use ten different templates manually composed by FLAN and randomly apply one of them for every example. As mentioned in , there are some templates that “turned the task around” to increase learning diversity, e.g., for sentiment classification, the model is prompted to generate a movie review based on the given sentiment label “Positive”.

Most finetuning configurations are the same as in Section 3.3.1. We experiment on four task clusters, including natural language inference, sentiment classification, paraphrase detection, and reading comprehension. Following the evaluation protocol of , the paraphrase cluster is dropped when evaluating on inference cluster and vice-versa. We finetune MetaLM and GPT for 30k steps with a batch size of 512. The learning rate is set to 1e-4. The sequence length for each example is limited to 1024. We also use the data packing strategy as in Section 3.3 to improve efficiency. The detailed hyper-parameters is provided in Appendix A.2.

5.2 Results

Table 4 reports the full results of instruction tuning on four task clusters. For each dataset, we use five different templates for evaluation, and present both the average and the best score. We observe that MetaLM achieves large improvements over the GPT baseline, which indicates the effectiveness of semi-causal language modeling. Considering the natural language inference cluster, GPT fails to obtain reasonable zero-shot results on difficult datasets (such as ANLI and WNLI), while MetaLM consistently performs well on various datasets. We notice similar trends on the other task clusters, i.e., sentiment, paraphrase, and reading comprehension. In addition to the average results, MetaLM outperforms the GPT baseline in terms of the best performance.

The setting of instruction tuning requires the capabilities of both finetuning and zero-shot generalization. Experimental results indicate that our method combines the best of causal and non-causal language models. MetaLM not only achieves favorable finetuning performance because of bidirectional encoders, but also retains the causal language model’s intriguing capability of zero-shot generalization.

6 In-Context Learning

We compare the performance of in-context learning between MetaLM and GPT. Conditioned on the task instruction and several input-label pairs, language models are repurposed towards the desired downstream task, following the input pattern while without updating parameters. As illustrated in Figure 4(d), the demonstrations consist of two parts, the example input is passed through the non-causal encoder and the label token uses original embeddings. Then the target label of the test input is generated by the universal task layer.

We conduct experiments under zero-shot, one-shot, and four-shot settings. We follow the evaluation protocol of GPT-3 . We evaluate each test example by randomly sampling examples from the training set as demonstrations. The Winograd only has the test set, so we sample demonstrations directly from it. Under few-shot settings, all examples are delimited by the separator token .

We evaluate MetaLM and the GPT baseline on nine tasks, including cloze and completion tasks (i.e, StoryCloze, HellaSwag), Winograd-style tasks (i.e, Winograd, Winogrande), commonsense reasoning (i.e, ARC-easy, ARC-challenge, PIQA), and two datasets BoolQ and Copa from the SuperGLUE benchmark . The detailed descriptions of these datasets are provided in Appendix B.3.

6.2 Results

Table 5 reports accuracy results of in-context learning. Compared with GPT, MetaLM achieves better or comparable results. For Winograd and completion tasks (i.e, StoryCloze, and HellaSwag), the performance of MetaLM has consistent improvements over GPT. Considering the average result over these datasets, MetaLM is better in both zero-shot ($k=0$) and few-shot ($k=1,4$) settings. The findings indicate that MetaLM inherits the excellent in-context learning ability, and the contextualized representations of non-causal encoders tend to help the model to generalize better.

Experiments on Vision-Language Tasks

We conduct experiments under the vision-language setting. The underlying non-causal encoder is a pretrained vision-language foundation model, which docks with the general-purpose interface. The pretraining task is similar to the language-only setting, despite the use of image-text pairs. Specifically, given an image-text pair, the image tokens are prepended to the text tokens. As shown in Figure 2, the non-causal encoder produces bidirectional fused representations of the image and a text prefix of random length. The causal decoder is pretrained to autoregressively predict the remaining tokens conditioning on the bidirectional fused representations. Text-only data is also leveraged and follows the same preparation protocol. We jointly pretrain on both image-text data and text-only data during the vision-language MetaLM pretraining.

Table 6 summarizes what capabilities we would like to evaluate and the corresponding vision-language datasets. We conduct experiments on zero-shot generalization in Section 4.3, in-context learning in Section 4.4, and finetuning in Section 4.5. The tasks can be grouped into several categories, i.e., visual question answering, visual reasoning, image captioning, and explanation generation. The evaluation across nine datasets covers both understanding and generation.

Figure 6 illustrates how we evaluate MetaLM in different settings. The input image and prompts are fed to a vision-language encoder, while the target output is generated by the language model. All the tasks are formulated in an open-ended generative manner.

2 Pretraining Setup

We use a 12-layer non-causal vision-language encoder and a 24-layer language model. The universal task layer follows the same network architectures and configurations of GPT-2 . The hidden size is 1024, and there are 16 attention heads. We employ sinusoidal position embeddings . The number of parameters is 353M. For the non-causal encoder, we use a vision-language model pretrained as in VLMo . The number of parameters is 192M. We use 224x224 resolution during pretraining for images. The connector is a three-layer feed-forward network. More details about hyper-parameters can be found in Appendix D.1.

We pretrain MetaLM for 350k steps with 256 batch size. We use AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.98$. The learning rate is 1e-4 and weight decay is 0.01. We use linear decay and apply warm-up at the first 2,500 steps. The dropout rate is set to 0.1.

We pretrain MetaLM using image-text pairs and text documents. For image-text pairs, our pretraining data consists of Conceptual Captions , Visual Genome , COCO Caption , and SBU Caption datasets. Together, there are about 4M images and 10M image-text pairs. For text documents, following and , we use the OpenWebText corpus, which is an open-source recreation of the Reddit web text, as the pretraining data.

3 Zero-Shot Generalization

We evaluate the zero-shot generalization capability of MetaLM under vision-language settings. Specifically, we conduct experiments on two tasks, including image captioning, and visual question answering. For image captioning, only an input image is given, and the goal is to generate its description. For visual question answering, a question is asked for the given image, and the model needs to predict the correct answers.

We apply greedy decoding during inference. The input images are resized to 224x224. We describe the datasets and specific setups of two tasks as follows:

We evaluate zero-shot caption generation on MS COCO Caption , NoCaps , and Flickr30k . We evaluate on the test set of COCO Karpathy split , which re-partitions the train2014 and val2014 images into 113,287, 5,000, and 5,000 for train, validation, and test. For NoCaps and Flickr30k, following , we evaluate on their validation set and test set, respectively. We use BLEU , CIDEr , METEOR , and SPICE as caption generation metrics. We utilize COCOEvalCaphttps://github.com/tylin/coco-caption to compute scores. We prompt MetaLM with “Summarize this image:” for all zero-shot caption generation experiments.

Following , we evaluate the zero-shot performance on VQAv2 validation set and OK-VQA test set. VQA score is calculated using normalization rules of the VQAv2 evaluation code.https://github.com/GT-Vision-Lab/VQA Different from classification over a predefined set of candidate answers, MetaLM predicts answers in an open-ended generation manner. We prompt MetaLM with the template“question: question text answer:” for all visual question answering experiments.

3.2 Results

Table 7 and Table 8 show the zero-shot captioning results on COCO Karpathy test split, NoCaps validation set, and Flickr30k test set. MetaLM outperforms recent strong methods on three image captioning datasets. To be specific, the compared model FewVLM leverages different prompts for image captioning, and we report its best results. By contrast, we use the same prompt “Summarize this image:” for comparisons in all the experiments. Our model robustly follows the instruction to produce readable captions in the zero-shot manner.

Table 9 reports the results of zero-shot visual question answering on VQAv2 and OK-VQA. On both datasets, MetaLM achieves better zero-shot results than Frozen and VLKD , even though Frozen has significantly more parameters. In addition, the OK-VQA dataset is designed for visual question answering that is supposed to require external knowledge. For example, the input image is a train, and the asked question is “When is it invented?”. The reasonable performance on OK-VQA indicates that the language model of MetaLM tends to serve as a knowledge source. Once object information is perceived by the vision encoder, the universal task layer generates the answer as language modeling.

The experimental results across five datasets show that MetaLM has the capabilities of zero-shot generalization and open-ended generation. We can use prompts to re-purpose the pretrained vision-language model to image captioning and visual question answering.

4 In-Context Learning

We evaluate the capability of in-context learning on visual question answering. We conduct $k$-shot learning, where $k$ demonstrations are used to guide the prediction of new examples without finetuning the parameters.

Following , we carry out few-shot experiments on the VQAv2 validation set and OK-VQA test set. We randomly sample up to four full examples from the training set for each test instance. The predicted answers are evaluated against the ground-truth answers following the normalization rules from the VQAv2 evaluation code. We use an image resolution of 224x224 during inference.

As shown in Figure 6(b), we put several examples before the test input and directly obtain the prediction from the universal task layer. Specifically, a full example is denoted as $e=[i,q,a]$, where i, q, a denote image, question, and answer, respectively. Similarly, a test input t is denoted as $t=[i,q]$. For $k$-shot in-context learning, the whole input sequence is $e_{1},...,e_{k},t$. Moreover, we use “Question: [question text] Answer:” as the prompt to instruct MetaLM. Then MetaLM uses greedy decoding to generate answers.

4.2 Results

Table 10 reports the in-context learning results on the visual question answering datasets VQAv2 and OK-VQA. The results show that adding in-context demonstrations improves the performance over zero-shot generalization as shown in Table 9. Besides, adding more examples brings larger improvements to both datasets. Compared with Frozen , MetaLM obtains better performance despite the use of relatively small model size. We find that MetaLM can conduct in-context learning on visual question answering without modifying the underlying vision-language model. Although the non-causal encoder only sees one example each time, the language model successfully adapts the model according to the $k$ demonstrations. In addition, with the help of the universal task layer, we can augment the existing foundation models with the general capability of in-context learning.

5 Finetuning on Downstream Tasks

We finetune the pretrained MetaLM on a wide range of vision-language tasks, including image captioning , visual question answering , visual reasoning , and explainable visual reasoning . We compare the finetuned MetaLM with both the strong discriminative models and recent generative models.

For all tasks, we use the resolution of 384x384 during finetuning. We also apply RandAugment for image augmentation. We keep the learning rate 1e-5 fixed for all datasets. More detailed hyper-parameters can be found at Appendix D.2. We describe the setups of various tasks as follows.

We evaluate on VQAv2 , VQA Karpathy split , and OK-VQA . For VQAv2, models are finetuned on the training and validation sets. We report the VQA score on the test-dev and test-std sets. For VQA Karpathy split, models are finetuned on the training and validation sets. We report the VQA score on the in-domain and out-domain test set. We finetune MetaLM for 140k steps for both the above two datasets. For OK-VQA, models are finetuned on the training set. We report the normalized VQA score on the test set. We finetune MetaLM with 10k steps. We apply a “Question: [question text] Answer: [answer text]” prompt for generative finetuning.

We evaluate on the NLVR2 dataset . The example in NLVR2 consists of two images and one sentence, where the sentence describes the relations between the images. Following previous work , we re-split the data into two individual image-text pairs and get their representations respectively. Then we leverage the concatenation of representations to generate the yes or no predictions. We apply “it is [label]” for generative finetuning. We finetune MetaLM for 5 epochs.

We evaluate on the COCO caption dataset with Karpathy split . Following , we report BLEU-4, CIDEr, METEOR, and SPICE as the evaluation metrics. All reported results are from cross-entropy finetuning without reinforced CIDEr optimization . Object tags are not used during finetuning. We apply a “caption: [caption text]” prompt for generative finetuning and finetune MetaLM for 100k steps on the training split.

We evaluate on the E-SNLI-VE dataset , which requires the models to predict the entailment labels between an image-text pair and simultaneously generate explanations for the prediction. We finetune MetaLM for 7 epochs. This task is naturally compatible with the language generation manner. We apply a “it is [entailment label] because [explanation].” prompt for generative finetuning.

5.2 Results: Visual Question Answering and Visual Reasoning

Table 11 reports the finetuning results on VQAv2, VQA Karpathy, and NLVR2. The finetuning performance is strong across the datasets. More importantly, MetaLM not only outperforms previous models with generative prediction, but also achieves competitive or better results compared with discriminative vision-language models. The property is favorable as the nature of some tasks is generative. For example, visual question answering needs open-ended predictions, rather than restricting the output space. The advantages of open-endedness are shown on the out-domain set of the VQA Karpathy-test. The top answers of the out-domain set are not in the most common 3,129 VQA answers. As the discriminative models can only make predictions that appear in the training set, it is difficult to generalize to out-domain examples. Among all the models, MetaLM achieves the best out-domain results. In comparison, although previous generative models get better results on the out-domain set, they usually underperform on other datasets. By contrast, MetaLM consistently achieves competitive results.

As shown in Table 12, we report the finetuning results on OK-VQA . Different from VQAv2, the dataset requires models to draw upon external knowledge to answer questions. Previous methods typically leverage a knowledge base to filter candidate answers. In contrast, language models have acquired rich world knowledge during pretraining. MetaLM grants the flexibility of leveraging such knowledge from the causal language model. As a result, MetaLM obtains significant improvements on this task without relying on additional knowledge bases.

Table 13 reports the finetuning results on E-SNLI-VE entailment label prediction. MetaLM is trained to jointly generate the entailment label and explanation with the “it is [entailment label] because [explanation]” prompt. MetaLM achieves the best accuracy compared with previous methods. Moreover, an important advantage of the generative model is that MetaLM can leverage explanations to improve the performance of entailment label prediction. It indicates that the explanation is of help to entailment classification. The results demonstrate that MetaLM can be used to facilitate the interactions between users and foundation models. In other words, we can use natural language to guide model finetuning via the general-purpose interface.

The competitive results across the above datasets demonstrate that the bidirectional modeling benefits finetuning in MetaLM. So we can have good performance of finetuning and open-ended prediction at the same time.

5.3 Results: Visually Grounded Language Generation

Table 14 reports the finetuning results of caption generation on COCO Karpathy test split. We directly compare with the results without CIDEr optimization for fair comparisons. The results show that MetaLM obtains substantial improvements over other models.

Table 15 shows the explanation generation results on E-SNLI-VE. We jointly generate entailment labels and explanations. MetaLM outperforms previous strong models on most metrics. Together with the label accuracy results on the same dataset in Table 13, our model achieves good performance for both understanding and explanation generation. In contrast, the method of obtains competitive performance for explanation generation, while getting inferior accuracy for entailment classification.

The results of visually grounded language generation show that our architecture is general enough to be applied to various sequence-to-sequence learning problems. MetaLM can achieve good performance via finetuning for vision-language generation tasks.

Related Work

Large-scale language model pretraining has achieved strong performance across various downstream tasks and aroused extensive research interest. The difference between the models mainly lies in the pretraining objective and model architecture. GPT pretrains causal language models with decoder-only Transformers, demonstrating intriguing properties of few-shot and in-context learning. Recent efforts focus on scaling up in terms of data and model size. In order to implement bidirectional encoding, Devlin et al. propose the masked language modeling objective. Clark et al. introduce the replaced token detection task to improve pretraining efficiency. Furthermore, some efforts investigate frameworks that can handle both natural language understanding and generation tasks. T5 introduces an encoder-decoder framework that converts all tasks into a text-to-text format. BART is a sequence-to-sequence model pretrained by reconstructing the original text from corrupted documents. UniLM presents to jointly optimize unidirectional, bidirectional and sequence-to-sequence language modeling objectives controlled by different self-attention masks. Wang et al. , Tay et al. , and Artetxe et al. study the effects of different pretraining objectives and architectures on downstream generalization. Specifically, causal language models are good at zero-shot or in-context learning, while non-causal models perform better for finetuning. In our work, we combine the best of both worlds by introducing semi-causal language modeling. So we can obtain decent finetuning performance and benefit from the capability of in-context learning. Moreover, the unification enables us to build a general-purpose interface to various foundation models.

2 General-Purpose Modeling

Some efforts investigate the general-purpose model that supports multiple tasks, transformations, and modalities in a shared module. MT-DNN trains on many tasks through multitask learning. Specific to language-only general-purpose, UniLM and T5 unify understanding and generation ability in a single model. Moreover, language models are finetuned to follow instructions , i.e., aligning language models with user intentions to implement the general-purpose capability. There are some work that support not only multitask but also multimodality. Jaegle et al. introduce Perceiver IO, a general architecture across multiple domains including language/visual understanding, multimodal and symbolic representations for games. Baevski et al. propose a unified learning framework for different modalities but still use modality specific encoders. Tsimpoukelli et al. demonstrate that the in-context learning ability of frozen language models can be transferred to a vision-language setting. Alayrac et al. also implement general-purpose understanding of image, video, and text by a large frozen language model. Reed et al. build a generalist agent that works as a multi-modal, multi-task, multi-embodiment generalist policy.

Conclusion

We present MetaLM, a general-purpose interface to foundation models across tasks and modalities. MetaLM consists of a causal decoder as the universal task layer, and multiple pretrained non-causal encoders mounted to it. We pretrain MetaLM with a new objective called semi-causal language modeling. Experimental results show that MetaLM exhibits strong finetuning and in-context learning performance across a wide range of language-only and vision-language tasks.

In the future, we would like to scale up the model size. Moreover, we are interested in extending MetaLM to multilingual settings, and handling more modalities (including language, vision, audio, and multimodality) simultaneously. Another strand of work is to extend the universal task layer to vision tasks, such as object detection, and semantic segmentation. We will also investigate parameter-efficient finetuning with MetaLM.

References

Appendix A Hyperparameters of Language-Only Experiments

We provide the detailed pretraining hyperparameter settings of language-only MetaLM. Model hyperparameters are shown in Table A.1 and optimization hyperparamters are shown in Table A.1.

Hyperparameters of non-causal and semi-causal models for language-only pretraining.

Optimization hyperparameters for language-only pretraining.

A.2 Multitask Finetuning and Instruction Tuning

We provide the detailed settings of language-only multitask finetuning and instruction tuning with MetaLM in Table 16.

Appendix B Datasets Used for Language-Only Experiments

Language-only MetaLM is pretrained on Pile , which is an 800 GB English text corpus combining 22 diverse sources. We exclude data sources of GitHub, arXiv and PubMed Central from the original Pile. Thus the pretraining corpus we used is composed of 19 sources, divided into the following five categories:

Academic: FreeLaw, USPTO Backgrounds, PhilPapers, NIH Exporter, PubMed Abstracts

Internet: Pile-CC, OpenWebText2, StackExchange, Wikipedia (English)

Prose: BookCorpus2, Books3, Gutenberg [80, PG-19]

Dialogue: OpenSubtitles , Youtube Subtitles, EuroParl , Hacker News, Ubuntu IRC

Miscellaneous: Enron Emails , DM Mathematics

B.2 Multitask Finetuning and Instruction Tuning

We list the datasets we used for language-only multitask finetuning and instruction tuning.

Natural Language Inference is to determine whether a hypothesis is true (entailment), false (contradiction) or undetermined (neutral) given a premise. We use the following datasets: ANLI , CB , MNLI , QNLI , RTE , SNLI and WNLI .

Sentiment Classification is to determine the emotional tone of a piece of text, whether it is positive or negative: IMDB , SST-2 , Sentiment140 , Yelp .

Paraphrase Detection is to detect the semantic similarity of two sentences: QQP , MRPC , Paws Wiki .

Coreference Resolution is to determine if two expressions refer to the same entity in a text: DPR , Winogrande , WSC .

Commonsense Reasoning evaluates the ability to perform physical or social commonsense: HellaSwag , PiQA , COPA .

Reading Comprehension is to answer some questions conditioned on a given passage: DROP , SQuADv1 , SQuADv2 , OBQA , BoolQ .

Miscellaneous consists of some additional datasets: CoLA , WiC , TREC .

Closed-Book QA is to answer a question without external knowledge: ARC-easy , NQ .

Struct to Text is to construct a natural language description for some structured data: CommonGen , E2ENLG .

Summarization is to generate a summary of a given passage: AESLC , SamSum , XSum .

Furthermore, we utilize the hand-crafted templates from FLAN , which composes ten templates for each dataset. For multitask finetuning, we apply only the first template for each dataset. For instruction tuning, we apply all the ten templates.

B.3 In-Context Learning

We conduct experiments of in-context learning on four categories:

Cloze and completion tasks: StoryCloze , HellaSwag

Winograd-style tasks: Winograd , Winogrande

Commonsense reasoning: ARC-easy/ARC-challenge , PIQA

Two datasets from SuperGLUE benchmark : BoolQ , Copa

Appendix C Detailed Results of Multitask Finetuning in Section 3.3

We list the full results of language-only multitask finetuning for all task clusters in our experiments. Results of natural language inference are shown in Table 17. Results of sentiment classification are shown in Table 18. Results of paraphrase detection are shown in Table 19. Results of reading comprehension are shown in Table 20. Results of coreference resolution are shown in Table 21. Results of miscellaneous cluster are shown in Table 22. Results of commonsense reasoning are shown in Table 23. Results of struct to text are shown in Table 24. Results of closed-book QA are shown in Table 25. Results of text summarization are shown in Table 26.

Appendix D Hyperparameters of Vision-Language Experiments

We report the detailed pretraining hyperparameter settings of the vision-language MetaLM in Table D.1 and report the optimization hyperparameters in Table D.1.

Hyperparameters of non-causal and semi-causal models for vision-language pretraining.

Optimization hyperparameters for vision-language pretraining.

D.2 Hyperparameters in Vision-Language Finetuning

We report the finetuning settings along with the prompts in Table 27. The vision-language MetaLM applies a 384x384 image size and greedy decoding for all finetuning tasks.