Instruction Tuning with GPT-4

Introduction

Large Language Models (LLMs) have shown impressive generalization capabilities such as in-context-learning (Brown et al. 2020) and chain-of-thoughts reasoning (Wei et al. 2022). To enable LLMs to follow natural language instructions and complete real-world tasks, researchers have been exploring methods of instruction-tuning of LLMs. This is implemented by either finetuning the model on a wide range of tasks using human-annotated prompts and feedback (Ouyang et al. 2022), or supervised finetuning using public benchmarks and datasets augmented with manually or automatically generated instructions (Wang et al. 2022b). Among these methods, Self-Instruct tuning (Wang et al. 2022a) is a simple and effective method of aligning LLMs to human intent, by learning from instruction-following data generated by state-of-the-art instruction-tuned teacher LLMs. It turns out that the line of instruction-tuning research has produced effective means to improve the zero and few-shot generalization abilities of LLMs. The recent success of ChatGPT (OpenAI 2023a) and GPT-4 (OpenAI 2023b) offers tremendous opportunities to improve open-source LLMs using instruction-tuning. LLaMA (Touvron et al. 2023) is a series of open-sourced LLMs, which match the performance of proprietary LLMs such as GPT-3. To teach LLaMA to follow instructions, Self-Instruct tuning has been quickly adopted given its superior performance and low cost. For example, Stanford Alpaca (Taori et al. 2023) uses 52K instruction-following samples generated by GPT-3.5, while Vicuna (Vicuna 2023) uses around 700K instruction-following samples (70K conversions) shared user-ChatGPT (ShareGPT 2023).

To advance the state of the art of instruction-tuning for LLMs, we propose for the first time to use GPT-4 as a teacher for self-instruct tuning. Our paper makes the following contributions:

GPT-4 data. We release data generated by GPT-4, including the 52K instruction-following dataset in both English and Chinese, and the GPT-4-generated feedback data that rate the outputs of three instruction-tuned models.

Models & Evaluation. Based on the GPT-4-generated data, we have developed instruction-tuned LLaMA models and reward models. To evaluate the quality of instruction-tuned LLMs, we use three metrics evaluated on test samples (i.e., unseen instructions): human evaluation on three alignment criteria, automatic evaluation using GPT-4 feedback, and ROUGE-L on un-natural instructions (Honovich et al. 2022). Our empirical study validates the effectiveness of using GPT-4-generated data for LLM instruction-tuning, and suggests practical tips of building a general-purpose instruction-following agent powered by LLMs.

Dataset

We reuse 52K unique instructions in the instruction-following data collected in the Alpaca dataset (Taori et al. 2023). Each instruction describes the task the model should perform. We follow the same prompting strategy to consider cases with and without input, which is the optional context or input for the task. The output answers to the instruction instance using LLMs. In the Alpaca dataset, the output is generated using GPT-3.5 (text-davinci-003) but we instead consider GPT-4 (gpt-4) for data generation. Specifically, we generate the following four datasets with GPT-4:

English Instruction-Following Data: For the 52K instructions collected in Alpaca (Taori et al. 2023), one English GPT-4 answer is provided for each. The details are described in Algorithm 1. We leave it as future work to follow an iterative process to construct our own instruction set using GPT-4 and self-instruct (Wang et al. 2022a).

Chinese Instruction-Following Data: We use ChatGPT to translate the 52K instructions into Chinese and ask GPT-4 to answer them in Chinese. This allows us to build a Chinese instruction-following model based on LLaMA, and study cross-language generalization ability of instruction-tuning.

Comparison Data: We ask GPT-4 to rate its own response from 1 to 10. Furthermore, we ask GPT-4 to compare and rate the responses from the three models, including GPT-4, GPT-3.5 and OPT-IML (Iyer et al. 2022). This is used to train reward models.

Answers on Unnatural Instructions: The GPT-4 answers are decoded on the core dataset of 68K instruction-input-output triplets (Honovich et al. 2022). The subset is used to quantify the gap between GPT-4 and our instruction-tuned models at scale.

We compare the English output response sets of GPT-4 and GPT-3.5 in Figure 1. For each output, the root verb and the direct-object noun are extracted; The frequency over the unique verb-noun pairs are computed over each output set. The verb-noun pairs whose frequency are higher than 10 are displayed in Figure 1(a) and (b), and the most frequent 25 pairs of two sets are compared in Figure 1(c). The frequency distributions of the sequence length are compared in Figure 1(d). GPT-4 tends to generated longer sequences than GPT-3.5. The GPT-3.5 data in Alpaca exhibits an output distribution with a longer tail than our GPT-4-generated output distribution, probably because the Alpaca dataset involves an iterative data collection process to remove similar instruction instances at each iteration, which is absent in our current one-time data generation. Despite this simple process, the GPT-4 generated instruction-following data demonstrates more favorable alignment performance, as shown in experiments later.

Instruction-Tuning Language Models

We train two models using supervised finetuning using the LLaMA 7B checkpoint: $(i)$ LLaMA-GPT4 is trained on 52K English instruction-following data generated by GPT-4, which distribution is displayed in Figure 1. $(ii)$ LLaMA-GPT4-CN is trained on 52K Chinese instruction-following data from GPT-4. We follow the training schedule in (Taori et al. 2023) for fair comparisons. These models are used to study the data quality of GPT-4 and the cross-language generalization properties when instruction-tuning LLMs in one language.

2 Reward Models

Reinforcement Learning from Human Feedback (RLHF) aims to align the LLM behavior with human preferences in order to make it more useful. One key component of RLHF is reward modeling, where the problem is formulated as a regression task to predict a scalar reward given a prompt and a response (Askell et al. 2021; Ouyang et al. 2022). This approach typically requires large-scale comparison data, where two model responses on the same prompt are compared Ouyang et al. 2022. Existing open-source works such as Alpaca, Vicuna, and Dolly (Databricks 2023) do not involve RLHF due to the high cost of labeling comparison data. Meanwhile, recent studies show that GPT-4 is capable of identifying and fixing its own mistakes, and accurately judging the quality of responses(Peng et al. 2023; Bai et al. 2022; Madaan et al. 2023; Kim et al. 2023). Therefore, to facilitate research on RLHF, we have created comparison data using GPT-4, as described in Section 2.

To evaluate data quality, we train a reward model based on OPT 1.3B (Iyer et al. 2022) to rate different responses. For each instance of the comparison data involving one prompt $\boldsymbol{x}$ and $K$ responses, GPT-4 assigns a score $s\in$ for each response. There are $C_{2}^{K}$ unique pairs constructed from this instance, each pair is $(y_{l},y_{h})$, whose corresponding scores follow $s_{l}<s_{h}$. A reward model $r_{\boldsymbol{\theta}}$ parameterized by $\boldsymbol{\theta}$ is trained with the objective: $\min\log(\sigma(r_{\boldsymbol{\theta}}(x,y_{h})-r_{\boldsymbol{\theta}}(\boldsymbol{x},y_{l})))$, where $\sigma$ is the sigmoid function. The distribution of the comparison data is shown in Figure 2.

Experimental Results

It is known that LLM evaluation remains a significant challenge. Our goal is to evaluate self-instruct tuned models on GPT-4 data on unseen instructions, to study their ability to follow instructions for arbitrary tasks. Specifically, we use three established datasets in our study:

User-Oriented-Instructions-252 https://github.com/yizhongw/self-instruct/blob/main/human_eval/user_oriented_instructions.jsonl (Wang et al. 2022a) is a manually curated set involving 252 instructions, motivated by 71 user-oriented applications such as Grammarly, StackOverflow, Overleaf, rather than well-studied NLP tasks.

Vicuna-Instructions-80 https://github.com/lm-sys/FastChat/blob/main/fastchat/eval/table/question.jsonl (Vicuna 2023) is a dataset synthesized by gpt-4 with 80 challenging questions that baseline models find challenging. Beside generic instructions, there are 8 categories, including knowledge, math, Fermi, counterfactual, roleplay, generic, coding, writing, common-sense.

Unnatural Instructions https://github.com/orhonovich/unnatural-instructions (Honovich et al. 2022) is a dataset of 68,478 samples synthesized by text-davinci-002 using 3-shot in-context-learning from 15 manually-constructed examples.

2 Human Evaluation with Alignment Criteria

To evaluate the alignment quality of our instruction-tuned LLMs, we follow alignment criteria from Anthropic Askell et al. 2021: an assistant is aligned if it is helpful, honest, and harmless (HHH). These criteria are used to evaluate how well an AI system is aligned with human values.

Helpfulness: whether it helps humans achieve their goals. A model that can answer questions accurately is helpful.

Honesty: whether it provides true information, and expresses its uncertainty to avoid misleading human users when necessary. A model that provides false information is not honest.

Harmlessness: whether it does not cause harm to humans. A model that generates hate speech or promotes violence is not harmless.

Based on HHH alignment criteria, we used Amazon Mechanical Turk to perform human evaluation on the model generation results. Please find the interface in Appendix Section A.1. Following (Wang et al. 2022a; Taori et al. 2023), we consider 252 user-oriented instructions for evaluation. We display the human evaluation results in pie charts in Figure 3.

First, we compare the quality of generated responses from two instruction-tuned LLaMA models, which are fine-tuned on data generated by GPT-4 and GPT-3, respectively. Note that aligning LLaMA to GPT-3 corresponds to the Stanford Alpaca model. From Figure 3(a), we observe that ($i$) For the “Helpfulness” criterion, GPT-4 is the clear winner with 54.12% of the votes. GPT-3 only wins 19.74% of the time. ($ii$) For the “Honesty” and “Harmlessness” criteria, the largest portion of votes goes to the tie category, which is substantially higher than the winning categories but GPT-3 (Alpaca) is slightly superior.

Second, we compare GPT-4-instruction-tuned LLaMA models against the teacher model GPT-4 in Figure 3(b). The observations are quite consistent over the three criteria: GPT-4-instruction-tuned LLaMA performs similarly to the original GPT-4. We conclude that learning from GPT-4 generated data can lead to very comparable performance with the original GPT-4 on the unseen instructional tasks, which suggests a promising direction to developing state-of-the-art instruction-following LLMs.

3 Comparisons with SoTA using Automatic Evaluation

Following (Vicuna 2023), we employ GPT-4 to automatically evaluate the generated responses of different models on 80 unseen questions in (Vicuna 2023). We first collect answers from two chatbots, including LLaMA-GPT-4 (7B) and GPT-4, and use the release answers of other chatbots from (Vicuna 2023), including LLaMA (13B), Alpaca (13B), Vicuna (13B), Bard (Google 2023), and ChatGPT. For each evaluation, we ask GPT-4 to rate the response quality between two models with scores from 1 to 10. We compare all models against a strong competing model such as ChatGPT and GPT-4, respectively. The results are shown in Figure 4.

For LLaMA instruction-tuned with GPT-4, we provide two sets of decoding results: $(i)$ One response per question, which is considered the baseline decoding result. $(ii)$ Five responses per questions. For the latter, the reward model is used to rank the responses which are then grouped into five subsets ranked from top 1 to top 5. We compare the five ranked groups against the baseline, and show the relative scores in Figure 4 (a,b). The ChatGPT and GPT-4 evaluation is consistent with the orders suggested by our reward model, which demonstrate the value of the feedback data and effectiveness of the reward model.

We compare all the chatbots in Figure 4(c,d). Instruction tuning of LLaMA with GPT-4 often achieves higher performance than tuning with text-davinci-003 (i.e., Alpaca) and no tuning (i.e., LLaMA): The 7B LLaMA_GPT4 outperforms the 13B Alpaca and LLaMA. However, there is still a gap compared with large commercial chatbots such as GPT-4.

We further study the performance of all the chatbots in Chinese in Figure 5. We first translate English responses of chatbots into Chinese using GPT-4. We also translate English questions into Chinese to obtain answers with GPT-4. The comparisons against translated and generated Chinese responses from GPT-4 are shown in Figure 5 (a) and (b), respectively. There are two interesting observations: $(i)$ we find that the relative score metric of GPT-4 evaluation (Vicuna 2023) is quite consistent, both in terms of different opponent models (i.e., ChatGPT or GPT-4) and languages (i.e., English or Chinese). $(ii)$ For GPT-4 results alone, the translated responses show superior performance over the generated response in Chinese, probably because GPT-4 is trained in richer English corpus than Chinese, which leads to stronger English instruction-following ability. In Figure 5 (c), we show results for all models who are asked to answer in Chinese.

We compare LLaMA-GPT4 with GPT-4 and Alpaca unnatural instructions in Figure 6. In terms of the average ROUGE-L scores, Alpaca outperforms the other two models. We note that LLaMA-GPT4 and GPT4 is gradually performing better when the ground truth response length is increasing, eventually showing higher performance when the length is longer than 4. This means that they can better follow instructions when the scenarios are more creative. Across different subsets, LLaMA-GPT4 can closely follow the behavior of GPT-4. When the sequence length is short, both LLaMA-GPT4 and GPT-4 can generate responses that contains the simple ground truth answers, but add extra words to make the response more chat-like, which probably leads to lower ROUGE-L scores.

Related Work

Instruction tuning of LLMs is an increasingly popular research direction in NLP (Zhong et al. 2021; Ouyang et al. 2022; Wei et al. 2021). Existing works aim to improve the quality and scale of three factors in the development pipeline, including instruction-following data, foundation language models and evaluation benchmarks. Each group typically maintains its own pipeline. For example, scaling instruction-finetuned language models (Chung et al. 2022) is built on top of FLAN (Wei et al. 2021). PromptSource contains a growing collection of prompts (which is also called P3: Public Pool of Prompts) (Bach et al. 2022). T0 is a series of models trained on P3 via multitask prompted training (Sanh et al. 2021). Instruction-tuning of OPT models is considered in (Iyer et al. 2022), where a larger and more comprehensive benchmark OPT-IML Bench is employed, covering FLAN (Wei et al. 2021), Super-NaturalInstructions (Wang et al. 2022b), and UnifiedSKG (Xie et al. 2022).

Given the broad capabilities of LLMs exhibited by ChatGPT, open-source models have drawn a significant interest and promoted work towards open, general-purpose, text-based assistants that are aligned with human values. Early attempts on foundation LLMs include BLOOM (Scao et al. 2022), GPT-J (Wang & Komatsuzaki 2021), GPT-NEO (Black et al. 2021) OPT (Zhang et al. 2022) and LLaMA (Zhang et al. 2023). To align LLMs with chat-based assistance, Open-Assistant (LAION-AI 2023) is built on GPT-J, and Alpaca/Vicuna are built on LLaMA. Furthermore, OpenFlamingo (Awadalla et al. 2023) and LLaMA-Adapter (Zhang et al. 2023) connect LLaMA with image inputs, paving a way to build open-source multi-modal LLMs.

Conclusions

This paper demonstrates the effectiveness of instruction tuning using GPT-4. We release 52K English and Chinese instruction-following instances generated using GPT-4 as well as model checkpoints finetuned from LLaMA, We hope our empirical observations and resource will benefit the development of open-source and general-propose LLMs that can better align with human values to complete tasks.

This represents work in progress, and several directions can be explored: $(i)$ Data and model scale. The GPT-4 data size is 52K and the base LLaMA model size is 7B. Vicuna collects around 700K conversion turns (approximated from the multi-turn ShareGPT data), and uses the 13B LLaMA model. Therefore, it would be promising to continue collecting more GPT-4 instruction-following data, combine with ShareGPT data, and train larger LLaMA models for higher performance. $(ii)$ RLHF. The reward model is only used in the decoding stage, which suggests that comparison data is promising to provide useful feedback for LLM training. It is natural to continue to train LLMs with reward models, for example for reinforcement learning using machine-generated feedback.

We thank Guoyin Wang, Haotian Liu and Hao Cheng for valuable discussions and insightful experience sharing on instruction-tuning language models. We thank the LLaMA team for giving us access to their models.

References

Appendix A Implementation Details

We implemented the HHH alignment criteria (Askell et al. 2021), and used Amazon Mechanical Turk to evaluate the model generated responses, the interface screenshot is shown in Figure 7.