Instruction-tuned Language Models are Better Knowledge Learners
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, Srinivasan Iyer
Introduction
Large language models (LLMs) store vast amounts of factual knowledge in their parameters through large-scale pre-training, and this knowledge can be used to answer various questions such as “where is the world’s largest ice sheet located” Brown et al. (2020); OpenAI (2023); Chowdhery et al. (2022); Zhang et al. (2022); Touvron et al. (2023a, b); Gemini Team (2023). However, this factual knowledge is static, meaning that it can become outdated as the world evolves, or prove insufficient when LLMs are used in specialized or private domains.
To keep LLMs up-to-date, it is common to continue pre-training on new documents to store knowledge in parameters, which allows LLMs to effectively answer queries that require up-to-date information Jang et al. (2022). A widely held view is that the factual knowledge stored in parameters can be elicited through prompting Brown et al. (2020); Petroni et al. (2019); Roberts et al. (2020), and that instruction-tuning (also known as supervised fine-tuning or alignment) makes this elicitation more effective Sanh et al. (2022); Wei et al. (2022); Ouyang et al. (2022). In the first part of this paper (§ 4), we conduct extensive experiments using Llama-2 Touvron et al. (2023b) to answer the following question: to what extent can we augment the knowledge stored in modern LLMs by continued pre-training on new documents, either with or without subsequent instruction-tuning? We find that, as we train LLMs repeatedly over documents to the extent that perplexity is minimized to one, the percentage of questions regarding those documents that LLMs answer correctly increases consistently to 27.6%. Subsequent instruction-tuning further improves it to 30.3%, confirming that this widely used practice is useful to elicit more knowledge from LLMs.This capacity might be underestimated by previous works due to using relatively small LMs or randomly initialized transformers, or lack of exhaustive training or instruction-tuning Wang et al. (2021); Hu et al. (2023); Zhu and Li (2023a). However, the amount of elicited knowledge is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the “perplexity curse”.Inspired by the “reversal curse” of Berglund et al. (2023).
In the second part of the paper (§ 5), we study methods to mitigate the perplexity curse by making LLMs more adept at absorbing knowledge from documents. Zhu and Li (2023a) presented an intriguing finding that training a randomly initialized transformer from scratch on a mix of biographies and related questions resulted in strong generalization to new questions. However, understanding the reasons behind this finding and exploring ways to practically apply it for absorbing knowledge from new documents requires further investigation. We found that question-answer (QA) pairs are generally straightforward and easily digestible, while documents tend to be more complex and cluttered, often weaving many factual statements together in a more intricate manner. Therefore, we hypothesize that it is beneficial to deliberately expose LLMs to QA data before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. We refer to this as pre-instruction-tuning (PIT) and conduct comprehensive experiments to benchmark different variations of this method. As shown in Fig. 1, our best-performing variation starts with training exclusively on QA pairs (e.g., “who handled the editing of Oppenheimer”) to grasp how knowledge is accessed. This is followed by training on a combination of these QA pairs and associated documents (e.g., “who handled the editing of Oppenheimer” and a document about “Oppenheimer”). In this phase, LLMs enhance their ability to absorb knowledge from information-dense documents, building upon the QA pairs that they have already mastered. To study continual knowledge acquisition, we build a dataset named Wiki2023, which includes a collection of documents from Wikipedia that are relevant to the year 2023. Comprehensive experiments on Wiki2023 demonstrate that after PIT, LLMs exhibit an enhanced ability to absorb knowledge from new documents (e.g., a document about “Barbie”). Detailed ablation studies reveal that this ability primarily stems from prioritizing learning how to access knowledge over learning to encode knowledge from documents. Overall, PIT significantly outperforms the standard instruction-tuning approach (§ 5.1 and § 5.2), improving QA accuracies by 17.8% on Llama-2 7B (30.3% 48.1%) and 16.3% on Llama-2 70B (46.4% 62.7%). Moreover, PIT also enhances the ability to absorb knowledge from documents of a different domain, shedding light on the potential to scale this method up to a wider variety of documents and instructions for more robust generalization (§ 5.4).
Building a Dataset to Study Continual Knowledge Acquisition
To assess the ability of LLMs to learn knowledge from new documents, it is essential to use a document corpus with minimal overlap with the original pre-training corpus. This ensures that when an LLM correctly answers questions, we can confidently attribute this capability to its learning from the new documents, rather than encountering similar questions in its original pre-training corpus. In this section, we describe a methodology for building such a corpus from Wikipedia.
In the following experiments (§ 4 and § 5), we use Llama-2 (7B and 70B) Touvron et al. (2023b) since it is one of the best-performing LLMs. We use Wikipedia articles classified under the “2023” Category including topics from diverse domains such as films, arts, economics, politics, events, etc.https://en.wikipedia.org/wiki/Category:2023 The likelihood that this factual information is not included in the original training corpus is supported by the low QA performance in Tab. 1 (9.5%/17.2% for 7B/70B).It is important to note the difficulty in completely avoiding factual overlap between Wiki2023 and the pre-training corpus of Llama-2. For example, a film released in 2023 might have had information available before 2023. Data duplication detection is an active research direction, which falls beyond the focus of this study. To accelerate the training process, we only use the first section of each article, which offers a thorough summary and contains many factual statements. The number of collected documents and an example document about “Oppenheimer” can be found in Fig. 2 and Fig. 3. We refer to this as the Wiki2023 dataset.
2 Wiki2023 Question-answer Pairs
To collect QA pairs for either instruction-tuning or performance evaluation, we employ publicly available LLMs to generate diverse questions and their respective answers given the article as context, following the Prompt § 2.2. On average, 4.93 questions are generated for each article. Fig. 2 and Fig. 3 show the detailed statistics and example QA pairs about “Oppenheimer”, respectively.
3 Splits
Among all domains, we select the film domain for evaluation and randomly select 256 articles as the test split (Wiki2023-film-test). We continually train LLMs on documents from the test split (Wiki2023-film-test-doc), and assess their performance based on the accuracy of corresponding questions (Wiki2023-film-test-QA). The remaining 1720 articles and corresponding QA pairs (Wiki2023-film-train) will be used to study different training strategies, which corresponds to the in-domain setting in Fig. 2. We also train on other domains before evaluation on the film domain to study the effectiveness of different methods across domains, which corresponds to the cross-domain setting in Fig. 2.
Experimental Settings
When training on documents, we prepend a
2 Hyperparameters
We use AdamW Loshchilov and Hutter (2019) with , , and a weight decay of 0.1. We decay the learning rate to 10% of its initial value using a cosine scheduler without warm-up. When pre-training on documents, we use a batch size of 256 documents and an initial learning rate of 3e-5. During instruction-tuning on QA pairs, we use the same batch size of 256 QA pairs, but opt for a reduced initial learning rate of 5e-6 because the number of tokens in a single batch used for computing losses is lower. The number of epochs varies depending on the setting and is detailed in the corresponding sections.
3 Evaluation Metrics
At inference time, we use greedy decoding to generate answers given questions as context, following the format in Fig. 3. To evaluate the original Llama-2, we add 5 QA pairs as in-context exemplars to make sure it follows the QA format. Since most questions are simple factoid questions and most answers are relatively short, we use exact match (EM) as our primary metric Kwiatkowski et al. (2019), which measures whether the model’s output matches the gold answer exactly after normalization (e.g., remove articles and punctuations). To assess longer responses and accommodate minor lexical differences, we also report answer recall, which assesses if the gold answer appears in the model’s output, and ROUGE-L, which measures the longest common subsequence between the model’s output and the gold answer.
How Much Knowledge Can LLMs Absorb via Continued Pre-training Followed by Instruction-tuning?
Factual knowledge stored in the parameters of LLMs can be accessed and applied to answering questions through prompting without additional training Brown et al. (2020); Petroni et al. (2019); Jiang et al. (2020); Roberts et al. (2020). With additional instruction-tuning (also known as supervised fine-tuning) on high-quality data Sanh et al. (2022); Wei et al. (2022), knowledge seems to be more effectively elicited from LLMs. However, when LLMs correctly answer a question, the source of the knowledge is unclear due to the diversity of the pre-training data. For instance, when answering the question “where is the world’s largest ice sheet located”, do LLMs derive their response by recalling and generalizing information from a seen document about the Antarctic ice sheet, or do they merely repeat answers from similar questions encountered in the training data? This distinction is crucial, as the former scenario implies an ability to comprehend documents and effectively store knowledge within parameters in a way that can be elicited later, whereas the latter is mere rote memorization.
Several works have studied this problem and the predominant finding is that LMs struggle to answer questions about documents they have been trained on Wang et al. (2021); Zhu and Li (2023a). It is important to note, however, that these experiments were mainly conducted using relatively small LMs such as BART, T5, or GPT-2 Wang et al. (2021); Jang et al. (2022); Hu et al. (2023), using randomly initialized transformers Zhu and Li (2023a), or without instruction-tuning Ovadia et al. (2023). This makes us wonder what are the actual limits of modern LLMs to absorb knowledge from new documents and answer questions about them using the standard continued pre-training followed by instruction-tuning recipe. In this section, we run extensive experiments using Llama-2 7B and 70B on Wiki2023-film to test their limits.
We experiment with two standard settings and assess their performance by answering associated questions.
Continued pre-training: train on test documents without instruction-tuning (Fig. 4 ➀).We found that LLMs struggle to adhere to the QA format after training on raw documents for multiple epochs. Therefore, we include a small set of QA pairs (64) during continued pre-training to prevent LLMs from forgetting the QA format.
Standard instruction-tuning: train on both train and test documents before instruction-tuning on train QA pairs (Fig. 4 ➁).
We perform instruction-tuning for a single epoch since more epochs usually result in diminished performance. For training on documents, we opt for multiple epochs (10/5 for a 7B/70B model), which allows for effective knowledge acquisition and remains affordable for corpora of moderate sizes.
Experimental results
As shown in Tab. 1, the relatively low performance of the original Llama-2 model (9.5%/17.2% for 7B/70B) indicates that most knowledge in the test documents is not included in the original pre-training corpus. After continued pre-training on documents, performances increase to 27.2%/41.7%, indicating that LLMs can absorb some amount of knowledge. Instruction-tuning further increases the performance to 30.3%/46.4%, confirming the effectiveness of this standard recipe. This observation is different from Zhu and Li (2023a), which demonstrates that instruction-tuning after pre-training is ineffective on a randomly initialized GPT-2-like transformer. The difference probably arises because Llama-2, through its pre-training on diverse corpora comprising raw documents and QA data, has developed a certain degree of proficiency in extracting knowledge from its parameters via questions. We also report the performance where the corresponding document is directly provided to Llama-2 as context (“open-book w/ doc” in Tab. 1). The significant gap between closed-book and open-book settings suggests that retrieving knowledge from the parameters of LLMs is still challenging.
2 Analyzing the Training Dynamics: Perplexity and Generalization
How does lower perplexity of documents lead to generalization to answering related questions? We vary the number of epochs (5(a)) and learning rate (5(b)) for continued pre-training on documents and monitor three metrics to study the training dynamics.Since we always decay the learning rate to 10% of its initial value, training for more epochs is not the same as continuing training from a checkpoint obtained after fewer epochs.
Knowledge acquisition QA accuracies on test questions measured by exact match.
Perplexity of documents We compute perplexity (PPL) on all tokens within the documents.
Knowledge retention We approximate the retention of accumulated knowledge during pre-training by assessing the QA accuracy on the Natural Questions (NQ) dataset. NQ was released in 2019, and primarily includes questions based on Wikipedia articles from that time.
As shown in 5(a), QA accuracy consistently improves as perplexity approaches one, indicating that factual knowledge learning necessitates exhaustive loss minimization over all tokens. This contrasts with learning general skills, where overly optimizing leads to overfitting.
As shown in 5(a) and 5(b), among all cases where LLMs have minimized perplexity on documents, cases trained with more epochs or larger learning rates typically exhibit superior QA performance. We hypothesize that more aggressive training leads to less overfitting to deceptive patterns in documents and better generalization when responding to questions.
In summary, lower perplexity does lead to stronger generalization when responding to questions, but it comes at the expense of forgetting previously acquired knowledge.
Improving LLMs in Absorbing Knowledge from Documents
The amount of knowledge elicited through the standard instruction-tuning is still limited, even though the perplexity of documents is minimized, a phenomenon we refer to as the “perplexity curse”. Our next question is how can we improve the ability of LLMs to absorb knowledge from documents to mitigate the perplexity curse. The main challenge is the gap between the way knowledge is presented in raw documents and how it is accessed through question-answering. We found that QA pairs are generally straightforward, while documents tend to be more complex and cluttered, weaving many factual statements together in a more intricate manner. Using Fig. 3 as an example, the answer to the question “who handled the editing of Oppenheimer” is included in a sentence in the middle of the article “Editing was handled by Jennifer Lame …”, which does not explicitly mention “Oppenheimer”. During training, LLMs must understand the context and deduce that “editing” refers to “the editing of Oppenheimer” to effectively encode this knowledge in the parameters.
Zhu and Li (2023a) studied this problem by training a randomly initialized GPT-2-like transformer from scratch on synthetic biographies and evaluated its ability to answer questions about the individuals. They found that training on a mix of biographies and questions related to half of those biographies led to strong generalization when answering questions about the remaining half of biographies, which resembles setting ➃ in Fig. 4. In contrast, training on biographies and QA pairs sequentially failed. However, the key contributor to the success remains uncertain because the data were blended together, and it is unclear how to apply this practically to absorb knowledge from new documents. Inspired by our observation of the different difficulty levels between QA pairs and documents, and the finding from Zhu and Li (2023a), we hypothesize that it is beneficial to deliberately expose LLMs to instruction-tuning data before continued pre-training so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed. We refer to this as pre-instruction-tuning (PIT) and study various implementations of PIT prior to continued learning (§ 5.1), followed by detailed ablations identifying the keys contributor to performance (§ 5.2 and § 5.3), and finally assess how well PIT performs across domains (§ 5.4). We adhere to the hyperparameters outlined in § 3.2 and perform PIT for 3 epochs unless specified otherwise.
We start with exposing instruction-tuning data before continued pre-training on documents—training on topically related QA pairs before training on test documents (Fig. 4 ➄). This can be directly compared with the continued pre-training setting (Fig. 4 ➀). The intuition is that questions help LLMs recognize key types of information, enabling LLMs to focus on important information during pre-training on subsequent documents, even though the questions are not directly tied to the documents. For example, training on a question like “who handled the editing of Oppenheimer” could help LLMs pay attention to screenwriters when training on new documents like “Barbie”. As shown in Tab. 1, this method outperforms continued pre-training, especially on larger LLMs (27.6%/41.7% 28.6%/49.7% for 7B/70B). The ablation that trains on QA data after training on documents (“instruction-tuning w/o train doc” in Tab. 2) is ineffective, confirming the importance of training on questions as a warm-up before encoding documents.
Pre-instruction-tuning on QA and documents sequentially
Our second implementation trains on QA and associated documents sequentially (Fig. 4 ➅), with the intuition that the ability to absorb knowledge from documents can be strengthened if an LLM is trained on the complex documents after it has grasped the associated simpler QA pairs. For instance, if an LLM has already learned that “Jennifer Lame” is the answer to “who handled the editing of Oppenheimer”, training on the document “Editing was handled by Jennifer Lame” can more efficiently refine its storage of knowledge in its parameters. As shown in Tab. 1, PIT on QA pairs and documents sequentially surpasses the QA-only variant (Fig. 4 ➄) and standard instruction-tuning (Fig. 4 ➁) (30.3%/46.4% 32.5%/54.6% for 7B/70B), demonstrating its effectiveness.
Pre-instruction-tuning
The effectiveness of PIT depends on ensuring that the associated QA pairs are already learned before encoding the respective documents. However, we observed that after training on documents (train doc in Fig. 4 ➅), the accuracy for corresponding questions (train QA in Fig. 4 ➅) dropped from almost perfect to 30%, indicating severe forgetting. To fix this, we train on the associated QA pairs and documents together (Fig. 4 ➆). As shown in Tab. 1, this significantly improves the performance, outperforming all other approaches, including mixing all data together (Fig. 4 ➃), by a large margin (39.4%/57.1% 45.5%/62.7% for 7B/70B). Training on both QA pairs and documents prevents forgetting, but it also obscures how the learning process works. It is unclear whether LLMs grasp QA pairs before encoding knowledge from documents, or if it works the other way around. In the following section, we deliberately arrange the order of QA pairs and documents during training to examine this, which leads us to propose an improved version of PIT.
2 Pre-instruction-tuning++
We first study how the performance varies with different numbers of epochs. As shown in Tab. 2, training for 1 epoch is insufficient, and the performance of 3, 5, or 10 epochs is similar. We fix the number of epochs to 3 and arrange the order of QA pairs and corresponding documents as shown in Fig. 6. The interleaved arrangement cycles through all the data 3 times, ensuring that in each epoch, questions either precede or follow their associated documents. On the other hand, the grouped arrangement clusters each example’s 3 appearances together, guaranteeing that the repeated questions are positioned either before or after their respective repeated documents. As shown in Tab. 2, positioning QA pairs before corresponding documents achieves better performance in both grouped and interleaved arrangements, indicating that during PIT, the learning mechanism prioritizes understanding how to access knowledge before learning to absorb information from the more complex and information-dense documents.
Based on this, we propose an improved variant called pre-instruction-tuning++, which trains exclusively on QA pairs to understand patterns of knowledge access, then progresses to training on a combination of QA and document data to align knowledge access through questions and knowledge encoding from documents (Fig. 4 ➇). As shown in Tab. 2, PIT++ significantly outperforms PIT (Fig. 4 ➆) from 45.4% to 48.1%, while training on QA data after on the mix (PIT– in Tab. 2) does not yield additional benefits. This reinforces our hypothesis that understanding how knowledge is accessed aids in absorbing knowledge from documents, and therefore, should be prioritized.
3 Ablation Studies
A drawback of standard instruction-tuning is that knowledge in test documents might be forgotten after training on QA pairs (a phenomenon also known as the “alignment tax” Ouyang et al. (2022)). To show that the lower performance of standard instruction-tuning is not due to forgetting, we add a setting where we mix train QA with test documents during instruction-tuning to prevent forgetting (Fig. 4 ➂). As shown in Tab. 2, this does not help, confirming our hypothesis.
Pre-instruction-tuning is not simply upweighting salient tokens from documents
We include an ablation inspired by Hu et al. (2023) which upweights tokens when pre-training on documents to focus on salient information. We assign a weight of 1.0 to tokens in documents that are included in the answers (e.g., “Jennifer Lame” in the sentence “Editing was handled by Jennifer Lame”), and assign a lower weight of 0.5 to other tokens. As shown in Tab. 2, this weighted continued pre-training is ineffective, confirming our hypothesis.
4 Cross-domain Generalization
We validated the effectiveness of PIT by training and evaluation on data from the same domain (Wiki2023-film). Can PIT make LLMs better at absorbing knowledge from documents of a different domain? To this end, we follow the cross-domain setting outlined in Fig. 2—training on other domains (Wiki2023-other-train) and testing on the film domain (Wiki2023-film-test). The results of standard instruction-tuning and PIT, in both in-domain and cross-domain settings, are detailed in Tab. 3. Even though it is not as effective as the in-domain counterparts, cross-domain PIT still significantly outperforms instruction-tuning, demonstrating that it can generalize across different domains. This finding sheds light on the potential to scale this method up to a broader range of documents and instructions for more robust generalization.
We also evaluate the effectiveness of PIT in two other scenarios: (1) when applied to non-Wikipedia documents, and (2) when addressing questions asked by real users. For the first scenario, we take the Llama-2 7B model trained with PIT on 2023Wiki-other and further train it on biographies synthesized in Zhu and Li (2023a) (bioS). Then, we evaluate based on questions about the individuals. For the second scenario, we manually search Google using questions generated by LLMs from Wiki2023-film-test, collect a total of 93 similar questions from real users by leveraging Google’s “People Also Ask” feature, and then evaluate Llama-2 7B on these questions. As shown in Tab. 4, PIT outperforms baselines in both scenarios, demonstrating its generalization ability.
Related Work
Several works have studied whether LMs can answer questions about information in documents they have been trained on. Wang et al. (2021); Jang et al. (2022); Hu et al. (2023) use relatively small LMs such as BART Lewis et al. (2020a), T5 Raffel et al. (2020), or GPT-2 Radford et al. (2019). Ovadia et al. (2023) focus on the comparison between RAG and continued pre-training approaches without using instruction-tuning. Zhu and Li (2023a, b) examine this problem from a similar angle as ours using a GPT-2-like transformer trained from scratch on synthetic biographies and fine-tuned on QA pairs related to the individuals. They examined a mixed training setting on both biographies and QA pairs, which is our major motivation to study different strategies to incorporate QA data before continued pre-training. Other works study adapting LLMs to new domains via various strategies Zhang et al. (2023); Cheng et al. (2023); Han et al. (2023); Wu et al. (2023); Nguyen et al. (2023); Zhao et al. (2023).
2 Instruction-tuning or Alignment
Instruction-tuning (also known as supervised fine-tuning) on high-quality annotated data Sanh et al. (2022); Wei et al. (2022); Mishra et al. (2022); Iyer et al. (2022); Kopf et al. (2023); Zhou et al. (2023); Sun et al. (2023b, a) and/or data generated by proprietary models Taori et al. (2023); Chiang et al. (2023); Wang et al. (2023b); Ivison et al. (2023), or alignment with reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) Ouyang et al. (2022); Touvron et al. (2023b); Rafailov et al. (2023); Tian et al. (2023) has been a central topic recently because it elicits knowledge from LLMs and enhances various abilities to handle questions from users. We focus on factuality and study the best way to perform instruction-tuning to elicit factual knowledge from LLMs.
3 Analyzing the Training Dynamics of LMs
Many works study the training dynamics of LMs from different perspectives. Carlini et al. (2022) quantifies memorization across model sizes and the frequency of data duplication. Tirumala et al. (2022) finds that larger LMs memorize training data faster with less overfitting. Xia et al. (2023) show that perplexity is more predictive of model behaviors than other factors. Dery et al. (2022) studies end-task aware pre-training using classification tasks and RoBERTa models. Our work differs in that we specifically focus on the capacity of recalling and generalizing information from a seen document to answer questions.
4 Retrieval-augmented Generation
Retrieval-augmented generation (RAG) is a widely used approach to incorporate new knowledge into LLMs by augmenting fixed LLMs with retrieved information from external sources Chen et al. (2017); Guu et al. (2020); Lewis et al. (2020b); Borgeaud et al. (2022); Wang et al. (2023a); Alon et al. (2022); He et al. (2021); Sachan et al. (2021); Izacard et al. (2023); Lee et al. (2022); Jiang et al. (2022); Shi et al. (2023); Jiang et al. (2023); Asai et al. (2023); Nakano et al. (2021); Qin et al. (2023); Lin et al. (2023). While RAG is effective in reducing hallucinations commonly experienced when relying solely on knowledge stored in parameters, its retrieval and generation process adds extra latency and complexity. In contrast, continued pre-training to store knowledge in parameters and utilizing the stored knowledge to answer questions in a closed-book manner are simpler and faster at inference time. Enhancing this capability is also scientifically significant, as it represents a fundamental step in employing LLMs as dependable assistants for accessing information. Therefore, this paper focuses on exploring parametric approaches.
Conclusion
We study the best way of continued training on new documents with the goal of later eliciting factual knowledge. We propose pre-instruction-tuning that learns how knowledge is accessed via QA pairs prior to encoding knowledge from documents. Extensive experiments demonstrate the superiority of pre-instruction-tuning versus standard instruction-tuning. Future directions include scaling this method up to a broader range of documents and instructions for more robust generalization.
Limitations
The Wiki2023 dataset provides a relatively clean testbed for studying continual knowledge acquisition. However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning on QA data in this paper. The effectiveness of pre-instruction-tuning with different types of data for enhancing other skills like reasoning or comprehension is something that needs to be explored in future studies.
Acknowledgements
We would like to thank Zeyuan Allen-Zhu, Zexuan Zhong, Shuyan Zhou, Frank F. Xu, Qian Liu, and Ruohong Zhang for their help with the experiments and constructive feedback.