M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models

Introduction

Large Language Models (LLMs) Raffel et al. (2020); Xue et al. (2021); Zhang et al. (2022); Brown et al. (2020); Touvron et al. (2023); Scao et al. (2022); Zhao et al. (2023); Zhou et al. (2023) have achieved remarkable progress in recent years, especially with the release of ChatGPThttps://openai.com/blog/chatgpt, which is widely acknowledged to revolutionize the world of natural language processing and to transform AI and society Altman (2023); Bubeck et al. (2023); Huang et al. (2023); Cao et al. (2023). Generally, LLMs are trained via self-supervised learning Balestriero et al. (2023) on a huge amount of unlabeled data Zhu et al. (2015); Liu et al. (2019b); Zellers et al. (2019); Gokaslan et al. (2019), which cover a wide range of genres, e.g., encyclopedias, news, books, social medias, etc. Many studies have demonstrated that LLMs are able to acquire broad knowledge of many types and subjects Zhao et al. (2023); Paperno et al. (2016); Hoffmann et al. (2022); Touvron et al. (2023); Rae et al. (2021); Raffel et al. (2020); Du et al. (2022a).

The paradigms that elicit and apply the acquired knowledge in LLMs onto downstream tasks have shifted from fine-tuning to instruction-tuning. Early LLMs usually adopt fine-tuning, which, however, suffers from lack of cross-task generalization as the fine-tuned LLMs are often task-specific and not being parameter-efficient as all pre-trained LLM parameters are usually required to be updated on downstream tasks. As LLMs reach the scale of billions of parameters, a more efficient alternative to elicit knowledge, in-context Learning (ICL) Brown et al. (2020); Xie et al. (2022); Dong et al. (2023) has emerged, which uses only a few demonstration examples concatenated in a prompt. In order to enhance the cross-task generalization of LLMs to a variety of downstream tasks, instruction-tuning Wei et al. (2022); Bach et al. (2022); Wang et al. (2022b), which is performed via multi-task learning Chung et al. (2022); Liu et al. (2019a) has been proposed. In instruction-tuning, the instructions for different tasks are different, but in a unified form. Supervised Fine-tuning (SFT) Ouyang et al. (2022) and Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017); Stiennon et al. (2020); Ouyang et al. (2022) are successful methods of instruction-tuning, which not only achieve generalization to unseen instructions but also align LLMs with human values and intents Sanh et al. (2022); Wei et al. (2022); Chung et al. (2022).

As the capability of knowledge acquisition and application in LLMs is constantly and rapidly evolving, a natural question which arises, is how we can assess such knowledge. Traditional single-task evaluation benchmarks Rajpurkar et al. (2016); Khot et al. (2020) are no longer adequate for evaluating them. Multi-task benchmarks like GLUE Wang et al. (2018), SuperGLUE Wang et al. (2019) and BIG-bench Srivastava et al. (2022) aggregate multiple NLP tasks to evaluate LLMs, which, however, are not sufficient either to assess knowledge acquired by LLMs. To address this issue, Hendrycks et al. (2021) propose MMLU, a widely used benchmark to test the knowledge acquisition and application capability of LLMs, which uses test questions across multiple subjects that humans lean to assess LLMs in zero- and few-shot settings. As MMLU is an English benchmark, it cannot be directly used for measuring LLMs trained with data in other languages. Even if it is translated into other languages, like the way used in evaluating GPT-4 OpenAI (2023), there are still gaps in knowledge across different languages as they usually have different education systems and knowledge structures.

Similar to LLMs in English, LLMs dedicated in Chinese have also achieved rapid advances recently Du et al. (2022b); Zeng et al. (2021); Zhang et al. (2021); Sun et al. (2021); Zeng et al. (2022); Ren et al. (2023); Wu et al. (2021); Wang et al. (2021); Chen et al. (2023). However, a massive knowledge evaluation benchmark that measures Chinese LLMs in line with Chinese education system is a desideratum. To bridge this gap, we propose M3KE, a Massive Multi-Level Multi-Subject Knowledge Evaluation benchmark, which is designed to measure the knowledge acquired by Chinese LLMs by testing their multitask accuracy in zero- and few-shot settings. M3KE contains 20,477 questions collected from 71 tasks. In particular, unlike recent benchmarks MMCU Zeng (2023) and AGIEval Zhong et al. (2023), M3KE covers all major levels of Chinese education system, ranging from primary school to college, as well as a wide variety of subjects, including humanities, history, politics, law, education, psychology, science, technology, art and religion. All questions are multiple-choice questions with four options, hence ensuring a standardized and unified assessment process. Table 1 shows the comparison between M3KE and other related benchmarks.

With M3KE, we have tested recently released Chinese LLMs , to track the progress of Chinese LLMs in knowledge acquisition and application. The evaluated models are either pre-trained on massive data or pre-trained + fine-tuned with SFT or RLHF. The model sizes vary from 335M to 130B parameters.

With extensive experiments, we observe that most evaluated Chinese LLMs have near random-chance accuracy, even for primary school tasks. The best performance is achieved by an SFT model built on the open-source BLOOM Scao et al. (2022), which is 14.8 points lower than the accuracy of GPT-3.5-turbo.

Our main contributions are summarized as follows.

We propose M3KE, a knowledge evaluation benchmark for Chinese LLMs, which to date covers the largest number of tasks in line with Chinese education system.

We have tested a wide range of open-source Chinese LLMs, with model sizes varying from 335M to 130B, against GPT-3.5-turbo.

We have analyzed the performance of each model on different subject clusters and education levels in both zero- and five-shot settings.

Related Work

Recent years have witnessed a rapid development of Chinese LLMs, following the efforts of their English counterparts, e.g., GPT-3 Brown et al. (2020), Gopher Rae et al. (2021), LLaMA Touvron et al. (2023). Chinese LLMs, such as Pangu-$\alpha$ with 200B parameters Zeng et al. (2021), Yuan 1.0 with 245B parameters Wu et al. (2021), ERNIE 3.0 Titan with 260B parameters Sun et al. (2021), have been trained on Chinese textual data that contain tokens ranging from 180B to 329B. These models are developed in industry, which are usually not open-source. With the success of open-source LLMs Taori et al. (2023); Peng et al. (2023) based on LLaMA, Chinese versions, such as ChatGLM-6Bhttps://github.com/THUDM/ChatGLM-6B, MOSShttps://github.com/OpenLMLab/MOSS, Phoenix Chen et al. (2023), have emerged very recently. These models usually contain less than 20 billion parameters and are supervised fine-tuned on instructions that are either distilled from models of GPT-3.5 or learned in a self-instructing manner Wang et al. (2022a).

Benchmarks.

The capability of eliciting and applying knowledge acquired during training is an important indicator for measuring LLMs. However, existing evaluation benchmarks Wang et al. (2018, 2019); Srivastava et al. (2022); Xu et al. (2020) are normally designed to evaluate LLMs on various NLP tasks, not tailored for knowledge acquisition and application assessment. To comprehensively measure knowledge in LLMs, MMLU Hendrycks et al. (2021) is proposed, which collects multiple-choice questions from 57 tasks that humans learn. As a different education system is used, on the one side, knowledge in Chinese LLMs may not exhibit in the translated-into-Chinese version of MMLU, e.g., Chinese Medicine, Chinese Legal System. On the other side, knowledge to be assessed in MMLU may be absent in Chinese textual data used to train Chinese LLMs.

Our work is related to 3 datasets that have been developed concurrently with M3KE. MMCU Zeng (2023) is a Chinese benchmark that assesses knowledge in four domains: medicine, education, law, and psychology. AGIEval Zhong et al. (2023) is a bilingual benchmark that measures the capability of LLMs on tasks of the Chinese college entrance exam and American college admission test, for high-school graduates. DomMa Gu et al. (2023) is another Chinese benchmark that focuses on domain-specific knowledge. In contrast to these benchmarks, M3KE is a comprehensive Chinese benchmark that spans major stages of Chinese education system, from primary school to college with a broader range of subject categories, such as art, religion, traditional Chinese medicine, and classical literature.

M3KE

M3KE covers major Chinese education levels, including primary school, middle school, high school, college and professional exams, as well as multiple tasks as shown in Figure 1 while the detailed subjects are listed in Appendix A. We collect and organize multiple-choice questions from public websites. To ensure the quality and comprehensiveness of the questions, entrance exam questions are selected as much as possible. For the primary school, middle school and high school education level, we choose the subjects according to the corresponding entrance exams for Chinese students. For the college level, we select subjects according to the national entrance exam for master’s degree in China. In addition to subjects under the major Chinese education system, we also collect comprehensive tasks to expand the knowledge coverage in M3KE, including computer grade exam, ancient Chinese language, novels and Chinese national civil service exam which covers commonsense knowledge, arts, religion, etc.

In total, we have 71 tasks and 20,477 questions. We divide each task into a test set and a few-shot set, where the few-shot set includes 5 questions for each task for the few-shot evaluation setting. The test set includes 20,122 questions, and each task contains at least 100 questions. Instances of M3KE are listed in Table 2.

Arts & Humanities comprise a range of disciplines that cover Chinese, literature, arts and history. These disciplines focus on the analysis and interpretation of literary and cultural artifacts, rather than on practical applications. For instance, the Chinese in primary school aims to evaluate the students’ proficiency in language use and literary appreciation for ages 7 to 13, such as the usage of synonyms and antonyms. The historical studies cover both Chinese and world history from ancient to modern times. M3KE also incorporates artistic subjects, such as dance, fine arts, music and film, because we believe that art is an essential aspect of human culture and should be relevant to LLMs as well.

2 Social Sciences

Social sciences differ from Arts & Humanities in that they emphasize practical aspects of humanistic studies, such as law, politics, education and psychology. These subjects are mainly taught at the college level. Although ideological and political courses are also part of the Chinese middle school and high school curriculum, they primarily involve moral education. Social sciences also encompass economic and management studies, which largely consist of questions from the joint exams for graduate students majoring in these fields in China. These studies include microeconomics, macroeconomics, management and logic at the undergraduate level.

3 Natural Sciences

Natural sciences encompass engineering, science, medicine and fundamental disciplines such as math, physics, chemistry, biology and so on. These subjects often require a high degree of computation, analysis and logical reasoning skills. The same subject may assess different types of knowledge at different levels according to the Chinese education system. For instance, primary school math mainly tests the basic arithmetic operations, while high school math covers more advanced mathematical concepts, such as sequences, derivatives and geometry.

4 Other

Other types of tasks include religion, Chinese civil service exam, and specialized tasks, like ancient Chinese language and novel reasoning task. These tasks require knowledge that is not limited to a single level or subject as described above. The Chinese civil service exam involves knowledge in commonsense, humanities, logic and other domains, which we can consider as an assessment of the comprehensive knowledge for LLMs. Similarly, in the novel task, these questions involve a lot of information from many classical novels.

5 Overall Statistics

Table 3 shows the overall statistics of M3KE. The numbers of tasks in the four subject clusters described above are 12, 21, 31 and 7, respectively, while the numbers of questions in the four subject clusters are 3,612, 6,222, 8,162 and 2,126, respectively. The maximum number of questions is 425 while the minimum number is 100. Questions in social and natural sciences are usually longer than those in arts & humanities and other while their answer choices are shorter.

Experiments

We assessed state-of-the-art large language models recently developed for Chinese on M3KE, attempting to understand and track the progress of Chinese LLMs in learning and applying knowledge from massive data.

The assessed Chinese LLMs can be divided into two categories: models being only pre-trained and models that are instruction-tuned with SFT/RLHF. For the former, we selected GLM-335M Du et al. (2022b), GLM-10B Du et al. (2022b), GLM-130B Zeng et al. (2022) and BLOOM-7.1B Scao et al. (2022). For the latter, we included ChatGLM-6Bhttps://github.com/THUDM/ChatGLM-6B, MOSS-SFT-16Bhttps://huggingface.co/fnlp/moss-moon-003-sft, BELLE-7B Yunjie Ji and Li (2023), where BELLE-7B is the SFT version based on BLOOMZ-7.1B-MT Muennighoff et al. (2022). We used the two variants of BELLE fine-tuned on 200K and 2M instructions, namely BELLE-7B-0.2Mhttps://huggingface.co/BelleGroup/BELLE-7B-0.2M and BELLE-7B-2Mhttps://huggingface.co/BelleGroup/BELLE-7B-2M. We also evaluated GPT-3.5-turbohttps://openai.com/product from OpenAI as a reference.

2 Prompts

All models were tested using the $n$-shot setting with a unified prompt, where $n$ is an integer from 0 to 5. For the zero-shot setting (i.e., $n=0$), the unified prompt provided to all models is “Please choose the correct option from ‘A’, ‘B’, ‘C’, ‘D’ based on the following question”. For few-shot setting (i.e., $n>0$), the unified prompt is “Please choose the correct option from ‘A’, ‘B’, ‘C’, ‘D’ based on the following examples and question”. The input to all LLMs consists of the prompt, question, answer choices and suffix, which is “the correct option is: ”. Even we tell models to only output the correct answer choice indicator (i.e., $\in\{A,B,C,D\}$) in the prompt, not all models can follow this instruction. Sometimes they output both answer choice and rationale to the answer choice (the order of these two types of outputs are random). We hence keep only the output answer choice indicator as the final answer to calculate accuracy.

3 Results

We compared the zero-shot accuracy of each model in Table 4 in terms of subject clusters. For the pre-trained models, there is a clear positive correlation between accuracy and model size, where the model with 130B parameters significantly outperforms the models with 335M/7B/10B parameters, even though they have different backbones. The accuracy of GPT-3.5-turbo is significantly higher than those of the evaluated Chinese LLMs, which currently provides an upper bound for open-source Chinese LLMs. All pretrained LLMs with $\leq 10B$ parameters achieve an accuracy lower than random-chance accuracy (i.e., 25$\%$), indicating that knowledge acquired by these models is not adequate for M3KE. In addition, we observe that the number of instructions used for SFT is an important factor, as the BELLE model fine-tuned with 2M instructions is significantly better than that with 0.2M instructions. The zero-shot performance of GPT-3.5-turbo is much higher than the compared open-sourced Chinese LLMs, but still lower than 50$\%$ accuracy, suggesting that M3KE is a very challenging benchmark.

We further compared the accuracy of different models under the 5-shot setting. Results are shown in Table 5. For pre-trained models, ICL in the few-shot setting significantly improves the performance and the smaller the pretrained model is, the larger the achieved improvement is. The exception is GLM-130B, which performs significantly worse under the 5-shot setting than the zero-shot setting. We conjecture that GLM-130B already has the ability to understand questions without examples because it uses instances in the instruction format as part of the pre-training corpus Zeng et al. (2022), and demonstrations may bring interference to the final prediction of the model. The 5-shot results of the SFT models are mixed in comparison to those in the zero-shot setting. We find that for ChatGLM-6B and BELLE-7B-2M, 5-shot is worse than zero-shot setting, similar to the results observed on GLM-130B. In contrast, 5-shot has a positive impact on MOSS-SFT-16B and BELLE-7B-0.2M. As these models are different from each other in terms of model size, training data, instruction data, etc., we leave the in-depth analysis on the mixed results to our future work.

We finally provide the results of each model on different education levels in Table 6 for the zero-shot setting and Table 7 for the few-shot setting. Interestingly, we observe that LLMs do not reach higher performance at lower education levels than higher education levels, even for GPT-3.5-turbo. This suggests that tasks from lower education levels remain challenging for these state-of-the-art Chinese LLMs.

Conclusion

We have presented a new benchmark M3KE, to assess the capability of Chinese LLMs in learning and applying knowledge in multiple subjects at multiple levels of Chinese education system. M3KE contains 71 tasks and 20,447 questions. We find that all evaluated state-of-the-art open-source Chinese LLMs significantly lag behind GPT-3.5. We hope that this benchmark can be used to track and promote further progress in Chinese LLMs.

References

Appendix A All Subjects

See Table LABEL:summary for all 71 tasks.