ARB: Advanced Reasoning Benchmark for Large Language Models

Introduction

In recent years, models such as GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022), and Chinchilla (Hoffmann et al., 2022) have shown increasing performance across a wide variety of natural language tasks ranging from translation to reasoning (Bubeck et al., 2023; Laskar et al., 2023). This rapid progress has been closely tracked and assessed by evaluating LLMs on benchmarks, which test model capabilities on a set of standardized problems. The GLUE benchmark (Wang et al., 2019b) for language understanding was first released in April 2018; but models such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) in the following year were already powerful enough to necessitate the “SuperGLUE” benchmark (Wang et al., 2019a). Since then, the race between language models and benchmarks has increasingly favored the former.

Scaling up, model sizes and datasets alike, has led to rapid improvements on various natural language tasks on benchmarks like BIG-bench (Srivastava et al., 2022) and HELM (Liang et al., 2022). Neural scaling laws (Kaplan et al., 2020; Caballero et al., 2023; Alabdulmohsin et al., 2022) have been used to predict the behavior of large scale models on various metrics. Nevertheless, LLM performance often increases unpredictably (Wei et al., 2022a), especially on tasks that require reasoning abilities. Predictions of performance on ML benchmarks often underestimate the rate of progress (Steinhardt, 2022). Since progress has been faster than anticipated, new benchmarks need to be more difficult.

Models such as ChatGPT have shown the ability to pass entry-level examinations in fields such as law (Bommarito II and Katz, 2022), medicine (Kung et al., 2023), economics (Caplan, 2023), and mathematics (Shakarian et al., 2023). Nevertheless, LLM understanding of many fields is reportedly shallow and unreliable (Shapira et al., 2023). Expert reasoning in domains with specialized knowledge is essential for automated systems to augment skilled professionals (Noy and Zhang, 2023).

In this paper, we introduce a new benchmark dataset, ARB (Advanced Reasoning Benchmark), designed to evaluate expert reasoning abilities in mathematics, physics, chemistry, biology, and law. To make the benchmark more challenging than previous benchmarks, we extract graduate-level tasks from resources intended for domain professionals. The performance of current models such as GPT-4 on the quantitative parts of ARB is very low using standard prompting methods.

Our dataset offers improvements over existing benchmarks:

Hundreds of problems requiring expert reasoning in quantitative subjects, where LLMs are known to underperform;

A large percentage of the problems are short-answer and open response questions, in contrast to the multiple-choice questions that dominated earlier benchmarks.

In addition, we propose an automated rubric-based method allowing self-evaluation of intermediate reasoning steps. While not currently a substitute for human evaluation, rubrics generated by GPT-4 have good coverage, and self-evaluation scores track human grading surprisingly well.

We provide the instructions to access the dataset in the supplementary material.

Related Work

Improving the reasoning capabilities of LLMs has been a subject of recent interest, with a particular focus on advanced prompting techniques (Wei et al., 2022b; Kojima et al., 2023; Wang et al., 2023; Yao et al., 2023; Nye et al., 2021). Such techniques have seen increasingly successful applications in solving reasoning problems involving commonsense reasoning and mathematics, by promoting active reasoning processes within the LLMs before yielding final answers.

Model architectures such as Minerva (Lewkowycz et al., 2022) have exemplified the enhancement of reasoning capabilities through fine-tuning on extensive datasets covering math and reasoning tasks. This has yielded improved performance across several benchmarks, including MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), and MMLU (Hendrycks et al., 2020). Concurrently, other lines of research (Li et al., 2023; Lightman et al., 2023; Cobbe et al., 2021) have investigated the application of verification techniques to augment and enhance LLM performance.

Most of the aforementioned work has typically evaluated techniques against math benchmarks (e.g., GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), SVAMP (Patel et al., 2021), ASDiv (Miao et al., 2020), AQuA (Ling et al., 2017), MAWPS (Koncel-Kedziorski et al., 2016), MultiArith (Roy and Roth, 2016)) and commonsense reasoning tasks (e.g., CSQA (Talmor et al., 2018), StrategyQA (Geva et al., 2021), HotpotQA (Yang et al., 2018)). Recently, several new benchmarks have been introduced for reasoning and planning tasks, such as the GPT-Planning Benchmark (Valmeekam et al., 2023), ALERT Reasoning Benchmark (Yu et al., 2022), JEEBench (Arora et al., 2023)), and (Gendron et al., 2023). Additionally, comprehensive evaluation suites like the Chain-of-Thought Hub (Fu et al., 2023) have been proposed.

Despite their utility, existing benchmarks are limited in difficulty, represent a restricted range of reasoning challenges, and do not necessarily mirror real-world tasks demanding complex reasoning. Moreover, recent advancements such as Minerva (Lewkowycz et al., 2022) have revealed that these benchmarks may not offer sufficient challenge.

The rapid progress in LLM capabilities has led many to explore using LLMs in the LLM evaluation pipeline. Apart from using LLMs to generate evaluation tasks (Zhang et al., 2022; Perez et al., 2022), LLMs have increasingly been used as a proxy for human evaluation (Chiang and Lee, 2023; Liu et al., 2023; Fu et al., 2023; Kocmi and Federmann, 2023). Useful LLM-based evaluation for alignment has been done using rubrics (Bai et al., 2022). We explore the efficacy of rubrics for evaluation when applied to highly complex math and physics problems.

Benchmark

The key considerations when building a machine learning benchmark are:

Difficulty. Most tasks have to be out of reach of current models; a benchmark where many models score over 95% is not useful for tracking differential AI development.

Usefulness. The tested skills should correlate with generally useful human skills.

Ease of evaluation. It should be straightforward for the model creators to compare the performances of different models. The scores should be interpretable.

Minimizing data contamination. A consistent issue with popular benchmarks is that the recent LLMs contain some tasks in their training data (OpenAI, 2023). This leads to overestimation of true model capabilities.

Connection to general capabilities. If a model is trained on data similar to the benchmark, it is possible it achieves high performance without generalization or “intelligence”, failing to solve novel tasks of similar difficulty (Chollet, 2019). Conversely, problems should not be pathological or overly adversarial, to avoid the dangers of underclaiming (Bowman, 2021).

The benchmark consists of three types of questions: multiple choice, short answer, and open response, in descending order of proportion in the dataset.

Multiple choice questions consist of a question and four to five possible answers, and the correct answer is the one that best answers the question. They were sourced from standardized tests, such as the MCAT and bar exam prep, and make up a large proportion of the dataset due to their ease of grading.

Short answer questions, on the other hand, ask for final answers in the format of a short phrase or mathematical expression. They were sourced from problem books such as Souza and Silva (2008), Gelca and Andreescu (2017), and physics book series Lim and Qiang (2001), Lim (2007), Lim (1998), Lim et al. (2019), and Lim (1996). We generally avoided algebraic expressions, because of technical difficulties in the grading process.

A given algebraic expression may have several equivalent forms (e.g. nontrivial functional relations for the functions appearing in the final answer), and a grading scheme which accounts for all possible variations across our entire dataset is not feasible. Moreover, physics problems often require answers introducing new notation that is not explicitly mentioned in the problem statement.

Open response questions are more challenging: they consist of a question and a blank space for the answer. They were sourced from problem books and exams, such as the Harvard PhD comprehensive exams in mathematics (Harvard University, 2021). Such tasks require manual grading. These questions are aspirational in nature, as current systems (e.g. ChatGPT) cannot produce satisfactory responses, even for the “elementary” problems.

2 Mathematics

This part of the dataset is the most diverse. It includes contest mathematics problems as well as “university mathematics” (i.e. mathematics traditionally taught in universities at the undergraduate and beginning graduate level). The contest problems are sourced from Gelca and Andreescu (2017) and Brayman and Kukush (2018), and the university mathematics problems are sourced from Souza and Silva (2008) and Harvard University (2021). The dataset does not include high school contest problems because those are already present in other well-known benchmarks (Hendrycks et al., 2021). The Putnam and Brayman books both contain official solutions, which we also include in the dataset. This can be useful for fully automating the grading process, which we leave to future work.

For university mathematics, we pick Souza and Silva (2008) for its large selection of “standard” undergraduate mathematics problems, as well as many problems suitable for the short answer portions. We also select Harvard University (2021) because it covers topics that other collections of exams rarely not cover, such as representation theory of finite groups and algebraic topology.

3 Physics

The physics problems are structured similarly as the math problems. The main difference is that some physics problems contain figures, and there are more problems with numerical answers. The problems were sourced from the Major American Universities PhD Qualifying Questions and Solutions series (Zhongguo-Kexue-Jishu-Daxue, 1990).

4 MCAT

The MCAT test contains multiple choice problems testing biology, psychology, chemistry, physics, and reading comprehension. The MCAT problems are sampled from the third edition of McGraw-Hill Education 3 MCAT Practice Tests (Campbell et al., 2017) and cover both science and reading questions. This book was chosen as very few of these problems appear in standard web-searchable sources, limiting contamination. As in the previous categories, we pick problems which are self-contained. Because some MCAT science questions are accompanied by images, we accompany such questions with corresponding image files.

5 Law

Applying law involves the application logical reasoning, in addition to grasping legal knowledge. This makes assessments of legal skills an especially attractive type of language model benchmark, where we are attempting to assess the reasoning and intelligence of these models. Furthermore, if the models better understand law, they can be more reliable and ultimately more useful in real-world applications, potentially even increasing the efficiency and transparency of governments more broadly.

Most lawyers in the U.S. go to law school, graduate, then study for the Bar Examination, and then must pass the bar before going on to practice law professionally. To evaluate legal understanding of the models, we use an older Bar Examination practice set that, to the best of our knowledge, is not available online in a way that could have led to its inclusion in training data for the language models that we are assessing. The practice bar exam we administer to the various language models covers most major areas of law and therefore it tests legal reasoning and broad U.S. legal knowledge.

Evaluation

We evaluate current LLMs on all text-only problems in our dataset. Other LLM benchmark papers do not evaluate on multimodal tasks due to the lack of good multimodal models; we follow suit. Given public communications about GPT-4 (OpenAI, 2023) and Gemini (Ghahramani, 2023), it is likely the physics and MCAT image problems will be useful for testing multimodal LLMs soon.

We evaluate ChatGPT (gpt3.5-turbo-0301), GPT 3.5 (text-davinci-003), GPT-4 with 8k context length (gpt-4-0314), and Claude (claude-v1.3-100k). We evaluate all question types using task-specific instructions and chain of thought. In chat models, we put the instructions as the system prompt; otherwise we put them at the beginning of the prompt.

In all problem types, in order to extract the model’s final answer, we instruct the model to write its final answer at the end of the response after the delimiter ANSWER: . We then parse the model generated final answer as the remaining text after the delimiter. The response is marked as incorrect if the delimiter is not found. Due to the differences in evaluation for multiple choice versus open-ended responses, we adopt multiple evaluation procedures.

Multiple choice

To evaluate multiple choice questions, we can simply compare the extracted final answer to the ground truth. A response is considered correct if the extracted choice matches the ground truth choice. With appropriate prompting, all models output a parsable answer > 97% of the time. We conduct a separate manual evaluation on a sampled subset of the questions to check that our parsing procedure is not mischaracterizing the true performance of the model.

Numerical

To evaluate problems with a numerical final answer, we first extract the delimited model answer as above. In the physics problems, many answers are in units; we prompt the model with information about the unit, and instruct it to fully simplify its answer and omit any units. However, sometimes the model forgets to do either or both, and so we apply a series of regexes to remove units. We then attempt to parse the result into a mathematical expression using Python’s SymPy library (Meurer et al., 2017). If this parsing fails, the answer is marked as incorrect. Once parsed, we score a the model answer as correct if $\frac{\left|\texttt{model\_answer}-\texttt{ground\_truth}\right|}{\texttt{ground\_truth}}<0.01$.

Symbolic

Problems with symbolic answers are less structured and harder to parse. To do so, we again leverage SymPy, first normalizing expressions to contain a default set of variable names and then checking for equivalence up to a permutation of the variables. However this approach is error-prone and only works for the subset of symbolic responses in a function form. More advanced responses, such as those containing set notation, require human evaluation.

Proof-like

Natural language proofs cannot be evaluated automatically; the authors with training in mathematics grade the proofs. Further manual human evaluation requires a thorough inspection of the intermediate reasoning steps. This makes evaluation expensive in practice.

Model-based evaluation

To address the difficulties in developing automated metrics for evaluating more advanced problems, we experiment with two model based approaches. First, we prompt ChatGPT to grade the equivalence of two symbolic expressions with score options when the totally incorrect, $0.5$ when the symbolic expressions are nearly the same e.g. equivalent up to a constant, and $1$ when they are an exact match. Our prompting strategy can be found in the supplementary material.

More generally, we evaluate the capabilities of GPT-4 to grade intermediate reasoning chains via a rubric-based evaluation approach. For symbolic and proof-like problems, we few-shot prompt GPT-4 to create a 10-point rubric. This is done by handwriting a small set of initial rubrics for proof-like problems and prompting the model with these examples and the ground truth reference solution. The model assigns point values to intermediate steps using the reference solution as a guide. This process is illustrated in the supplementary material.

With model generated rubrics in hand, we then evaluate each question against its rubric. This is done by again prompting GPT-4 to go step by step through the model answer and assign partial credit based on the rubric. This provides a denser automatic evaluation metric on increasingly unstructured answers. As a nice side benefit, it makes human evaluation of complex symbolic questions much easier, significantly reducing the amount of time required per question.

1 Results

We now evaluate gpt-4, gpt-3.5-turbo, text-davinci-003, and claude-v1.3 on ARB. The results for the mechanically scored subjects are in Figure 1.

We see models generally do quite well on the multiple choice Law and MCAT subsets, but struggle significantly on questions with numerical final answers. GPT-4 is the only model capable of reliably simplifying complex expressions, but even GPT-4 struggles to reliably perform arithmetic and symbolic manipulations over long contexts.

On the multiple-choice questions, the only model that cannot reliably follow the answer formatting instructions is gpt-3.5-turbo. This happens for a variety of reasons, including the model refusing to answer or to commit to a single answer choice. On the Law benchmark, gpt-3.5-turbo does not output a parsable answer around 25% of the time. The other models exhibit this failure in less than 5% of multiple-choice questions, with GPT-4 being correctly parsed over 99% of the time.

We see a similarly low performance profile across models on symbolic problems, reported in Table 2.

2 What Kind of Errors Do LLMs Make?

The GPT-4 evaluation paper (Bubeck et al., 2023) classified errors GPT-4 makes in single-pass evaluation on GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) into three types: arithmetic mistakes, misunderstood statement, and wrong approach. We make a more fine-grained analysis and extend it to math and physics problems in our dataset. The results are in Table 3.

The errors current LLMs make on the Mathematics part of ARB fall into five general types:

Misunderstanding / answering only a part of the question / misread problem;

Wrong approach: the model’s early chain of thought does not guess the right approach;

Logical errors: the model uses a false implication between two statements;

Hallucinating facts or theorems: the model confabulates a statement that is false in general, or not applicable in context;

Arithmetic/calculation error: the model multiplies incorrectly, omits a term in an expression, gives a wrong numerical value for a fraction, and other similar mistakes.

We grade GPT-4 using the above as a guideline. Our grading of the model’s CoT answers is not mutually exclusive; if the model both uses an approach that doesn’t go anywhere and makes a calculation error in it, we count it towards both categories. Note that the errors might not be independent: arithmetic mistakes could be more or less frequent in wrong approach solutions as opposed to the solutions with correct idea. We notice that the model is likely to make incorrect simplifications to get to some final answer in approaches that cannot work; this is expected, as prompting the model to produce a solution with a final answer often leads it to produce some final answer by any means.

When the model outputs a chain of implications, it is not always clear whether some false statement is due to a logical error, or it is a straight-out confabulation. We merge those two error types in Table 3.

Some problems ask for multiple things to be proven or calculated. Our graders gave the model a score of 0.5 if it correctly derived at least half of the "subproblems" (for example, homology groups of a given manifold). With this more benevolent form of grading, the performance of GPT-4 on the Proof-like problems jumps to 16%. Where applicable, slight discrepancy with automatic evaluation is also possible due to the error tolerance.

We note that many of the problems in Physics Symbolic have correct symbolic answers even when there are flaws in the chain of thought reasoning of GPT-4. This is likely due to some kind of memorization, although not necessarily from the same sources: see Table 12 for an example.

It is possible that our graders underestimate the rate of arithmetic mistakes in some cases, especially when the approach is clearly wrong, or it is not clear whether a given error is due to faulty reasoning or due to a missed term in the calculations.

For the larger subsets (see Table 1), we subsample the problems to between 20 and 40 per subject area; this is enough for a ballpark estimate of the frequency of different errors, and is not worth increasing because attributing error types is inherently fuzzy.

Model-based Rubric Evaluation

As reasoning tasks increase in complexity, it gets harder to evaluate model performance. Symbolic final answers are in some cases difficult to grade automatically. Further, we are often more interested in the correctness of the reasoning used to produce the final answer; but evaluating intermediate reasoning steps requires expert human supervision. An ideal solution would be to use LLMs as evaluators based on a reference solution; unfortunately, there are major reliability issues.

To improve reliability, we propose generating rubrics as an important component of the evaluation process. The model generates the rubric from the reference solution, then evaluates any solution based on the generated rubric. To aid rubric generation, we give few-shot examples of human-written rubrics to the rubric-generating model run. We study this approach by conducting a human evaluation of GPT-4 generated rubrics and the GPT-4 grading of its own solutions using the generated rubrics.

We rate the quality of GPT-4 generated rubrics by hand in the first two rows of Table 4. Likert scores from 1-5 are assigned to both the coverage of the rubric, i.e. how well it captures key subproblems, and the point breakdown. Rubric quality scores are reported in Table 5 for symbolic and proof-like problems. We find GPT-4 designs rubrics which cover the crucial solution steps well, but struggles to properly allocate points to each step based on relative importance. However, it is much better than GPT-3.5-turbo, which tends to over-allocate points to only one or two solution steps.

The obvious limitation of rubric scoring is the case of correct solutions not covered by the rubric. We find that on our benchmark, GPT-4 rarely generates a fully or even mostly partially correct solution that does not follow the rubric. Once done rating the model generated rubrics, we then manually grade GPT-4’s solutions according to each rubric and compare the result to GPT-4’s evaluation. We also annotate, for each problem, both whether GPT-4 assigns credit inappropriately or fails to assign credit when it should.

We find a moderately high correlation between GPT-4’s evaluation score and the manual score. In some cases, the model, assigns an extra point or two when compared to the annotated rubric score. However, the self-eval score almost never deviates more than two points from the ground truth. The main failure mode we detect is the assignment of partial credit to attempted solutions completely outside the problem rubric, where the human evaluation score is always zero. Taken together, we believe these results suggest that rubric-based evaluation is a promising automated evaluation method.

Having established rubric-based evaluation as a (imperfect) proxy for correctness, we now comment on the GPT-4 performance graded by the rubric. Table 5 shows GPT-4 is best at generating correct intermediate reasoning steps for physics questions. Inspecting the model outputs suggests that GPT-4 is good at recalling relevant and useful concepts in physics for solving the relevant problem; however, it can struggle with the mathematical manipulations required to solve the problem. The model is worse at recognizing the correct concepts and formulating an appropriate plan for the math questions, particularly for proof-like problems.

Limitations and Conclusion

In this paper, we presented ARB, a novel benchmark for evaluating advanced reasoning capabilities in large language models. Our dataset is composed of various problems from the sciences and law, sourced from graduate-level exams and professional resources. Despite advancements in current LLMs, their performance remains very low on the quantitative subjects, in ARB’s tasks. We also proposed a rubric-based self-evaluation method, enabling LLMs to grade their own reasoning. This method is not yet reliable enough to replace human grading. We hope this method can be extended to more reliable and cheap testing of complex model outputs.

As with all other benchmarks that are not created anew and kept secret, it is possible there is data contamination. For example, the MCAT books are not available for free in most jurisdictions, but it certainly possible that some model creators have trained on it anyway.

Finally, the benchmark does not remotely cover all aspects of human ability; a model solving this benchmark perfectly could still be much worse than most educated people in many aspects. Nevertheless, we hope that increasing the difficulty standards helps the research community ground the performance of increasingly powerful models more accurately.

Acknowledgments and Disclosure of Funding

We thank Jeffrey Deng for developing and documenting the API, and building the project website. We would also like to thank Raunak Chowdhuri for helpful comments, and Zhangir Azerbayev for useful discussions early on in the project. TS is supported by NSF grant 1745583.

References

Appendix A Datasheet

We present the data card, following the format proposed by Pushkarna et al. (2022).

Risk and Mitigation. There is little risk associated with this dataset, as it is intended for benchmarking reasoning capabilities of models, and it is too small to be used for advancing capabilities.

Maintenance Status. Limited Maintenance. The data will not get major updates, but any technical issues will be addressed.

Maintenance Plan. Any technical issues will be addressed.

Updates. Updates are limited to bug and error fixes.

Errors. Error handling will be considered case by case.

Example: Typical Data Point. Each data point of the dataset consist of a pair of problem statement and ground truth solution. Table 7, Table 9 and Table 10 include problem statement and ground truth solution of typical data points.

Sensitive Human Attributes. We have not found any sensitive human attributes in our dataset.

Data Distributions. Table Table 1 shows the number of problems for each subject area and answer type. Text entries (problem statement, ground truth solution, ground truth answer) for all categories are in LaTeX (although obviously, the non-quantitative subjects have very few mathematical expressions).

Appendix B Dataset format

The benchmark dataset is available in .jsonl format, containing problem statements, ground truth solutions, and final ground truth answers for each entry. We additionally include metadata such as subject names and problem topics, where available.

We chose the four subject areas discussed earlier for several reasons. Primarily, the dataset focuses on math and physics, as these subjects present more challenging problems than existing benchmarks. However, to ensure a comprehensive evaluation of models, we also included subjects like Law and MCAT. This inclusion allows for assessing model performance across a wider range of technical domains, beyond the quantitative sciences.

Although previous works have evaluated recent models on law (Katz et al., 2023), we draw upon the established importance of broad benchmarks like SuperGLUE (Wang et al., 2019a). Making a benchmark more comprehensive expands the evaluation scope while enhancing the dataset’s significance in the wider AI research context.

Appendix C Accessing the dataset

We facilitate access to the dataset through API callsThe API can be accessed through the following: https://arb.duckai.org/api/lib. The documentation can be found here: https://app.swaggerhub.com/apis-docs/arb-dataset/arb-api/1.0.5. The problems can be accessed by the different splits and subject areas.

We use an API to host the data (rather than making it publicly available on popular platforms such as HuggingFace or Github) out of concern for data contamination. Many models released in the last few years have been trained on benchmark datasets OpenAI (2023), an approach that could inadvertently inflate the model’s performance on the associated benchmark. There are several proposed strategies to mitigate this problem, including dataset poisoning and canary text methods.However, restricting dataset accessibility to web crawlers seemed to be the only way to ensure integrity of our dataset for future uses.

Appendix D Model-based Rubric Eval

Table 7 presents a GPT-4 generated rubric and self-evaluation for symbolic math questions. GPT-4 does a good job breaking the problem into important sub-parts of the reference solution. Further, it ensures a well-balanced point distribution among sub-problems, assigning similar point values to each sub-part. In contrast, less powerful models like GPT-3.5-turbo struggle to generate good rubrics, as they tend to under-specify sub-problems and allocate too many points to irrelevant steps.

The main downside on evaluating solutions only on the final answer is that the metric is not smooth; a small mistake in the middle of the solution always leads to an incorrect final answer. Grading using a rubric shows that GPT-4 generates some useful ideas, for example the correct formula for parameterizing an ellipse. The model also correctly identifies that the question’s area is optimized by an isosceles triangle. Despite this, it is unable to correctly compute the final answer due to an earlier mistake in the response. This indicates that GPT-4 has some problem-solving abilities, but struggles to detect or recover from earlier errors in generation.

Unlike GPT-4, GPT-3.5-turbo is not to write rubrics with good coverage of the reference solution and a fair point breakdown. Often the model will over-simplify the rubric and allocate far too many points to non-essential parts of the problem. However, GPT-3.5-turbo does possess some ability to reason about complex symbolic expressions. Motivated by this, we asked the model to grade the final answers to symbolic math and physics problems. While much easier to grade than intermediate reasoning steps, more involved symbolic expressions still require human evaluation to compare accurately. Using cheap models like GPT-3.5-turbo to automate this symbolic equivalence boosts our abilities to evaluate models on this more complicated class of reasoning problems. We prompt GPT-3.5-turbo to compare extracted model generated symbolic final answers from GPT-4 to the reference answer and record results in Table 8.

GPT-3.5-turbo is surprisingly good at not equating non-equivalent symbolic statements, achieving a false positive rate of 0. However, this comes at of the model often not assigning credit when it should, leading to an underestimate of the true performance of the graded answers on symbolic tasks. In particular, GPT-3.5-turbo often fails when comparing expressions with more than 3 variables. Furthermore, we observe in several cases the student model GPT-4 is able to correctly generate the final answer, but does not present it in the expected format, making it impossible for GPT-3.5-turbo to correctly compare against the ground truth reference. These errors could potentially be avoided by giving the model access to the entire student generated reasoning trace, but we expect this to be a minimal help and perhaps even harm performance.

Appendix E Interesting Examples

One common way GPT-4 gets numerical questions wrong is when it do some formal manipulations follows by a blackbox numerical calculation after which the model spits out an (incorrect) answer. (Table 10) The formal manipulation does not simplify the problem, and so all the work of solving the problem is done in the final step of the calculation.

MCAT Example.

GPT-4 get’s confused when meanings of words are implicit in prerequisite knowledge or contexts. In one example about DNA replication (Table 9), the model correctly identifies that the radioactive thymine is present in the two strands of nucleotides from the original DNA, it fails to deduce that both of the resulting double helices are radioactive. This seems to be because the model confuses the word "DNA" with "strands" of the DNA. When looking at choice C, the model (incorrectly) assumes that each of the four strands in the new double helices are radioactive, when it is clear from context that the choice is referring to the radioactive molecule being present somewhere in each double helix (not necessarily in each strand). Because of this misconception, the model chooses D.

Law Example.

An unexpected mistake from GPT-4 in answering law questions is where the model reads too much into an answer choice. For example, GPT-4 incorrectly produced this answer and reasoning:

“B: This answer choice suggests that when two crossing offers are identical, one will be treated as an offer and the other as an acceptance. This accurately reflects the situation between Mom and Huck, as they both agreed on the same price.”

“Yes, because when two crossing offers are identical in import, one will be treated as an offer and the other as an acceptance.”

The error GPT-4 made is treating the statement in the answer choice (“when two crossing offers are identical, one will be treated as an offer and the other as an acceptance”) as a fact, and then making a choice based on that fact being true. Better prompting could likely precent errors of this type.

Appendix F Memorization

While it is difficult to quantify potential problem leakage and memorization of similar problems, some outputs suggest this might be a relevant factor. In Table 12, GPT-3.5-turbo appears to be memorizing the answer to this symbolic physics problem, given that it is getting the formula for capacitance $C$ correct despite having the wrong reasoning. In contrast, GPT-4 stays faithful to its reasoning chain, resulting in a slightly inaccurate final answer. Note that the overall performance of all models we test is still somewhat low, hence not majorly affected by memorization of similar problems.

Appendix G Prompts for Sections 4 and 5

For OpenAI chat models (gpt-4 and gpt-3.5-turbo), following best practices, we prepend the system prompt. We do not use the system prompt for the other models due to there not being a well-established way to do so, and out of concerns of hurting performance on the other models. We note that omitting the system prompt should have little effect on the performance of the other models, as the directions of the system prompt are essentially restated in our user prompts.

Appendix H Checklist Information.

Dataset Intended Uses. The ARB benchmark dataset, documented within the paper, is primarily intended for research purposes. We do not advocate for this dataset to train models that help students cheat on exams. We hope that the research community will use this benchmark to better assess reasoning capabilities of language models.

Author Statement and License. We bear all responsibility in case of violation of rights. The ARB dataset is licensed under CC BY 4.0, and all helper code we release is under the MIT license. For all problems originating in books listed in Section 3, following (Hendrycks et al., 2021), we abide by Fair Use §107: "the fair use of a copyrighted work, including such use by … scholarship, or research, is not an infringement of copyright", where fair use is determined by "the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes" and "the effect of the use upon the potential market for or value of the copyrighted work".