MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

Introduction

This work focuses on mathematical reasoning, a critical capability of modern large language models (LLMs) (OpenAI, 2023; Anil et al., 2023). Despite the recent advances in this field, a noticeable gap exists between closed-source and open-source LLMs—closed-source models like GPT-4 (OpenAI, 2023), PaLM-2 (Anil et al., 2023), and Claude 2 (Bai et al., 2022) dominate popular mathematical reasoning benchmarks such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b), while open-source models like Llama (Touvron et al., 2023a; b), Falcon (Penedo et al., 2023), OPT (Zhang et al., 2022) lag behind on all benchmarks by a wide margin.

Current efforts to bridge this gap are twofold: (1) Continued pre-training like Galactica (Taylor et al., 2022) and MINERVA (Lewkowycz et al., 2022), which continues to train an LLM on math-related web data of more than 100B tokens. This approach improves a model’s general scientific reasoning capability but incurs a high computation cost. (2) Dataset-specific fine-tuning like rejection sampling fine-tuning (RFT) (Yuan et al., 2023) and WizardMath (Luo et al., 2023), which fine-tunes LLMs using supervised data specific to certain datasets. Although such approaches improve in-domain performance, they cannot generalize to a wider range of math reasoning tasks beyond their fine-tuning data. For instance, both RFT and WizardMath can increase the accuracy on GSM8K (Cobbe et al., 2021) by 30%+, one of their fine-tuning datasets, but hurt the accuracy on out-of-domain datasets like MMLU-Math (Hendrycks et al., 2021a) or AQuA (Ling et al., 2017) by up to 10%.

In this paper, we aim to propose a lightweight yet generalizable math instruction-tuning approach to enhance the general (i.e., not limited to the fine-tuning tasks) mathematical reasoning capabilities of LLMs. Existing methods (Luo et al., 2023; Yuan et al., 2023; Taylor et al., 2022) primarily focus on Chain-of-Thought (CoT) approaches (Wei et al., 2022b; Nye et al., 2022) to solve math problems through step-by-step natural language descriptions. This approach excels in its generality to cover most math subjects but struggles with computation precision, and complex mathematical or algorithmic reasoning procedures (e.g., solving quadratic equation roots and calculating matrix eigenvalues). In contrast, prompts in the format of code like Program-of-Thought (PoT) approaches (Chen et al., 2022) and PAL (Madaan et al., 2022; Gao et al., 2023) utilize external tools (i.e., Python interpreter) to greatly simplify the math solving process. This approach advocates offloading the computation process to the external Python interpreter to solve complex mathematical and algorithmic reasoning procedures (e.g., solving quadratic equations with sympy or calculating matrix eigenvalues with numpy). However, PoT falls short in dealing with more abstract reasoning scenarios, like common-sense reasoning, formal logic, and abstract algebra, especially when there exist no built-in APIs.

To leverage the strengths of both CoT and PoT approaches, we introduce a new math hybrid instruction-tuning dataset MathInstruct, which has two main characteristics: (1) broad coverage of different math fields and complexity levels, and (2) hybrid CoT & PoT rationales. MathInstruct is based on seven existing math rationale datasets and six newly-curated datasets (see details in Table 1). We use MathInstruct to fine-tune Llama (Touvron et al., 2023a; b; Rozière et al., 2023) models of different scales ranging from 7B to 70B. The resulting MAmmoTH models ( Figure 1) demonstrate unprecedented potential in serving as math generalists.

We evaluate MAmmoTH on a spectrum of datasets, including in-domain (IND) test sets—GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), AQuA-RAT (Ling et al., 2017), NumGLUE (Mishra et al., 2022b)—and out-of-domain (OOD) test sets—SVAMP (Patel et al., 2021), SAT (Zhong et al., 2023), MMLU-Math (Hendrycks et al., 2021a), Mathematics (Davies et al., 2021), and SimulEq (Koncel-Kedziorski et al., 2016). Compared with existing methods, our models generalize better to OOD datasets and substantially improve the performance of open-source LLMs in mathematical reasoning. Notably, on the popular competition-level MATH dataset (Hendrycks et al., 2021b), our 7B model can beat WizardMath (open-source MATH SoTA) (Luo et al., 2023) by 3.5x (35.2% vs 10.7%), and our 34B MAmmoTH-Coder (fine-tuned on Code Llama (Rozière et al., 2023)) can even beat the result of GPT-4 (using CoT).

We highlight our contributions from two perspectives: (1) From the data engineering perspective, we present MathInstruct, a high-quality math instruction tuning dataset, combining a variety of math problems and hybrid rationales. (2) From the modeling perspective, we investigate the impact of various data sources and input-output formats through training and evaluating over 50 different models and baselines ranging from 7B to 70B. Our models, including MAmmoTH and MAmmoTH-Coder, achieve substantial accuracy gains over existing open-source models.

Our Approach

Mathematical reasoning serves as a vital gauge for assessing the ability of LLMs to execute complex multi-hop and quantitative reasoning. Previously, this has been a challenging task for neural networks, which struggle to solve even basic addition and subtraction problems (Yang et al., 2023). However, recent LLMs have considerable advancements in mathematical reasoning. Key breakthroughs have been made through CoT prompting (Wei et al., 2022b; Nye et al., 2022) and PoT prompting (Chen et al., 2022; Gao et al., 2023). CoT prompting encourages LLMs to solve problems incrementally on a scratchpad, enhancing both accuracy and explainability in mathematical reasoning. This approach contrasts with traditional methods that generate answers directly. PoT prompting, on the other hand, formulates the intermediate reasoning process as a program, executed with an external tool like Python, to compute the answer. This method improves robustness in solving complex mathematical problems by offloading the calculations to external tools. However, most existing work (Zhou et al., 2023a) in PoT is limited to proprietary models like GPT-4 (OpenAI, 2023) and Codex (Chen et al., 2021). The PoT potential of open-source models is yet to be seen. Our work aims at optimizing LLMs’ CoT and PoT reasoning capabilities through instruction tuning.

2 Curating a Diverse and Hybrid Instruction Tuning Dataset

Our study aims to compile a list of high-quality and diverse math instruction-tuning datasets, standing out with three main characteristics: (1) broad coverage of different mathematical fields and complexity levels, and (2) hybrid CoT & PoT rationales.

Broad Coverage of Different Math Fields and Complexity Levels: We aim for a broad representation of math fields and complexity levels in our dataset. This ensures exposure to a diverse set of mathematical knowledge, fostering versatility in our models. Based on these criteria, we narrow down our choices to a few high-quality datasets that are widely adopted and encompass different math fields and complexity levels, such as GSM8K, MATH, AQuA, Camel, and TheoremQA. Furthermore, we notice a lack of coverage for college-level math knowledge, such as abstract algebra and formal logic, in existing datasets. To rectify this, we use GPT-4 to synthesize CoT rationales for questions in TheoremQA and create question-CoT pairs through Self-Instruct (Wang et al., 2023h), utilizing a few seed exemplars found online.

Hybrid CoT and PoT Rationales: Contrary to previous work (Yuan et al., 2023; Luo et al., 2023; Lee et al., 2023; Wang et al., 2023g) that focus on CoT, our dataset strategically combines both. This integration enhances the dataset’s versatility, catering to varying mathematical problem-solving approaches. However, most existing datasets provide limited program rationales, leading to an imbalance between CoT and PoT rationales. To fill the gap, we utilize GPT-4 to supplement the PoT rationales for selected datasets, including MATH, AQuA, GSM8K, and TheoremQA. We then filter these GPT-4 synthesized programs by comparing their executed results with human-annotated ground truth, which ensures the high quality of the added rationales.

Following these guidelines, our instruction dataset, detailed in Table 1, encompasses 260K (instruction, response) pairs, covering a wide range of core mathematical fields (arithmetic, algebra, probability, calculus, and geometry, etc.), including hybrid CoT and PoT rationales, and offering diversity in both language and difficulty levels. This attests to its high quality and unique characteristics.

3 Training Setup

We unify all the subsets in our MathInstruct to conform to the structure of an Alpaca-like instruction dataset (Taori et al., 2023). This standardization ensures that the fine-tuned models can process data consistently, regardless of the original dataset formats. We choose the open-source models Llama-2 (Touvron et al., 2023b) and Code Llama (Rozière et al., 2023) as our base models. We fine-tune these models including 7B, 13B, 34B, and 70B on MathInstruct, which allows us to validate our MathInstruct at multiple scales. We fine-tune all the models with Huggingface transformers library (Wolf et al., 2019). We use a learning rate of 2e-5 for the 7B and 13B models, and 1e-5 for the 34B and 70B models. We set the batch size at 128 and used a cosine scheduler with a 3% warm-up period for three epochs. To efficiently train the computationally intensive 34B and 70B models, we employ DeepSpeed training with ZeRO-3 stage (Rajbhandari et al., 2020).

4 Evaluation Setup

Our hybrid training enables models to solve problems using either the CoT or PoT approach. By default, the model provides the CoT solution. To switch to the PoT approach, one can add the trigger phrase “Let’s write a program to solve the problem” following the question.

Our preliminary evaluation reveals that PoT generally outperforms CoT, notably in open-form questions like GSM8K and MATH, as programmable solutions are better at solving complex mathematical and algorithmic reasoning procedures. However, PoT struggles with abstract reasoning scenarios such as commonsense reasoning, formal logic, and abstract algebra, particularly in the absence of built-in APIs. To further combine the power of both approaches, we introduce a simple hybrid decoding strategy: The model first attempts PoT prompting. If the program is not executable, we falls back to CoT prompting. This heuristic significantly enhances our model’s overall performance (see more discussions in subsection 3.4).

Experiments

We have selected diverse evaluation datasets (Table 2), encompassing a variety of in-domain and out-of-domain samples across diverse fields of mathematics, to assess the models’ capabilities in general mathematical reasoning.

For the in-domain datasets, we consider GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), AQuA-RAT (Ling et al., 2017), and NumGLUE (Mishra et al., 2022b). For the out-of-domain datasets, we choose SVAMP (Patel et al., 2021), Mathematics (Davies et al., 2021), SimulEq (Koncel-Kedziorski et al., 2016), SAT-Math (Zhong et al., 2023), and MMLU-Math (Hendrycks et al., 2021a). The wide selection of evaluation datasets includes math problems from elementary, high school, and college levels. Some of the datasets even include formal logic and commonsense reasoning. The choice of these datasets is to ensure a comprehensive evaluation of the models’ capabilities to generalize to unfamiliar situations and different math fields. The chosen evaluation datasets consist of both open-formed questions and multi-choice questions.

2 Baselines

We partition our baselines into the following four categories:

Closed-source LLMs: We consider 4 closed-source LLMs including GPT-4 (OpenAI, 2023), GPT-4 (Code Interpreter), PaLM-2 Unicorn (Anil et al., 2023), Claude-2 (Bai et al., 2022) and Codex (Chen et al., 2021). GPT-4, PaLM-2, and Claude-2 use CoT prompting while GPT-4 (Code Interpreter) and Codex use PoT prompting.

Llama Base: For the base models, we consider Llama-1/2 (Touvron et al., 2023a; b), Llama-2-Chat (Touvron et al., 2023b).

Coder Model: To compare with different coder models, we choose Code-Llama (Rozière et al., 2023), CodeT5+ (Wang et al., 2023i) and CodeGen (Nijkamp et al., 2023).

STEM Pre-training: We cover Galactica (Taylor et al., 2022) mainly to understand the performance of models specialized in STEM knowledge.

Instruction Tuning: We include Orca-Platypus (Mukherjee et al., 2023), Vicuna-1.5 (Zheng et al., 2023b), Tulu (Wang et al., 2023g), Platypus-2 (Lee et al., 2023) and Guanaco (Dettmers et al., 2023). We cover a wide spectrum of models trained with different types of datasets.

Dataset-Specific Tuning: We include both RFT (Yuan et al., 2023) and WizardMath (Luo et al., 2023), which specifically tune the models to adapt to GSM8K and MATH datasets. We include them to understand their generalization.

For most baselines, we choose CoT prompting to maximize their performance due to their incompetence in program generation. All the ‘Code Model’ use PoT prompting. For GSM8K, MATH, AQuA, and NumGLUE, we will evaluate both 8-shot in-context-learning and zero-shot setups to report the highest score. For SVAMP, Mathematics, SimulEq, SAT, and MMLU, we use 5-shot in-context-learning to maintain consistency with prior work (Wei et al., 2022b; Chen et al., 2023). Our few-shot exemplars are mostly taken from PHPhttps://github.com/chuanyang-Zheng/Progressive-Hint (Zheng et al., 2023a). For MAmmoTH and MAmmoTH-Coder, we always evaluate under 0-shot setting. For all models, we allow a maximum sequence length of 2048 tokens for decoding. For multiple-choice questions, if the generated answer lacks an option, we map it by re-prompting the model: “Please find the closest option to [generated answer]. The options are [options]”.

3 Main Results

We report our in-domain and out-of-domain results in Table 3 and Table 4 respectively. Overall, we can see that MAmmoTH and MAmmoTH-Coder are able to outperform the SoTA model at different scales. In general, the performance gain for OOD datasets is more significant than IND datasets. These results show us the potential of our models as a mathematical generalist. On several datasets, MAmmoTH-Coder-34B and MAmmoTH-70B are even surpassing closed-source LLMs.

From Table 3, we can observe that our main competitors for IND datasets are WizardMath (Luo et al., 2023) and Platypus (Lee et al., 2023). WizardMath’s training is heavily rooted in GSM8K and MATH datasets. Therefore, WizardMath’s results are highly competitive on these two datasets. However, the dataset-specific training can be detrimental to OOD datasets like AQuA. In contrast, Platypus fine-tunes LLMs on a wide range of text and math reasoning datasets. it improves the open-source SoTA on several datasets. Similarly, MAmmoTH can achieve universal improvement across the board. A major observation is that MAmmoTH is particularly strong at solving more complex math problems in MATH, where the gain of our model over WizardMath (open-source SoTA on MATH) can exceed 25% at different scales.

From Table 4, we can observe that our main competitor for OOD datasets is Platypus (Lee et al., 2023). Similar to in-domain results, Platypus is able to yield gains over the baseline models universally across the board, especially on the MMLU-Math dataset, which is tied with MAmmoTH-70B. It is worth noting that the performance gains of our model on OOD datasets are even more significant than on in-domain datasets. This demonstrates our models’ remarkable generalizability to unseen math problems. Notably, MAmmoTH-7B also boosts the CoT performance of WizardMath-7B greatly on MMLU-Math by 9%, which contains a substantial number of questions beyond the subjects we covered in our training dataset.

Comparison between Different Base Models. In our experiments, we experimented with both Llama-2 and Code-Llama as the base models. From the two tables, we can observe that Code-Llama is consistently better than Llama-2, especially on OOD datasets. The gap between MAmmoTH and MAmmoTH-Coder can even reach up to 5%. Surprisingly, the average performance on OOD datasets of MAmmoTH-Coder (34B) is actually higher than MAmmoTH (70B). We believe MAmmoTH-Coder benefits greatly from the continuous code training of Code-Llama, which not only enhances the PoT capabilities but also improves Llama’s general reasoning skills.

4 Ablation Study on Data Source

Ablation of the Data Source. In order to better understand what factors contribute to the great gain of MAmmoTH over existing baselines, we set up a group of control experiments in Figure 2. We study the following setups:

(1) MAmmoTH (MathInstruct- CoT): This experiment aims to understand how much our curated CoT data could improve the generalization over the SoTA model WizardMath (Luo et al., 2023) trained specifically on GSM + MATH. As can be seen, while sacrificing accuracy on GSM + MATH by 3%, our CoT subset fine-tuning improves the overall nine-dataset accuracy from 27% to 32%.

(2) MAmmoTH (MathInstruct- PoT): This experiment aims to understand the advantage of our PoT subset. As can be observed, our PoT subset fine-tuning can significantly improve the overall accuracy from 27% to 41%. This ablation reflects the importance of unlocking the program generation capabilities of our model.

(3) MAmmoTH (MathInstruct- Hybrid): We further combine CoT and PoT as the hybrid training data to achieve the best overall performance of 47.9%. This combined gain comes from two aspects:

The CoT subset helps maintain generic language-based reasoning skills to handle scenarios where PoT cannot handle well, e.g., abstract reasoning multi-choice questions in AQuA and MMLU.

The PoT subset can teach the model how to utilize Python APIs to solve complex math problems with high precision, e.g., the MATH problems requiring complex computation.

We put some case studies in Appendix B to demonstrate the respective advantages of PoT and CoT in solving different types of math problems. To summarize, we attribute our substantial gain to: 1) diverse data sources covering different math fields and complexity levels and 2) a hybrid of CoT & PoT instruction tuning and decoding strategy.

Given the diverse sources of MathInstruct used in training MAmmoTH, it is important to understand how each dataset contributes to the overall performance of the model. We focus on five significant subsets: GSM8K, MATH, Camel, AQuA and NumGLUE. We conduct an experiment gradually adding each dataset into training and compare the performance with the one fine-tuned on the whole MathInstruct. As we can see from Table 5, when the data is not very diverse in training at the beginning (e.g., GSM8K only), the overall generalization performance is very bad: the model only fits in-distribution data and struggles to answer questions beyond GSM questions. And when gradually adding other major subsets, besides seeing the improvements on its own test sets overall, we could observe MAmmoTH becomes a better math generalist.

These results underscore the significant impact of diverse data sources on MAmmoTH performance, a core aspect of making MAmmoTH a math generalist. The results also provide valuable insights for future data curation and collection efforts (e.g., we should always collect diverse data and avoid collecting only specific types of data).

To help understand the contribution of the 6 newly curated datasets as shown in Table 1, we remove them from MathInstruct, and train a model on the existing data. As shown in the last two rows of Table 5, our new curated data substantially improves the performance on many datasets and leads to a 9% overall increase, which reflects the importance of the NEWLY curated dataset.

Influence of Hybrid Decoding.

To demonstrate the effectiveness of the hybrid decoding method, we conduct an experiment as outlined in subsection 2.4. By default, we initially attempt the PoT decoding method for a given question. If it fails to generate an executable query, we then transition to the CoT decoding method. The performance of different decoding methods (CoT, PoT, and Hybrid) is shown in Table 7. This hybrid decoding improves performance on every test set, showcasing that our model can effectively leverage the strengths of both CoT and PoT decoding strategies.

Conclusion

In this paper, we propose a novel math instruction tuning approach to activate open-source LLMs’ mathematical reasoning capabilities. Through a comprehensive study, we show that our models can outperform the SoTA performance at different scales by a huge margin. Our models benefit massively from: 1) the broad coverage of different math fields and complexity levels, and 2) a hybrid of CoT and PoT training. Our instruction tuning dataset contains 260K samples, which makes fine-tuning highly affordable even for academic labs. Our work paves the road for future studies to activate LLMs’ core capabilities in specialized domains.

References

Appendix A Related Work

Our work builds upon the existing mathematical reasoning literature. Early on, mathematical reasoning is mostly focused on solving synthetic basic math problems like AddSub (Hosseini et al., 2014) and other arithmetic reasoning datasets (Koncel-Kedziorski et al., 2015; Roy & Roth, 2015; Patel et al., 2021). Later on, more difficult math word problem datasets (Cobbe et al., 2021; Amini et al., 2019; Ling et al., 2017; Hendrycks et al., 2021b) have been proposed to focus on addressing realistic math word problems. NumGLUE (Mishra et al., 2022b) and LiLA (Mishra et al., 2022a) compile the existing literature to build a more diversified dataset collection. However, these datasets are mostly focused on grade school math problems. To further test LLMs’ limits in addressing more complex math problems, MMLU (Hendrycks et al., 2021a) includes college math problems in its evaluation suite. More recently, (Chen et al., 2023; Wang et al., 2023e) have proposed to tackle more challenging college-level science and math problems. Our instruction tuning dataset is built upon existing work to include a diversified collection of math problems from different subfields.

A.2 Reasoning with Large Language Models

LLMs have demonstrated great capabilities to reason with the help of Chain-of-Thought prompting (Wei et al., 2022b; Kojima et al., 2022; Wang et al., 2023f). Suzgun et al. (2022) have shown that CoT can already surpass human performance on challenging BIG-Bench tasks. Later on, several other works (Drozdov et al., 2023; Zhou et al., 2023c; Nye et al., 2022; Wang et al., 2022a; 2023a; Li et al., 2023b; Wang et al., 2023d; Yu et al., 2023) also propose different approaches to utilize LLMs to solve reasoning tasks by allowing intermediate steps. ReAct Yao et al. (2023) proposes to leverage external tools like search engines to enhance LLM reasoning skills. Another trend is to enable LLMs’ capabilities to use programs as thought processes like PoT (Chen et al., 2022). Some follow-up works include self-critic (Gou et al., 2023), self-eval (Xie et al., 2023), plan-and-solve (Wang et al., 2023c). These methods propose to enhance LLMs’ capabilities to solve math problems with PoT. Self-critic (Gou et al., 2023) and self-eval (Xie et al., 2022) both adopt self-evaluation to enhance the robustness of the generated program. Plan-and-solve (Wang et al., 2023c) instead adopts more detailed planning instructions to help LLMs create a high-level reasoning plan. These methods all prove to bring decent improvements over PoT.

A.3 Instruction Tuning in Language Models

Instruction tuning is part of a line of work designed to “align” language models with more useful objectives and human preferences. The instruction tuning step is seen as a major step to activate LLMs’ certain capabilities to respond to human instructions. Previously, instruction tuning is mainly focused on enhancing LLMs’ general-purpose instruction following abilities. Since 2021, CrossFit (Ye et al., 2021) and NaturalInstruction (Wang et al., 2022b), FLAN (Wei et al., 2022a) and T0 (Sanh et al., 2022) are amongst the first wave of instruction tuning effort to understand LLMs’ generalization capabilities. Later on, FLAN-v2 (Chung et al., 2022; Longpre et al., 2023) have been proposed to understand the effect of scaling up the instruction datasets to understand its impact on model performance. These approaches mainly adopt human-annotated datasets to build the instruction following dataset. More recently, multiple works (Wang et al., 2023h; Xu et al., 2023; Peng et al., 2023; Zhou et al., 2023b; Wang et al., 2023g) propose to utilize synthetic instruction following data distilled from GPT-3/4 to align open-source LLMs. The most similar effort to ours is Platypus (Lee et al., 2023) which aims to utilize a domain-specialized dataset to construct a small-scale instruction following dataset to enhance LLMs’ reasoning capabilities.

Appendix B Case Study

We conduct a comparison between our PoT results vs. CoT results in Figure 3, Figure 4 and Figure 5. In the first example, even though PoT and CoT can both solve the problem, CoT gives a very tedious solution to derive the answer. Such solution is not only slow but also unstable. In the second and third case, we can further see the advantages of PoT over CoT by utilizing external tools and Python packages to greatly simplify the solution. Figure 6 shows some types of questions (especially the formal logic question) that are not easily handled by programs. In order to address these types of questions, CoT is a better choice.

Appendix C Limitations

Despite their training on a diverse set of mathematical rationale datasets, the MAmmoTH models might exhibit limitations when faced with problems outside their primary domain of expertise like mathematical analysis, complex analysis, graph theory, numerical analysis, etc. Thus, our models are not suitable for solving more complex problems in these fields. Also, they have not been trained with proof-type problems, thus their theorem-proving capability is also limited. In the future, we would like to expand the models’ skill set to cover more fields and theorem-proving problems.

There is also a risk of the MAmmoTH models generating potentially harmful, offensive, or biased content, especially if they are asked to answer questions beyond math. The MAmmoTH series could be misused for malicious purposes, such as spreading misinformation or probing sensitive topics. Developers should conduct safety testing and tuning tailored to their specific applications before deploying any MAmmoTH model. While we have made every effort to ensure the cleanliness and purity of our training data, we cannot guarantee absolute perfection. It is unlikely but not impossible that some inappropriate questions slipped through the curation process.

Future work may continue to explore how to further improve the robustness and generalizability of MAmmoTH in mathematical reasoning. For example, recent work identifies “sycophancy” and “Clever Hans effect” in reasoning: LLMs cannot maintain truthful solutions to reasoning tasks when challenged by the user’s absurdly invalid arguments and critiques (Wang et al., 2023b). Potential methods to improve the models’ reasoning robustness could involve the exploration of synthetic data intervention methods as explored in (Wei et al., 2023).