Reasoning with Language Model Prompting: A Survey

Introduction

Reasoning ability lies at the heart of human intelligence, yet in natural language processing (NLP), modern neural networks can hardly reason from what they are told or have already known (Duan et al., 2020; Wang et al., 2021; Bhargava and Ng, 2022). Fortunately, with the revolutionary development of pre-training (Brown et al., 2020; Chen et al., 2021; Chowdhery et al., 2022), scaling up the size of language models (LMs) has shown to confer a range of reasoning abilities, such as arithmetic (Wang et al., 2022e; Lewkowycz et al., 2022), commonsense (Jung et al., 2022; Liu et al., 2022b), symbolic (Zhou et al., 2023; Khot et al., 2023) reasoning. As shown in Figure 1, such abilities may be unlocked by prompting strategies Liu et al. (2022d) (e.g., chain-of-thought (CoT) prompting Wei et al. (2022b), generated knowledge prompting Liu et al. (2022c)), which can dramatically narrow the gap between human and machine intelligence. Likewise, a vast amount of work has been proposed in the NLP community; however, these approaches, scattered among various tasks, have not been systematically reviewed and analyzed.

Organization of This Survey: In this paper, we conduct the first survey of recent progress in reasoning with language model prompting. We first give some preliminaries on this direction (§2) and then propose to organize relevant works by taxonomy (§3). We further provide in-depth comparisons with discussion for insights (§4). To facilitate beginners who are interested in this field, we highlight some open resources (§5) as well as potential future directions (§6).

Preliminaries

In this section, we introduce preliminaries of reasoning with LM prompting. For standard prompting, given the reasoning question $\mathcal{Q}$, prompt $\mathcal{T}$ and parameterized probabilistic model $p_{\rm LM}$, we aim to maximize the likelihood of answer $\mathcal{A}$ as:

where $a_{i}$ and $|\mathcal{A}|$ denotes the $i$-th token and the length of the final answer respectively. For few-shot prompting, $\mathcal{T}$ is comprised of $\mathcal{K}$ exemplars of $(\mathcal{Q},\mathcal{A})$ pair. CoT approaches further add reasoning steps $\mathcal{C}$ into prompt where $\mathcal{T}=\{(\mathcal{Q}_{i},\mathcal{C}_{i},\mathcal{A}_{i})\}_{i=1}^{\mathcal{K}}$, thus Equation 1 can be reformed to:

where $p(\mathcal{C}\mid\mathcal{T},\mathcal{Q})$ and $p(\mathcal{A}\mid\mathcal{T},\mathcal{Q},\mathcal{C})$ are defined as follows:

with $c_{i}$ is one step of total $|\mathcal{C}|$ reasoning steps.

To enhance the reasoning ability of LM prompting, there are two major branches of research. The first one focuses on optimizing the reasoning strategy with prompting as shown in Figure 2, including prompt engineering (§3.1.1), process optimization (§3.1.2) and external engine (§3.1.3).

For prompt engineering (§3.1.1), many methods try to improve the quality of prompt $\mathcal{T}$, and we call those works single-stage methods, while others append $c_{i}$ into the context of $(\mathcal{T},\mathcal{Q})$ at each reasoning stage or design specific $\mathcal{T}_{c_{i}}$ for each $c_{i}$, and we regard those as multi-stage methods. Note that one stage here refers to one input-output process. For process optimization (§3.1.2), the simplest ways are to bring in an optimizer with parameters $\boldsymbol{\theta}$ to calibrate $\mathcal{C}$ when generating $\mathcal{A}$, and we call those works self-optimization methods. Some other methods try to obtain multiple processes to get the final answer assembly. We regard those works as ensemble-optimization methods. Moreover, the overall optimization process can be iteratively integrated with fine-tuning the $p_{\rm LM}$ on generated triplet $(\mathcal{Q},\mathcal{C},\mathcal{A})$, which are regarded as iterative-optimization methods. Besides, some works leverage external reasoning engines (§3.1.3) to produce $\mathcal{T}$, to directly execute $\mathcal{C}$ or by implanting tool API calls in $\mathcal{C}$ for reasoning.

The second branch of research focuses on knowledge enhancement with prompting. Note that rich implicit “modeledge” Han et al. (2021) in LMs can generate knowledge or rationales as knowledge-informed prompt $\mathcal{T}$ (§3.2.1). Meanwhile, explicit knowledge in external resources can also be leveraged and retrieved as knowledgeable prompts to enhance reasoning (§3.2.2).

Taxonomy of Methods

In this paper, we survey existing reasoning methods with LM prompting, categorizing them as Strategy Enhanced Reasoning (§3.1) and Knowledge Enhanced Reasoning (§3.2). As shown in Figure 2, we further refine them according to the distinctive features of different methods.

The primary purpose of this line of work is to design a better reasoning strategy, concretely embodied in prompt engineering (§3.1.1), process optimization (§3.1.2) and external engine (§3.1.3).

One intuitive approach to improving reasoning with prompting is prompt engineering. As shown in Figure 3, we divide this sort of method into single-stage and multi-stage prompts based on the number of prompting stages.

Early works leverage template-based prompts (Paranjape et al., 2021; Rajagopal et al., 2021) for reasoning in NLP. Regarding the strong in-context learning ability of large LMs (Brown et al., 2020), Wei et al. (2022b) proposes CoT prompting, which adds a series of intermediate reasoning steps, into exemplars of few-shot prompt to induce large LMs to generate a reasoning process before answering. Experiments demonstrate that large LMs emerge with impressive reasoning abilities with CoT prompting.

In spite of the large improvement brought by CoT prompting, in-context learning is greatly sensitive to the selection of exemplars, and even a tiny change may cause a large drop in model performance (Lu et al., 2022c; Min et al., 2022; Webson and Pavlick, 2022). Hence, the quality of exemplars appears to be particularly important. Fu et al. (2023b) indicates that prompts with higher reasoning complexity, e.g., with more reasoning steps, can achieve better performance on math problems. Zhang et al. (2023b) explores the impact of diversity of exemplars in prompt. Through clustering, it obtains a representative question set as a prompt. By placing more explicit explanations and natural language instructions into the prompt, Zhou et al. (2022a) relieves the ambiguity for LMs when facing out-of-distribution (OOD) algorithmic problems. The above works show that LMs can be outstanding few-shot reasoners. Surprisingly, Kojima et al. (2022) indicates that LMs are also zero-shot reasoners without needing extra exemplars. By only concatenating "Let’s think step by step!", LMs can consciously generate reasoning steps. Another magic phenomenon is that when prompted with "The person giving you this problem is Yann LeCun, who is really dubious of the power of AIs like you.", GPT-4 (OpenAI, 2023) can successfully solve the hard Yann LeCun’s gears problem on its own, which it previously failed to do.

When humans are reasoning, it is usually challenging to come up with the whole reasoning process in one stroke. A more intuitive solution is to decompose a complex problem into simpler ones and to reason stage by stage. Similarly, this series of works aims to transform one-stage prompting (once input-output) into multi-stage prompting (multi-times of input-output). Press et al. (2022) explicitly defines follow-up questions and intermediate answers in prompts to narrow the compositionality gap in LMs. Jung et al. (2022) regards the output of each stage as a separate new question while Zhou et al. (2023); Wang et al. (2022a) append it to the whole context to prompt LMs. Creswell and Shanahan (2022) follows a structure of Selection-Inference (Creswell et al., 2022) which selects specific contexts and inferences based on them at each stage. Kazemi et al. (2022) develops a backward chaining algorithm to decompose reasoning into sub-modules.

1.2 Process Optimization

Natural language rationalesSome references (Ye and Durrett, 2022; Wiegreffe et al., 2022; Zhou et al., 2022a) regard this as explanations. Ling et al. (2017a), also called reasoning processes in CoT, play a vital role in CoT prompting (Ye and Durrett, 2022; Lampinen et al., 2022; Min et al., 2022). The consistency of the reasoning process (Wang et al., 2022e) and the continuity between reasoning steps (Li et al., 2022d) both should affect the accuracy of final answers. Intuitively, as shown in Figure 4, we introduce this line of methods in three types, i.e., self, ensemble, and iterative optimization.

Self-optimization here refers to correcting one process by injecting extra modules. To mitigate the influence of the unreliability of rationales, Ye and Durrett (2022) utilizes a calibrator to tune the probabilities of a prediction based on the score which reflects the factuality of a rationale. During free-text rationales generation, Wiegreffe et al. (2022) fine-tunes a sequence-to-sequence model as a filter to predict whether the rationale is acceptable.

Due to the limitation of only one reasoning path, the following works rely on ensemble calibration among multiple processes. Wang et al. (2022e) introduces sampling strategies (Ackley et al., 1985; Fan et al., 2018) commonly used in natural language generation to obtain multiple reasoning processes and generate the most consistent answer by majority vote. Based on the motivation of when a reasoning process reaches a wrong answer, not all the steps may undertake the final incorrectness, Li et al. (2022d) proposes a step-aware voting verifier to score each reasoning path. When disorientated majority processes overwhelm reasonable minority processes, the step-aware voting verifier can alleviate the limitation of vanilla majority vote (Wang et al., 2022e). Besides, Wang et al. (2022d) empirically observes that decoder sampling in the output space is the key to robustly improving performance because of the brittleness of manual prompt engineering.

Note that LMs can achieve excellent performance in few-shot (Wei et al., 2022b) or zero-shot (Kojima et al., 2022) manners with prompts, another paradigm is to calibrate reasoning processes iteratively with LM finetuning. Specifically, iterative-optimization-based methods try to repeat the process of prompting LMs to generate reasoning processes and use the instances with generated reasoning processes to finetune themselves. Zelikman et al. (2022) initiates with a small set of exemplars to push LMs to produce reasoning steps and answers themselves. Questions and reasoning steps with the correct answers will be directly added to the dataset for finetuning. Incorrect ones will be fed into the model again by being tagged on a hint that labels the correct answer. Compared with Zelikman et al. (2022), Huang et al. (2022) does not need gold labels during self-teaching. Following Wang et al. (2022e), it generates multiple reasoning processes and finetunes the most consistent self-generated answers. Shinn et al. (2023); Madaan et al. (2023); Paul et al. (2023) uncover the emergent ability of LLMs to self-reflect, by continuously correcting reasoning chains through iterative self-reflection.

1.3 External Engine

When reasoning with LM prompting, the models should have the ability of semantic understanding (e.g., questions) and complex reasoning (e.g., by generating reasoning processes); however, we cannot have both fish and bear’s paw (Hendrycks et al., 2021; Nogueira et al., 2021; Lewkowycz et al., 2022). To tear up the obstacle, external reasoning engines lend a helping hand to LMs (see Figure 5).

Given a physical reasoning question, Liu et al. (2023) utilizes a computational physics engine (Todorov et al., 2012) to simulate the physical process. The simulation results are treated as prompts to help LMs reason, making up for the lack of physical knowledge in LMs.

With the emergence of LMs of code (Chen et al., 2021; Xu et al., 2022), collaborating LMs and codes to tackle specific tasks has recently sprung up (Wang et al., 2022c; Cheng et al., 2022; Wu et al., 2022b). Note that programs yield advantage behaviors in robustness and interpretability and can better illustrate complex structures and deduct complex calculations. Intuitively, Madaan et al. (2022) reframes structured commonsense reasoning tasks as code generation tasks, replacing the natural language with python class code to represent structured graph both in few-shot prompts and LM outputs. Gao et al. (2022) decomposes solution steps from LMs to a programmatic runtime and remains the only learning task for the LMs. In few-shot prompts and LM outputs, the reasoning processes are replaced by a mixture of natural and programming language, where natural language is treated as annotations to aid the generation of the program. Similar to Gao et al. (2022), Chen et al. (2022b) proposes program of thoughts (PoT) prompting which disentangling computation from reasoning. The main difference is that it also puts forward a zero-shot format of PoT prompting.

Despite possessing remarkable generation and decision-making capabilities, LLMs struggle with some basic functionalities where much simpler and smaller tools excel (Qin et al., 2023). Building on this insight, Schick et al. (2023) trains models by integrating the usage of various tools, including calculators, Q&A systems, search engines and etc. Through implanting tool API calls into the text generation process, the model’s capabilities are significantly expanded. Paranjape et al. (2023) designs the tool-use for LLMs as an automated schema, which eliminates the need for hand-crafting task-specific demonstrations and carefully scripted interleaving of model generations with tool use. Lu et al. (2023a) harnesses the powerful decision-making abilities of LLMs, enabling them to combine various external tools to tackle compositional reasoning tasks.

2 Knowledge Enhanced Reasoning

As noted in Manning (2022), knowledge plays a vital role in AI reasoning systems. Knowledge enhanced methods aim to prompt LMs with implicit (§3.2.1) or explicit (§3.2.2) knowledge to assist in reasoning (see Figure 6).

Researchers have shown that LMs contain considerable implicit knowledge (Davison et al., 2019; Petroni et al., 2019; Jiang et al., 2020). The following works try to induce such “modeledge” as knowledge-informed prompts for reasoning.

Liu et al. (2022c) applies GPT-3 (Brown et al., 2020) with few-shot prompting to generate knowledge and prompts the downstream LM. Liu et al. (2022b) draws support from reinforcement learning (Schulman et al., 2017) to further calibrate the knowledge. Different from the approaches using few-shot prompting in the knowledge generation stage, Sun et al. (2022) proposes a two-stage generative prompting which additionally includes answer generation prompts. Other works (Li et al., 2022b; Wang et al., 2023; Shridhar et al., 2022; Magister et al., 2022; Ho et al., 2022) follow knowledge distillation that generates reasoning samples by prompting a larger LM and teaches smaller LMs.

2.2 Explicit Knowledge

Although large LMs have shown strong generation ability (Wiegreffe et al., 2022; Li et al., 2022b; Wang et al., 2023), they still have the tendency to hallucinate facts (Rohrbach et al., 2018) and generate inconsistent knowledge (Liu et al., 2022b). Recent works show that retrieving prompts for in-context learning is a nice means to achieve good performance (Liu et al., 2022a; Rubin et al., 2022).

Due to the instability of common retrieval approaches to measure the similarity of structured information, Lu et al. (2023b) proposes a dynamic prompt retrieval method based on policy gradient strategy, without brute-force searching. He et al. (2023) retrieves relevant knowledge based on the reasoning steps of CoT to provide more faithful explanations. Trivedi et al. (2022) augments CoT prompting by persistently retrieving wiki documents for open-domain knowledge-intensive tasks that require complex multi-step reasoning.

Comparison and Discussion

Table 1 shows four comparison scopes of different methods. We further illustrate the performance comparison of LMs with different scales on GSM8K (Cobbe et al., 2021) of arithmetic reasoning in Figure 7. Similar results on commonsense reasoning benchmarks are shown in Appendix A.3.

Wei et al. (2022b) systematically demonstrates that few-shot prompting performs better in almost all tasks as model scale increases, which can be explained by the fact that LMs with larger model size contain more implicit knowledge for reasoning (Liang et al., 2022b). Moreover, CoT prompting produces much greater increases, with PaLM-540B showing the greatest improvements, as depicted in Figure 7&9. However, when the model scale declines to less than 100B, CoT prompting will yield no performance gain and may even be detrimental. Thus, CoT prompting elicits an emergent ability of model scale (Wei et al., 2022a). One possibility is that when the stored knowledge reaches a certain level, the reasoning ability of LMs undergoes a qualitative change from quantitative change, leading to the emergence of emergent capabilities. Additionally, Srivastava et al. (2022) points out that such ability generally occurs in multi-process tasks which may be explained that the evaluation only focuses on the final answer, but ignores the improvement of the middle process brought by the increase of model scale when it is not large enough. Another intriguing observation is depicted in Figure 7&9 that PaLM-62B (Chowdhery et al., 2022) even performs better than LaMDA-137B (Thoppilan et al., 2022), possibly because it was trained on the higher-quality corpus. This phenomenon leads us to speculate that such emergent ability is not solely determined by model parameter scale but also related to the quality of pre-training data.

Notably, Figure 7&9 also illustrate that holding the same parameter scale, Codex (Chen et al., 2021) outperforms GPT-3 significantlyNote that Codex and GPT-3 in our paper refer to code-davinci-002 and text-davinci-002 respectively in OpenAI API., even though the major difference between them is the training corpus (Codex is a GPT-3 variant training on code). This phenomenon can also be inspected in recent works (Zhou et al., 2023; Li et al., 2022d; Zhang et al., 2023b; Madaan et al., 2022; Liang et al., 2022b), indicating that pre-training on code branch not only enables the ability of code generation/understanding but may also trigger the reasoning ability with CoT. The exact cause is still elusive, but one intuition is that code is a form of text more similar to reasoning, thinking about procedure-oriented programming is analogous to solving problems step by step, and object-oriented programming is analogous to decomposing complex tasks into simpler ones (Yao et al., 2022). In addition, Prystawski and Goodman (2023) finds that CoT is beneficial only when the training data exhibits local structure. Due to its expertise in reasoning by navigating through multiple variables, CoT excels in deducing the relationship between two variables that have seldom been encountered in the same context. However, it may not perform better than simple statistical estimators when it comes to reasoning with variables that frequently co-occur in the training data.

2 Comparison of Prompts

Table 1 shows the comparison of different methods of reasoning with LM prompting. There are three main sources of prompts for existing methods: 1) Manual construction is suitable for template-based prompts and few-shot prompting where the prompt is uncomplicated. 2) LM Generated prompt makes up for the shortcomings of manual construction prompt. It can customize specific rationales for each question and provide sufficient knowledge with the prompt for fine-tuning or self-training. 3) Retrieval-based prompt often relies on well-annotated external resources (e.g., Wikipedia) and consumes expensive information retrieval, but it can alleviate the unstable issue of the generation.

We observe that no matter how prompt is produced, CoT prompting only works on large LMs. Smaller LMs work by fine-tuning with rationales. Combined with the empirical conclusion in Ye and Durrett (2022), these phenomena reveal that high-quality reasoning rationales contained in the input context are the keys for reasoning with LM prompting. Although some works have attempted to explore the in-context learning ability on large LMs (Xie et al., 2022; Min et al., 2022; Akyürek et al., 2022), the reason why CoT prompting can succeed is still intriguing to the community and not well-understood. One possible hypothesis is that CoT is a magical side product of training on code that can be unlocked by prompt. Note that exemplars containing CoT in few-shot prompts can be viewed as a kind of instruction that arouses the reasoning ability hidden in large LMs. Chung et al. (2022) verifies the similar result using CoT in instruction fine-tuning to advance model performance further. In fact, in-context learning can be seen as an intermediate state of evolution from general prompts to human-readable instructions. Following this trend, prompts may grow into an essential interface of human-machine interaction.

Benchmarks and Resources

In this section, we will give a brief overview of reasoning benchmarks and tasks. More details of datasets, as well as reasoning with ChatGPT can be found in Appendix A.4 and A.5.

Arithmetic reasoning, also referred to as mathematical reasoning, is the ability to perform reasoning on math word problems (MWP). Early works on this task (Hosseini et al., 2014; Kushman et al., 2014; Roy et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015) focus on relatively small datasets consisting of grade school single-step or multi-step MWP. Later works increase in complexity, difficulty, and scale. Most recently, Mishra et al. (2022a) extends existing datasets to construct a unified benchmark concerning mathematical abilities, language diversity, and external knowledge.

Commonsense knowledge and commonsense reasoning are some of the major issues in machine intelligence (Storks et al., 2019; Bhargava and Ng, 2022). When answering a question, people often draw upon their rich world knowledge. For LMs, the major challenge of performing commonsense reasoning lies in how to involve physical and human interactions under the presumption of general background knowledge (Bhargava and Ng, 2022). Many benchmark datasets and tasks (Clark et al., 2018; Mihaylov et al., 2018; Talmor et al., 2019; Bisk et al., 2020; Geva et al., 2021) are designed, and the most widely used benchmark today is CommonsenseQA (Talmor et al., 2019).

Common forms of logical reasoning include deductive reasoning and inductive reasoning, deductive reasoning and abductive reasoning Sinha et al. (2019); Bao et al. (2022); Young et al. (2022); Bao et al. (2023). Deductive reasoning is performed by going from general information to specific conclusions. Typical datasets in this field consist of synthetic rule bases plus derived conclusions (Clark et al., 2020; Tafjord et al., 2021). Dalvi et al. (2021) creatively proposes a dataset containing multi-step entailment trees together with rules and conclusions. As opposed to deductive reasoning, inductive reasoning aims to draw conclusions by going from specific observations to general principles (Yang et al., 2022c).

Symbolic reasoning here only refers to a narrow collection of simple tasks that test a diverse set of symbolic manipulation functions, rather than symbolic AI, which is a more general concept. Typical symbolic reasoning tasks include last letter concatenation, reverse list and coin flip (Wei et al., 2022b).

Except for textual modality, humans utilize the information available across different modalities when performing reasoning. To this end, multimodal reasoning benchmarks (Zellers et al., 2019; Park et al., 2020; Dong et al., 2022) are presented to narrow this gap. Recently, Lu et al. (2022a) presents ScienceQA, a large-scale multimodal multiple choice dataset that consists of diverse questions of science topics with corresponding answers and explanations. Zhang et al. (2023a) proposes a new task of multimodal analogical reasoning over knowledge graphs.

2 Resources

Thanks to the open-source spirit of the NLP community, numerous resources are publicly available alongside papers for researchers to experiment with. ThoughtSource is a central, open resource and community around data and tools related to CoT reasoning in large language modelshttps://github.com/OpenBioLink/ThoughtSource. The LangChain library is designed to help developers build applications using LLMs combined with other sources of computation or knowledgehttps://github.com/hwchase17/langchain. $\lambda$prompt allows for building a complete large LM-based prompt machines, including ones that self-edit to correct and even self-write their own execution codehttps://github.com/approximatelabs/lambdaprompt. Recently, Ou et al. (2023) develops EasyInstruct, a Python package for instructing LLMs like GPT-3 in research experiments. A test case for reasoning using EasyInstruct can be found in Appendix A.6.

Future Directions

LMs have been demonstrated to have emergent zero-shot learning and reasoning abilities (Wei et al., 2022b; Wang et al., 2022e; Wei et al., 2022a). To uncover the mystery of such a success, many researchers have empirically explored the role of in-context learning (Ye and Durrett, 2022; Liu et al., 2022a) and rationales (Min et al., 2022; Lampinen et al., 2022). Some works concentrate on unraveling the principles of machine learning algorithm behind in-context learning (Xie et al., 2022; Akyürek et al., 2023) and prompt tuning (Wei et al., 2021). Another line of works tries to investigate the architecture of Transformers via knowledge neurons Dai et al. (2022) or skill neurons Wang et al. (2022b). More recent works (Wang et al., 2022c; Madaan et al., 2022) demonstrate that pre-trained LMs of code are better handling structured commonsense reasoning and prediction than LMs of natural language, even when the downstream task does not involve source code at all. However, the code-based pre-training (or re-structured pre-training Yuan and Liu (2022)) still has limitations since it has to utilize off-the-shelf structure (e.g., existing aligned corpus or build from scratch via syntax tree or AMR Banarescu et al. (2013)) to reformulate plain texts. Thus, the truth may be close, and we argue that it is beneficial to study the theoretical principle to advocate for a transparent view of reasoning with LM prompting and further decipher the dark matter of intelligence by highlighting the counterintuitive continuum across language, knowledge, and reasoningKeynote talk on ACL 2022 entitled “2082: An ACL Odyssey: The Dark Matter of Intelligence and Language”.. Note that reasoning in NLP has the potential advantages of complex problem-solving and should better utilize dark matters in cross-disciplines (e.g., Theory of Mind Sap et al. (2022); Moghaddam and Honey (2023); Zhou et al. (2022b); Shapira et al. (2023)).

To be noted, existing methods mainly depend on large LMs, which may consume high computing resources. Regarding practicality, it is necessary to study reasoning with small LMs or develop efficient reasoning methodologies which pay attention to carbon emission and energy usage during model training and inference Xu et al. (2021). One feasible way may be developing models that can enable generalization across a range of evaluation scenarios such as Flan-T5 (Chung et al., 2022), which finetune both with and without exemplars (i.e., zero-shot and few-shot) and with and without CoT. Recently, an intuitive approach has been proposed to transfer the reasoning capabilities of large LMs to smaller LMs via knowledge distillation (Shridhar et al., 2022; Magister et al., 2022; Ho et al., 2022). Other promising directions include retrieval augmentation (Li et al., 2022a), model editing (Cao et al., 2021; Mitchell et al., 2022a, b; Cheng et al., 2023), delta-tuning (He et al., 2022; Mao et al., 2022; Pal et al., 2022; Ding et al., 2022), etc.

Robustness, faithfulness and interpretability have long been pursued by the field of deep learning, especially in tasks that require strong logic, like reasoning. Shaikh et al. (2022) demonstrates that zero-shot CoT will produce undesirable toxicity and biases, indicating the necessity of robust, faithful and interpretable reasoning. Creswell and Shanahan (2022) leverages a selection-inference Creswell et al. (2022) multi-stage architecture for faithful reasoning, but there is still a lack of interpretability within each stage. Code-based works Madaan et al. (2022); Gao et al. (2022); Chen et al. (2022b) reach robustness and interpretability to some extent, but they have the aid of an external engine. There is still a long way to achieve true robustness, faithfulness and interpretability with LMs. Fortunately, Dohan et al. (2022) provides a new idea for utilizing a probabilistic program to tackle various reasoning problems. Other solutions may be neural-symbolic approaches Du et al. (2021); Li et al. (2022c); Ouyang et al. (2021); Feng et al. (2022) or human feedback (Ouyang et al., 2022).

Textual reasoning is restricted to what can be expressed through natural language. A more promising direction is multimodal reasoning regarding the information diversity of the real world of human reasoning. Lu et al. (2022a) generates CoT when dealing with a multimodal dataset; however, it simply extracts textual descriptions from images, and it is still a textual reasoning task indeed. Intuitively, it is beneficial to integrate multimodal information into reasoning processes such as images, audio, videos, etc., and design a unified multimodal CoT. Apart from unified multimodal models, it is also promising to model chains (Wu et al., 2022a) to conduct interactive reasoning among models of different modalities. Besides, Sap et al. (2022) shows that one of today’s largest language models (GPT-3 Brown et al. (2020)) lacks the skill to reason about the mental states, and reactions of all people involved. Thus, interactive reasoning methodologies should be noted by inspiring from other domains (e.g., Cognitive Science Hollenstein et al. (2019), Social Intelligence Krishna et al. (2022)), which may have potential guidance for reasoning in NLP since only increasing the scale of LMs is likely not the most effective way to create AI systems.

Generalization is one of the most significant symbols of models to attain true reasoning abilities. Given a reasoning task, we hope LMs can handle not only the problem itself but solve a group of similar reasoning tasks (not seen during training). Zhou et al. (2022a); Anil et al. (2022) explore the OOD problem on the length of reasoning questions, but the true generalization is still far from satisfactory. Meanwhile, Kejriwal et al. (2022) highlights that more comprehensive evaluation methods grounded in theory (e.g., naive physics Gardin and Meltzer (1989) and commonsense psychology Gordon and Hobbs (2004)) should be proposed. We argue that the generalizable reasoning may be closely related to analogy reasoning (Chen et al., 2022a; Webb et al., 2022), causal reasoning (Feder et al., 2022), compositional reasoning (Yang et al., 2022a), etc.

Conclusion and Vision

In this paper, we provide a review of reasoning with language model prompting, including comprehensive comparisons, and several research directions. In the future, we envision a more potent synergy between the methodologies from the NLP and other domains and hope sophisticated and efficient LM prompting models will increasingly contribute to improving reasoning performance.

Acknowledgment

We would like to express gratitude to the anonymous reviewers for their kind comments. We thank Yicong for correcting an inappropriate Equation in the paper. This work was supported by the National Natural Science Foundation of China (No.62206246 and U19B2027), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G), CAAI-Huawei MindSpore Open Fund, and NUS-NCS Joint Laboratory (A-0008542-00-00). The computing resources were supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

Limitations

In this study, we provide a survey of reasoning with language model prompting. We discuss the related surveys in Appendix A.1 and will continue adding more related approaches with more detailed analysis. Despite our best efforts, there may be still some limitations that remain in this paper.

Due to the page limit, we may miss some important references and cannot afford all the technical details. We mainly review the cutting-edge methods within two years (mostly in 2022) in §3, mainly from the ACL, EMNLP, NAACL, NeurIPS, ICLR, arXiv, etc., and we will continue to pay attention to and supplement the latest works.

Most of the reasoning benchmarks mentioned in §5 are gathered and categorized from the experimental part of mainstream works. The definition and boundary of each task may not be accurate enough. Besides, our work may miss some kind of reasoning tasks such as reasoning with generics Allaway et al. (2022), default inheritance reasoning Brewka (1987), non-monotonic reasoning Ginsberg (1987) in NLP, and will try our best to fulfill this gap.

We give detailed comparisons and discussions of language models and prompts in §4, and list some promising future directions in §6. All the conclusions are proposed and further speculated upon empirical analysis of existing works which may not be macroscopic enough. As the field evolves faster, we will update the latest opinions timely.

References

Appendix A Appendix

As this area is relatively nascent, only a few surveys exist. Closest to our work, Huang and Chang (2022) gives a survey towards reasoning with large language models. Dong et al. (2023) organizes and discusses the advanced techniques of in-context learning. Zhao et al. (2023) reviews the latest advancements in Large Language Models (LLMs) and delves into the unresolved challenges that will shape future developments. Bhargava and Ng (2022) covers methods for commonsense knowledge reasoning and generation with pre-trained LMs. Lu et al. (2022b) reviews the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. Liang et al. (2022a) surveys knowledge graph reasoning tracing from static to temporal and then to multi-modal knowledge graphs. Mialon et al. (2023) reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. Hamilton et al. (2022) conducts a survey of studies implementing neural-symbolic (NeSy) NLP approaches for reasoning and so on. Guo et al. (2022) provides a survey of several popular works dealing with uncertainty reasoning. Qin et al. (2023) concentrates on the leverage of external tools by LLMs which is also called Tool Learning. Other surveys focusing on prompt learning Liu et al. (2022d) or pre-trained models Qiu et al. (2020); Du et al. (2022) are also related to our work.

Unlike those surveys, in this paper, we conduct a review of reasoning with LM prompting, hoping to systematically understand the methodologies, compare different methods and inspire new ideas.

A.2 Taxonomy of Methods and Tasks

We list the complete taxonomy of reasoning with language model prompting from methods and tasks in Figure 8.

A.3 Performance Comparison of LMs with Different Scales

To show the generalization of discussions in §4.1 on different reasoning tasks, we additionally show the performance comparison of LMs with different scales on CommonsenseQA (Talmor et al., 2019) of commonsense reasoning in Figure 9.

A.4 Detailed Information of Reasoning Benchmarks

In § 5, we give a brief overview on benchmarks and tasks requiring various reasoning skills. We list more benchmarks and show their key statistics in Table 2. Apart from the above-mentioned specific reasoning tasks in § 5, there are some benchmarks (Lake and Baroni, 2017; Srivastava et al., 2022; Yu et al., 2022) that can evaluate the model’s more diverse and generalized reasoning capabilities, which can also be included in the category of reasoning tasks.

A.5 Reasoning with ChatGPT

Recently, OpenAI (2022) develops ChatGPT, an AI chatbot system that has attracted tremendous users. ChatGPT is trained on a massive dataset of text and is able to generate human-like responses to a wide variety of prompts, the promising approach for which is called Reinforcement Learning from Human Feedback (Ouyang et al., 2022). The backbone of ChatGPT is from a model in the GPT-3.5 large LM serieshttps://beta.openai.com/docs/model-index-for-researchers. In order to savor the reasoning ability of large LMs more realistically, we conduct some case tests on ChatGPT. Concretely, we pick out a piece of data from GSM8K (Cobbe et al., 2021), CommonsenseQA (Talmor et al., 2019) and Last Letter Concatenation (Wei et al., 2022b) which respectively represent arithmetic reasoning, commonsense reasoning, and symbolic reasoning. Then we test each of the selected data on ChatGPT directly. Results can be seen in Figure 10-12.

Figure 10 shows that given a math problem in GSM8K (Cobbe et al., 2021), ChatGPT outputs a reasoning process and a correct answer without in-context exemplars. This blazes its powerful arithmetic reasoning ability. The reasoning process has the same format as the gold label in GSM8K, indicating that GSM8K may be contained in the training corpus of ChatGPT.

In Figure 11, we test ChatGPT on a piece of data in CommsonsenseQA (Talmor et al., 2019). It not only gives the correct answer but additionally details why each option is right or wrong, which does not appear in the gold label of the dataset. This demonstrates the strong commonsense reasoning ability of ChatGPT.

Figure 12 is a case in Last Letter Concatenation (Wei et al., 2022b). We observe that although ChatGPT gives a detailed and accurate description of last letter concatenation, it fails to answer the given question, showing that its symbolic reasoning capability is not as excellent as the above two.

A.6 Reasoning using EasyInstruct