V-STaR: Training Verifiers for Self-Taught Reasoners

Introduction

Learning to recognize and correct mistakes is a feature of human intelligence (Metcalfe, 2017). When dealing with complex tasks, such as coding or solving a math problem, we can recognize errors in reasoning and explore alternative paths to a solution. To improve the reasoning performance of LLMs, several approaches exploit the ability of LLMs to produce solutions and check the correctness of these solutions during training, for example, using test cases for code generation. These self-improvement approaches, such as STaR (Zelikman et al., 2022), RFT (Yuan et al., 2023), and ReSTEM (Singh et al., 2023), improve LLMs by fine-tuning them on their self-generated solutions and optionally iteratively running this process. However, all these approaches are data-inefficient in that they use only correct solutions, and discard incorrect solutions, which is often a large portion of model-generated solutions, especially for challenging reasoning tasks.

Orthogonal to self-improvement, another promising direction to improve LLM reasoning is to use learned LLM verifiers at test-time (Cobbe et al., 2021; Wang et al., 2023b). Specifically, the LLM generates multiple candidate solutions at test time and the verifier is used for ranking these solutions and selecting the best one. Such verifiers are trained by fine-tuning an LLM on a dataset of solutions generated from a frozen LLM, labeled with either final correctness (Cobbe et al., 2021) or step-by-step human-generated annotations (Lightman et al., 2024).

To combine the best of both worlds, we propose Verification for Self-Taught Reasoners (V-STaR). The key idea in V-STaR is to utilize both the correct and incorrect LLM-generated solutions during the self-improvement process to train a verifier using DPO (Rafailov et al., 2023), in addition to training a LLM as generator using correct solutions. V-STaR iteratively improves the reasoning ability of a generator by generating solutions for training problems. Correct solutions are added to the generator’s training data, while all generated solutions, labeled with their correctness, are used to train a verifier (Fig. 1). This iterative process yields progressively improved generators, trained on augmented data, and leads to higher quality completions and more challenging negative examples for the verifier. At test time, the verifier ranks multiple candidate solutions from the generator and selects the best one.

We empirically evaluate V-STaR on math reasoning using GSM8K (Cobbe et al., 2021) and a subset of MATH (Hendrycks et al., 2021), and on code-generation using MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). Fine-tuning LLaMA2 (Touvron et al., 2023) and CodeLLaMA (Rozière et al., 2023), we compare V-STaR to other self-improvement and verification-based methods, as well a non-iterative V-STaR baseline that uses the same number of generation samples to bootstrap a generator and verifier. We observe 6% to 17% absolute improvement in test accuracy over best performing self-improvement and verification-based methods for math reasoning, and 4% to 12% in code generation. Notably, in terms of performance, 7B models fine-tuned with V-STaR surpass base LLaMA2 70B (8-shot) on GSM8K, and nearly match CodeLLaMA 34B (zero-shot) on HumanEval. Our contributions are:

We propose V-STaR, which utilizes iteratively generated correct and incorrect solutions to train a better generator and verifier. As shown in Fig. 2, V-STaR substantially outperforms existing approaches for math reasoning and code generation tasks.

As a secondary contribution, we find DPO to be more effective for training verifiers than the prevalent approach by Cobbe et al. (2021). We also propose a formula for Verifier$@k$ (§4.4), akin to Pass$@k$, to reliably evaluate performance with test-time verification.

Preliminaries

Given a pretrained language model $G$ and the original training data of a task $\mathcal{D}_{\text{SFT}}=\{(x_{1},y_{1}),(x_{2},y_{2}),\cdots(x_{N},y_{N})\}$, where $x$ is typically a description of a problem and $y$ is the solution, such as chain-of-thought rationale or generated code. The de facto approach for such tasks with causal language models is supervised fine-tuning (SFT) with the negative log-likelihood objective on the training data:

where $G$ is also referred to as generator in reasoning tasks. LLMs can be used to generate high quality chain-of-thought rationales or solutions for a range of tasks. This observation has motivated using correct generations from the model itself to bootstrap problem solving (Zelikman et al., 2022; Singh et al., 2023; Yuan et al., 2023).

Self-Taught Reasoner (STaR; Zelikman et al., 2022) corresponds to an iterative approach where a language model improves itself using correctness feedback. In each iteration, we generate one solution $\hat{y}$ using greedy decoding with the language model generator $G$ for each problem $x$ in training dataset $\mathcal{D}$. Having access to test cases or ground truth answers, generated solutions can be checked for their binary correctness label $z$ by:

A completion $\hat{y}$ is labeled correct if it has the same final answer as the ground truth answer for math problems, or if it passes all the test cases for code generation problems. Only correct solutions ($z=1$) are included in the dataset at iteration $j$ where $\mathcal{D}_{j}=\{(x_{1},\hat{y}_{1}),(x_{2},\hat{y}_{2}),\cdots(x_{N},\hat{y}_{N})\}$. Then, the generator is fine-tuned on this new dataset using (Eq. 1) where $\mathcal{D}_{\text{SFT}}=\mathcal{D}_{j}$. This fine-tuned generator is used in subsequent iterations.

Rejection Sampling Fine-tuning (RFT; Yuan et al., 2023) first fine-tunes a pretrained LM on the training dataset $\mathcal{D}_{\text{SFT}}$ to obtain $G$. For each problem $x_{i}$, we then sample $k$ solutions $\{\hat{y}_{i,j}\sim G(y|x_{i})\}_{j=1}^{k}$ and similar to STaR, we only keep correct generated solutions ($z=1$). In RFT, the original dataset is then augmented with the correct completions to $\mathcal{D}_{j}$, and $G$ is fine-tuned on the new $\mathcal{D}_{j}$ to obtain $G_{\text{RFT}}$. Unlike STaR, RFT is not an iterative approach.

$\text{STaR}^{{\dagger}}$. Each STaR iteration can be performed similarly to RFT, akin to ReSTEM (Singh et al., 2023). Since there could be multiple correct solutions for a problem, one could sample $K$ solutions per problem at each STaR iteration, but this is not prescribed in the original STaR paper. For the rest of the paper, we denote this variant of STaR as $\text{STaR}^{{\dagger}}$. Performing only 1 iteration of $\text{STaR}^{{\dagger}}$ corresponds to RFT.

2 Test-time verification

Cobbe et al. (2021) trained verifiers that assess the probability that a candidate solution is correct for a given problem. At test time, the language model $G$ generates many candidate solutions and the one ranked highest by the verifier is selected as the final answer. To train a verifier model $V$, similar to RFT, $k$ candidate solutions are sampled from a generator $G$ for each training problem and labeled for their correctness $z$ to make the verifier training data $\mathcal{D_{\text{VER}}}=\{(x_{i},\hat{y}_{i,j},z_{i,j})\}_{i=1}^{N}$, where $z_{i,j}$ is a binary label indicating whether $\hat{y}_{i,j}$ is a correct or incorrect solution.

To train the verifier $V$, which they refer to as an outcome-supervised reward model (ORM), Cobbe et al. (2021) fine-tune a LLM on $\mathcal{D_{\text{VER}}}$ using a combination of language modeling (Eq. 1) and binary classification. The model is trained to predict $\hat{y}_{i,j}$ given $x_{i}$ and $z_{i,j}$ given $\{x_{i};\hat{y}_{i,j}$} with the language modeling objective and the classification objective, respectively. See §5 for more details.

3 Preference learning with DPO

Fine-tuning pretrained LLMs from human feedback can result in large gains in performance for downstream tasks (Ouyang et al., 2022; Bai et al., 2022). The typical framework to do so is to collect paired human preferences for a set of input prompts $\mathcal{D}_{\text{pref}}=\{(x_{i},y_{i}^{+},y_{i}^{-})\}_{i=1}^{N}$, train a reward model using $\mathcal{D}_{\text{pref}}$, and then fine-tune the LLM using this reward (Stiennon et al., 2020).

More recently, Rafailov et al. (2023) proposed Direct Preference Optimization (DPO) that does not use a separately trained reward model during fine-tuning. DPO requires supervised fine-tuning (SFT) a pretrained LLM on the downstream task to obtain $G_{\text{SFT}}$, which is also called the reference policy in this method. Given the preference dataset $\mathcal{D}_{\text{pref}}$ and $G_{\text{SFT}}$, DPO’s objective increases the relative log probability of preferred $y^{+}$ to dispreferred $y^{-}$ completions.

V-STaR: Verifiers for self-taught reasoners

Existing self-improvement methods, such as RFT, STaR, and $\text{ReST}^{EM}$, throw away model generated solutions that are incorrect. However, incorrect solutions can also contain valuable information: a language model could learn from discrepancies between correct and incorrect solutions for a given problem, and identify error patterns in generations, enhancing its ability to provide more accurate solutions. In this work, we propose V-STaR that utilizes both incorrect and correct generated solutions in an iterative process and use them to train a better generator and verifier (see Alg. 1).

First, we fine-tune a pretrained LLM $G_{\text{base}}$ on the original training data $\mathcal{D_{\text{SFT}}}$ to obtain generator $G_{\text{SFT}}$.

Next, we sample $k$ completions for each problem in the training data from the generator $\{\hat{y}_{i,j}\sim G(y|x_{i})\}_{j=1}^{k}$, where $x\in\mathcal{D}_{\text{query}}$ (see §C for an example).

Generated solutions are labeled for their correctness $z$ using ground truth answers or test cases. We use only correct generated solutions ($z=1$) to augment the generator training data $\mathcal{D_{\text{GEN}}}$ as $(x_{i},\hat{y}_{i,j})$. Both correct and incorrect generated solutions are added to verifier data $\mathcal{D_{\text{VER}}}$ with their correctness label as $(x_{i},\hat{y}_{i,j},z_{i,j})$, so the verifier can learn from generator’s mistakes.

In the next iteration $t$, the generator $G^{t}$ is obtained by fine-tuning the pretrained model $G_{\text{base}}$ on the augmented $\mathcal{D_{\text{GEN}}}$. We can sample solutions again from this generator $G^{t}$. This process is repeated for up to $T$ iterations to augment $\mathcal{D_{\text{GEN}}}$ and $\mathcal{D_{\text{VER}}}$ iteratively.

The final generator $G^{T}$ is obtained by using $\mathcal{D_{\text{GEN}}}$ to fine-tune a pretrained model $G_{\text{base}}$. The verifier $V^{T}$ is obtained by using $\mathcal{D_{\text{VER}}}$ to further train a model $G_{\text{SFT}}$ which was fine-tuned on the original $\mathcal{D_{\text{SFT}}}$.

In our approach, the original training data is also included as correct solutions in both generator data and verifier data. The main difference between our verifier training method and Cobbe et al. (2021) is that our verifier training data is collected iteratively, each iteration from a better generator, while ORM only collects data from a fixed generator that is only fine-tuned on the original SFT data.

Following Cobbe et al. (2021), current LLM verifiers are trained with a combination of language modeling and binary classification loss (§2.2). These two objectives can be unified via offline preference learning methods, such as DPO (Rafailov et al., 2023), where the proximity to the reference policy is a proxy for the language modeling objective while the classification loss is a proxy for reward modelling.

To use DPO for training verifiers, we construct a preference pair dataset using collected solutions in $\mathcal{D_{\text{VER}}}$. We treat correct solutions as preferred and incorrect solutions as not preferred completions given the problem. Specifically,

where $m$ is the number of preference pairs which are from the Cartesian product of correct and incorrect solutions

We train our verifier $V$ using this constructed $\mathcal{D_{\text{VER}}}$ and the SFT policy $G_{\text{SFT}}$ using the DPO objective, $\mathcal{L}_{\text{DPO}}(V;G_{\text{SFT}})$:

where $\hat{r}(x,y)=\beta\log\frac{V(y|x)}{G_{\text{SFT}}(y|x)}$, $\sigma$ is the logistic function, and $\beta$ is a hyper-parameter controlling the proximity to the reference policy $G_{\text{SFT}}$. The DPO objective steers the verifier towards increasing the likelihood of correct solutions $y^{+}$ and decreasing the likelihood of incorrect solutions $y^{-}$ for a problem $x$. We found DPO verifiers to be better than ORM-style verifiers (Cobbe et al., 2021) when using LoRA adapters (Hu et al., 2022). See §4.8 for more details.

Empirical results

To demonstrate the effectiveness of V-STaR, we conduct experiments on two widely used datasets: GSM8K (Cobbe et al., 2021) for solving math problems, and MBPP (Austin et al., 2021) for code-generation problems. We also evaluate the transfer generalization performance of V-STaR using Hendrycks’ MATH (Hendrycks et al., 2021) HumanEval (Chen et al., 2021). Specifically, for math reasoning, we only train our generators and verifiers using GSM8K training data and evaluate them on the whole GSM8K test set and a subset of MATH test set, Counting & probability, prealgebra and number theory where the final answer is a number and no latex exists in the question.. For code generation, we train our models using the MBPP training data and evaluate them on the full test sets of MBPP and HumanEval, formatted using the MBPP prompt template.

We run our experiments by training LLaMA2 (Touvron et al., 2023) and CodeLLaMA (Rozière et al., 2023) 7B and 13B models using LoRA adapters (Hu et al., 2022). Generators are trained with a causal language modeling objective, and our baseline (V-STaR[1 Iter]) and V-STaR verifiers are trained using DPO.

The reference policy $G_{\text{SFT}}$ for DPO is trained on the original training data for 2 and 3 epochs for GSM8K and MBPP, respectively. See §3.1 for more details on using DPO to train the verifier. At inference, we use the likelihood of a (generated) solution given a problem under the trained verifier (i.e. $V(\hat{y}|x)$) as scores to rank candidate solutions.

2 Data generation

For each iteration, $k=16$ completions are sampled per query from the previous iteration’s generator. For GSM8K, the first iteration samples are from a generator trained solely on the original GSM8K training data for 2 epochs. For MBPP, this data is from a pretrained CodeLLaMA model with three in-context examples (see §A). Completions are labeled for correctness by checking the final answer for math problems and running test cases for coding problems.

3 Baselines and metrics

We run V-STaR for 3 iterations and sample $K=16$ solutions at each iteration to augment $\mathcal{D}_{\text{GEN}}$ and $\mathcal{D}_{\text{VER}}$. To assess the gains from our iterative approach, we compare against a number of baselines (Table 1):

SFT: Standard fine-tuning (Eq. 1) on training data without any self-improvement or verifier training.

$\text{STaR}^{{\dagger}}$: A generator is bootstrapped by sampling K=16 completions per query for 3 iterations, see §2.1.

RFT: Running $\text{STaR}^{{\dagger}}$by sampling $3\times 16$ completions for only 1 iteration, see §2.1.

Verification (SFT + Verifier): Generating $3\times 16$ completions using SFT generator to train a verifier with DPO, as described in §2.3.

V-STaR [Iter 1]: Bootstrapping a generator and training a verifier for 1 iteration only with $k=3\times 16$ completions sampled from $G_{\text{SFT}}$, so that the total generation budget matches V-STaR.

At inference, we generate 128 candidate solutions for each test problem using the generator. We report Pass$@1$ for the generators and Verifier$@64$ for verification-based methods, using (Eq. 2). We also report majority voting (Wang et al., 2023c) performance as a strong baseline, following Cobbe et al. (2021); Lightman et al. (2024).

4 Reliable estimation of Verifier@​k@𝑘@k

To estimate verifier$@k$ accuracy, one would repeat the following procedure several times and average the results: sample $k$ solutions, rank them using a verifier and take the top scoring one as the predicted answer (Cobbe et al., 2021; Lightman et al., 2024). However, computing verifier$@k$ this way can have high variance. Instead, to measure the verifier$@k$ reliably, we propose a formula, akin to how Pass$@k$ is computed (Chen et al., 2021). To do so, we estimate the probability that out of $k$ samples drawn without replacement from a fixed set of $N$ (for $N>k$) samples, the one with the highest verifier score is correct, using the following formula:

where $[\alpha_{0},\dots,\alpha_{N-1}]$ are the binary correctness values (0 or 1) for the $N$ candidates $y_{i}$ sorted in decreasing order by their verifier score. The numerator in (Eq. 2) can be derived by considering subsets where the top-ranked candidate is $y_{i}$ for all possible values of $i\in\{0,\dots,N-k-1\}$.

5 V-STaR on math and code reasoning

As shown in Fig. 2, V-STaR shows consistent gains across GSM8K, MBPP, MATH subset and HumanEval test sets for LLaMA2 7B and 13B models (Fig. 8) over baselines. In math, we report absolute improvement of 6% to 17% in test accuracy over $\text{STaR}^{{\dagger}}$ and Verification, and 4% to 12% in code generation tasks. The gains over V-STaR [1 iter] in Fig. 3 show that iteratively generating solutions to collect verifier training data results in a better distribution and quality compared to a non-iterative approach with the same generation budget. We also tried collecting all the verifier training data from the generator at iteration 3. Although this variant uses a larger sampling budget, it resulted in a 2% lower absolute test accuracy on GSM8K than V-STaR.

To test the out-of-domain performance of V-STaR, the generators and verifiers trained on MBPP are evaluated on HumanEval, while those trained on GSM8K are evaluated on a subset of MATH test set (see Fig. 2 and Fig. 4). In general, we observe lower absolute Pass$@1$ and Verifier$@64$ scores for all methods as these two tasks are considered to be more difficult than GSM8K and MBPP. That said, Iterative V-STaR outperforms baselines, and V-STaR [1 iter] on both tasks and across model sizes. Utilizing incorrect solutions to train verifiers results in large improvements than just bootstrapping with correct model generated solutions using $\text{STaR}^{{\dagger}}$or RFT. While we use LoRA adapters due to compute constraints, we hypothesize that gains from V-STaR could potentially be larger with full parameter fine-tuning.

Verifier$@k$ accuracy. Fig. 5 shows test accuracy for $k=1$ to $k=64$, calculated from 128 candidate solutions per test problem, for 7B models on both tasks. Verifier$@0$ is equivalent to Pass$@1$ and ignores verifier scores. Verifier$@k$ saturates for $k\geq 16$ and the gap between V-STaR [1 Iter] and V-STaR stays consistent.

6 Should the verifier be in the training loop?

Optionally, one could train intermediate verifiers at each iteration and filter correct solutions to include in $\mathcal{D_{\text{GEN}}}$ and $\mathcal{D_{\text{VER}}}$ to provide feedback. This step seems more reasonable with sufficient exploration, that is larger values of $k$, when sampling $k$ solutions from the generator in each iteration.

We tried putting the verifier in the training loop to filter correct solutions from the generator for the next training iteration. To do so, we sampled $k=64$ completions per query from the generator, labeled their correctness, and took only the top 8 based on their verifier score. We take as many samples from the incorrect set so that the total number of correct and incorrect completions per query is 16 or less. After running three iterations with verifier in the loop for MBPP, the final verifier$@64$ accuracy, Pass$@1$ and Pass$@10$ are 53.2, 46.34 and 69.57 respectively.

Our results suggest that having the verifier in the training loop does not provide a substantial gain for this task. V-STaR is simpler without the verifier in the loop and there is no need to train a verifier at each iteration; however we did not experiment with other tasks and different sampling strategies from the generator at each iteration. We leave a more detailed study of this question to future work.

7 How many completions can V-STaR be extended to?

Fig. 6 shows the performance of V-STaR 7B on GSM8K measured by Verifier$@k$ as a function of $k$. V-STaR outperforms majority voting (Wang et al., 2023c) at searching over a large number of candidate solutions. While V-STaR is far more effective than majority voting for $k\leq 64$, the performance gap starts to slightly decrease for larger value of $k$, similar to performance decay reported in Cobbe et al. (2021). Furthermore, V-STaR can be used for any problem solving task where we can verify correctness while majority voting is not applicable to tasks such as code generation. We also tried combining verifier scores with reranking strategies, such as weighted reranking and weighted majority voting (Liu et al., 2023), but did not observe performance gains.

8 Comparing DPO vs. ORM verifiers

We trained ORM style verifiers, as described in §2.2, with LoRA adapters. These verifiers did seem to achieve relatively poor performance compared to DPO-based verifiers. 5(a) shows the comparison between the V-STaR [1 Iter] trained with DPO and an ORM style verifier on the same training data. ORM fails to effectively search through generated candidate solutions for number of candidates above 4 in the GSM8K task. The ORM style verifier is also performing worse than our DPO based verifier in MBPP for number of candidate solutions above 16.

9 Evaluating DPO verifier as a generator

Since DPO fine-tuned models can also be used as generators, we evaluate how good is the generation ability of DPO verifiers. Fig. 7 shows Pass$@1$ and Verifier$@64$ for V-STaR verifier as a function of training updates, for three different $\beta$ coefficients for proximity to SFT policy in DPO objective (§3.1). The verifier’s solving ability starts degrading only after a small number of training updates. In contrast, using the DPO objective for verification seems to be sample efficient as the model’s Verifier$@64$ increases significantly with only $2k$ training updates.

Related work

Challenging multi-step reasoning tasks has driven innovative research on LLMs, such as generating answers given questions via intermediate steps (Wei et al., 2022; Kojima et al., 2022). A large volume of recent work studies ways to improve the correctness of these intermediate steps and reducing the cost of arriving at a correct solution.

One family of methods, beginning with STaR (Zelikman et al., 2022), reinforced self-training (Gulcehre et al., 2023), and rejection fine-tuning (Yuan et al., 2023), relies on solutions generated by the LLM to update itself. These methods fine-tune the model on generated solutions that yield a correct answer. ReSTEM (Singh et al., 2023) view this fine-tuning as expectation-maximization based RL fine-tuning of a solution-generating agent. Wang et al. (2023a) propose a contrastive loss to make correct solutions more likely than incorrect ones, while Ni et al. (2023a) propose to use intermediate states of successful solutions as supervision to improve credit assignment. Discovery of successful solutions is a difficult exploration problem, and Luong et al. (2024) has shown that RL-based fine-tuning of a LLM is difficult unless it is initialized by some steps of supervised fine-tuning. In An et al. (2023), a more powerful LLM was used to edit the incorrect rationales generated by a smaller model and provide positive data for its fine-tuning. However, Huang et al. (2023) argued that LLMs are limited in their ability to correct their own reasoning. V-STaR is similar to self-improvement methods in that it uses its self-generated solutions for fine-tuning, but also trains a verifier using these solutions, including both correct and incorrect ones.

Training verifiers.

Verifiers – models that score or rank reasoning chains with the aim of favouring successful rationales – were introduced for mathematical reasoning tasks by Cobbe et al. (2021), who proposed to collect correct and incorrect rationales from a tuned generator and train a verifier. They noted the importance of a large training set for the success of the method. Uesato et al. (2022) found that process supervision – correctness of the rationale – enhances the performance of fine-tuned LLMs relative to outcome supervision – whether the answer is correct or not. Subsequent work correspondingly studied ways of deriving reward signals for individual reasoning steps (Li et al., 2023; Lightman et al., 2024; Yu et al., 2023), combining solution-level and step-level verifiers (Zhu et al., 2023), and augmenting verifiers with auxiliary information, such as results of program execution (Ni et al., 2023b). In Ma et al. (2023); Wang et al. (2023b), rationale generation is treated as a graph search problem, either using a stepwise verifier to guide the search or estimating the quality of steps by Monte Carlo rollouts. In V-STaR, the verifier is trained with DPO, which enjoys a high sample efficiency (see Fig. 7), and is used for ranking LLM-generated solutions at test-time.

The manner of training the verifier varies between works. The verifier can be viewed a reward model trained on human annotations – making the training of a generator that satisfies the verifier as an instance of RL with human feedback (Ziegler et al., 2019) – or on synthetic data, leading to forms of RL with AI feedback (Bai et al., 2022; Yang et al., 2023). The verifier can alternatively be viewed as a generative model, such as by conditioning on control tokens indicating a positive or negative label of a solution (Korbak et al., 2023) or by extracting the score as the likelihood of a special token following the candidate solution (Liu et al., 2023). V-STaR takes the unique approach of using DPO (Rafailov et al., 2023) to contrast the likelihoods of correct and incorrect solutions under the verifier (see §3.1).

Conclusion

We propose V-STaR, a data-efficient and simple to implement approach that utilizes correct and incorrect generated solutions from an iteratively trained generator to train a strong verifier. We find training verifiers with DPO to be more effective than the common method by Cobbe et al. (2021). Our empirical results show the effectiveness of V-STaR over existing self-improvement and verification-based methods. V-STaR has the potential to improve existing self-improvement loops on a wide range of problems with access to correctness feedback during training.

Acknowledgements

We would like to thank Denny Zhou, Azade Nova and Adam Kosiorek for providing valuable feedback to an early draft. We acknowledge material support from NVIDIA Corporation in the form of computational resources. In addition, we thank Google and Microsoft for their invaluable financial support.

Impact statement

This work aims to improve the reasoning skills of LLMs in a self-refinement loop, making use of existing datasets and the models’ own solution-generation abilities. As such, V-STaR makes a step towards improving the data and compute efficiency of LLM inference, which promotes their accessibility and use in real applications. On the other hand, we encourage researchers and practitioners who build upon our work to be cautious of the possible misuse of LLMs with enhanced reasoning capability.

References

Appendix A The prompt used for MBPP few-shot generation.

Following Ni et al. (2023b), we use the following prompt to sample completions per problem for data generation during training.

Appendix B Test accuracy of 13B V-STaR and baselines

x = Andy walks 50 meters from his house to school. After school, he comes back to the house and goes to the market. If he walks 140 meters in total, how many meters is the distance between the house and the market? $\hat{\textbf{y}}_{1}$ = He walks to school and back, which is $2\times 50$ meters = 100 meters. So he walks 140 meters - 100 meters = 40 meters to the market. Answer=40 $\hat{\textbf{y}}_{2}$ = If he walks 50 meters from his house to school, and 140 meters in total, he walks 140 - 50 = 90 meters from the school to the market. Answer=90