Understanding R1-Zero-Like Training: A Critical Perspective

Introduction

DeepSeek-R1-Zero (Guo et al., 2025) revolutionizes the pipeline of large language model (LLM) post-training by introducing the R1-Zero-like training paradigm: directly applying RL to base LLMs without relying on supervised fine-tuning (SFT) as a preliminary step. This new paradigm is appealing due to its simplicity and the demonstrated RL scaling phenomenon: the model reasoning capabilities improve along with a continual increase in model’s response length. This phenomenon is also accompanied by the “Aha moment”, at which the model learns emergent skills such as self-reflections.

In this paper, we aim to understand R1-Zero-like training by studying two essential components: base models and RL. In the first part, we investigate various attributes of base models, with the focus on the Qwen2.5 model family (Yang et al., 2024a; b), which has been used in recent attempts to reproduce R1-Zero (Pan et al., 2025; Zeng et al., 2025; Liu et al., 2025b; Hu et al., 2025), as well as DeepSeek-V3-Base (Liu et al., 2024), from which the real R1-Zero model was RL-tuned. In the second part, we identify the bias in optimization of GRPO (Shao et al., 2024), which may lead to progressively longer incorrect responses. To this end, we propose a simple modification to eliminate the bias, i.e., to get GRPO Done Right (Dr. GRPO), which leads to better token efficiency (highlighted in Fig. 1).

Our analysis on base models and RL suggests a minimalist recipe for R1-Zero-like training: we RL-tune Qwen2.5-Math-7B using the (unbiased) Dr. GRPO algorithm on MATH (Hendrycks et al., 2021) level 3-5 questions with the Qwen-Math template, and achieve state-of-the-art performance (Fig. 2) with only $27$ hours compute on $8\times$ A100 GPUs. We hope our findings presented in this paper, models released, and the codebase open-sourced could benefit future research in the field.

As an overview, we summarize the takeaways of this paper below:

Analysis on Base Models

In this section, we scrutinize a wide range of base models, including the Qwen-2.5 family (Yang et al., 2024a; b), Llama-3.1 (Grattafiori et al., 2024) and DeepSeek series (Liu et al., 2024; Shao et al., 2024; Guo et al., 2025), asking them $500$ questions sampled from the MATH (Hendrycks et al., 2021) training set and analyzing their responses.

Since training from a base model is a fundamental setting of the R1-Zero-like paradigm, we first investigate whether widely used open-source base models, which are typically trained for sentence completion (i.e., $p_{\theta}({\mathbf{x}})$), can have their question-answering capabilities effectively elicited through appropriate templates, thereby functioning as a question-answering base policy $\pi_{\theta}(\cdot|{\mathbf{q}})$. In addition to the R1 template (Template 1) in Guo et al. (2025), we consider the Qwen-Math template (Template 2) used by Zeng et al. (2025), as well as No template (Template 3):

Template 1 (R1 template). A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within $<$think$>$ $<$/think$>$ and answer is enclosed within $<$answer$>$ $<$/answer$>$ tags, respectively, i.e., $<$think$>$ reasoning process here $<$/think$>$ $<$answer$>$ answer here $<$/answer$>$.\nUser: {question}\nAssistant: $<$think$>$ Template 2 (Qwen-Math template). $<$|im_start|$>$system\nPlease reason step by step, and put your final answer within \\boxed{}.$<$|im_end|$>$\n$<$|im_start |$>$user\n{question} $<$|im_end|$>$\n$<$|im_start|$>$assistant\n Template 3 (No template). {question} Experimental settings. We include Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Llama-3.1-8B, DeepSeek-Math-7B and DeepSeek-V3-Base-685B for experiments. For each model, we first apply No template to get the model responses, then let GPT-4o-mini to judge whether the model responses are in an answering format (regardless of quality) or in a sentence-completion pattern. We record the percentage of responses that tend to answer the question as the metric. We then apply both R1 template and Qwen-Math template to obtain model responses, and determine the most suitable template for each model based on the metric. Finally, we evaluate the pass@8 accuracy of each model with the corresponding template to assess whether the base policies can explore rewarding trajectories for RL improvement.

Results. The left plot of Fig. 3 shows how well base models (with or without templates) answer the provided questions. We observe that Llama and DeepSeek models all improve the answering ability by employing the proper template (R1 template). However, Qwen2.5 models work best (with $100\%$ answering rate) when no template is used. This intriguing property motivates further investigation, as discussed in Sec. 2.2. Meanwhile, the lowest answering rate with no template suggests that DeepSeek-V3-Base is a nearly pure base model. This observation motivates us to explore whether a pure base model like DeepSeek-V3-Base demonstrates the Aha moment (Sec. 2.3). The middle plot of Fig. 3 shows the pass@8 accuracy of different base models (with template) at different sampling temperatures. This metric can serve as an indicator of base policy’s exploration ability. For example, if a base policy cannot even sample a single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there is no reward signal. Our results demonstrate that all tested models are exploratory (thus ready for RL), with Qwen2.5 models performing the best (even surpassing DeekSeek-V3-Base). This might partially explain that most R1-Zero projects (Zeng et al., 2025; Hu et al., 2025) are based on Qwen2.5 models.

2 Qwen-2.5 Models Unlock the Best Performance When Discarding Template

We next dig into the intriguing observation (c.f. Fig. 3(Left)) that all Qwen2.5 base models readily serve as chat models even without any template. We take a step further to evaluate the reasoning ability of Qwen2.5-Math models on five standard benchmarks: AIME 2024 (Li et al., 2024), AMC (Li et al., 2024), MATH500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). Following common practice, we use greedy decoding and limit the sampling budget to 3000 tokens.

As shown in Table 1, not using any template can drastically boost the average performance, resulting in an improvement of about $60\%$ compared to the traditional 4-shot prompting. Since Qwen2.5-Math (Yang et al., 2024b) uses chat model’s data (question-answer pairs) during the pretraining stage, we hypothesize that they might pretrain on the concatenated text to maximize $\log p_{\theta}({\mathbf{q}};{\mathbf{o}})$ directly. If our hypothesis turns out true, we shall be more careful about using Qwen2.5 models to reproduce DeepSeek-R1-Zero, since the base models are already SFT-like without templates.

3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base

One of the most inspiring results of DeepSeek-R1-Zero is the emergence of self-reflection behaviors, a.k.a., Aha moment, through pure RL training. A few prior studies (Liu et al., 2025b; Yeo et al., 2025) have suggested that there may not be Aha moment in open-source R1 replications because the base models they use already exhibit self-reflection keywords. However, they have not tested DeepSeek-V3-Base, on which the real R1-Zero model was RL-tuned. We complete this missing piece by hosting DeepSeek-V3-Base-685B ourselves and investigating its responses of $500$ MATH questions with the R1 template. From the right plot of Fig. 3, we can observe that DeepSeek-V3-Base also generates a decent amount of self-reflections, further validating claims by Liu et al. (2025b). We also show examples in Fig. 4 where DeepSeek-V3-Base generates keywords such as “Aha”, “wait”, and “verify the problem”.

An additional important question is whether self-reflection behaviors are associated with improved model performance after RL training. To investigate this, we host DeepSeek-R1-Zero and analyze its responses of the same questions from MATH dataset. While self-reflection behaviors occur more frequently in R1-Zero, we observe these behaviors are not necessarily imply higher accuracy. Detailed analysis can be found in App. D.

Analysis on Reinforcement Learning

Language model generation can be formulated as a token-level Markov Decision Process (MDP) ${\mathcal{M}}=({\mathcal{S}},{\mathcal{A}},r,p_{{\mathcal{Q}}})$. At each generation step $t$, the state $s_{t}\in{\mathcal{S}}$ is the concatenation of the input question and the output response generated so far: $s_{t}={\mathbf{q}};{\mathbf{o}}_{<t}=[q_{1},\dots,q_{M},o_{1},\dots,o_{t-1}]$. The policy $\pi_{\theta}(\cdot|s_{t})$ will select the next token $o_{t}$ from the vocabulary ${\mathcal{A}}$, resulting in a deterministic transition to the next state $s_{t+1}=s_{t};[o_{t}]$. The generation process starts from sampling an initial state $s_{1}={\mathbf{q}}\sim p_{{\mathcal{Q}}}$ from a set of questions, and stops when the autoregressive policy generates the [eos] token or exhausts the budget.

Typically, we maximize the entropy-regularized objective (Schulman et al., 2017a):

where $R({\mathbf{q}},{\mathbf{o}})=\sum_{t=1}^{|{\mathbf{o}}|}r(s_{t},o_{t})$ is the return (Sutton & Barto, 2018) of the trajectory ${\mathbf{q}};{\mathbf{o}}$, and $\pi_{\text{ref}}$ is a reference policy. The KL regularization term is usually adopted ($\beta>0$) for reinforcement learning from human feedback (Christiano et al., 2017), where $r$ is a reward model learned from data collected by $\pi_{\text{ref}}$. In this case, regularization helps prevent $\pi_{\theta}$ from deviating too far from the distribution where the reward model is accurate (Jaques et al., 2019; Stiennon et al., 2020). However, RL-tuning reasoning models typically employs rule-based verifiers as $r$ (Lambert et al., 2024), eliminating the concerns of distributional shift. This allows us to remove the KL term, which not only saves the memory and computation required by $\pi_{\text{ref}}$ during training, but also potentially leads to better performance for R1-Zero-like training (Hu et al., 2025). We will assume $\beta=0$ throughout this paper.

Policy optimization algorithms. To optimize $\pi_{\theta}$ with the above objective (Eq. 1 with $\beta=0$), Proximal Policy Optimization (PPO) (Schulman et al., 2017b) maximizes the following surrogate objective:

where $\pi_{\theta_{\text{old}}}$ is the policy before the update, $\epsilon$ is the clipping hyperparameter, and $\hat{A}_{t}$ is an estimator of the advantage function of the $t$-th token. A standard way to estimate $\hat{A}_{t}$ is to compute the Generalized Advantage Estimation (GAE) (Schulman et al., 2015) with a learned value model $V_{\phi}$. However, in the context of LLM RL-tuning, learning the value model is computationally expensive, so methods that estimate $\hat{A}_{t}$ without $V_{\phi}$ are practically preferred. For example, Shao et al. (2024) proposed GRPO, which first samples a group of responses $\{{\mathbf{o}}_{1},\dots,{\mathbf{o}}_{G}\}$ per question and computes their returns $\mathbf{R}=\{R_{1},\dots,R_{G}\}$, then sets the advantage of all tokens from ${\mathbf{o}}_{i}$ as $\hat{A}_{t}=\frac{R_{i}-\operatorname{mean}(\mathbf{R})}{\operatorname{std}(\mathbf{R})}$.

In Deepseek-R1-Zero (Guo et al., 2025), a notable trend is the consistent increase in response length throughout the training process. This is frequently interpreted as an indication of the development of advanced reasoning abilities such as self-reflection. Recent studies (Pan et al., 2025; Zeng et al., 2025; Hu et al., 2025) have replicated this phenomenon using various algorithms and implementations. However, we argue that the observed increase in response length may also be attributed to a bias inherent in the GRPO (Shao et al., 2024) objective function:

with the return $R({\mathbf{q}},{\mathbf{o}}_{i})$ typically only including the outcome verifiable reward in LLM reasoning (the analysis also applies to process reward cases).

Compared to the objective function in Eq. 2, GRPO introduces two biases (see also Fig. 5):

Response-level length bias: This arises from dividing by $|{\mathbf{o}}_{i}|$. For positive advantages ($\hat{A}_{i,t}>0$, indicating a correct response), this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages ($\hat{A}_{i,t}<0$, indicating an incorrect response), longer responses are penalized less due to their larger $|{\mathbf{o}}_{i}|$, causing the policy to prefer lengthier responses among incorrect ones.

Question-level difficulty bias: This is caused by dividing the centered outcome reward by $\operatorname{std}(\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\})$. Questions with lower standard deviations (e.g., those that are too easy or too hard, with the outcome rewards being almost all 1 or 0) are given higher weights during policy updates. While advantage normalization is a common trick in RL (Andrychowicz et al., 2021), it is typically computed across an entire batch. In contrast, question-level normalization results in varying weights in the objective for different questions, leading to a difficulty bias in optimization.

Length Bias Also Exists in Open-Source PPO Implementations. We also examined several popular open-source implementations of vanilla PPO algorithms for LLM post-training. To our surprise, all of these implementations normalize the loss by response length (see LABEL:lst:ppo_impl and Table 2), which misaligns with the PPO objective as defined in Eq. 2. This formulation-implementation misalignment was present even before the publication of GRPO. We speculate that the misalignment might originate from the pretraining stage (Shoeybi et al., 2019), where all tokens are packed into a fixed-length context and normalizing the loss by the context length (i.e., computing loss.mean(-1)) improves the numerical stability. However, in the RL-tuning stage, typical implementations (von Werra et al., 2020) normalize the loss by the response length, which is not a constant, introducing an unintended length bias.

2 Dr. GRPO: Group Relative Policy Optimization Done Right

To avoid the aforementioned optimization bias in GRPO, we propose to simply remove the {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\frac{1}{|{\mathbf{o}}_{i}|}} and {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\operatorname{std}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})} normalization terms. Meanwhile, to faithfully implement the unbiased optimization objective, we could replace the mask.sum(axis=dim) with a constant value (e.g., generation budget) in the masked_mean function in LABEL:lst:ppo_impl, as highlighted by the line in green. Notably, these simple modifications recover the PPO objective in Eq. 2, with the advantage estimated by Monte Carlo return with an unbiased baseline (Sutton & Barto, 2018). We give detailed derivations in App. A. We refer to our new optimization algorithm as Dr. GRPO. We next experimentally validate its effectiveness.

Experimental settings. We implement our algorithm using Oat (Liu et al., 2025a), a modular, research-friendly and efficient LLM RL framework. We adopt the Qwen2.5-1.5B base model and the R1 template (Template 1) for online RL-tuning. We implement the verification-based reward function using Math-Verifyhttps://github.com/huggingface/Math-Verify., with the following minimalistic rule:

We run RL on questions sampled from the MATH (Hendrycks et al., 2021) training dataset, and compare the vanilla GRPO with the proposed Dr. GRPO. We evaluate the online model on five benchmarks: AIME2024, AMC, MATH500, Minerva Math and OlympiadBench. More experimental details including hyperparameters can be found in our open-sourced codebase.

Results. We report various metrics in Fig. 6 to demonstrate that Dr. GRPO can effectively mitigate the optimization bias and lead to better token efficiency. In particular, we first note that both GRPO and Dr. GRPO exhibit similar trend to DeepSeek-R1-Zero (Guo et al., 2025), namely their response length increases along with training reward (Plots 1 & 2). However, we observe that GRPO tends to continually generate longer responses even when the reward improvement slows down (Plot 2). Although such a phenomenon is often referred to as the “emergence” of long-CoT through RL (Zeng et al., 2025; Hu et al., 2025), we argue that it is also confounded by the response-level length bias (Sec. 3.1) during optimizationWe note that both Zeng et al. (2025) and Hu et al. (2025) employ PPO, which is unbiased by formulation. However, their loss implementations still introduce the length bias (see LABEL:lst:ppo_impl).. In contrast, by computing the unbiased policy gradients, Dr. GRPO prevents the response length from growing wildly during training (Plot 2). Moreover, on evaluation benchmarks, the length of incorrect responses is substantially reduced by Dr. GRPO compared to the baseline (Plot 4), suggesting that an unbiased optimizer also mitigates overthinking (Chen et al., 2024).

3 A Duet of Template and Question Set Coverage in RL dynamics

Recall that the Qwen2.5-Math base models can readily answer questions with high accuracy without any prompt template (Sec. 2.2). Based on this intriguing observation, we are interested in how different templates affect the RL training. Furthermore, given the general belief that larger question set coverage leads to better performance (Luo et al., 2025; Hu et al., 2025), we also study the interaction between different templates and different levels of question coverage.

Experimental settings. Starting from the Qwen2.5-Math-1.5B base model, we apply R1 template, Qwen-Math template and No template respectively to run RL using Dr. GRPO. All experiments are repeated for different question sets that are detailed in Table 3.

Results. Fig. 7 shows the RL curves of different runs, from which we can make several interesting observations: 1) Templates determine the performance of the initial policies, but RL can improve all policies to a comparable performance of ~$40\%$ (given a proper question set); 2) When using the R1 template, question sets have a significant impact on the dynamics of RL, with too narrow coverage leading to lower plateau performance. However, when using the Qwen-Math template, the best final performance is attained by RL on GSM-8K, demonstrating that training on much simpler (and o.o.d.) questions can largely improve (nearly double) the test accuracy on harder questions. From these observations, we draw the following insights:

The Qwen2.5-Math-1.5B base model already possesses strong math-solving capabilities (see the starting point in the right plot of Fig. 7). Applying templates in fact destructs the capability before RL reconstructing it. This implies that we shall be more conservative in claiming the huge gains brought about by pure RL.

When there is a large mismatch between base models and templates (e.g., R1 template mismatches Qwen2.5-Math-1.5B), the policy improvement mainly comes from RL-tuning, thus requiring question set to have good coverage (left plot of Fig. 7). Otherwise, even a small and completely o.o.d. question set could induce the reasoning ability equally well, by reinforcing correct reasoning behaviors instead of infusing new knowledge.

4 Domain-Specific Pretraining Improves RL Ceiling

Recent successful R1-Zero-like replications of math reasoners mostly employ Qwen2.5 base models as the initial policies (Zeng et al., 2025; Cui et al., 2025; Hu et al., 2025), which are already strong math solvers and exhibit self-reflection patterns (Sec. 2.2 and 2.3). In this section we hope to explore the other side: can R1-Zero-like training succeed on originally weak (in terms of math reasoning) base models? We answer this question affirmatively, with the observation that math pretraining would improve the ceiling of RL.

Experimental settings. We adopt the Llama-3.2-3B base model as our starting point, and use the unbiased Dr. GRPO algorithm for RL-tuning with the R1 template. We hypothesize that domain-specific pretraining would help RL, hence we adopt the Llama-3.2-3B-FineMathhttps://huggingface.co/HuggingFaceTB/FineMath-Llama-3B., which is continual pretrained on the FineMath dataset (Allal et al., 2025). Moreover, as we hypothesize that Qwen2.5 models are likely to be pretrained on concatenated question-response texts (Sec. 2.2), we similarly prepare a concatenated dataset from NuminaMath-1.5 (LI et al., 2024), and continual pretrain Llama-3.2-3B-FineMath for 2 epochs with learning rate 1e-5. We refer to the concatanated continual pretrained model as Llama-3.2-3B-NuminaQA.

Results. We present the RL curves of different base models in the left plot of Fig. 8. We observe that RL can even improve the vanilla Llama base model, but the gain is minimal. After continual pretraining (and concatenated continual pretraining) to embed math domain knowledge, Llama models can show much stronger RL performance, validating our hypothesis. We also revisit the GRPO’s optimization bias with the Llama base model. The right plot of Fig. 8 compares the model performance and response length trained with GRPO and Dr. GRPO. We can clearly see that GRPO can produce the “double-increase” phenomenon, potentially leading to a misperception that long-CoT can also emerge on Llama models after math pretraining. Unfortunately, the increase of length might be due to the optimization bias (Sec. 3.1), which can be effectively mitigated by the proposed Dr. GRPO (Sec. 3.2 & right plot of Fig. 8).

Closing Remarks

We have taken a critical perspective to examine base models used for R1-Zero-like training, as well as algorithms used for RL. Through the analysis, we demystified how pretraining biases influence RL outcomes and how optimization choices, like GRPO, can unintentionally shape model behavior. With the proposed Dr. GRPO, we offer a simple fix that improves token efficiency while preserving reasoning performance. Our results show that scaling RL can be both effective and efficient—sometimes, less really is more.

References

Appendix A Policy Gradient Derivations

In the context of RL for LLM post-training, we typically maximize the value of

where $R({\mathbf{q}},{\mathbf{o}})=\sum_{t=1}^{|{\mathbf{o}}|}r({\mathbf{q}},{\mathbf{o}}_{\leq t})$ is the return (Sutton & Barto, 2018) of the trajectory ${\mathbf{q}};{\mathbf{o}}$, and $r({\mathbf{q}},{\mathbf{o}}_{\leq t})$ represents the token-level reward for $t$-th token in response ${\mathbf{o}}$.

The Monte Carlo policy gradient (Sutton & Barto, 2018) of Eq. 4 is

where $B({\mathbf{q}},{\mathbf{o}}_{<t})$ is a variance reduction term, which is invariant with respect to $o_{t}$ so that

By setting $B({\mathbf{q}},{\mathbf{o}}_{<t})=\operatorname{mean}({\{R({\mathbf{q}},{\mathbf{o}}_{1}),\dots,R({\mathbf{q}},{\mathbf{o}}_{G})\}})$, the policy gradient of Eq. 5 becomes

We adopt the PPO (Schulman et al., 2017b) objective to compute Eq. 6:

from which we conclude that both $\operatorname{std}$ and $|{\mathbf{o}}|$ should not appear in the RL objective.

Appendix B Detailed Benchmark Results

We show the detailed benchmark results for three scales (1.5B, 3B and 7B) in Table 4. We also include the instruct models at the same scale and R1-Distill models for comparison. Note that since we employ the Qwen2.5-Math base models, which have a context length of 4k, we thus limit the generation budget at 3k for all baselines compared. For models that are trained for a longer context (OpenReasoner-Zero end R1-Distill-Qwen), we also report their performance at 8k generation budget.

Appendix C Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors

We construct a pool of carefully selected keywords and phrases that signal self-reflection behaviors in the LLM’s responses. However, LLM-generated responses often contain hallucinations and off-topic content, leading to the presence of simple, ambiguous keywords that do not necessarily indicate genuine self-reflection. For instance, terms like “wait” and “try again” frequently result in false positive detections. To reduce false positives, we maintain a small, highly selective keyword pool consisting of terms that are strongly indicative of self-reflection. In our experiment, the keyword pool is limited to: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, and go over the steps.

We present the occurrences of various keywords in the responses generated by different models in Figure 9. Interestingly, different model families emphasize different keywords. For instance, phrases such as “check again”, “double-check”, “re-evaluate”, “re-examine”, “recheck”, “reconsider”, and “verify again” appear most frequently in the Qwen2.5 family. In contrast, “re-evaluate”, “re-examine”, and “verify again” do not appear in the responses of the DeepSeek family, while Llama models frequently use the phrase “think again.” We hypothesize that this phenomenon results from differences in the pretraining data, particularly in relation to reasoning and mathematics.

Although we meticulously select the keyword pool, it may still be insufficient to identify some implicit behaviors of self-reflection that do not contain a specific keyword. Additionally, it can lead to false positives, as illustrated in Case (a) of Figure 10. To address these limitations and more accurately assess the self-reflection capability of base models, we leverage stronger LLMs (GPT-4o-mini in our experiments) to analyze the responses and determine whether they exhibit explicit self-reflection (e.g., keywords like ”recheck” and ”reevaluate”) or implicit self-reflection (e.g., more sophisticated patterns that cannot be easily captured through keyword matching). This approach helps distinguish true self-reflection behaviors from superficial or incidental use of related terms.

While LLM-based detection effectively filters out false positives from keyword-based detection and identifies implicit self-reflection behaviors, it can still misclassify responses, particularly when they are lengthy and complex. For instance, Case (b) in Figure 10 shows a false positive in LLM-based detection, where the response is categorized as self-reflection by the LLM but does not actually exhibit self-reflection. This type of error can be filtered out by keyword-based detection. To enhance robustness, we integrate keyword-based and LLM-based detection through cross-validation. The combined detection results, along with the individual results from keyword-based and LLM-based methods, are presented in Figure 11.

Appendix D Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero

We analyze DeepSeek-V3-Base and DeepSeek-R1-Zero to understand changes in model behavior during R1-Zero training. In Fig. 12, we present the breakdown of response categories across difficulty levels for 500 MATH questions evaluated on both models. The results indicate that most incorrect responses are corrected after RL training, demonstrating substantial performance gains from R1-Zero training. Meanwhile, we find an increase in unformatted responses, which aligns with the observation in Liu et al. (2025b).

In Table 5, we report the average response lengths across categories. Note that truncated responses would fall into any of the other three categories if a larger context size were used; thus, we exclude them from the table. The results show a substantial increase in response lengths across all categories, including correct responses, consistent with the results in the Fig. 3 of Guo et al. (2025). However, the average length of incorrect responses is notably longer than that of correct responses. We hypothesize this is because more challenging questions generally require longer responses due to increased reasoning complexity, and incorrect responses are more likely to originate from harder questions, resulting in a longer average length.

Self-reflection does not necessarily imply higher accuracy. To investigate whether self-reflection behaviors are associated with model performance during the inference (acknowledging that self-reflection may improve exploration during training—a potential positive effect outside this section’s scope), we analyze questions that elicit at least one response with self-reflection from DeepSeek-R1-Zero across eight trials. For each question, we sample 100 responses and divide them into two groups: those with self-reflection and those without. We then compute the accuracy difference between these two groups for each question. As shown in Fig. 13, the results indicate that nearly half responses with self-reflection do not achieve higher accuracy than those without self-reflection, suggesting that self-reflection does not necessarily imply higher inference-stage accuracy for DeepSeek-R1-Zero.

Appendix E Prompts Used for GPT-As-A-Judge

Prompt for LLM-based detection to determine whether a response contains self-reflection behaviors.

LLM-based Detection for Self-Reflection I will send you a mathematical question along with a detailed response. Your task is to determine whether the response is attempting to answer the question. If the response is off-topic, hallucinated, random talk, or otherwise irrelevant, mark it as 0. Otherwise, assess whether the response exhibits self-reflection. Categorization Rules: 1. Category 0: The response is off-topic, nonsensical, incoherent, overly repetitive, or lacks logical reasoning. • Example cases: – The response does not relate to the question. – It contains meaningless or hallucinated content. – It consists of excessive repetition without coherence. 2. Category 1: The response attempts to answer the question but does not exhibit self-reflection. • Example cases: – The response directly solves the problem without revisiting steps. – No attempt is made to verify the correctness of the answer or explore alternative solutions. 3. Category 2: The response demonstrates self-reflection at any level. • This may include: – Explicit self-reflection keywords, such as: *recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, go over the steps*, etc. – Implicit self-reflection behaviors, such as revisiting the solution, questioning assumptions, or considering alternative approaches without explicit keywords. • If any form of self-reflection is present, always categorize it as 2, regardless of correctness or answer quality. 4. Category 3: The response consists solely of Python code for calculations without exhibiting self-reflection. • Example cases: – The response only provides a Python script to compute the solution without any verification, re-evaluation, or alternative considerations. Output Format: Your response should first provide a very brief explanation of your analysis, followed by a single category number (0, 1, 2, or 3) at the end. You must include the category number at the end of your response. Example outputs: • ‘The response is off-topic and does not attempt to answer the question. 0.‘ • ‘The response provides a direct solution without self-reflection. 1.‘ • ‘The response demonstrates self-reflection. 2.‘ • ‘The response consists solely of Python code without any self-reflection. 3.‘ Question: {question} Response: {response} Prompt for checking the model’s question-answering ability.