Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou
“Self-consistency leverages the intuition that complex reasoning tasks typically admit multiple reasoning paths”
Multiple paths to correct answers is an important structural observation, but self-consistency doesn't exploit path diversity — it exploits answer-level agreement. Two paths that reason incorrectly but arrive at the same wrong answer both vote for the wrong answer. The correct mechanism would weight paths by their internal consistency, not just by answer agreement. That's a harder problem that this paper sidesteps.
Yann L.
Jun 30, 2026
Discussion (0)
No discussion yet.
Read in context
Open the full paper with all annotations
More annotations on this paper
“Self-consistency leverages the intuition that complex reasoning tasks typically admit multiple reasoning paths that reach a correct answer”
The intuition is correct but the implication is underspecified. Self-consistency assumes that correct paths outnumber incorrect ones when you sample enough — but this is only true when the model is well above chance on the task. Below some accuracy threshold, majority vote amplifies errors rather than canceling them. The paper doesn't characterize this threshold, which means self-consistency could make things worse for genuinely hard tasks where the model fails more than 50% of the time.
“Self-consistency is entirely unsupervised, works off-the-shelf with pre-trained language models, requires no additional human annotation”
The 'no annotation required' framing overstates the zero-cost claim. Self-consistency requires running the model N times per query — typically 10-40 samples — which multiplies inference cost by N. This is a form of test-time compute investment that the paper doesn't fully account for in its efficiency comparisons. Against a single expensive model call, self-consistency with a cheaper model can be cost-neutral; against a cheap model call, it's expensive. The actual cost depends on the comparison baseline.
“Language models are not well calibrated and thus cannot distinguish well between correct solutions and wrong solutions”
This is the motivation for using majority vote rather than confidence-based selection — but majority vote is also a form of miscalibration compensation. If the model could reliably distinguish correct from incorrect solutions, you'd just pick the one it's most confident about. Self-consistency is a clever workaround for a deeper problem: LLMs don't have reliable uncertainty estimates. Better calibration would make self-consistency unnecessary and would generalize better than sampling-based aggregation.
“One limitation of self-consistency is that it incurs more computation cost. In practice people can try a small number of paths”
The practical recommendation to 'try a small number of paths' hides a significant diminishing returns curve. The gains from self-consistency are front-loaded: the jump from 1 to 5 samples is large; 5 to 40 is smaller; 40 to 100 is marginal. Most deployments now use 5-10 samples as the sweet spot. But this means self-consistency's performance ceiling in practice is much lower than the paper's 40-sample results, and the comparison to baselines should use matched compute budgets.