AI 译本 · Equalang
新上架PDF · 2026年11月翻译
加载中…
加载中…
NeurIPS · 2025
强化学习真的能激发大语言模型超越基础模型的推理能力吗？
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao et al.
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.
带可验证奖励的强化学习（RLVR）近期在提升大语言模型（LLM）的推理表现方面取得了显著成效，尤其是在数学和编程任务上。传统强化学习有助于智能体探索并学习新策略，与此类似，人们认为 RLVR 能让 LLM 持续自我提升，从而获得超越相应基础模型的新推理能力。本研究通过系统考察多种模型系列、强化学习算法，以及数学、编程和视觉推理基准上的 RLVR 训练模型，评估其推理能力边界，并采用较大 k 值下的 pass@k 作为评价指标，从而对 RLVR 的现状进行审慎检验。令人意外的是，我们发现当前的训练设置并未诱发根本性的新推理模式。虽然 RLVR 训练模型在较小的 k 值下（例如 k = 1）优于基础模型，但当 k 值较大时，基础模型的 pass@k 得分更高。覆盖率和困惑度分析表明，观察到的推理能力源自基础模型，且受其限制。将基础模型视为上限后，我们的定量分析显示，六种常见的 RLVR 算法表现相近，在挖掘基础模型潜力方面仍远未达到最优。相比之下，我们发现，蒸馏能够引入来自教师模型的新推理模式，并真正拓展模型的推理能力。总体而言，我们的研究结果表明，当前的 RLVR 方法尚未发挥强化学习的潜力，无法在 LLM 中激发真正新颖的推理能力。这凸显了改进强化学习范式的必要性，例如持续扩展训练规模，以及进行多轮智能体与环境交互，以释放这一潜力。
PDF · 2026年11月翻译