[ICML-2025] R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2505.13417 |
> | Code & Data | https://github.com/THU-KEG/AdaptThink |
> | Public      | arXiv |
> | Date        | 2025.06.20 |


# Summary Overview

作者认为对于简单的问题，没有必要使用 thinking 模式，而是直接进行回答也能获得很高的准确率。从效率上来说，模型应该有选择的使用 thinking，而不是所有的都开。通过强化学习训练模型，使得模型能够在推理时，自动地选择是否使用 thinking。

*AdaptThink*, a novel RL algorithm to teach reasoning models to choose the optimal thinking mode adaptively based on problem difficulty.

1. a constrained optimization objective that encourages the model to choose *NoThinking* while maintaining the overall performance;
2. an importance sampling strategy that balances *Thinking* samples during on-policy training, thereby enabling cold start and allowing the model to explore and exploit both thinking modes throughout the training process.

![image.png](/static/img/0dfb2411dced00d77d9029b6dc49bdc1.image.png)

# Main Content

*Can the reasoning model learn to select Thinking or NoThinking mode adaptively based on the difficulty of the input problem, thereby achieving more efficient reasoning without sacrificing or even improving performance?*

![image.png](/static/img/59189518bafb33c06b2839abf40282f7.image.png)

**Constrained Optimization Objective**

an ideal selection policy should prefer to choose *NoThinking* as long as the overall performance is not diminished. Let $\pi_{\theta_\text{ref}}$ denote the reference model, which is the initial $$

## Related Work

# Experiments

## Metrics

## Models

## Datasets

## Baselines

## Results

## Case Study

# 🤖

:::info{title=" "}

:::

# Others

[arXiv-2025] AdaptThink: Reasoning Models Can Learn When to Think

# Raise the Ceiling: Clip-Higher

> Cite from: DAPO: An Open-Source LLM Reinforcement Learning System at Scale



In our initial experiments using naive PPO or GRPO, we observed the entropy collapse phenomenon: the entropy of the policy decreases quickly as training progresses. The sampled responses of certain groups tend to be nearly identical. This limited exploration and early deterministic policy, which can hinder the scaling process.

We propose the **Clip-Higher** strategy to address this issue. Clipping over the importance sampling ratio is introduced in Clipped Proximal Policy Optimization (PPO-Clip) to restrict the trust region and enhance the stability of RL. We identify that the upper clip can restrict the exploration of the policy, where making an 'exploitation' token more probable is much easier yet the probability of an unlikely 'exploration' token is too tightly bounded to bu uplifted.

Concretely, when $\varepsilon = 0.2$ (the default value of most algorithms) and $\hat{A}_{i,t} > 0$ (the system tries to increase the probability), consider two actions with probabilities $\pi_{\theta_{\text{old}}}(o_i \mid q) = 0.01$ and $0.9$. The upper bounds of the increased probabilities $\pi_\theta(o_i \mid q)$ are $0.012$ and $1.08$, respectively ($\pi_{\theta_{\text{old}}} \cdot (1 + \varepsilon)$). This implies that 'exploitation' tokens with a higher probability (e.g., $0.9$) are not constrained to get even extremely larger probabilities like $0.999$. Conversely, for low-probability 'exploration' tokens, achieving a non-trivial increase in probability is considerably more challenging. Empirically, we also observe that the mean probability of up-clipped tokens is low: $\pi_\theta(o_i \mid q) < 0.2$ (Figure 3a). This finding supports our intuition that the upper clipping threshold indeed restricts the probability increase of low-probability 'exploration' tokens, thereby potentially constraining the exploration of the system.

![image.png](/static/img/13364ebf33a5bb1d1cebcbddc9b3ce34.image.png)

Adhering to the **Clip-Higher** strategy, we decouple the lower and higher clipping range as $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$, as highlighted in Equation 10:

$$
\begin{aligned}
\mathcal{J}_{\text{DAPO}}(\theta) = & \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \\
& \left[\frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min\left(r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}\left(r_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right) \hat{A}_{i,t}\right)\right] \\
\text{s.t.} \quad & 0 < |\{o_i \mid \text{is\_equivalent}(a, o_i)\}| < G.
\end{aligned}
$$

We increase the value of $\varepsilon_{\text{high}}$ to leave more room for the increase of low-probability tokens. As shown in Figure 2, this adjustment effectively enhances the policy's entropy and facilitates the generation of more diverse samples. We keep $\varepsilon_{\text{low}}$ as it is, because increasing it will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space.

Raise the Ceiling: Clip-Higher

> | Resource | Info |
> | :---        |:--- |
> | Paper       | https://arxiv.org/abs/2409.12183 |
> | Code & Data | https://github.com/Zayne-sprague/To-CoT-or-not-to-CoT |
> | Public      | ICLR |
> | Date        | 2025.06.10 |


# Summary Overview

探讨了 CoT 是否会在各类任务上都提升模型性能，但是发现 CoT 仅在数学和逻辑相关的任务上提升大，其他类型的任务收益很小。

![image.png](/static/img/5e037193e349f2c7f5427052588004d5.image.png)

# Main Content
 
 **Findings:**
 1. CoT only helps substantially on problems requiring mathematical, logical, or algorithm reasoning.
 2. CoT primarily helps with the execution step that performs computation and symbolic manipulation, but falls short of what LLMs with tool augmentation can do.

![image.png](/static/img/6393e07b799184b591157d51f1e8a25d.image.png)

![image.png](/static/img/a5c7cd415d28d5733b071a3326c15887.image.png)

**Where does zero-shot CoT improve over direct prompts?**
*On datasets that require math (MATH, GSM8K) or formal logic (ContextHub, MuSR to a lesser degree) to answer the problem.*

**Does the answer format impact where CoT will help?**
*Not much. Free response capabilities required for BigGen Bench may not benefit from pre-planning.*

**Are the gains in Knowledge, Soft Reasoning, and Commonsense significant?**
*Mostly no, except for MMLU, StrategyQA, and MuSR.*

# 🤖

:::info{title=" "}

:::

# Others

[ICLR-2025] To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

> | Resource | Info |
> | :---        |:--- |
> | Paper       | http://arxiv.org/abs/2506.02397 |
> | Code & Data | https://github.com/AgenticIR-Lab/OThink-R1 |
> | Public      | arXiv |
> | Date        | 2025.06.05 |


# Summary Overview

造了一批 Fast / Slow Thinking 的混合微调数据，然后使用了一个增加了 KL 散度的 Loss 进行 SFT。

![image.png](/static/img/4bb7ef46e53f6c870a4605f353dd6019.image.png)

$$\mathcal{L}_{\text{hybrid}} = -\mathbb{E}_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}_{\text{SFT}}} \left[ \log \pi_\theta (y|\mathbf{x}) \right] + \beta_1 \text{KL}(\pi_\theta \| \pi_{\text{LRM}}) + \beta_2 \text{KL}(\pi_\theta \| \pi_{\text{LLM}})$$

# Main Content

**Two challenges:**
1. How to idnetify which scenarios that fast thinking is sufficient?
2. How to empower LRMs to autonomously select the appropriate reasoning mode?

**Three characteristic patterns of redundant reasoning:**
1. Multi-Solution Exploration: Persistent exploration of alternative solutions despite determining a correct answer.
2. Repeated Self-Validation: Excessive re-validation of every intermediate steps in final solutions.
3. Defensive Assumptions: Being overly cautious, take extraneous hypotheses into consideration based on internal knowledge, rather than problem-specific constraints.

![image.png](/static/img/f108c13f899deaf75f40f2c0d4765a94.image.png)

![image.png](/static/img/bbc10491bbfc03f6a15419cee81ff0a5.image.png)

# 🤖

:::info{title=" "}

## 核心创新点

### 动态思维模式切换机制
OThink-R1的最大创新在于提出了一种**内在的快/慢思维模式切换机制**，这是首次将认知科学中的双系统理论成功应用于大型推理模型（LRMs）。该机制能够让模型自主判断问题复杂度，并动态选择适当的推理模式。

### 推理冗余识别方法
论文系统性地分析了LRMs的推理轨迹，并提出了利用识别范式和LLM-Judge将推理轨迹分类为"冗余推理"或"必要推理"的方法。这种分类方法为优化推理效率提供了理论基础。

### 计算效率的显著提升
实验结果表明，OThink-R1在保持准确率的同时，平均减少了约23%的推理冗余。这一成果直接解决了LRMs产生比非推理LLMs多9.78倍token的效率问题。

## 关键学习点

1. **混合范式的必要性**：单一的静态推理模式无法适应不同复杂度的任务，灵活的混合模式是未来发展方向。

2. **认知科学启发的AI设计**：人类认知理论可以有效指导AI系统设计，特别是在推理策略选择方面。

3. **效率与性能的平衡**：通过智能的模式切换，可以在不牺牲性能的前提下大幅提升计算效率。

2. 论文中存在的问题及改进建议：

## 主要问题

### 模式切换标准的模糊性
论文虽然提出了使用LLM-Judge进行分类，但具体的判断标准和阈值设置缺乏详细说明。这可能导致在实际应用中难以复现和调优。

### 缺乏多样化任务评估
实验主要集中在数学和问答任务上，对于其他类型的推理任务（如代码生成、逻辑推理、创意写作等）的适用性未得到充分验证。

### 动态切换的开销分析不足
论文未详细分析模式切换本身带来的计算开销，这在实际部署中可能是一个重要考量因素。

## 改进建议

1. **建立更精确的复杂度评估体系**：开发基于任务特征的多维度复杂度评分系统，包括逻辑深度、知识广度、推理步骤等指标。

2. **扩展评估基准**：在更多样化的任务集上进行评估，包括多模态推理、长文本理解等场景。

3. **引入自适应学习机制**：让模型能够从历史决策中学习，不断优化模式切换的决策边界。

3. 基于论文的内容和研究结果，提出的创新点或研究路径：

## 创新研究方向一：多粒度推理模式
不仅限于快/慢两种模式，而是设计包含超快速（直觉）、快速（模式识别）、中速（结构化推理）、慢速（深度分析）的多层次推理体系。

## 创新研究方向二：协同推理网络
构建多个专门化的推理模块，通过动态路由机制实现模块间的协同工作，不同模块可以并行处理问题的不同方面。

## 创新研究方向三：元推理优化器
开发一个专门的元模型，用于预测和优化主模型的推理路径，实现推理过程的实时优化和资源分配。

## 创新研究方向四：认知负载感知推理
引入认知负载理论，让模型能够评估用户的理解能力和需求，动态调整输出的详细程度和推理深度。

4. 为新的研究路径制定的研究方案：

## 研究方案一：多粒度推理模式系统

### 研究方法
1. **理论建模**：基于认知科学和信息论，建立多粒度推理的数学模型
2. **架构设计**：设计支持四种推理模式的统一神经网络架构
3. **训练策略**：开发分层强化学习算法，让模型学习选择最优推理粒度

## 研究方案二：协同推理网络架构

### 研究方法
1. **模块化设计**：开发专门化的推理模块（逻辑、数学、语言、常识等）
2. **动态路由**：设计基于注意力机制的智能路由系统
3. **协同训练**：开发新的多任务学习范式

## 研究方案三：元推理优化器

### 研究方法
1. **元学习框架**：采用MAML等元学习算法训练优化器
2. **推理轨迹分析**：收集和分析大量推理轨迹数据
3. **在线优化**：开发实时推理路径优化算法

## 研究方案四：认知负载感知推理系统

### 研究方法
1. **用户建模**：构建用户认知能力评估模型
2. **自适应生成**：开发根据用户需求调整输出的算法
3. **交互式优化**：通过用户反馈持续改进

:::

# Others