Resource	Info
Paper	https://arxiv.org/abs/2505.15778
Code & Data	https://github.com/eric-ai-lab/Soft-Thinking
Public	arXiv
Date	2025.07.11

Main Content

作者提出了 Soft Thinking，一种无需训练的方法来在连续的概念空间中进行推理的方法。

A fundamental limitation of standard CoT reasoning is its inherently unidirectional and sequential nature: at each step, the model samples a single token, committing to one specific branch of the reasoning path.

In this work, we propose a new perspective: instead of constraining LLMs to reason within the discrete, sequential space of language tokens, we aim to enable LLMs to reason with soft, abstract concepts, which encompass more general and fine-grained semantics and retain information about multiple possible paths. This retains the original distribution of the next step. At each step, we construct a new embedding from a concept token by probability-weighting all tokens embeddings, which form the continuous concept token. This approach allows the model to represent and process abstract concepts, endowing each output token with more nuanced and fine-grained semantics, and enabling the processing of multiple paths conceptually.

As a result, we introduce a Cold Stop mechanism to further boost efficiency and address the challenge of generation collapse (e.g., repetition) caused by out-of-distribution(OOD) inputs, where certain concept tokens may be unseen during training. To be specific, Cold Stop monitors the entropy of the model's output distribution at each step and terminates the reasoning process early when the model demonstrates high confidence (i.e., low entropy) over several consecutive steps. This mechaism prevents unnecessary computation and mitigates the risk of model collapse when dealing with OOD inputs, ensuring more robust and efficient reasoning.

Two major advances:

1. by operating in the continuous concept space formed as a convex combination of all token embeddings, the model can capture and manipulate abstract concepts and detailed semantic information;
2. because each concept token keeps a probability distribution from all possible next tokens, the model can implicitly and efficiently explore multiple reasoning paths in parallel, rather than being limited to a single trajectory.

In language models with fewer than 7B parameters, the input embedding layer and the output language model head are typically weight-tied, enabling continuous-space reasoning by aligning the input and output spaces after extensive training. In contrast, for models with more than 7 billion parameters, these components are typically decoupled, meaning that the hidden states and input embeddings reside in different spaces. Directly using hidden states as input embeddings leads to siginficant representational mismatch, which is difficult to bridge even with extensive retraining.

Soft Thinking: Reasoning in a Continuous Concept Space

Definition 1 (Concept Token). At any intermediate thinking step, let $p\in\Delta^{|V|-1}$ be the LLM-produced probability distribution over the vocabulary. We call this probability vector a concept token, denoted by

Unlike a traditional step that collapses the distirbution to a single token id, the concept token preserves the full distribution of every possible next step.

Definition 2 (Continuous Concept Space). Let $E\in\mathbb{R}^{|V|\times d}$ be the embedding matrix and $e(k)=E[k]$ the embedding of the $k$-th vocabulary item. The continuous concept space is the convex combination of all embedding vectors.

i.e. the set of all probability-weighted mixtures of token embeddings. Note that this is different from the usual semantic space, which is modeled as a $d$-dimensional real vector space.

Reasoning Process. Soft Thinking only replaces the intermediate thinking step of the standard CoT. At each step of soft thinking, the model generates a concept token. Then, in the next step, the concept token $ct$ is injected back into the LLM by the embedding of concept token:

When the most probable token for a certain concept token is the end-of-thinking, the intermediate reasoning process stops, and the model switches to generating the output. All output stage tokens $y_j$ are sampled in the usual discrete manner; only the intermediate thinking phase flows through the continuous concept space defined above.

Why methodname Helps. Using concept tokens allows the model to avoid making hard decisions too early. Instead of selecting a single token at each step, the model keeps the full probability distribution over vocabulary. This gives it the flexibility to explore different reasoning paths, especially when it's unsure. By working in this continuous concept space, the model can represent more abstract concepts that don't map cleanly to a single word. These abstract concepts can later evolve into more concrete thoughts as reasoning continues. This flexibility helps the model think more clearly, avoid early mistakes, and better handle complex multi-step problems.

Cold Stop While concept tokens enable more abstract reasoning, feeding in continuous concept tokens during inference places the model in an out-of-distribution (OOD) regime. This can lead to model collapse if the reasoning process continues for too long without correction. To mitigate this, we propose a Cold Stop mechanism that dynamically stops intermediate reasoning when the model becomes overconfident. At each step, we compute the entropy of the concept token:

Since $Soft Thinking$ preserves the entire probability distribution at each step, the entropy serves as a natural signal for uncertainty, which is often used in LLMs to evaluate the quality of generation. Low entropy, typically represents "cold" in physics, indicates that the model is confident in its prediction~cite{entropy}, and thus can conclude soon. Given an entropy threshold $\tau$ and a required number of consecutive confident steps $k$, we apply the following rule:

• If $H(p) < \tau$, increment a low-entropy step counter; otherwise, reset the counter.
• When the counter reaches $k$, we insert an end-of-thinking token $\langle/\mathrm{think}\rangle$ to conclude reasoning and begin final answer generation.

This strategy avoids unnecessary computation and prevents the model collapse under OOD conditions, while preserving the benefits of soft thinking through an entropy-based confidence measure.

🤖

Others