Source-linked AI summary
CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
Dachuan Shi, Hanlin Zhu, Xiangchi Yuan, Wanjia Zhao, Kejing Xia, Wen Xiao, Wenke Lee
TL;DR
COPT addresses the cost and delayed answers of thinking-first chain-of-thought by drafting an answer before reflective thinking. It uses contrastive continuous-embedding verification and dynamic draft visibility, improving accuracy and token efficiency across evaluated tasks.
Problem
Chain-of-thought can delay plausible answers and consume unnecessary tokens because LLMs may continue reasoning after identifying an answer.
Method
COPT drafts an answer first, uses discrete-versus-continuous support to estimate reliability, and conditionally performs on-policy thinking with dynamic draft visibility.
Results
COPT consistently improves accuracy and token efficiency across mathematics, coding, and agentic reasoning tasks, with reported gains up to 23% and token reductions up to 57%.
Takeaways & Limitations
Training-free contrastive verification makes reversed, on-policy reasoning a practical approach to more cost-effective inference.
Takeaways & Limitations
The theoretical interpretation assumes a finite latent-state model and a mixture-linear continuous-prefix assumption.
Abstract
from arXiv · showhide
Chain-of-thought (CoT) is a standard approach for eliciting reasoning capabilities from large language models (LLMs). However, the common CoT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative reasoning. In this paper, we introduce CopT, a reformulated reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous embeddings as inference-time contrastive verifiers. Specifically, it contrasts the model's support for the same generated tokens under discrete-token inputs and continuous-embedding inputs, yielding a sequence-level reverse KL estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, CopT performs further on-policy thinking, where a second KL estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic reasoning tasks, CopT improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.
1 Introduction
COPT reverses chain-of-thought by drafting an answer before thinking, then uses contrastive verification to decide whether correction and controlled draft exposure are needed.
- Performative reasoning can make LLMs complete extensive reasoning even after internally identifying a plausible answer.
- COPT first elicits a draft answer, then performs on-policy thinking conditioned on it for reflection and correction.
- Continuous embeddings are recast as inference-time verifiers by contrasting support for identical generated tokens under discrete and continuous inputs.
- COPT retains explicit, readable reasoning while using continuous-space uncertainty information without directly generating reasoning in continuous embeddings.
- A sequence-level reverse KL estimator assesses draft reliability and triggers subsequent thinking when the draft appears insufficiently reliable.
- A second KL estimator controls draft-answer visibility during thinking, preserving useful partial information while reducing exposure to potentially misleading content.
2 Related Work
Related work includes explicit natural-language reasoning and latent reasoning with continuous embeddings, which preserve different amounts of information during generation.
- Explicit reasoning uses natural-language traces to improve LLM performance on complex tasks and is elicited through prompting or post-training.
- Latent reasoning operates on continuous embeddings instead of committing to discrete tokens at every reasoning step.
- Continuous representations can encode information from the full next-token distribution, whereas discrete decoding retains only the sampled token.
3 Methodology
COPT uses a low-cost draft stage, contrastive reliability estimation, and chunk-wise on-policy thinking with dynamically controlled draft visibility.
- COPT reformulates reasoning into a leading draft-answer stage followed, when necessary, by trailing on-policy thinking.
- Reliability estimation: The method estimates draft reliability with a normalized sequence-level reverse KL estimator before selectively triggering on-policy thinking.
- Reliability estimation: During draft generation, COPT caches chosen-token probabilities and probability-weighted continuous embeddings that preserve uncertainty information.
- Reliability estimation: Reliability estimation compares the discrete-prefix distribution with a continuous-prefix distribution using the cached embeddings and original answer-token probabilities.
- On-Policy Thinking Conditioned on the Draft Answer: When κa exceeds the reliability threshold τa, COPT initiates a subsequent thinking process to correct or support the draft answer.
- On-Policy Thinking Conditioned on the Draft Answer: COPT partitions on-policy thinking into fixed-length chunks and evaluates chunk-level continuation reliability with a second reverse KL estimator.
- On-Policy Thinking Conditioned on the Draft Answer: The draft answer is made visible or hidden across chunks according to stability estimates, allowing useful information while limiting misleading content.
4 Theoretical Analysis
The analysis interprets COPT’s reverse-KL estimator as measuring answer-relevant uncertainty under a mixture-prefix assumption, rather than latent-state uncertainty alone. It extends this interpretation across answer positions and notes that cached probabilities and embeddings make estimator computation low-overhead.
- COPT’s reverse-KL estimator is designed to measure answer-relevant uncertainty rather than uncertainty over latent reasoning states themselves.The estimator responds to uncertainty only when it affects the next answer-token distribution.
- The reliability and visibility estimators use cached chosen-token probabilities and continuous embeddings, incurring only small overhead after online generation.The estimators are calculated on sequences that have already been generated.
- Under the mixture-linear continuous-prefix assumption, the expected local reverse-KL contribution equals the mutual information between latent state S and emitted answer token A.This connects the estimator to information carried by answer tokens about unresolved reasoning states.
- High latent-state entropy contributes zero when all plausible states induce the same next-token distribution, because the emitted token then carries no information about the selected state.Thus, uncertainty is harmless when plausible latent states agree on the next answer.
- Applying the argument token by token, the normalized draft score estimates the average answer-relevant uncertainty across positions conditioned on preceding context.The position-wise interpretation extends the single-position result to the complete draft answer.
5 Experiments
COPT is evaluated across model families and scales, while Table 1 compares it with reasoning baselines on mathematics, coding, and STEM benchmarks using Qwen3-8B. The experiments vary reasoning effort to target comparable or higher accuracy.
- Experimental Settings: COPT is evaluated on pure Transformer and hybrid Gated-DeltaNet Qwen models at 2B, 8B, and 35B scales.The selection spans dense and sparse mixture-of-experts architectures as well as multiple model families.
- Experimental Settings: Table 1 compares COPT with standard COT and greedy COT on mathematics, coding, and STEM reasoning benchmarks using Qwen3-8B.The table reports both accuracy and generation length.
- Experimental Settings: COPT reports settings targeting accuracy comparable to or higher than COT, with increased reasoning effort used for higher-accuracy results.Green blocks denote the increased-effort COPT settings.
& STEM
COPT improves the accuracy–token trade-off across mathematics, coding, STEM, and agentic reasoning by selectively allocating additional thinking and controlling draft visibility. Its estimators identify unreliable drafts and adjust correction effort, yielding lower token use and latency at comparable or higher accuracy.
- Mathematics, coding, and STEM: COPT improves accuracy while reducing generation length across most mathematics, coding, and STEM benchmark settings.The evaluation uses Qwen3-8B and compares COPT with standard and greedy COT.
- Mathematics, coding, and STEM: +0.23% GSM8K accuracy with 55.1% fewer generated tokens, and +0.20% Math500 accuracy with 27.6% fewer tokens.With increased reasoning effort, accuracy gains reach +3.34% on AIME24 and +2.92% on AIME25.
- Mathematics, coding, and STEM: +1.83% HumanEval accuracy with 56.8% fewer tokens at matched accuracy levels.With increased reasoning effort, accuracy gains are +3.66% on HumanEval, +6.67% on LeetCode-Contest, +1.39% on MBPP, and +2.02% on GPQA Diamond.
- Comparison with continuous-generation methods: COPT achieves the best accuracy with the fewest generated tokens compared with Soft-Thinking and SwiReasoning.Compared with SwiReasoning, COPT uses 18.3%, 8.0%, 36.4%, and 18.0% fewer tokens across four benchmarks.
- Efficiency and controllability: COPT reduces latency by 37% at comparable or higher accuracy while sweeping τa and τr to control the accuracy–token trade-off.The latency result was measured on a single H200 GPU.
- Agentic reasoning: On agentic tasks, COPT maintains comparable or higher accuracy with 40.6% fewer tokens on Qwen3.5-2B and 44.7% fewer tokens on Qwen3.5-35B-A3B.On ZebraArena, accuracy gains reach +23.03% on the large split, while tokens fall 19.6%.
- Ablation studies: κa concentrates stricter selections on erroneous drafts, while κr increases correction of erroneous drafts as draft visibility becomes stricter.The estimators provide reliability-based allocation rather than uniform additional thinking.
6 Conclusion
COPT reverses the usual reasoning order by drafting an answer first and invoking on-policy thinking when the draft appears unreliable. The training-free method uses continuous embeddings as inference-time verifiers and improves accuracy and token efficiency across reasoning tasks.
- Conclusion: COPT first elicits a draft answer, then invokes conditioned on-policy thinking for reflection and correction when the draft appears unreliable.This reverses the usual order of thinking and answering.
- Conclusion: COPT contrasts discrete-token and continuous-embedding inputs to verify draft reliability during inference.The method keeps reasoning explicit while using continuous embeddings as verifiers.
- Conclusion: Experiments across mathematics, coding, and agentic reasoning show consistent improvements in accuracy and token efficiency.The paper presents this combination as a practical path toward more cost-effective inference-time reasoning.
- Impact and scope: COPT is an inference-time method whose broader benefits and risks depend on how reasoning LLMs are deployed.The impact statement notes possible usefulness for beneficial applications and harmful applications without safeguards.
C Limitations
COPT’s limitations concern estimator variance, probability-access requirements, task-dependent answer-span extraction, and draft-length calibration. The method uses default settings to remain broadly applicable, with a smaller cap for multi-turn ZebraArena.
- COPT’s KL estimators use a single realized inference trajectory, which may have higher variance than multisample estimates.Token averaging mitigates this variance; more adaptive or lower-variance estimators remain future work.
- COPT requires next-token probabilities, making it less convenient for closed APIs that return only text.This access is natural for open-weight models and systems exposing logits.
- Answer-content granularity: Answer-span-specific κa estimation improves GSM8K accuracy from 95.98% to 96.36% and reduces average generation length from 961 to 885 tokens.The finer-grained setting is useful when a compact answer region can be identified.
- Answer-content granularity: Task-dependent answer extraction is not always available beyond mathematics, so the paper computes κa over the whole draft by default.This preserves generality across domains despite potentially less precise reliability estimation.
- Draft-answer length: A 1,024-token draft cap is used by default because some challenging tasks produce repetitive or uninformative drafts before extended thinking.The cap acts as a practical safeguard, while ZebraArena uses 512 tokens to limit accumulated multi-turn context growth.
D.3 Detailed Results on the LeetCode-Contest Benchmark
On LeetCode-Contest, COPT improves accuracy across splits while offering controllable accuracy–generation-length trade-offs. The largest gain occurs on hard problems, where further reflection and correction are especially useful.
- COPT improves overall LeetCode-Contest accuracy from 59.44% to 66.11% with a similar token budget.
- On the hard split, COPT raises accuracy from 43.18% to 54.55%, an absolute gain of +11.37%.The results associate the largest benefit with problems requiring further draft-answer reflection and correction.
- Accuracy-efficiency trade-offs: The increasing-effort COPT setting improves all three splits over standard COT with only a small increase in generation length.
- Accuracy-efficiency trade-offs: The lower-token COPT setting reduces token usage by 4.3% while improving overall accuracy by +1.67%.It lowers easy-split accuracy but preserves the hard-split gain and improves the medium split.
- Accuracy-efficiency trade-offs: The two COPT settings demonstrate controlled trade-offs between generation length and accuracy.
E.2 Benchmark Details
The evaluation covers 10 mathematics, coding, and agentic reasoning benchmarks, using execution-based, answer-based, function-calling, and multi-turn tool-augmented assessments. KL estimators are defined over fixed-length draft and thinking sequences partitioned into chunks.
- Benchmark coverage: COPT is evaluated on 10 benchmarks spanning mathematics and STEM, coding, and agentic reasoning.The suite includes GSM8K, Math500, AIME 2024, AIME 2025, GPQA Diamond, HumanEval, LeetCode-Contest, MBPP, BFCL v4, and ZebraArena.
- Mathematics and STEM: The mathematics benchmarks range from grade-school arithmetic and high-school competition problems to expert-verified STEM questions and competition-level AIME tasks.
- Coding: HumanEval, LeetCode-Contest, and MBPP evaluate coding through Python execution or tests over generated solutions.LeetCode uses 180 contest problems, while MBPP’s sanitized test split contains 257 Python problems.
- Agentic reasoning: BFCL v4 evaluates accurate function and tool invocation across non-live and live splits containing 2,501 problems.
- Agentic reasoning: ZebraArena evaluates multi-turn tool-augmented reasoning under missing clues, requiring environment queries before uniquely verifiable constraint solving.
- KL-estimator setup: The draft-answer and on-policy trajectories define sequence distributions for KL estimation, with on-policy thinking partitioned into chunks and draft visibility controlled per chunk.The continuous-prefix distributions replace selected discrete content with cached continuous embeddings.
F Additional Derivations for Answer-Relevant Uncertainty
The derivation shows that COPT’s reverse-KL estimator measures uncertainty relevant to the next answer token rather than latent-state entropy. It vanishes when latent states disagree neither in their induced answer distributions nor in answer-relevant behavior.
- Under the derivation’s mixture-linear model, the reverse-KL estimator measures uncertainty affecting the next answer token rather than uncertainty over latent reasoning states.
- Harmless latent-state uncertainty: If all supported latent states induce the same next-token distribution, the reverse-KL score is zero even when latent-state entropy H(S) is large.
- Harmless latent-state uncertainty: Many latent states can remain encoded in a continuous prefix without harming reliability when they agree on the next-token distribution.
- The derivation’s nonnegativity argument establishes the estimator’s answer-relevant uncertainty interpretation under the stated assumptions.
- Deterministic-token case: In the deterministic-answer setting, the reverse-KL score equals the entropy of the induced answer token g(S), not the entropy of latent state S.