Source-linked AI summary
Recursive Think-Answer Process for LLMs and VLMs
Byung-Kwan Lee, Youngchae Chee, Yong Man Ro
TL;DR
Single-pass Think–Answer models may stop with uncertain or incorrect reasoning because existing methods do not use confidence to trigger self-correction. R-TAP adds confidence-guided recursive Think–Answer cycles with dedicated rewards, and reports consistent gains across language and vision-language reasoning benchmarks alongside fewer self-corrections. Its current implementation generates all T trajectories in parallel, creating computational and memory overhead.
Problem
Single-pass Think–Answer inference can leave uncertain or incorrect reasoning uncorrected because confidence is not used to decide whether further refinement is needed.
Method
R-TAP uses a learned Confidence Generator and Recursively Confidence Increase Reward plus Final Answer Confidence Reward to guide recursive Think–Answer refinement.
Results
R-TAP reports consistent performance improvements across diverse LLM and VLM reasoning benchmarks and fewer “Oops!”-style self-corrections during inference.
Takeaways & Limitations
Confidence-guided recursion supports more reliable, stable, and faster inference-time reasoning across language and multimodal models.
Takeaways & Limitations
R-TAP’s current batch-dependent implementation generates all T recursive trajectories in parallel, causing computational and memory overhead even when confidence emerges early.
Abstract
from arXiv · showhide
Think-Answer reasoners such as DeepSeek-R1 have made notable progress by leveraging interpretable internal reasoning. However, despite the frequent presence of self-reflective cues like "Oops!", they remain vulnerable to output errors during single-pass inference. To address this limitation, we propose an efficient Recursive Think-Answer Process (R-TAP) that enables models to engage in iterative reasoning cycles and generate more accurate answers, going beyond conventional single-pass approaches. Central to this approach is a confidence generator that evaluates the certainty of model responses and guides subsequent improvements. By incorporating two complementary rewards-Recursively Confidence Increase Reward and Final Answer Confidence Reward-we show that R-TAP-enhanced models consistently outperform conventional single-pass methods for both large language models (LLMs) and vision-language models (VLMs). Moreover, by analyzing the frequency of "Oops"-like expressions in model responses, we find that R-TAP-applied models exhibit significantly fewer self-reflective patterns, resulting in more stable and faster inference-time reasoning. We hope R-TAP pave the way evolving into efficient and elaborated methods to refine the reasoning processes of future AI.
1. Introduction
R-TAP addresses the unreliability of single-pass Think–Answer reasoning by using confidence-guided recursive cycles for self-correction. It applies a confidence generator and two recursive rewards to improve reasoning across language and multimodal models.
- Motivation: Single-pass Think–Answer models can leave inaccurate or uncertain reasoning uncorrected despite self-reflective cues such as “Oops!”Existing reinforcement-learning methods optimize single trajectories without using confidence to trigger further refinement.
- Approach: R-TAP enables LLMs and VLMs to iteratively refine reasoning through confidence-guided Think–Answer cycles.The model evaluates confidence after each cycle and runs another cycle when confidence is low.
- Approach: The Confidence Generator estimates the correctness of individual reasoning paths and guides recursive refinement.It evaluates question–Think–Answer pairs and produces confidence scores used for reward construction.
- Approach: R-TAP combines Recursively Confidence Increase Reward with Final Answer Confidence Reward to train higher-confidence reasoning trajectories.These rewards encourage confidence to improve across cycles and remain high for the final prediction.
- Results: R-TAP improves performance across diverse language and vision-language reasoning benchmarks while reducing “Oops!”-style self-corrections during inference.The reported outcome is more reliable and faster inference-time reasoning with fewer failures along the trajectory.
2. Related Work
Prior reasoning methods improve search or refinement but generally lack intrinsic confidence assessment within Think–Answer inference. R-TAP introduces confidence-aware recursive reasoning for both LLMs and VLMs.
- Prior approaches: LLMs and VLMs have advanced reasoning through scaling, instruction tuning, and Think–Answer separation.Think–Answer models such as DeepSeek-R1 and OpenAI o1 demonstrate strong mathematical and programming performance.
- Prior approaches: Structured reasoning methods often rely on external sampling or reranking rather than intrinsic self-evaluation.Tree- and graph-based approaches explore larger reasoning spaces, while sampling-based strategies use mechanisms outside the model’s internal confidence assessment.
- Research gap: Existing Think–Answer systems usually terminate after one reasoning trajectory even when reasoning is uncertain or contains self-reflective cues.Sampling-based reranking only partially mitigates this limitation and does not let the model decide internally whether more reasoning is needed.
- Research gap: Prior iterative-refinement methods lack a principled confidence predictor that dynamically invokes additional reasoning cycles.The related-work discussion identifies this as an unmet capability in Think–Answer models.
- R-TAP: R-TAP provides a confidence-aware recursive process that measures confidence increases across cycles and final-prediction confidence for LLMs and VLMs.This positions the method as a unified mechanism for self-corrective reasoning beyond static single-pass inference.
3. R-TAP: Recursive Think–Answer Process
R-TAP extends Think–Answer reasoning with confidence-guided recursive cycles, allowing models to continue when uncertain and terminate when sufficiently confident. Its framework combines recursive generation, confidence estimation, and rewards for refinement and final confidence while preserving inference-time efficiency.
- Overall Framework: R-TAP recursively generates Think–Answer responses, with recursion depth T fixed during training but continuation or termination determined internally at inference.The recursive trajectory is represented as O = {o(1), o(2), ..., o(T)} conditioned on question q.
- Overall Framework: The training objective uses GRPO over recursive trajectories, with total trajectory rewards encouraging further reasoning under low confidence and stopping at satisfactory confidence.The total reward includes confidence-based terms alongside format, answer-correctness, and length rewards.
- Confidence Generator: The Confidence Generator Cϕ estimates the reliability of each recursive response and is removed at inference, so R-TAP adds no inference-time cost.It receives a question and Think–Answer response and produces a confidence score; its architecture replaces the language head with a confidence head followed by a sigmoid.
- Confidence Generator: Confidence Generator training uses binary correctness labels from sampled Think–Answer responses to learn reliability estimates for recursive trajectories.The method samples N responses per question, separates correct and incorrect responses, and optimizes a binary classification objective.
- Confidence-Based Reward Design: R-TAP uses two complementary rewards: recursive confidence increase encourages meaningful refinement, while final answer confidence encourages sufficiently confident termination.The effective recursion depth M is bounded by the preset maximum T, and the final-confidence reward uses a preset threshold τ.
- Evaluation Scope: The framework is presented as applicable to LLMs and VLMs, with evaluations covering coding, mathematical, multimodal, and math-related reasoning benchmarks.The cited comparisons include HumanEval, MBPP, LiveCodeBench, AIME, AMC, MATH500, MinervaMath, OlympiadBench, R1-OneVision, MathVista, MathVerse, MathVision, and WeMath.
4. Experiments
Experiments evaluate R-TAP across language and vision-language models, component choices, confidence estimators, recursion depths, and inference efficiency. The reported results show consistent performance gains, reduced erroneous reasoning, and shorter inference time.
- Overall performance: R-TAP produces dramatic and consistent performance improvements across multiple LLMs and VLMs, including multimodal benchmarks.The experiments apply R-TAP to several backbone models and datasets in both language and vision-language settings.
- Ablations: Confidence estimation with the proposed method achieves the highest accuracy among the compared approaches, while recursion depths from 1 to 4 improve accuracy.The comparisons cover alternative confidence estimation methods and recursion-depth settings for both LLMs and VLMs.
- Reasoning stability: The number of “Oops”-style tokens decreases monotonically during training, with deeper recursion producing faster and greater reductions.The tokens serve as a proxy for erroneous reasoning.
- Inference efficiency: R-TAP lowers erroneous reasoning across all evaluated datasets and yields significantly shorter inference time than baseline and other state-of-the-art models.The reported inference-time comparison is presented in log-hours across benchmarks.
5. Discussion and Conclusion
R-TAP augments LLMs and VLMs with learned confidence-guided recursion and improves performance, but its parallel generation of all recursive trajectories creates computational and memory overhead.
- R-TAP uses a learned confidence generator with recursive confidence-increase and final-answer confidence rewards to refine Think–Answer trajectories.
- R-TAP supports up to T recursive steps, but current batch-dependent implementations generate all T trajectories in parallel even when confidence emerges early.This design simplifies parallelization while introducing significant computational and memory overhead.
A. Full Responses for Multimodal Example
The multimodal example illustrates recursive correction: the model revises an initial counting error, verifies visual details, and identifies the only flower satisfying both criteria.
- The example asks the model to choose a flower with five petals and a stem containing three leaves for the ladybird.
- The model refines its answer over multiple reasoning cycles by correcting counting errors and checking object-level visual details.
- After recursive refinement, the model identifies the only flower meeting both visual criteria.
B. Algorithm of Recursive Think-Answer Process
The R-TAP algorithm samples recursive outputs, retains responses through early correct answers, computes confidence-based rewards, and alternately updates the confidence generator and reasoning model.
- R-TAP initializes a reference model, training model, and frozen previous model from pretrained LLMs or VLMs.
- The algorithm samples multiple outputs through a maximum of M recursive steps and extracts responses up to early correct answers.
- It computes rewards and advantages for recursive outputs using the confidence generator and answer parsing.
- Training alternates between updating the confidence generator and updating the LLM or VLM.
C. Detailed Comparison between Performance and Computation Complexity
R-TAP compares favorably with refinement, verification, and self-consistency methods while reducing inference computation. Its confidence-guided training shapes recursive reasoning and supports adaptive continuation or termination.
- Comparison with Related Refinement and Verification Methods: R-TAP learns when to continue or terminate reasoning under the same output token budget, unlike methods relying on explicit feedback loops, majority voting, or external verification.The comparison includes Reflexion, Self-Consistency, Self-Refine, and Self-Verification.
- Token Efficiency and Computational Cost: R-TAP reduces output tokens by approximately 2–3× compared to self-consistency, self-refine, and verification-based baselines under the same output token budget, while achieving higher accuracy.The comparison covers inference-time token efficiency despite recursive sampling during training.
- Effect of Majority Voting: R-TAP-trained models achieve strong performance with N = 1, while additional self-consistency voting yields only marginal gains.Increasing N improves performance across all models, but the added benefit is limited for R-TAP-trained models.
- Recap: R-TAP combines GRPO-based trajectory training, recursive rewards from confidence and intermediate results, suppression of unnecessary Oops-style refinement, and improved inference efficiency.These components connect the training objective with reduced refinement steps at inference.
- Confidence-Guided Reasoning: Confidence is used as an internal reinforcement signal during training, continuously modulating recursive reasoning depth rather than performing posthoc correction at inference time.This distinguishes R-TAP from approaches that use confidence for reranking, filtering, or verification after generation.
- Future Works: Future work targets adaptive recursion strategies that dynamically determine whether and how deeply to continue Think–Answer cycles while minimizing redundant computation.The planned direction replaces fixed recursion depth or static confidence thresholds with adaptive mechanisms.