PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Think-Answer Process for LLMs and VLMs

Byung-Kwan Lee, Youngchae Chee, Yong Man Ro

arXiv:2603.02099v2cs.CL

TL;DR

Think–Answer models typically stop after one reasoning pass, even when uncertain, without internal confidence assessment or further refinement. R-TAP adds confidence-guided recursive cycles; experiments report consistent gains across language and vision-language benchmarks, alongside fewer “Oops!”-style corrections.

  • Problem

    Existing Think–Answer systems generally terminate after one reasoning pass, even when reasoning is uncertain or self-reflective cues suggest further assessment may be needed.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    R-TAP uses a confidence generator and two complementary rewards to guide iterative Think–Answer cycles that refine reasoning.

    Source: §1 · p. 2

  • Results

    R-TAP delivers consistent performance improvements across diverse language and vision-language reasoning benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    R-TAP reduces “Oops!”-style self-corrections, indicating more reliable and faster inference-time reasoning.

    Source: §1 · p. 3

    R-TAP generates all T recursive trajectories in parallel, introducing computational and memory overhead even when a confident answer emerges early.

    Source: §5 · p. 8

Abstract

from arXiv · show

Think-Answer reasoners such as DeepSeek-R1 have made notable progress by leveraging interpretable internal reasoning. However, despite the frequent presence of self-reflective cues like "Oops!", they remain vulnerable to output errors during single-pass inference. To address this limitation, we propose an efficient Recursive Think-Answer Process (R-TAP) that enables models to engage in iterative reasoning cycles and generate more accurate answers, going beyond conventional single-pass approaches. Central to this approach is a confidence generator that evaluates the certainty of model responses and guides subsequent improvements. By incorporating two complementary rewards-Recursively Confidence Increase Reward and Final Answer Confidence Reward-we show that R-TAP-enhanced models consistently outperform conventional single-pass methods for both large language models (LLMs) and vision-language models (VLMs). Moreover, by analyzing the frequency of "Oops"-like expressions in model responses, we find that R-TAP-applied models exhibit significantly fewer self-reflective patterns, resulting in more stable and faster inference-time reasoning. We hope R-TAP pave the way evolving into efficient and elaborated methods to refine the reasoning processes of future AI.

1. Introduction

R-TAP addresses the limits of one-pass Think–Answer reasoning with confidence-guided recursive refinement, and the paper reports improved reasoning performance with fewer self-corrections and faster inference. It uses a confidence generator and two recursive rewards to train LLMs and VLMs to refine their answers across cycles.

1. Introduction

  • Think–Answer models such as OpenAI’s o1 [42] and DeepSeek-R1 [35] separate reasoning from answering and have shown problem-solving gains, including in mathematical reasoning and competitive programming.The paradigm has also been extended to vision-language reasoning [24] [40] [78] [82] [105] [118].Source: §1 · pp. 1–2
  • Single-pass models stop after one Think–Answer pair even when reasoning is uncertain, while GRPO-style methods optimize one trajectory without confidence-based self-evaluation or correction.This leaves inaccurate or inconsistent reasoning unrefined, a concern for complex or high-stakes tasks.Source: §1 · p. 2
  • R-TAP lets LLMs and VLMs repeat Think–Answer cycles when confidence is low, using a Confidence Generator to assess and guide refinement.The model revisits and improves previous reasoning instead of stopping after one pass.Source: §1 · p. 2
  • Its two rewards encourage confidence to rise across cycles and remain high in the final answer.The rewards are Recursively Confidence Increase Reward and Final Answer Confidence Reward.Source: §1 · pp. 2–3
  • Training first fine-tunes the Confidence Generator on binary correctness labels, then applies GRPO with recursive rewards; the generator adds no inference-time cost.During RL, the Confidence Generator is also trained to predict confidence scores for updated model responses.Source: §1 · p. 3
  • R-TAP delivers strong, consistent improvements across language and vision-language reasoning benchmarks, with fewer “Oops!”-style self-corrections and faster inference-time reasoning.The paper attributes these results to confidence-guided recursive refinement.Source: §1 · p. 3

2. Related Work

Earlier LLM/VLM and Think–Answer approaches expanded reasoning capabilities but largely remained single-pass or lacked internal confidence-guided recursion; R-TAP introduces selective recursive refinement.

Evolution of LLMs and VLMs

  • LLMs and VLMs advanced from GPT-3 and RLHF-aligned InstructGPT [80] and ChatGPT to multimodal GPT-4 [6], alongside open-source architectures and scaled VLM reasoning.Source: §2 · p. 3

Think-Answer Reasoning

  • Program-of-Thoughts [18], Tree of Thoughts [106], and Graph of Thoughts [12] explore larger reasoning spaces, but typically rely on external sampling or reranking rather than intrinsic self-evaluation.Source: §2 · p. 4
  • DeepSeek-R1 [35] and OpenAI o1 [42] demonstrate Think–Answer reasoning for math and programming, while LMM-R1 [82], Vision-R1 [40], R1-V, and R1-Zero [118] apply slowthinking to visual problem solving.Source: §2 · p. 4
  • Existing Think–Answer systems usually stop after one pair despite uncertainty or “Oops!” cues, and reranking cannot assess confidence or trigger further reasoning internally.Source: §2 · p. 4

Recursive and Confidence-Guided Refinement

  • No prior Think–Answer model explicitly used a confidence predictor to invoke additional reasoning cycles, despite earlier external verification [96] and heuristic feedback [87].Source: §2 · p. 4
  • R-TAP adds confidence-aware recursive Think–Answer inference that detects low-confidence reasoning and selectively reengages, using cycle-wise confidence increases and final-answer confidence.Source: §2 · p. 4

3. R-TAP: Recursive Think–Answer Process

R-TAP extends single-pass Think–Answer reasoning with confidence-guided recursive responses, allowing models to refine uncertain reasoning or stop when sufficiently confident. A confidence generator and complementary rewards support this process while the generator’s removal at inference preserves inference-time efficiency.

3.1. Problem Definition and Overall Framework

  • R-TAP generates recursive Think–Answer responses, using a fixed recursion depth during training and allowing the model to continue or terminate internally at inference.This addresses single-pass models’ tendency to stop even when reasoning is uncertain or incorrect.Source: §3.1 · p. 5

3.2. Confidence Generator

  • The Confidence Generator scores each response’s reliability on a 0-to-1 scale and is trained to predict correctness, then removed at inference to avoid added inference-time cost.It is initialized from the reference model structure with a confidence head and sigmoid activation; pre-training uses balanced binary classification.Source: §3.2 · p. 5

3.3. Confidence-Based Reward Design

  • R-TAP combines a recursive confidence-increase reward for meaningful refinement with a final-answer confidence reward that encourages stopping above a preset threshold.The effective recursion depth is bounded by the training depth T; the total reward also includes format, answer-correctness, and length terms.Source: §3.3 · pp. 5–6
  • R-TAP is presented as applicable to LLMs and VLMs, enabling internal uncertainty assessment and selective refinement while preserving standard single-pass inference-time efficiency.The confidence generator is used only during training, supporting stable and faster reasoning without its inference-time cost.Source: §3.2 · p. 5, §3.3 · p. 6

4. Experiments

R-TAP improves reported LLM and VLM performance, while ablations support the contributions of its reward components and confidence estimation. The experiments also report fewer erroneous-reasoning indicators and shorter inference times with R-TAP.

4.1. Implementation Details

  • Training used T=4 recursion depths and G=12 outputs, while evaluation removed the Confidence Generator and used only the trained LLMs or VLMs.The described training setup also used NVIDIA A100 80GB GPUs and DeepSpeed with ZeRO-3.Source: §4.1 · pp. 6–7

4.2. Validating R-TAP

  • R-TAP consistently improves LLM performance across evaluated models, with reported gains for Phi-4-reasoning models approaching OpenAI-o1 and -o3.The paper also reports dramatic improvements for recently released Qwen3 and AM-Thinking-v1 models against open- and closed-source reasoning LLMs.Source: §4.2 · p. 7
  • R-TAP consistently improves performance across the reported VLM evaluations, including multimodal benchmarks.The paper reports gains for Skywork-R1V2 and for Noisy-Rollout-7B, R1-OneVision-7B, and MM-Eureka-7B and -32B.Source: §4.2 · p. 7

4.3. Ablation Studies

  • Removing any of R-TAP’s confidence-increase, final-answer-confidence, or answer reward components lowers peak accuracy in both LLM and VLM settings.The component ablations use Phi-4-reasoning-plus [5] for LLMs and Skywork-R1V2 [100] for VLMs.Source: §4.3 · p. 7, Table 10 · p. 8, Table 11 · p. 8
  • R-TAP’s Confidence Generator achieves the highest reported accuracy among the compared confidence-estimation methods for both LLMs and VLMs.Comparisons include calibration via ECE [31] [34], Semantic Equivalence Entropy [49], the “IDK” token [21], and iterative LLM generation [64] [115].Source: §4.3 · pp. 7–8, Table 12 · p. 8, Table 13 · p. 8
  • Increasing Think–Answer recursion depth from 1 to 4 improves accuracy for both LLMs and VLMs.The paper reports these depth comparisons in the LLM and VLM recursion-depth tables.Source: Table 14 · p. 8, Table 15 · p. 8, §4.3 · p. 8
  • “Oops”-style tokens, used as a proxy for erroneous reasoning, decline during training and with deeper recursion, while evaluation errors and inference time fall across benchmarks.The reported inference-time reduction is in log-hours.Source: Figure 6 · p. 8, §4.3 · p. 8

5. Discussion and Conclusion

R-TAP adds a learned confidence generator and two confidence rewards to LLMs and VLMs, enabling iterative refinement that yields substantial performance gains and narrows the gap between smaller and much larger models.

5. Discussion and Conclusion

  • R-TAP yields substantial performance gains and narrows the gap between smaller models and much larger ones across LLMs and VLMs.The method supports up to T recursive steps.Source: §5 · p. 8
  • A learned confidence generator and recursive-confidence-increase and final-answer-confidence rewards enable iterative refinement of models’ think–answer trajectories.Source: §5 · p. 8
  • Because current deep-learning implementations are batch-dependent, all T trajectories must run in parallel, causing significant computational and memory overhead even when an answer becomes confident early.The authors hope R-TAP encourages research on confidence-aware iterative reasoning and more efficient, trustworthy, general-purpose systems for LLMs and VLMs.Source: §5 · p. 8

A. Full Responses for Multimodal Example

A multimodal example shows R-TAP refining an answer across reasoning cycles by correcting counting errors and checking visual details to identify the only flower meeting both criteria.

A. Full Responses for Multimodal Example

  • Across multiple reasoning cycles, the model corrects initial counting errors, verifies object-level visual details, and identifies the only flower meeting both criteria.The example illustrates iterative answer refinement in a multimodal task.Source: Figure 7 · p. 14

B. Algorithm of Recursive Think-Answer Process

R-TAP trains from pretrained confidence generators and LLMs/VLMs through recursive response sampling, confidence-based reward computation, and iterative updates to both models.

B. Algorithm of Recursive Think-Answer Process

C. Detailed Comparison between Performance and Computation Complexity

R-TAP reduces inference output tokens while achieving higher accuracy than refinement and verification baselines under the same output token budget. Its learned confidence signal regulates recursive reasoning depth, supporting a more stable single-sample policy.

C. Detailed Comparison between Performance and Computation Complexity

  • Approximately 2–3× fewer output tokens than self-consistency, self-refine, and verification-based baselines under the same output token budget, while achieving higher accuracy.The comparison covers LLM and VLM benchmarks and reports output-token or decoding-token counts alongside benchmark results.Source: C. Detailed Comparison between… · p. 15
  • With identical output token budgets of 32768 for Phi-4-reasoning-plus and 12000 for R1V2-38B, increasing voting numbers improves performance, but R-TAP models perform strongly at N = 1 and gain only marginally from more voting.The authors describe R-TAP as learning a stable single-sample policy that reduces reliance on majority voting, not as replacing self-consistency.Source: C. Detailed Comparison between… · p. 15
  • R-TAP suppresses unnecessary Oops-style refinement and reduces inference refinement steps.The recap identifies these as components of the approach and sources of improved inference efficiency.Source: C. Detailed Comparison between… · p. 16
  • R-TAP uses GRPO-based Think-Answer training and confidence-derived recursive rewards to regulate reasoning depth, rather than using confidence only for post-generation reranking or verification.Its confidence generator produces a continuous signal in [0, 1] despite binary supervision, and the recap also identifies intermediate results as part of the recursive reward.Source: C. Detailed Comparison between… · pp. 15–16
  • Future work targets adaptive recursion and lightweight gating or early-exit policies, parameter-efficient training, and extending confidence-guided reasoning to smaller models.The stated aims are to reduce redundant computation while preserving accuracy, lower memory and training overhead, and maintain competitive performance in smaller-scale models.Source: C. Detailed Comparison between… · p. 16

Related papers

Loading 2603.02099v2…