PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Self-Correction in Vision-Language Models via Rollout Augmentation

Yi Ding, Ziliang Qiu, Bolian Li, Ruqi Zhang

arXiv:2602.08503v2cs.CVcs.CLcs.LG

TL;DR

Self-correction is important for strong VLM reasoning, but conventional RL entangles self-correction and direct-reasoning signals, while effective correction behaviors are rare. Octopus recombines rollouts into dense correction examples and uses response masking to separate the learning signals; Octopus-8B achieves best performance across seven benchmarks while requiring 0.72× training time per step compared to the best baseline.

  • Problem

    Effective self-correction behaviors emerge rarely, creating sparse learning signals, while conventional binary rewards entangle self-correction with direct reasoning during RL training.

  • Method

    Octopus recombines existing rollouts to construct dense self-correction examples, balances positive and negative supervision, and uses response masking to decouple self-correction from direct reasoning.

  • Results

    Octopus-8B achieves the best performance across seven benchmarks and requires only 0.72× training time per step compared to the best baseline.

  • Takeaways & Limitations

    Rollout augmentation enriches learning signals without additional generation cost, while response masking enables both self-correction and direct reasoning to be learned.

  • Takeaways & Limitations

    The rollout-pairing procedure assumes neither response is correct or wrong and uses pairing only to keep self-correction-style trajectories in-distribution.

Abstract

from arXiv · show

Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs). However, existing reinforcement learning (RL) methods struggle to learn it, as effective self-correction behaviors emerge only rarely, making learning signals extremely sparse. To address this challenge, we propose correction-specific rollouts (Octopus), an RL rollout augmentation framework that synthesizes dense self-correction examples by recombining existing rollouts. This augmentation simultaneously improves sample efficiency due to rollout reuse and stabilizes RL optimization through balanced supervision. Furthermore, we introduce a response-masking strategy that decouples self-correction from direct reasoning, avoiding signal conflicts and enabling both behaviors to be learned effectively. Building on this, we introduce Octopus-8B, a reasoning VLM with controllable self-correction capability. Across 7 benchmarks, it achieves SoTA performance among open-source VLMs, outperforming the best RLVR baseline by 1.0 score while requiring only $0.72\times$ training time per step.

1. Introduction

Self-correction is important for robust VLM reasoning, but current RL methods provide sparse, indirect signals for learning it. Octopus addresses this by recombining existing rollouts and decoupling self-correction from direct reasoning.

  • Self-correction resembles human problem solving and is important for strong, robust reasoning in vision-language models.
  • Current RL methods provide only outcome-level rewards, so effective self-correction emerges implicitly, remains difficult to control, and cannot be reliably triggered at inference.
  • Correct and incorrect reasoning trajectories often coexist in standard rollouts, allowing their contrast to synthesize explicit self-correction samples without additional computational overhead.
  • Octopus recombines rollouts to create dense examples, increase training samples from n to n^2, balance positive and negative examples, and stabilize policy updates.
  • Response masking separates self-correction and direct-reasoning signals, enabling the model to learn both capabilities without conflicting supervision.
  • Across 7 benchmarks, Octopus-8B exceeds Qwen3-VL-8B-Instruct by 9.5 average accuracy points, Qwen3-VL-8B-Thinking by 1.2 points, and GSPO by 1.0 point while using 0.72× training time per step.

2. Preliminaries

RLVR trains models on tasks with verifiable outcomes, commonly using GRPO to estimate relative advantages from grouped responses. GSPO addresses instability from off-policy scaling with sequence-level importance weighting.

  • RLVR trains language models on tasks whose outcomes can be easily verified, including mathematics and question answering.
  • GRPO estimates response advantages over groups of policy-generated outputs for reinforcement-learning optimization.
  • The GRPO advantage is normalized from rule-based rewards within each rollout group, while token-level importance ratios weight updates.
  • GSPO replaces token-level with sequence-level importance weighting to reduce high-variance noise and stabilize training on long reasoning trajectories.
  • Single-pass self-correction treats revision within one response as an intrinsic behavior marked by a special correction token.

3. Learning Self-Correction from Paired Rollouts

Standard RL rarely provides explicit wrong →correct trajectories for learning self-correction. Octopus recombines existing rollout responses to create dense correction examples while balancing supervision and reusing computation.

  • The Challenge: Self-Correction Signals Are Sparse: Only up to 0.3% of standard-RL samples exhibit effective wrong →correct transitions, rising to below 1% with prompt encouragement.Correct →wrong samples are also rare, and models tend to preserve their initial response.
  • The Challenge: Self-Correction Signals Are Sparse: Correct and incorrect responses often coexist within a rollout group, providing contrastive signals for constructing correction examples.Pairing these responses exposes how an incorrect trajectory can be revised into a correct one.
  • Correction-Specific Rollout Augmentation: Octopus recombines pre- and post-correction responses into n^2 paired rollouts spanning positive and negative correction categories.The four categories are wrong →correct, correct →correct, correct →wrong, and wrong →wrong.
  • Correction-Specific Rollout Augmentation: Wrong →correct pairs are the most informative because they directly encode effective self-correction behavior.The method retains generated rollouts and samples additional augmented pairs while balancing positive and negative examples.
  • Correction-Specific Rollout Augmentation: Octopus produces dense correction signals, balances positive and negative samples, and improves sample efficiency by reusing existing rollouts without additional computational cost.Its balanced supervision is intended to stabilize RL optimization.

4. Training Recipe

The training recipe first establishes an in-distribution self-correction format, then uses response masking to separate self-correction from direct reasoning. Mixed sampling avoids entropy collapse, while the two-stage strategy improves correction learning and stabilizes rewards.

  • 4.1. Cold-Start and Data Construction: Prompting alone can produce incomplete post-correction reasoning, so cold-start format learning ensures both responses contain complete, self-contained reasoning.The format is built around responses before and after the <sc> token.
  • 4.1. Cold-Start and Data Construction: Mixed sampling uses policy-model responses before <sc> and stronger-model responses after <sc> to obtain higher-quality corrections.The stronger model conditions on the input, ground truth, and initial response.
  • 4.1. Cold-Start and Data Construction: Mixed sampling maintains an entropy trajectory comparable to GSPO and achieves higher accuracy rewards than GSPO and in-distribution sampling.In-distribution sampling causes a larger entropy drop that limits later RL improvement.
  • 4.2. Conflicts Between Direct Reasoning and Self-Correction in RL Training Objective: Binary rewards leave pre- and post-correction accuracy nearly identical, while reward shaping induces reward hacking after approximately 200 training steps.The shaping approach can encourage deliberately incorrect first responses followed by trivial corrections.
  • 4.3. Response-Masking Strategy for Decoupled Learning: Stage I masks loss on o1 and updates only o2, while KL regularization constrains o1 toward the reference model.This isolates self-correction learning before joint optimization.
  • 4.3. Response-Masking Strategy for Decoupled Learning: Octopus augmentation keeps the self-correction reward stable by balancing positive and negative samples, whereas unaugmented training produces a sharp reward increase.The reward gap between o2 and o1 widens during Stage I, indicating improved self-correction capability.
  • 4.3. Response-Masking Strategy for Decoupled Learning: Stage II selectively unmasks o1 for samples with non-conflicting reward signals, allowing direct reasoning and self-correction to co-evolve.Samples with conflicting signals keep o1 masked.

5. Experiments

Across seven benchmarks, Octopus improves reasoning accuracy, training efficiency, and controllable self-correction through rollout augmentation and response-masking strategies.

  • Main Results: Octopus improves over Qwen3-VL-8B-Instruct by 9.5 average accuracy points across seven benchmarks and establishes a state of the art among comparable open-source VLMs.It also outperforms Qwen3-VL-8B-Thinking except on MathVerse.
  • Main Results: 1.0 average accuracy point separates Octopus from the GSPO RLVR baseline, while Octopus also outperforms SRPO across all evaluated tasks.The comparison attributes the improvement to explicit self-correction and increased effective self-correction reward signals.
  • Efficiency: 0.72× training time per step lets Octopus exceed GSPO accuracy with 16 rollouts while avoiding the roughly doubled cost associated with increasing baseline rollout samples.Octopus augments 8 generated rollouts to 16 during training without additional rollout-generation cost.
  • Ablation Study: SFT alone raises accuracy from 62.2 to 63.4, versus 71.7 for Octopus-8B, while removing Stage I causes a 1.9-point drop.The reported results characterize SFT as primarily learning the self-correction format and Stage I as decoupling self-correction learning.
  • More Results and Analysis: Appending <sc> tokens progressively improves accuracy and inference token efficiency, enabling additional correction steps beyond the single round used during training.The figure compares original responses with triggered responses and reports sequential test-time scaling and pass@k performance on MMStar.
  • More Results and Analysis: Octopus’s margin over GSPO on MMStar pass@k grows from 2.5 at pass@1 to 4.6 at pass@32, indicating a broader learned reasoning boundary.The paper attributes this effect to augmented training signals that encourage exploration beyond the original distribution.
  • More Results and Analysis: Octopus-8B outperforms baselines on text-only AIME2024 and MMLU-Pro despite training solely on vision-language inputs.The reported comparison uses o1 and o2 variants corresponding to Octopus without and with self-correction.

6. Related Works

Related work frames VLM reasoning around supervised fine-tuning and reinforcement learning, while controllable self-correction commonly relies on costly multi-pass prompting.

  • VLM Reasoning: VLM reasoning research commonly uses supervised fine-tuning with chain-of-thought trajectories or reinforcement learning to improve complex-task performance.The passage situates multimodal reasoning as an extension of successful text-based reasoning.
  • Self-Correction: Prior controllable self-correction methods typically use multi-pass prompting with explicit correction formats, requiring extensive interaction and long contextual histories.These approaches are described as token-inefficient at inference.

7. Conclusion

The paper presents Octopus as a reinforcement-learning framework that synthesizes self-correction examples and separates correction from direct reasoning. Across seven benchmarks, Octopus-8B achieves strong direct-generation and test-time-scaling performance with lower per-step training time than the best baseline.

  • Conclusion: Octopus pairs responses within rollout groups to construct dense, explicit self-correction examples without additional generation cost.Response masking decouples self-correction learning from direct reasoning to avoid objective conflicts.
  • Conclusion: Seven-benchmark experiments show Octopus-8B achieves the best reported direct-generation and test-time-scaling performance while using 0.72× the best baseline’s training time per step.The conclusion reports the result at the paper level rather than for an individual benchmark.
  • Conclusion: The results support explicitly learning self-correction as a route to more capable, efficient, and robust VLM reasoning.The paper also identifies structured supervision synthesized from policy samples as a promising broader direction.

Impact Statement

The paper states that it has no specific social impact to highlight or clarify.

  • Impact Statement: The impact statement identifies no specific social impact requiring discussion.

A.1. Training Details

Training uses fixed experimental settings across methods, with Octopus-specific prompts and a selection procedure for constructing augmented rollouts. The prompts explicitly separate reasoning, answers, and self-correction behavior.

  • Training setup: RL experiments use ViRL-39k, with Geometry-3k used to select the best training checkpoint.SFT uses LLaMAFactory, while RL uses Easy-R1.
  • Training setup: Learning rate, maximum sequence length, warm-up steps, and training epochs are fixed across methods for fair comparison.Rollouts are generated with vLLM, and the self-correction token is explicitly defined.
  • Octopus selection: Octopus selects samples by correctness, then balances wrong→correct signals with correct→correct supplementation and fills negatives from correct→wrong and wrong→wrong samples.The procedure initializes a training set and repeatedly selects samples until reaching the target size N.
  • Baseline prompting: Baseline prompts require internal reasoning followed by a boxed final answer, while SRPO additionally requests critical reflection and a revised answer.The SRPO prompt places the reflection after the initial answer and asks for a new response based on it.
  • Octopus prompting: Octopus prompts responses in the o1 ⊕<sc> ⊕o2 format, allowing a self-correction marker followed by regenerated reasoning and an answer.The prompt instructs the model to generate a new reasoning process and answer from scratch after self-correction.

B.1. Evaluation Setting

Octopus-8B is evaluated across seven benchmarks using VLMEvalKit and standardized sampling and answer-extraction procedures. Its two-response inference format still yields comparable overall latency after RL training, while random cross-component mixing can collapse training.

  • Evaluation setup: Octopus-8B is evaluated across seven benchmarks with VLMEvalKit and vLLM as the inference backend.Benchmark details are provided in Table 5.
  • Evaluation setup: Reasoning models use a 16,384-token inference budget and boxed-answer extraction, whereas instruct models use 4,096 tokens and full generated outputs.Sampling uses temperature 0.6, top-p 0.95, and no top-k restriction.
  • Inference efficiency: Octopus-8B generates o1 and o2 in one inference pass, yet final output lengths converge across methods during RL training, producing comparable overall inference latency.The efficiency comparison reports average reasoning latency and output token count on MathVista.
  • Failure attempts: Randomly mixing o1 and o2 components causes training collapse because the two components follow different distributions.Octopus therefore samples each component from its corresponding set to preserve the original response structure.
  • Failure attempts: Figure 9 visualizes the collapsed training curve from the random-mixing strategy.The figure is presented as a failure attempt during RL training.

D. Case Study

The case studies illustrate Octopus-8B revising reasoning after identifying arithmetic or visual errors, although one example shows that self-reflection can fail to detect an initial mistake. Other examples preserve an already correct solution while regenerating its reasoning.

  • Reward-hacking case: The arithmetic-cost example corrects an incomplete first response by calculating $36.32 for silk and $32.68 for canvas, totaling $69.00.The regenerated response supplies the missing multiplication and addition steps.
  • Case Study 2: A self-reflection attempt repeats the incorrect five-year count and fails to identify the 2010 mistake.The first response marks 2010 as not over 4% despite its value being 4.1%.
  • Case Study 1: Octopus-8B corrects an omitted 2010 value of 4.1%, changing the count of years above 4% from 5 to 6.The revised response explicitly identifies the missed comparison and reports 6 years.
  • Case Study 3: The visual-yard-line example revises an initial claim that yard line 1 is visible, concluding that only 30, 40, 50, and 60 are visible.The initial reasoning overlooked visual content near the goal line.
  • Case Study 4: For the ratio problem, both responses compute AB = 10 from AB:AC = 2:5 and AC = 25.The second response parameterizes the ratio as AB = 2x and AC = 5x before solving x = 5.
Loading 2602.08503v2…