PaperlayerView exact revision on arXiv

Source-linked AI summary

NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, Michael Qizhe Shieh

arXiv:2504.13055v4cs.CV

TL;DR

VLM reinforcement learning still lacks well-explored policy exploration and must contend with imperfect perception that affects reasoning. NoisyRollout mixes clean and distorted-image trajectories with annealed noise while preserving the RL objective, and it achieves strong generalization and robustness across multiple benchmarks and settings.

  • Problem

    Policy exploration for scaling VLM test-time compute remains underexplored, and imperfect visual perception negatively affects subsequent reasoning.

  • Method

    NoisyRollout mixes rollouts from clean and moderately distorted images and gradually reduces distortion strength during training without modifying the RL objective.

  • Results

    NoisyRollout achieves improved generalization and robustness with state-of-the-art performance across multiple visual reasoning and perception benchmarks, including five out-of-domain benchmarks.

  • Takeaways & Limitations

    The method provides a simple, lightweight approach that improves visual reasoning through more diverse policy exploration without additional training cost.

  • Takeaways & Limitations

    Performance gains over vanilla GRPO can be smaller for the 32B model than for the 7B model when the larger model’s initial policy was already RL-fine-tuned.

Abstract

from arXiv · show

Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to better scale test-time compute remains largely underexplored. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. We introduce NoisyRollout, a simple yet effective data augmentation method that addresses these issues by mixing training trajectories from both clean and moderately distorted images. This approach injects perceptual diversity, encouraging better policy exploration and leading to more robust reasoning. A noise annealing schedule gradually reduces distortion strength, aiding exploration early in training while ensuring later stability. Crucially, our method is easy-to-adopt--requiring no additional training cost and no modifications to the RL objective. Extensive experiments on 2 distinct training datasets demonstrate that NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models across 5 out-of-domain reasoning and perception benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes (7B and 32B), data scales (from 1K to 6K) and image augmentation types (Gaussion noise and rotation), highlighting its generalizability and scalability.

1 Introduction

NoisyRollout targets two VLM RL challenges: underexplored policy exploration and imperfect visual perception that affects reasoning. It mixes clean and distorted-image trajectories, then anneals distortion to balance early exploration with later stability.

  • VLM policy exploration remains underexplored, while higher rollout temperature can create superficial variability rather than robust behaviors.
  • Imperfect visual perception negatively affects subsequent reasoning, but many RL approaches adapted from LLMs do not address this challenge.
  • NoisyRollout generates clean- and moderately distorted-image rollouts for each image-query training sample to introduce meaningful rollout diversity.
  • Successful noisy-input trajectories provide alternative reasoning strategies, while clean–distorted reward differences supply implicit contrastive signals for perceptual refinement.
  • A noise annealing schedule reduces distortion strength over training, retaining early noisy signals while mitigating later policy-gradient instability and distributional mismatch.
  • 53.2% on MathVerse, 28.5% on MathVision, and 72.1% on HallusionBench were achieved across five out-of-domain reasoning and perception benchmarks.

2 NoisyRollout: A Free-Lunch with Noisy Reinforcement Learning

NoisyRollout augments GRPO with clean and distorted-image rollouts grouped for reward and advantage computation, while policy updates remain conditioned on clean inputs. Annealed distortion preserves exploration early and improves stability later without changing the RL objective or adding training cost.

  • Hybrid rollout strategy: NoisyRollout combines clean and distorted-image trajectories within GRPO, using their shared rewards to compute the group baseline and normalized advantages.
  • Hybrid rollout strategy: The current policy update remains conditioned solely on the clean image and query, while noisy trajectories expand the rollout group used for optimization.
  • Noise annealing schedule: At step t, the noise level is set as αt = η(α0, t, tmax), and the distorted image is generated as ˜I = Tαt(I).
  • Noise annealing schedule: Annealing retains diverse supervision early, then reduces distribution shifts and moves noisy rollouts progressively closer to the clean-input policy.
  • Summary: NoisyRollout preserves the original RL objective and requires no additional training cost or complex training protocol.
  • Summary: Positive distorted-input trajectories can improve robustness to challenging or out-of-domain visual conditions, while reward differences refine perceptual behavior.

3 Experiments

Experiments evaluate NoisyRollout across datasets, model sizes, training scales, rollout temperatures, noise strengths, and annealing strategies. Results show stronger generalization, targeted diversity, and sensitivity to distortion strength and policy-training stage.

  • Main Results: NoisyRollout consistently outperforms vanilla GRPO across five out-of-domain visual reasoning and perception benchmarks.On Geometry3K with Qwen2.5-VL-7B-Instruct, it achieves 53.2% on MathVerse, 28.5% on MathVision, and 69.6% on WeMath.
  • Main Results: 72.1% HallusionBench accuracy with the reasoning prompt exceeds vanilla GRPO’s 69.8% and avoids the baseline model’s direct-answer-to-reasoning drop.NoisyRollout also obtains a higher Bradley–Terry win rate for perception quality during reasoning.
  • Robustness Across Datasets and Model Sizes: 1.2%, 1.3%, and 1.9% gains over GRPO occur on MathVerse, MathVista, and HallusionBench for the 7B model trained on MMK12.The corresponding 32B gains are 0.4%, 0.7%, and 1.1% on the same benchmarks; gains are smaller for 32B on some tasks because its initial policy was already RL-finetuned.
  • Rollout Diversity: Early-training rollout diversity increases under NoisyRollout and precedes higher final accuracy, while later diversity falls below the control group.NoisyRollout with temperature 1.0 outperforms vanilla GRPO across temperatures 0.8–1.4 and mixed-temperature variants, indicating more targeted diversity than temperature adjustment alone.
  • Data Scale: NoisyRollout outperforms vanilla GRPO across data scales from 1.1K to 6.4K without diminishing gains as dataset size increases.The experiments use Geometry3K with 2.1K samples and MMK12 with 6.4K preprocessed samples.
  • Hyperparameters and Training Dynamics: Moderate initial Gaussian noise improves performance as α0 increases from 0 to 500, whereas stronger distortion produces near-zero-reward rollouts and destabilizes updates.Noisy rollouts contribute more to policy optimization early, when distortion is higher, and their contribution diminishes as training becomes more on-policy.

4 Related Work

Prior work has extended reinforcement learning from language models to vision-language models, including efforts focused on multimodal reasoning and rule-based optimization.

  • Multimodal reinforcement learning: Recent multimodal RL systems extend reinforcement learning to vision-language reasoning tasks.Examples include LMM-R1, Vision-R1, R1-V, OpenVLThinker, and MM-Eureka.
  • Reinforcement-learning foundations: RL research has evolved from human-feedback alignment toward rule-based rewards and optimization methods such as GRPO.DeepSeek-R1 and Kimi-1.5 exemplify this broader progression.

5 Conclusion

The paper introduces NoisyRollout for RL training of VLMs, using clean and distorted-image trajectories to improve rollout diversity and policy exploration without additional training cost. Experiments report stronger generalization, robustness, and sample efficiency across visual reasoning and perception benchmarks.

  • Conclusion: NoisyRollout mixes trajectories from clean and distorted inputs to promote vision-oriented rollout diversity during RL.The method is designed to enhance policy exploration without modifying the RL objective or adding training cost.
  • Conclusion: The method achieves improved generalization and robustness across multiple visual reasoning and perception benchmarks.The conclusion characterizes the reported performance as state of the art among the evaluated open-source RL-tuned models.
  • Conclusion: NoisyRollout demonstrates high sample efficiency in the reported experiments.The conclusion identifies sample efficiency as a central empirical property of the approach.

NeurIPS Paper Checklist

The checklist assessment states that the abstract and introduction accurately present NoisyRollout’s motivation, method, contributions, results, and scope.

  • Claims and scope: The introduction clearly outlines the motivation, problem setting, proposed method, and key contributions.The assessment identifies NoisyRollout as the proposed method and links the claims to the paper’s experimental scope.
  • Claims and scope: The checklist answer is Yes because the introduction’s claims are consistent with the experimental results and scope discussed in Section 3.The guidance emphasizes that claims should match results and expected generalization.

2. Limitations

The checklist reports that the paper documents its limitations, experimental setting, compute resources, reproducibility materials, and ethics compliance, but does not report error bars for its large-scale RL experiments.

  • Limitations: The paper states that NoisyRollout’s limitations are detailed in Appendix F.The checklist separately records this as the paper’s limitations discussion.
  • Reproducibility: The paper states that code and running instructions are provided in the supplementary materials for reproducing the experiments.The checklist marks open access to data and code with sufficient instructions as Yes.
  • Experimental documentation: Training and evaluation details are reported in Section 3 and Appendix J, including compute-resource information in Table 13.The checklist marks both experimental details and compute-resource disclosure as provided.
  • Statistical significance: The paper reports no error bars for large-scale RL results because of the required computational cost.Multiple independent runs are used for selected lower-cost ablation and analytical experiments, including Table 10 and Figure 8.
  • Ethics: The paper reports conformity with the NeurIPS Code of Ethics.The ethics checklist answer is Yes.

10. Broader impacts

The paper reports broader-impact information through its submission disclosures, including societal-impact discussion, asset licensing and documentation, and the absence of human-subjects research.

  • The submission answers yes to discussing both potential positive and negative societal impacts, with details placed in Appendix E.
  • The authors characterize the vision-language model and data as focused on understanding and reasoning, with minimal misuse risk.
  • The code and model are provided in supplementary materials, while licenses for existing assets are mentioned in Appendix K.
  • The paper states that it does not involve crowdsourcing or research with human subjects.

A Additional Ablation Studies

Additional ablations examine annealing schedules, rollout budgets, data seeds, initial noise strength, and clean/noisy rollout proportions, identifying conditions that preserve performance and stability.

  • Noise annealing strategy: 58.5% average accuracy makes sigmoid annealing the strongest schedule, exceeding power decay at 57.1% and exponential decay at 57.0%.
  • Total rollout number: NoisyRollout consistently outperforms vanilla GRPO when total rollout counts are held constant, whereas increasing vanilla GRPO from n = 8 to n = 16 yields only marginal out-of-domain gains.
  • Meaningful perceptual variance is required: severely corrupted inputs produced near-zero rewards, unreliable gradients, instability, and eventual divergence.
  • Initial noise step (MMK12): NoisyRollout outperforms standard GRPO on MMK12 with an appropriate initial noise step of α0 = 450.
  • Proportion of noisy rollouts: 58.5% average accuracy is achieved with a balanced 50/50 clean-and-noisy rollout mix, versus 56.3% for the no-noise baseline.

B Unsuccessful Attempts

The study documents unsuccessful design choices and evaluation procedures, showing that some rollout-conditioning and reward-penalty variants failed, while perception comparisons used isolated visual traces.

  • Optimizing noisy and clean trajectories on corresponding inputs: Conditioning policy updates on the same clean or distorted input used for each rollout did not meaningfully improve over original GRPO.
  • Optimizing noisy and clean trajectories on corresponding inputs: The proposed alternative instead groups clean and noisy rollouts for advantage calculation while anchoring policy optimization to clean inputs.
  • Reward penalty on noisy subgroup: Explicitly penalizing all noisy rollouts caused training divergence because the policy learned to distinguish noisy from clean trajectories.
  • Visual perception comparison: The perception study sampled 300 paired reasoning traces, extracted only visual components, reversed presentation order to reduce position bias, and computed Bradley-Terry win rates.

F Limitations

The paper’s limitations concern experimental scale, the restriction of NoisyRollout to RL fine-tuning, and the absence of a formal theory of exploration and convergence.

  • Experiments primarily cover models up to 32B parameters and datasets of a few thousand samples, leaving larger-scale validation for future work.
  • NoisyRollout is applied during RL fine-tuning of an already pre-trained VLM rather than during large-scale pre-training.
  • The study lacks formal analysis of how hybrid trajectory mixing and noise annealing affect exploration–exploitation trade-offs and RL convergence.

I Detailed Methodology for Gradient Contribution Analysis

The analysis measures how Clean and Noisy rollout subgroups contribute to the overall RL update by comparing isolated subgroup updates with an anchor gradient. It uses matched parameter states, five-step updates, and repeated runs for robust estimates.

  • Anchor gradient: The anchor gradient g_t is the actual parameter change from training step t to t + Δt under the standard mixed-rollout update.The analysis uses Δt = 5 steps and selected model parameters θ_t.
  • Subgroup isolation: Clean and Noisy subgroup effects are isolated by masking the other subgroup’s loss during otherwise matched Δt-step updates from θ_t.One update uses only Clean rollouts, while the other uses only Noisy rollouts.
  • Contribution measurement: Each subgroup’s contribution is quantified by projecting its effective gradient onto the anchor gradient and computing a projection ratio.The ratios estimate the proportion of the anchor gradient attributable to Clean and Noisy rollouts.
  • Robust estimation: Effective gradient quantities are averaged across 5 independent runs, each initialized from the same parameter state θ_t.This averaging is intended to provide robust estimates of subgroup contributions.

J Supplementary Implementation Details

The supplementary implementation section summarizes experiment configurations, including image-distortion and noise-annealing settings, and records the licenses for the paper’s software, datasets, and models.

  • Experimental settings: Table 13 summarizes hyperparameter configurations across model sizes and datasets, focusing on image-distortion parameters and noise-annealing schedules.The table is identified as a summary of hyperparameter configurations.
  • Licensing: The paper states that its codes, datasets, and models use standard community licenses.The listed materials include training and evaluation datasets, frameworks, pretrained models, and the Gemini API.
  • Licensing: Listed licenses include MIT, Apache License 2.0, Creative Commons variants, CC BY-NC 4.0, and Gemini API Additional Terms of Service.Specific assignments are provided for Geometry3K, MMK12, MathVerse, MathVision, MathVista, WeMath, verl, EasyR1, Qwen2.5-VL models, and Gemini API.
Loading 2504.13055v4…