PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing Multimodal Reasoning Against Visual Degradation

Rui Liu, Dian Yu, Haolin Liu, Yucheng Shi, Tong Zheng, Runpeng Dai, Haitao Mi, Pratap Tokekar, Leoweiliang

arXiv:2605.09262v1cs.CVcs.CL

TL;DR

RL-trained MLLMs remain brittle under real-world visual degradations, while existing augmentation and value-based robustness methods do not transfer cleanly to critic-free autoregressive fine-tuning. ROMA evaluates degraded views against clean-image trajectories and combines worst-case token-level regularization with clean-advantage anchoring and correctness conditioning. Across seven benchmarks, it improves robustness over GRPO on seen and unseen corruptions while matching clean accuracy.

  • Problem

    RL-trained MLLMs remain brittle under blurry, compressed, and low-resolution inputs, while prior robustness techniques do not transfer cleanly to critic-free autoregressive fine-tuning.

    Source: §1 · pp. 1–2

  • Method

    ROMA uses clean-image rollouts with teacher-forced evaluation of degraded views, worst-case token-level KL regularization, clean-advantage auxiliary policy gradients, and correctness conditioning.

    Source: §1 · p. 2

  • Results

    +2.4% on seen and +2.3% on unseen corruptions over GRPO, while matching clean accuracy across seven multimodal reasoning benchmarks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    ROMA reinforces robustness to visual degradation without sacrificing the clean-input reasoning performance of critic-free MLLMs.

    Source: §1 · p. 2, §4.2 · p. 6

Abstract

from arXiv · show

Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.

1 Introduction

ROMA addresses the brittleness of RL-trained MLLMs under visual degradation by modifying critic-free optimization without sampling degraded-input rollouts. It combines teacher-forced corrupted-view evaluation with worst-case, correctness-conditioned invariance and clean-advantage anchoring, improving robustness while preserving clean performance.

  • MLLMs that perform reliably on clean inputs can fail on blurry, compressed, or low-resolution visual content in deployment.Source: §1 · p. 1
  • Naively rolling out on degraded inputs can cause reward poisoning because perceptual occlusions trigger hallucinated trajectories and destabilize optimization.Source: §1 · p. 2
  • ROMA modifies RL optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance.Source: §1 · pp. 2–3
  • A dual-forward-pass strategy rolls out on clean images, then teacher-forces the same trajectories under multiple degraded views without sampling degraded-input rollouts.Source: §1 · p. 2, Figure 1 · p. 2
  • ROMA combines worst-case token-level KL invariance, clean-advantage-anchored auxiliary policy gradients, and correctness-conditioned regularization.Source: §1 · pp. 2–3
  • 68.7% clean accuracy accompanies 61.6% on seen and 56.3% on unseen perturbations, improving over GRPO by +2.4% and +2.3%, respectively.Source: §1 · pp. 2–3

2 Related Work

Prior visual-robustness methods use augmentation and regularization in vision and traditional actor-critic RL, but these approaches do not transfer directly to critic-free generative MLLM fine-tuning. ROMA reformulates visual invariance for this setting with token-level, correctness-conditioned constraints and clean-image advantage anchoring.

  • Vision and deep RL commonly pursue robustness through visual data augmentation, including cropping, cutout, flipping, and blurring.Source: §2 · p. 3
  • Traditional actor-critic robustness techniques are incompatible with modern critic-free MLLM fine-tuning because of architectural mismatch and multimodal semantic sensitivity.Source: §2 · p. 3
  • Robustness to visual degradation in RL-based multimodal reasoning remains underexplored despite advances in multimodal reasoning.Source: §2 · p. 3
  • ROMA targets this gap with a correctness-conditioned token-level invariance penalty tailored to critic-free frameworks.Source: §2 · p. 3
  • ROMA anchors advantage computation to clean visual inputs rather than reinforcing hallucinated reasoning under perceptual occlusion.Source: §2 · p. 3

3 Approach

ROMA reformulates critic-free RL fine-tuning to improve MLLM reasoning under visual degradation while retaining clean-image optimization. Its approach combines correctness-conditioned token-level invariance, worst-case augmentation selection, and an auxiliary degraded-view policy-gradient objective grounded in clean trajectories.

Motivation

  • ROMA targets the failure of clean-image RL policies to generalize under blur, sensor noise, and compression artifacts.Source: §3 · p. 4

Design rationale

  • The framework isolates policy invariance because critic-free methods such as GRPO make value-based regularization inapplicable.Source: §3 · p. 4

Correctness-conditioned invariance

  • ROMA penalizes token-level divergence between degraded-view and clean-view policy outputs, using stop-gradient on clean outputs to protect clean representations.Source: §3 · p. 4
  • Consistency regularization applies only to trajectories with positive reward, preventing the policy from becoming robustly incorrect on hallucinated or factually incorrect reasoning.Source: §3 · p. 4

Worst-case optimization

  • Worst-case multi-view optimization selects the augmentation producing the maximum divergence rather than averaging penalties across sampled views.Source: §3 · p. 4

Auxiliary policy gradient

  • An auxiliary clipped-surrogate policy-gradient objective evaluates a randomly sampled degraded view using clean-rollout trajectories and advantages, preventing reward poisoning and policy collapse.Source: §3 · pp. 4–5

Unified objective

  • The consolidated objective combines the main RL objective, auxiliary degraded-view optimization, and worst-case invariance regularization through coefficients α and β.Source: §3 · p. 5

4 Experiments

ROMA is evaluated on seven multimodal reasoning benchmarks using Qwen3-VL 4B and 8B models, with macro-averaged robustness results across seen and unseen degradations. It preserves clean-input performance while improving robustness, including under severe and unseen corruption, and ablations assess its optimization components and hyperparameters.

Experimental Setup

  • Experiments fine-tune Qwen3-VL 4B and 8B models on MMRL30k and evaluate seven multimodal reasoning benchmarks with macro-averaged seen and unseen degradation results.The evaluation covers mathematical, chart, visual, and logical reasoning tasks.Source: §4.1 · p. 5, §4.2 · p. 6

Clean Performance

  • 68.2% and 68.7% clean accuracy show ROMA remains comparable to GRPO for the 4B and 8B models, respectively.GRPO achieves 67.7% and 68.9% on the corresponding clean-input evaluations.Source: §4.2 · p. 6

Robustness to Visual Degradations

  • +2.4% seen-corruption robustness is achieved by the 8B model over GRPO, with a smaller clean-to-degraded drop of 7.1% versus 9.7%.ROMA reaches 61.6% under seen degradations, compared with GRPO’s 59.2%.Source: §4.2 · p. 6

Generalization to OOD Degradations

  • +2.3% unseen-corruption robustness is achieved by the 8B model over GRPO, while the clean-to-OOD drop is reduced to 12.4% from 14.9%.ROMA reaches 56.3% under unseen degradations, compared with GRPO’s 54.0%.Source: §4.2 · pp. 6–7

Ablation Studies

  • Worst-case multi-view optimization outperforms mean aggregation, with mean penalties reducing accuracy by 1.6% on seen and 1.8% on unseen degradations.The ablation supports penalizing the hardest augmented view rather than averaging divergence across views.Source: §4.3 · p. 8
  • Removing auxiliary policy gradients reduces accuracy by 1.6% on seen and 1.8% on unseen degradations, while unconditional invariance reduces both by 2.2%.Sensitivity analyses identify α = 0.10, K = 3, and β = 0.10 as effective settings; larger auxiliary or invariance weights can over-regularize.Source: §4.3 · p. 8, §4.4 · p. 9

5 Conclusions

ROMA integrates visual robustness into critic-free RL fine-tuning while maintaining reasoning stability across corruptions without sacrificing clean-data performance.

  • ROMA integrates adversarial visual robustness directly into the MLLM reasoning pipeline.Source: §5 · p. 9
  • A dual-forward-pass strategy teacher-forces corrupted views against clean-image trajectories rather than sampling degraded-input rollouts.Source: §5 · p. 9
  • Token-level KL regularization on worst-case augmentations is paired with an auxiliary policy-gradient loss to preserve reward signals under degradation.Source: §5 · p. 9
  • The method maintains reasoning stability across diverse visual corruptions without sacrificing clean-data performance.Source: §5 · p. 9

A.1 Implementation Details

Models are trained on MMRL30k with structured reasoning outputs and a fixed 120-step optimization configuration.

  • Training uses the approximately 30K-sample MMRL30k dataset.Source: §A.1 · p. 13
  • Responses use <thinking></thinking> tags for reasoning and \boxed{} for the final answer.Source: §A.1 · p. 13
  • Training runs for 120 steps with learning rate 1e−6 and weight decay 0.01.Source: §A.1 · p. 13
  • The global batch size is 128, the rollout batch size is 256, and each input produces 8 rollouts.Source: §A.1 · p. 13

A.2 Degradation Details and Severity Levels

The evaluation protocol samples visual-corruption parameters continuously during training and uses three severity levels, with the strongest level testing magnitude OOD generalization.

  • Training samples visual-corruption parameters continuously from their corresponding distributions.Source: §A.2 · p. 13
  • Evaluation uses three severity levels, from Level 1 through Level 3, following the ImageNet-C framework.Source: §A.2 · p. 13
  • Level 3 applies severities beyond the training-distribution bounds to evaluate out-of-distribution magnitude generalization.Source: Table 9 · p. 13, §A.2 · p. 13
  • The unseen pool is strictly out of distribution across all severity levels.Source: Table 9 · p. 13

A.3 Experiments

The 8B model's detailed evaluation results are reported separately for the base model, GRPO baseline, and ROMA across seen and unseen degradation types.

  • Tables 10, 11, and 12 report detailed 8B-model results for the base model, GRPO baseline, and ROMA, respectively.Source: §A.3 · p. 13
  • The results cover all seen and unseen degradation types.Source: §A.3 · p. 13

A.4 Discussions and Future Work

The discussion distinguishes seen degradations used during training from unseen degradations used to test out-of-distribution generalization. It also identifies extensions to temporal reasoning and severity-aware objective weighting as future directions.

Future work

  • Future work could extend worst-case multi-view optimization from visual inputs to temporal modalities such as video-based reasoning.Source: §A.4 · p. 13
  • Future research could adaptively weight the auxiliary policy-gradient objective and invariance penalty according to inferred degradation severity.Source: §A.4 · p. 13

Degradation settings

  • Seen degradations include Gaussian noise, Gaussian blur, JPEG compression, and resolution downscaling, while unseen degradations evaluate OOD generalization.Unseen examples include motion blur, salt-and-pepper noise, speckle noise, posterization, and pixelation.Source: Figure 3 · p. 14

Detailed analysis

Related papers

Loading 2605.09262v1…