Source-linked AI summary
Reinforcing Multimodal Reasoning Against Visual Degradation
Rui Liu, Dian Yu, Haolin Liu, Yucheng Shi, Tong Zheng, Runpeng Dai, Haitao Mi, Pratap Tokekar, Leoweiliang
TL;DR
RL-trained MLLMs remain brittle under real-world visual degradations, while existing augmentation and value-based robustness methods do not transfer cleanly to critic-free autoregressive fine-tuning. ROMA evaluates degraded views against clean-image trajectories and combines worst-case token-level regularization with clean-advantage anchoring and correctness conditioning. Across seven benchmarks, it improves robustness over GRPO on seen and unseen corruptions while matching clean accuracy.
Problem
RL-trained MLLMs remain brittle under blurry, compressed, and low-resolution inputs, while prior robustness techniques do not transfer cleanly to critic-free autoregressive fine-tuning.
Method
ROMA uses clean-image rollouts with teacher-forced evaluation of degraded views, worst-case token-level KL regularization, clean-advantage auxiliary policy gradients, and correctness conditioning.
Results
+2.4% on seen and +2.3% on unseen corruptions over GRPO, while matching clean accuracy across seven multimodal reasoning benchmarks.
Takeaways & Limitations
ROMA reinforces robustness to visual degradation without sacrificing the clean-input reasoning performance of critic-free MLLMs.
Abstract
from arXiv · showhide
Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.
1 Introduction
ROMA addresses the brittleness of RL-trained MLLMs under visual degradation by modifying critic-free optimization without sampling degraded-input rollouts. It combines teacher-forced corrupted-view evaluation with worst-case, correctness-conditioned invariance and clean-advantage anchoring, improving robustness while preserving clean performance.
- MLLMs that perform reliably on clean inputs can fail on blurry, compressed, or low-resolution visual content in deployment.
- Naively rolling out on degraded inputs can cause reward poisoning because perceptual occlusions trigger hallucinated trajectories and destabilize optimization.
- ROMA modifies RL optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance.
- A dual-forward-pass strategy rolls out on clean images, then teacher-forces the same trajectories under multiple degraded views without sampling degraded-input rollouts.
- ROMA combines worst-case token-level KL invariance, clean-advantage-anchored auxiliary policy gradients, and correctness-conditioned regularization.
- 68.7% clean accuracy accompanies 61.6% on seen and 56.3% on unseen perturbations, improving over GRPO by +2.4% and +2.3%, respectively.
2 Related Work
Prior visual-robustness methods use augmentation and regularization in vision and traditional actor-critic RL, but these approaches do not transfer directly to critic-free generative MLLM fine-tuning. ROMA reformulates visual invariance for this setting with token-level, correctness-conditioned constraints and clean-image advantage anchoring.
- Vision and deep RL commonly pursue robustness through visual data augmentation, including cropping, cutout, flipping, and blurring.
- Traditional actor-critic robustness techniques are incompatible with modern critic-free MLLM fine-tuning because of architectural mismatch and multimodal semantic sensitivity.
- Robustness to visual degradation in RL-based multimodal reasoning remains underexplored despite advances in multimodal reasoning.
- ROMA targets this gap with a correctness-conditioned token-level invariance penalty tailored to critic-free frameworks.
- ROMA anchors advantage computation to clean visual inputs rather than reinforcing hallucinated reasoning under perceptual occlusion.
3 Approach
ROMA reformulates critic-free RL fine-tuning to improve MLLM reasoning under visual degradation while retaining clean-image optimization. Its approach combines correctness-conditioned token-level invariance, worst-case augmentation selection, and an auxiliary degraded-view policy-gradient objective grounded in clean trajectories.
- Motivation: ROMA targets the failure of clean-image RL policies to generalize under blur, sensor noise, and compression artifacts.
- Design rationale: The framework isolates policy invariance because critic-free methods such as GRPO make value-based regularization inapplicable.
- Correctness-conditioned invariance: ROMA penalizes token-level divergence between degraded-view and clean-view policy outputs, using stop-gradient on clean outputs to protect clean representations.
- Correctness-conditioned invariance: Consistency regularization applies only to trajectories with positive reward, preventing the policy from becoming robustly incorrect on hallucinated or factually incorrect reasoning.
- Worst-case optimization: Worst-case multi-view optimization selects the augmentation producing the maximum divergence rather than averaging penalties across sampled views.
- Auxiliary policy gradient: An auxiliary clipped-surrogate policy-gradient objective evaluates a randomly sampled degraded view using clean-rollout trajectories and advantages, preventing reward poisoning and policy collapse.
- Unified objective: The consolidated objective combines the main RL objective, auxiliary degraded-view optimization, and worst-case invariance regularization through coefficients α and β.
4 Experiments
ROMA is evaluated on seven multimodal reasoning benchmarks using Qwen3-VL 4B and 8B models, with macro-averaged robustness results across seen and unseen degradations. It preserves clean-input performance while improving robustness, including under severe and unseen corruption, and ablations assess its optimization components and hyperparameters.
- Experimental Setup: Experiments fine-tune Qwen3-VL 4B and 8B models on MMRL30k and evaluate seven multimodal reasoning benchmarks with macro-averaged seen and unseen degradation results.The evaluation covers mathematical, chart, visual, and logical reasoning tasks.
- Clean Performance: 68.2% and 68.7% clean accuracy show ROMA remains comparable to GRPO for the 4B and 8B models, respectively.GRPO achieves 67.7% and 68.9% on the corresponding clean-input evaluations.
- Robustness to Visual Degradations: +2.4% seen-corruption robustness is achieved by the 8B model over GRPO, with a smaller clean-to-degraded drop of 7.1% versus 9.7%.ROMA reaches 61.6% under seen degradations, compared with GRPO’s 59.2%.
- Generalization to OOD Degradations: +2.3% unseen-corruption robustness is achieved by the 8B model over GRPO, while the clean-to-OOD drop is reduced to 12.4% from 14.9%.ROMA reaches 56.3% under unseen degradations, compared with GRPO’s 54.0%.
- Ablation Studies: Worst-case multi-view optimization outperforms mean aggregation, with mean penalties reducing accuracy by 1.6% on seen and 1.8% on unseen degradations.The ablation supports penalizing the hardest augmented view rather than averaging divergence across views.
- Ablation Studies: Removing auxiliary policy gradients reduces accuracy by 1.6% on seen and 1.8% on unseen degradations, while unconditional invariance reduces both by 2.2%.Sensitivity analyses identify α = 0.10, K = 3, and β = 0.10 as effective settings; larger auxiliary or invariance weights can over-regularize.
5 Conclusions
ROMA integrates visual robustness into critic-free RL fine-tuning while maintaining reasoning stability across corruptions without sacrificing clean-data performance.
- ROMA integrates adversarial visual robustness directly into the MLLM reasoning pipeline.
- A dual-forward-pass strategy teacher-forces corrupted views against clean-image trajectories rather than sampling degraded-input rollouts.
- Token-level KL regularization on worst-case augmentations is paired with an auxiliary policy-gradient loss to preserve reward signals under degradation.
- The method maintains reasoning stability across diverse visual corruptions without sacrificing clean-data performance.
A.1 Implementation Details
Models are trained on MMRL30k with structured reasoning outputs and a fixed 120-step optimization configuration.
- Training uses the approximately 30K-sample MMRL30k dataset.
- Responses use <thinking></thinking> tags for reasoning and \boxed{} for the final answer.
- Training runs for 120 steps with learning rate 1e−6 and weight decay 0.01.
- The global batch size is 128, the rollout batch size is 256, and each input produces 8 rollouts.
A.2 Degradation Details and Severity Levels
The evaluation protocol samples visual-corruption parameters continuously during training and uses three severity levels, with the strongest level testing magnitude OOD generalization.
- Training samples visual-corruption parameters continuously from their corresponding distributions.
- Evaluation uses three severity levels, from Level 1 through Level 3, following the ImageNet-C framework.
- Level 3 applies severities beyond the training-distribution bounds to evaluate out-of-distribution magnitude generalization.
- The unseen pool is strictly out of distribution across all severity levels.
A.3 Experiments
The 8B model's detailed evaluation results are reported separately for the base model, GRPO baseline, and ROMA across seen and unseen degradation types.
- Tables 10, 11, and 12 report detailed 8B-model results for the base model, GRPO baseline, and ROMA, respectively.
- The results cover all seen and unseen degradation types.
A.4 Discussions and Future Work
The discussion distinguishes seen degradations used during training from unseen degradations used to test out-of-distribution generalization. It also identifies extensions to temporal reasoning and severity-aware objective weighting as future directions.
- Future work: Future work could extend worst-case multi-view optimization from visual inputs to temporal modalities such as video-based reasoning.
- Future work: Future research could adaptively weight the auxiliary policy-gradient objective and invariance penalty according to inferred degradation severity.
- Degradation settings: Seen degradations include Gaussian noise, Gaussian blur, JPEG compression, and resolution downscaling, while unseen degradations evaluate OOD generalization.Unseen examples include motion blur, salt-and-pepper noise, speckle noise, posterization, and pixelation.
- Detailed analysis: The appendix provides detailed performance breakdowns for the Base 8B model, the GRPO baseline, and the proposed approach across specific degradations.