PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?

Jiaqi Tang, Jianmin Chen, Youyang Zhai, Wei Wei, Runtao Liu, Mengjie Zhao, Xiangyu Wu, Qingfa Xiao, Qifeng Chen

arXiv:2606.08063v1cs.CVcs.AIcs.CL

TL;DR

MLLMs are vulnerable to visual corruptions, while existing text-based approaches cannot restore lost pixel details. Robust-U1 enables visual self-recovery through reconstruction, dual-reward reinforcement learning, and joint reasoning over corrupted and recovered images, achieving superior robustness across real-world and adversarial corruption benchmarks.

  • Problem

    MLLMs suffer substantial performance degradation under real-world visual corruptions, while existing text-based methods cannot represent lost pixel-level details.

  • Method

    Robust-U1 trains MLLMs to reconstruct corrupted visuals using supervised fine-tuning, SSIM- and CLIP-based reinforcement learning, and joint reasoning over both images.

  • Results

    Robust-U1 achieves superior robustness across real-world and adversarial corruption benchmarks, with recovered images directly improving reasoning performance.

  • Takeaways & Limitations

    Visual self-recovery is a critical mechanism for robust visual understanding by coupling explicit reconstruction with multimodal reasoning.

  • Takeaways & Limitations

    Recovery may remain difficult for highly severe corruptions, and performance on rare or adversarial-specific distortions remains underexplored.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions. While existing robustness enhancement approaches exist, they are limited: black-box feature alignment lacks interpretability, and white-box text-based reasoning cannot restore lost pixel-level details. This work investigates a fundamental research question: Can MLLMs recover corrupted visual content by themselves? To address this, we propose Robust-U1, a novel framework that equips MLLMs with explicit visual self-recovery capability for robust understanding. The approach comprises three core stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards (pixel-level SSIM and semantic-level CLIP similarity) for aligning high visual quality, and multimodal reasoning that jointly considers both the corrupted input and the recovered image. Extensive experiments demonstrate that Robust-U1 achieves state-of-the-art robustness on the real-world corruption benchmark and maintains superior performance under adversarial corruptions on general VQA benchmarks. Analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. The source code is available at https://github.com/jqtangust/Robust-U1.

1. Introduction

Robust-U1 addresses MLLMs’ vulnerability to real-world visual corruptions by enabling explicit visual self-recovery and reasoning over both corrupted and recovered images. Its three-stage pipeline combines supervised reconstruction, SSIM- and CLIP-based reinforcement learning, and multimodal reasoning, achieving strong robustness across corruption benchmarks.

  • MLLMs achieve strong visual understanding but remain vulnerable to real-world visual corruptions, hindering practical deployment.
  • Existing robustness methods use black-box feature alignment or text-based reasoning, but neither explicitly restores corrupted pixel-level visual details.
  • Robust-U1 equips MLLMs with explicit visual self-recovery, enabling pixel-level reconstruction beyond implicit feature alignment or text-only reasoning.
  • Its three-stage pipeline uses supervised fine-tuning, reinforcement learning with SSIM and CLIP similarity rewards, and multimodal reasoning over corrupted and recovered content.
  • Robust-U1 outperforms existing robust MLLMs across R-Bench corruption intensities and maintains superior robustness with minimal performance decrease under adversarial corruptions on general VQA benchmarks.

2. Related Works

Prior robustness methods for MLLMs primarily use implicit alignment or text-based reasoning, but textual approaches cannot restore lost visual information. Robust-U1 extends these paradigms with explicit visual self-recovery, reconstructing corrupted images to support robust understanding.

  • Corruption Robustness of MLLMs: MLLMs remain vulnerable to environmental perturbations that impair visual perception and cause significant performance drops.
  • Corruption Robustness of MLLMs: Existing robustness methods primarily fall into implicit alignment and text-based reasoning, with the latter describing corruption types and semantic consequences textually.
  • Corruption Robustness of MLLMs: Text-based reasoning remains constrained to language and cannot restore visual information lost through corruption.
  • Contribution: Robust-U1 equips MLLMs to generate recovered corrupted images and use them to facilitate robust visual understanding.
  • Think with Images: Earlier reasoning used text-only chains of thought, while recent methods incorporate visual representations, intermediate features, or descriptions, including multi-agent coordination.

3. Methodology

Robust-U1 equips a unified MLLM with explicit visual self-recovery and multimodal reasoning over corrupted and recovered images. Its three stages combine supervised latent reconstruction, dual-reward reinforcement learning, and joint visual understanding.

  • Framework formulation: Robust-U1 approximates the corruption inverse D−1 to reconstruct a recovered image Ir, then combines Ir with the corrupted input Ic for final reasoning.The robust reasoning function uses both images to compensate for information loss while preserving awareness of corruption characteristics.
  • Framework overview: The framework has three stages: supervised fine-tuning for recovery, dual-reward reinforcement learning for recovery quality, and multimodal reasoning over both image versions.The stages are presented as Stage I, Stage II, and Stage III in the framework overview.
  • Stage I: supervised recovery: A pre-trained unified MLLM, BAGEL, is specialized into a self-recovery module using a recovery prompt and rectified-flow denoising in latent space.The model encodes the corrupted image as Zc and denoises a noisy clean latent Zo conditioned on Zc and the recovery prompt.
  • Stage II: dual-reward reinforcement learning: Flow-GRPO reinforcement learning optimizes structural fidelity with SSIM and semantic consistency with a CLIP-based reward.SSIM compares local luminance, contrast, and structure, while the semantic reward uses cosine similarity between recovered and clean image embeddings.
  • Stages II–III: optimization and reasoning: The reinforcement-learning stage produces recoveries that are visually and semantically close to clean images, while joint reasoning uses the recovered image for content and the corrupted image to resolve ambiguities.A KL divergence penalty constrains policy updates to prevent reward hacking and maintain generation quality.

4. Experiment

Robust-U1 is evaluated on real-world and adversarial visual corruptions, achieving state-of-the-art robustness while demonstrating that high-fidelity recovery and multimodal reasoning improve visual understanding. Ablations further show that both pixel-level and semantic-level rewards are important, especially under severe degradation.

  • Real-World Corruption Robustness: Robust-U1 achieves state-of-the-art performance across MCQ, VQA, and CAP tasks at all R-Bench degradation intensities.Its advantage over general-purpose and robust MLLMs becomes more pronounced as corruption severity increases.
  • Recovery Case Study: Robust-U1 recovers a clean image revealing the vehicle’s correct orientation, whereas prior methods are misled or generate erroneous recoveries.The case study illustrates that accurate pixel-level recovery supports correct visual reasoning.
  • Adversarial Corruption Robustness: 83.18 on MMMB with 100% corruption surpasses BAGEL’s 78.48 and Robust-R1’s 75.35.Robust-U1’s advantage is also consistent on the more challenging MMStar and RealWorldQA benchmarks.
  • Visual Recovery Quality: Each training stage improves recovery quality: SFT establishes reconstruction, Rpix primarily improves PSNR and SSIM, and Rsem achieves the best LPIPS.Qualitatively, Rpix sharpens edges and text, while Rsem better preserves natural textures and color fidelity.
  • Multimodal Reasoning: Removing multimodal reasoning causes a significant decline in overall performance because reasoning without the restored image cannot reliably interpret corrupted scenes.A counting case study shows text-only reasoning produces an incorrect answer, while reasoning over corrupted and recovered images succeeds.
  • Reward Ablations: Ablating Rsem causes the most severe degradation at high corruption levels, while removing Rpix harms tasks requiring precise visual understanding.Without Rsem, recoveries may be visually coherent but semantically erroneous; removing Rpix can occasionally marginally improve CAP scores.

5. Conclusion

Robust-U1 introduces a visual self-recovery paradigm for robust multimodal understanding, enabling MLLMs to reconstruct clean visual content from corrupted inputs. This explicit reconstruction advances beyond implicit feature alignment and text-only reasoning toward more intrinsic and generalizable resilience.

  • Conclusion: Robust-U1 pioneers visual self-recovery for robust multimodal understanding.The framework equips Multimodal Large Language Models with explicit reconstruction capability.
  • Conclusion: The framework enables MLLMs to actively reconstruct clean visual content from corrupted inputs.
  • Conclusion: Explicit reconstruction advances beyond implicit feature alignment or textual reasoning alone toward more intrinsic and generalizable resilience.

Impact Statement · Summary of Appendix · A. Implementation Details

Robust-U1 is presented as a step toward more reliable and interpretable vision-language systems, with particular relevance to safety-critical settings. The appendices document implementation and evaluation protocols alongside broader evidence for the framework’s effectiveness and robustness.

  • Impact Statement: Robust-U1’s self-recovery capability is intended to improve the reliability and interpretability of vision-language systems.The impact statement frames self-recovery as the framework’s contribution.
  • Impact Statement: The proposed improvement is particularly relevant to safety-critical applications, including autonomous navigation and medical image analysis.These settings may suffer significant consequences when real-world noise degrades performance.
  • Impact Statement: The paper acknowledges that image-reconstruction capability could be misapplied.The supplied impact passage explicitly recognizes this potential misuse.
  • A. Implementation Details: Appendix A reports per-stage training costs, including GPU type, time, memory, and trainable parameters.It also covers evaluation protocols on R-Bench and the three anti-degradation benchmarks MMMB, MMStar, and RealWorldQA.
  • Summary of Appendix: The appendices provide extended comparisons with external restoration modules and a discriminative MLLM.The listed restoration baselines are DFPIR, EVSSM, MambaIRv2, and BiLaLoRA.
  • Summary of Appendix: Together, the appendix materials are described as comprehensive evidence for Robust-U1’s effectiveness and robustness.This summary covers the collective role of the appendix materials rather than a single experiment.

A.1. Training Cost … B. Extended Quantitative Comparisons

The supplementary sections detail Robust-U1’s three-stage training costs and task-specific evaluation protocols, then position the framework against external restoration and selective recovery pipelines.

  • A.1. Training Cost: Training uses NVIDIA L20 (48GB) GPUs across three sequential stages with distinct computational requirements, data sources, and trainable parameters.Table 7 summarizes the three-stage cost breakdown.
  • A.1. Training Cost: 1920 GPU hours: reconstruction SFT is the dominant training-cost stage, using large-scale image–image pair training on ImageNet-C.The reported hardware is NVIDIA L20 (48GB) GPUs.
  • A.1. Training Cost: 160 GPU hours and 64 GPU hours: the RL and joint-reasoning stages are substantially lighter than reconstruction SFT.These correspond to Stages II and III, respectively.
  • A.2. Evaluation Protocol on R-Bench: R-Bench MCQ tasks use Accuracy as the primary indicator of whether the model identifies the correct option.The supplied protocol defines Accuracy formally using test samples, ground-truth answers, predictions, and an indicator function.
  • A.2. Evaluation Protocol on R-Bench: R-Bench VQA and image-captioning tasks use GPT-3.5-turbo as a proxy evaluator for semantic alignment with reference answers.Aggregate performance is represented by the mean score, emphasizing completeness, accuracy, and relevance.
  • A.3. Evaluation Protocol on Anti-Degradation Benchmarks: The three Anti-Degradation Benchmarks—MMMB, MMStar, and RealWorldQA—are evaluated uniformly in Multiple-Choice Question format.Performance is quantified with the standardized accuracy metric, while GPT-3.5-turbo parses intended choice labels through VLMEvalKit.
  • B. Extended Quantitative Comparisons: The extended comparisons examine Robust-U1 against external restoration preprocessing and a detect-then-recover pipeline that invokes recovery only when needed.These comparisons situate the framework against alternative recovery pipelines.

B.1. Comparison with External Restoration Modules … D.2. Sensitivity to the Reward Scaling Factor α

Across external comparisons, ablations, deployment analysis, and sensitivity studies, Robust-U1’s task-aligned self-recovery drives robustness gains, with trade-offs in inference cost and reward design. Its paired supervision is strongest, while recovery remains useful on clean inputs and relatively stable across moderate reward scaling.

  • B.1. Comparison with External Restoration Modules: Robust-U1 reaches 0.7398 overall on R-Bench, exceeding the best external-restoration baseline’s 0.5511.External modules precede Qwen2.5-VL-7B, but specialized restorers require degradation knowledge and can fail under unknown or compound corruptions.
  • B.2. Inference Cost and the Detect-then-Recover Variant: Detect-then-recover recovers most of the robustness gap at less than half the latency of full Robust-U1, whereas full recovery delivers the strongest robustness at highest latency.The standard MLLM is fastest but most fragile; inference costs are dominated by a 50-step rectified-flow denoising loop.
  • C.1. Isolating Reconstruction vs. CoT Supervision: Adding only CoT supervision raises BAGEL from 0.5770 to 0.6199 (+0.0429), while SFT+RL recovery raises it to 0.6623 (+0.0853).Combining reconstruction and CoT supervision further reaches 0.7398, indicating complementary gains.
  • C.2. Recovery Quality vs. Downstream Reasoning: SFT increases PSNR by +6.51 dB, from 14.37 to 20.88, but RL improves R-Bench by approximately +0.13 while increasing PSNR by no more than 0.6 dB.The results show that visually cleaner recovery is insufficient unless recovery is task-aligned for downstream reasoning.
  • C.3. Effect of Always-On Recovery on Clean Inputs: Always-on recovery improves R-Bench Overall by +0.1793 on corrupted inputs and by +0.0044 on clean inputs.The clean-input result indicates a small consistent benefit rather than harm from applying recovery universally.
  • D. Sensitivity Studies: The sensitivity studies vary semantic-reward supervision, α, and the frozen semantic encoder as training-time design choices.These studies assess reference-free rewards, reward scaling, and semantic-encoder substitution.
  • D.1. Reference-Free Semantic Reward: The reference-free semantic reward improves the SFT-only baseline by +0.0463, but paired structural and semantic supervision outperforms it by +0.1165 overall.The reference-free variant uses image–text consistency between the original image caption and recovered image, avoiding a paired clean reference.

D.3. Sensitivity to the Choice of Semantic Encoder · E. Reliability of Recovery and Evaluation

Robust-U1’s semantic reward remains effective across frozen vision-language encoders, while the reliability analysis examines hallucination risks and evaluator dependence. These results support encoder-insensitive semantic guidance and motivate explicit reliability checks.

  • D.3. Sensitivity to the Choice of Semantic Encoder: Robust-U1 evaluates whether its semantic reward depends on the frozen encoder used in Rsem.The default TinyCLIP encoder is replaced with CLIP-B/16, SigLIP-B/16, and a heavily distilled weaker CLIP variant.
  • D.3. Sensitivity to the Choice of Semantic Encoder: Even the substantially smaller distilled encoder retains 99.3% of the default performance.The result indicates that a weaker semantic encoder preserves nearly all default performance.
  • D.3. Sensitivity to the Choice of Semantic Encoder: The relative ordering of ablations with versus without Rsem is preserved across all tested encoders.Thus, the comparison does not change the observed ablation ranking.
  • D.3. Sensitivity to the Choice of Semantic Encoder: The semantic reward is consistently helpful, and Robust-U1 does not depend critically on a specific frozen vision-language encoder.This conclusion follows from the consistent ablation ordering and similar overall scores across encoder choices.
  • E. Reliability of Recovery and Evaluation: The reliability analysis examines whether recovery hallucinations mislead downstream reasoning and whether scores depend on the chosen LLM-based evaluator.This section shifts from performance results to two reliability questions.

E.1. Hallucination Risk Analysis … F.2. End-to-End Reasoning Case Studies

Robust-U1 rarely causes harmful hallucinations, and its robustness conclusions remain stable across evaluators. Qualitative comparisons and end-to-end cases show that explicit recovery restores critical visual details and supports accurate reasoning under severe corruption.

  • E.1. Hallucination Risk Analysis: The hallucination analysis categorizes possible recovery-branch hallucinations into three types and evaluates whether they mislead downstream reasoning.The study explicitly examines whether recovered content is absent from the original scene and whether such content affects answers.
  • E.1. Hallucination Risk Analysis: Only 4.1% of decisions become harmful flips, versus 7.2% for SFT-only and 15.6% for base BAGEL, while answer consistency reaches 92.3%.The analysis compares recovered-image answers with clean-image answers on R-Bench, classifying outcomes as consistent, harmful, beneficial, or neutral.
  • E.2. Sensitivity to the Evaluator Choice: Across evaluators, Robust-U1’s overall score ranges from 0.7121–0.7398, while method ordering remains preserved and scores stay above reported baselines.Qwen3-Max and GPT-4o are slightly stricter than GPT-3.5-turbo, especially on the harder captioning split.
  • F. Qualitative Results: Qualitative evidence combines side-by-side recovered-image comparisons with end-to-end reasoning traces on severely corrupted inputs.These complementary analyses cover both visual reconstruction quality and complete reasoning behavior.
  • F.1. Visual Comparisons of Recovered Images: Across eight representative examples, full Robust-U1 produces more faithful reconstructions than base BAGEL and SFT-only, restoring sharp edges, textures, and semantic content.Base BAGEL often remains blurry with artifacts, while SFT removes major corruption but may lack fine details.
  • F.2. End-to-End Reasoning Case Studies: End-to-end cases span traffic signals, object functions, counting, and scene understanding, with Robust-U1 explicitly recovering each severely corrupted input before answering.The case studies are presented in Tables 20, 21, 22, and 23 alongside ground-truth references and questions requiring detailed visual understanding.
  • F.2. End-to-End Reasoning Case Studies: Recovery enables accurate reasoning by reconstructing shapes, colors, and object identities, including recognizing a non-green left-turn arrow and counting green lights under degradation.The examples demonstrate reliable reasoning under motion blur, low illumination, and compression artifacts.

G. User Study · H. Limitations and Future Work

The user study found that participants strongly preferred Robust-U1’s restorations over BAGEL’s for both semantic faithfulness and overall visual quality. These perceptual gains support the connection between high-fidelity recovery and robust visual understanding.

  • G. User Study: The perceptual preferences were consistent with automated-metric improvements and support the claim that high-fidelity recovery preserves semantic content needed for reasoning.The study links visual recovery quality with more accurate and robust visual understanding.
  • G. User Study: The controlled study recruited 25 participants who evaluated 15 randomly selected R-Bench validation samples across three corruption intensity levels.Each sample showed the corrupted input and randomized BAGEL and Robust-U1 restorations.
  • G. User Study: Participants compared restorations using semantic faithfulness and overall visual quality criteria.The criteria covered reconstruction of objects, attributes, scene meaning, sharpness, artifacts, and naturalness.
  • G. User Study: A “No Preference” option allowed participants to mark restorations as indistinguishable for either criterion.This option accounted for comparisons where neither restoration was judged superior.
  • G. User Study: 92.3% of comparisons preferred Robust-U1 over 5.6% for BAGEL on Semantic Faithfulness.The result indicates a strong participant preference for Robust-U1’s semantic reconstruction.
  • G. User Study: 85.7% of comparisons preferred Robust-U1 over 10.1% for BAGEL on Overall Visual Quality.The low No Preference rate was 4.2% for this criterion.
  • G. User Study: 2.1% of Semantic Faithfulness comparisons and 4.2% of Overall Visual Quality comparisons received No Preference.These low rates indicate that participants usually distinguished the restorations’ quality.

H.1. Limitations · H.2. Future Work

Robust-U1 is limited by recovery quality on severe or uncommon corruptions and by its reliance on paired corrupted-clean training data, especially in specialized domains. Future work targets efficient recovery, corruption-specific priors, temporal extensions, and broader real-world benchmarks.

  • H.1. Limitations: The method requires paired corrupted-clean images for supervised fine-tuning and reinforcement learning, limiting applicability where realistic paired data are difficult to obtain.Synthetic data may have a domain gap from real-world corruptions, while specialized fields such as medical imaging and remote sensing pose substantial data-collection challenges.
  • H.2. Future Work: More efficient self-recovery architectures could reduce inference cost through lightweight modules, knowledge distillation, or conditional generation with fewer denoising steps.These approaches directly address the inference-cost trade-off quantified in Section B.2.
  • H.2. Future Work: Corruption-specific priors could improve recovery precision by combining data-driven methods with physics-based or statistical models such as blur kernels and noise models.This direction is particularly relevant to specialized applications including medical imaging and remote sensing.
  • H.2. Future Work: Extending self-recovery from single images to video requires modeling temporal consistency and motion dynamics for robust understanding under rain, fog, or low light.The current work focuses on single-image recovery and reasoning.
  • H.2. Future Work: Comprehensive benchmarks should cover diverse corruption types, severity levels, and multimodal tasks, emphasizing naturally occurring real-world corruptions over solely synthetic ones.Such benchmarks would support more rigorous evaluation of robust MLLMs.
  • H.2. Future Work: Addressing these limitations could help establish visual self-recovery as a fundamental capability for robust, reliable, and trustworthy multimodal AI in imperfect real-world conditions.The paper frames the four directions as a path toward this broader capability.
Loading 2606.08063v1…