PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEAR: Unlocking Generative Potential for Degraded Image Understanding in Unified Multimodal Models

Xiangzhao Hao, Zefeng Zhang, Zhenyu Zhang, Linhao Yu, Yao Chen, Yiqian Zhang, Haiyun Guo, Shuohuan Wang, Yu Sun

arXiv:2604.04780v1cs.CV

TL;DR

Image degradation exposes a pervasive weakness in multimodal understanding because unified models do not spontaneously use their generative pathways, and decode-reencode processing blocks effective joint optimization. CLEAR addresses both gaps with degradation-aware behavioral training, a direct latent bridge, and Interleaved GRPO; it substantially improves degraded-image performance while preserving clean-image accuracy, with task-driven optimization also improving intermediate perceptual quality.

  • Problem

    Unified multimodal models fail to use generation during reasoning on degraded images, while their decode-reencode pathway prevents effective joint optimization.

  • Method

    CLEAR combines degradation-aware supervised fine-tuning, a Latent Representation Bridge, and Interleaved GRPO to connect and jointly optimize generation and reasoning.

  • Results

    CLEAR substantially improves degraded-image understanding while preserving clean-image performance, and CLEAR-RL improves Bagel by 5.11 points (8.5% relative) on the reported evaluation.

  • Takeaways & Limitations

    The model learns to invoke generation selectively, and answer-correctness optimization can improve rather than reduce the perceptual quality of generated intermediate states.

  • Takeaways & Limitations

    After supervised fine-tuning, generated states remain constrained by the MSE target, whose regression-to-the-mean tendency limits sharpness and perceptual quality.

Abstract

from arXiv · show

Image degradation from blur, noise, compression, and poor illumination severely undermines multimodal understanding in real-world settings. Unified multimodal models that combine understanding and generation within a single architecture are a natural fit for this challenge, as their generative pathway can model the fine-grained visual structure that degradation destroys. Yet these models fail to leverage their own generative capacity on degraded inputs. We trace this disconnect to two compounding factors: existing training regimes never ask the model to invoke generation during reasoning, and the standard decode-reencode pathway does not support effective joint optimization. We present CLEAR, a framework that connects the two capabilities through three progressive steps: (1) supervised fine-tuning on a degradation-aware dataset to establish the generate-then-answer reasoning pattern; (2) a Latent Representation Bridge that replaces the decode-reencode detour with a direct, optimizable connection between generation and reasoning; (3) Interleaved GRPO, a reinforcement learning method that jointly optimizes text reasoning and visual generation under answer-correctness rewards. We construct MMD-Bench, covering three degradation severity levels across six standard multimodal benchmarks. Experiments show that CLEAR substantially improves robustness on degraded inputs while preserving clean-image performance. Our analysis further reveals that removing pixel-level reconstruction supervision leads to intermediate visual states with higher perceptual quality, suggesting that task-driven optimization and visual quality are naturally aligned.

1. Introduction

Image degradation is pervasive in real-world visual data and exposes a broad robustness vulnerability in multimodal models. CLEAR addresses the disconnect between generation and understanding by teaching generate-then-answer behavior, directly bridging latent representations into reasoning, and optimizing both capabilities with answer-correctness rewards.

  • Motivation: Real-world blur, noise, poor illumination, and compression damage visual cues and cause substantial accuracy losses across six standard multimodal benchmarks.The vulnerability spans commercial systems and open-source architectures, making degradation robustness important for practical deployment.
  • Motivation: Unified multimodal models fail to invoke their generative pathways to compensate for visual information destroyed by degradation.Generation and understanding coexist within these architectures but remain functionally disconnected during degraded-image question answering.
  • Motivation: CLEAR attributes this disconnect to training that never teaches generation during reasoning and a frozen decode-reencode pathway that blocks joint optimization.The behavioral gap concerns when and how to generate, while the structural gap prevents answer-level feedback from shaping generation.
  • Approach: CLEAR uses degradation-aware supervised fine-tuning, a Latent Representation Bridge, and Interleaved GRPO to connect visual generation with reasoning.The framework establishes generate-then-answer trajectories, injects generated latents directly into context, and jointly optimizes text and denoising under answer-correctness rewards.
  • Findings: CLEAR substantially improves degraded-image understanding while preserving clean-image performance, and removing pixel-level reconstruction supervision can improve generated-state perceptual quality.The authors interpret this as evidence that task-driven optimization and visual quality are naturally aligned.

2. Related Work

Prior work establishes that image degradation harms vision-language performance and that unified models can interleave understanding with generation. CLEAR targets the underexplored gap of using that generative capability to improve degraded-image reasoning through jointly optimized multimodal trajectories.

  • Robustness under Image Degradation: Existing robustness studies show that vision-language models remain sensitive to blur, noise, weather effects, and digital distortions across visual tasks.This literature includes visual question answering, captioning, and multimodal reasoning benchmarks.
  • Robustness under Image Degradation: Corruption-aware augmentation and external restoration pipelines mitigate degradation effects but do not directly connect restoration to unified-model reasoning.The related work contrasts these approaches with CLEAR’s internal capability-bridging objective.
  • Unified Vision-Language Models: Unified vision-language models combine image understanding and generation through discrete visual tokens or continuous VAE latent representations.These architectures trade off generation quality and understanding performance while sharing multimodal representations or backbones.
  • Unified Vision-Language Models: Interleaved multimodal generation offers richer reasoning trajectories, but its potential for degraded-image understanding remains largely unexplored.Current pipelines commonly require generated visual content to pass through pixel space and vision re-encoding before reasoning.
  • Reinforcement Learning for Vision-Language Reasoning: Existing reinforcement-learning methods optimize text reasoning or visual generation separately, whereas unified models require a shared reward for interleaved text-image trajectories.GRPO addresses text reasoning and Flow-GRPO addresses flow-based visual generation, but prior methods do not jointly optimize their contribution to final answers.

3. Method

CLEAR is implemented as a two-stage training pipeline that first establishes adaptive generate-then-answer behavior and a latent bridge, then jointly optimizes text reasoning and visual denoising with Interleaved GRPO. The bridge bypasses decode-reencode processing so answer-level rewards can shape generated visual states.

  • Base Architecture: CLEAR builds on Bagel-7B, whose shared Mixture-of-Transformer backbone supports understanding through SigLIP and generation through a VAE.Both pathways feed into a common language-model reasoning space.
  • Behavioral Initialization through SFT: Stage 1 supervised fine-tuning uses degradation-aware examples to teach direct answers for solvable inputs and generate-then-answer trajectories for failed degraded inputs.The dataset contains 24k samples split evenly between the two pathway types, with a separate 24k set reserved for Interleaved GRPO.
  • Behavioral Initialization through SFT: The SFT objective combines next-token prediction with MSE denoising and KL distillation losses to initialize textual reasoning and latent usage.The auxiliary losses encourage generated VAE latents to approximate clean-image latents and teach the model to read useful information from the latent path.
  • Behavioral Initialization through SFT: During SFT, generated VAE latents and ViT re-encoded features enter the reasoning context, while the ViT path is removed after the bridge is established.The vision encoder and VAE encoder/decoder remain frozen, and only the language-model backbone is updated.
  • Behavioral Initialization through SFT: The MSE target constrains post-SFT generated states through regression-to-the-mean, limiting their sharpness and perceptual quality.This motivates answer-correctness-based optimization beyond clean-image latent approximation.
  • Latent Representation Bridge: The Latent Representation Bridge concatenates generated VAE latent tokens directly with degraded-input ViT features and text, eliminating the pixel-space detour.This preserves complementary semantic and fine-grained structural evidence while creating a differentiable route from answer supervision to generation.
  • Interleaved GRPO: Interleaved GRPO jointly optimizes text tokens with GRPO and denoising steps with Flow-GRPO under shared answer-correctness rewards.The method samples complete interleaved trajectories and derives group-relative advantages from their final answers.
  • Interleaved GRPO: Randomly selecting one recorded denoising step per trajectory enables a single forward pass to produce both text logits and the denoising velocity field.This reduces the image-side optimization cost compared with retaining computation graphs across every denoising step.

4. Experiments

CLEAR is evaluated on degraded-image benchmarks, where its progressive training and latent bridge improve robustness, component effectiveness, and adaptive inference behavior while retaining clean-image performance.

  • Experimental Setup: 24k SFT samples are split evenly between direct-answer and generate-then-answer trajectories, and evaluation covers 16 corruption types across six MMD-Bench benchmarks plus R-Bench-Dis.The model is built on Bagel-7B, with only the language-model backbone updated while the SigLIP encoder and VAE components remain frozen.
  • Main Results: CLEAR-RL reaches 65.26 averaged across seven evaluation sets, improving Bagel by 5.11 points (8.5% relative) without additional parameters or external modules.CLEAR-SFT improves the average by 3.27 points over Bagel, while reinforcement learning provides the further improvement.
  • Main Results: A restoration model followed by Bagel reaches 65.05 on R-Bench, while CLEAR-RL reaches 67.07, exceeding it by 2.02 points.The comparison contrasts pixel-level restoration with CLEAR’s answer-correctness-driven intermediate representations.
  • Robustness Analysis: CLEAR-RL reduces Bagel’s clean-to-hard performance drop from 7.29 points (10.9% relative) to 5.56 points (7.8%), a 24% reduction in the robustness gap.Its advantage over Bagel widens from +4.11 on clean inputs to +5.39 on hard inputs.
  • Ablation Studies: SFT establishes the generate-then-answer behavior, while the latent bridge enables joint optimization; replacing it with decode-reencode limits the gain to 63.72.Applying GRPO directly to base Bagel is infeasible because it does not produce valid generate-then-answer trajectories.
  • Adaptive Generation Behavior: Generation triggering rises from 5.2% at low degradation to 12.2% at mid and 36.4% at high, with inference overhead tracking the triggering rate.CLEAR skips generation when available visual information is sufficient and triggers it when severe blur obscures critical details.

5. Conclusion

CLEAR bridges generation and understanding so unified multimodal models can use generated visual states during reasoning on degraded inputs. Its training and optimization combine text reasoning with visual generation under shared answer-based rewards.

  • CLEAR identifies a functional disconnect in unified multimodal models, where generation and understanding coexist but remain isolated under degraded inputs.
  • The framework establishes generate-then-answer behavior, connects generation to reasoning through a latent bridge, and jointly optimizes both capabilities with Interleaved GRPO.
  • CLEAR improves degraded-image performance while preserving clean-image accuracy, and answer-correctness-only rewards produce intermediate visual states with higher perceptual quality.
  • GRPO estimates completion advantages from within-group reward normalization, avoiding a separate value network and reducing memory and implementation complexity.
  • Flow-GRPO extends this optimization to image-generating flow-matching trajectories by sampling denoising paths and applying group-relative advantages.
  • Interleaved GRPO shares one final-answer reward across text and visual objectives while optimizing only one denoising step per trajectory to reduce memory cost.

B.1. Motivation and Comparison with R-Bench

MMD-Bench is designed to diagnose degraded-image robustness more systematically than R-Bench. It combines controlled severity, structured corruption categories, and diverse multimodal capabilities across established benchmarks.

  • Motivation and Comparison with R-Bench: R-Bench uses fixed pre-degraded images without clean counterparts or systematic severity control, limiting clean-to-degraded and severity-based robustness analysis.
  • Motivation and Comparison with R-Bench: MMD-Bench applies degradations to benchmarks with established clean-image performance, enabling direct computation of the clean-to-degraded performance gap.
  • Motivation and Comparison with R-Bench: MMD-Bench groups 16 corruption types into four real-world source categories for category-level and corruption-level vulnerability analysis.
  • Motivation and Comparison with R-Bench: Unlike R-Bench’s single combined score, MMD-Bench uses six benchmarks targeting different multimodal capabilities for fine-grained diagnosis.
  • Evaluation: R-Bench-Dis is included as an additional independently constructed evaluation set to test whether CLEAR generalizes beyond MMD-Bench.
  • Dataset Construction: Each image receives 48 degraded variants from 16 corruption types across Low, Mid, and Hard severity levels, with scores averaged across corruption types.

C.3. Dataset Statistics

The SFT dataset is approximately balanced across corruption types and severity levels, with pathway assignments introducing minor imbalances because harder corruptions more often trigger failures. Training and inference explicitly select between generation and direct answering based on image quality.

  • Dataset Statistics: The degradation distribution is approximately uniform across 16 corruption types and three severity levels.
  • Dataset Statistics: Minor distribution imbalances arise because harder corruptions are more likely to cause model failures and therefore receive generate-then-answer pathway assignments.
  • Dataset Statistics: The shared system prompt defines generate-then-answer for obscured details and direct answering when degraded visual information remains sufficient.
  • Dataset Statistics: The prompt requires explicit image-quality analysis before the model decides whether to invoke generation.

E. Full Severity-Level Results

CLEAR-RL degrades gracefully as corruption severity increases, with only a modest clean-to-low decline and the largest loss occurring from Mid to Hard. Stability varies across benchmarks.

  • Full Severity-Level Results: 70.27 to 64.97: CLEAR-RL’s average score declines 5.30 points, or 7.5% relatively, from Clean to Hard.
  • Full Severity-Level Results: 0.88 points: the Clean-to-Low decline is modest, indicating minimal accuracy loss under mild degradation.
  • Full Severity-Level Results: 3.40 points: the steepest decline occurs from Mid to Hard, when severe corruptions obscure critical visual details beyond full generative recovery.
  • Full Severity-Level Results: 9.22 points: MM-Vet has the largest absolute Clean-to-Hard drop, consistent with its reliance on integrated multi-cue reasoning.
  • Full Severity-Level Results: 61.43 to 61.05: RealWorldQA remains notably stable across severity levels, likely because its questions depend more on scene layout than fine texture.

F. Per-Corruption Analysis

CLEAR-RL improves over Bagel across all 16 corruption types, with the largest gains for degradations that uniformly destroy spatial structure and smaller gains for foreign-content overlays.

  • CLEAR-RL improves over Bagel consistently across all 16 corruption types.
  • +5.98 is the largest category-level gain for capture degradations, followed by +5.75 for environmental and +5.28 for transmission degradations.Post-processing degradations improve least, by +4.19.
  • +7.17 for motion blur and +6.41 for gaussian noise are the largest individual-corruption improvements.Lens blur and dark illumination also show sizable gains of +6.66 and +6.31.
  • +3.58 for graffiti is the smallest individual-corruption gain, as foreign visual content is harder to address through the same generative process.
  • The generate-then-answer strategy is broadly effective, benefiting uniform spatial degradation more than corruptions that introduce foreign visual content.

G. Inference Latency

CLEAR-RL inference time rises with degradation severity because its adaptive policy triggers generation more often, while untriggered cases retain text-reasoning overhead comparable to Text-only CoT.

  • Inference time closely tracks the generation triggering rate across degradation levels.
  • 5.2% triggering under Low degradation rises to 36.4% under High degradation, increasing total six-benchmark evaluation time to 3 hours 35 minutes.This represents a 74% increase.
  • 54m →1h 35m is the largest absolute benchmark-time increase, observed for MMBench with 2,974 samples.MMVP increases from 5m to 8m across the same severity comparison.
  • When generation is not triggered, overhead is comparable to Text-only CoT; triggered cases add a fixed 30-step denoising cost per sample.

H.2. Reward Weights

Reward weighting and denoising-step analyses identify balanced generation supervision and 30 denoising steps as favorable settings for CLEAR-RL accuracy and efficiency.

  • Reward weights: A 0.79-point Hard drop occurs when the decision reward is removed, because the model loses a direct signal for when to trigger generation.The model then tends to over-generate on easy inputs or under-generate on hard inputs.
  • Denoising steps: 30 denoising steps are a favorable accuracy-efficiency trade-off: 50 steps add only 0.07 points while increasing denoising time by 67%.At 10 steps, accuracy drops 1.29 points relative to the default.
  • Reward design: The accuracy reward uses GPT-4.1-mini to assign semantic correctness scores from 0 to 1, while the format reward prevents degenerate output structures.
  • Reward design: The asymmetric decision reward rewards successful generation and penalizes missed opportunities while treating two other outcomes as neutral.

J. Qualitative Results

Qualitative examples show CLEAR-RL selecting between direct answering and generate-then-answer reasoning, while also exposing failures when critical evidence remains too small or corrupted to recover.

  • Reasoning trajectories: In a generate-then-answer trajectory, the model diagnoses motion blur, triggers restoration, and extracts the recovered text.
  • Reasoning trajectories: In a direct-answer trajectory, the model judges that noisy visual information is sufficient and answers without generation.
  • Failure case: When critical evidence occupies a very small, severely corrupted region, 30-step denoising may improve overall quality without recovering enough detail for correct identification.The paper suggests region-aware or adaptive-resolution generation as a future direction.
  • Additional examples: Across additional examples, CLEAR-RL triggers generation for severe degradations, skips it for mild ones, and recovers task-relevant text, boundaries, and spatial layout.
Loading 2604.04780v1…