PaperlayerView exact revision on arXiv

Source-linked AI summary

JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration

Yunlong Lin, Zixu Lin, Haoyu Chen, Panwang Pan, Chenxin Li, Sixiang Chen, Yeying Jin, Wenbo Li, Xinghao Ding

arXiv:2504.04158v1cs.CV

TL;DR

Real-world weather restoration faces unknown, coupled degradations and a lack of paired labels for degraded images. JarvisIR uses a VLM to coordinate expert restoration models, with supervised fine-tuning and human-feedback alignment supported by CleanBench. The paper reports stronger decision-making and perception-restoration performance than strong baselines.

  • Problem

    Real-world weather images contain unknown, coupled degradations, while task-specific methods need degradation priors and synthetic-trained all-in-one methods face a domain gap.

    Source: §1 · p. 1

  • Method

    JarvisIR uses a VLM agent to coordinate expert restoration models and combines supervised fine-tuning on synthetic CleanBench data with human-feedback alignment on real data.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    JarvisIR outperforms strong baselines in decision-making and perception restoration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Human feedback enables unsupervised VLM fine-tuning on large-scale unlabeled real-world data for restoration in the wild.

    Source: §1 · p. 1, §12 · p. 20

    JarvisIR does not cover every real-world restoration scenario, including underwater images, old-photo enhancement, or blind face restoration; its current tools are limited in scope and capability.

    Source: §12 · p. 19

Abstract

from arXiv · show

Vision-centric perception systems struggle with unpredictable and coupled weather degradations in the wild. Current solutions are often limited, as they either depend on specific degradation priors or suffer from significant domain gaps. To enable robust and autonomous operation in real-world conditions, we propose JarvisIR, a VLM-powered agent that leverages the VLM as a controller to manage multiple expert restoration models. To further enhance system robustness, reduce hallucinations, and improve generalizability in real-world adverse weather, JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment. Specifically, to address the lack of paired data in real-world scenarios, the human feedback alignment enables the VLM to be fine-tuned effectively on large-scale real-world data in an unsupervised manner. To support the training and evaluation of JarvisIR, we introduce CleanBench, a comprehensive dataset consisting of high-quality and large-scale instruction-responses pairs, including 150K synthetic entries and 80K real entries. Extensive experiments demonstrate that JarvisIR exhibits superior decision-making and restoration capabilities. Compared with existing methods, it achieves a 50% improvement in the average of all perception metrics on CleanBench-Real. Project page: https://cvpr2025-jarvisir.github.io/.

1. Introduction

Real-world weather restoration must handle unknown, coupled degradations, but current approaches rely on degradation-specific priors, synthetic training data, or sensitive task ordering. JarvisIR uses a VLM to coordinate restoration experts, supported by CleanBench and a two-stage training framework.

1. Introduction

  • Task-specific methods rely on degradation-type priors, while supervised all-in-one methods trained on synthetic data face a real-world domain gap; expert collaboration is sensitive to task order.Even minor changes in execution sequence can cause significant performance degradation.Source: §1 · p. 1
  • Real degraded images are plentiful, but missing corresponding labels prevents supervised fine-tuning on them; human feedback offers an unsupervised training route.The paper frames reliable VLM control of specialized restoration models as a question requiring extensive paired data under conventional approaches.Source: §1 · p. 1
  • JarvisIR combines a VLM controller and expert restoration models with CleanBench, which contains 150K synthetic and 80K real instruction-response pairs.The dataset is designed to support training and evaluation.Source: §1 · p. 2
  • The framework uses supervised fine-tuning on synthetic CleanBench data, then human-feedback alignment on CleanBench-Real to improve robustness, reduce hallucinations, and generalize in real-world adverse weather.MRRHF adds hybrid sample generation, a regularization term, and multiple VLM-based IQA models as a unified reward model.Source: §1 · p. 2
  • JarvisIR outperforms strong baselines in decision-making and perception restoration.The paper presents it as a VLM-powered agent that autonomously manages multiple expert restoration models for coupled weather degradations in real-world environments.Source: §1 · p. 2

2. Related Work

Related work spans tool-using language models, preference alignment, and image restoration. The paper positions JarvisIR against restoration systems designed for synthetic conditions and describes its dataset workflow as three stages.

Tool-Augmented LLMs

  • Tool-augmented LLM research covers tool-call responses [53], multi-tool interaction [56], generalized tool use [65], and tool selection [60], while visual systems use Hugging Face models [61], Azure models [88], and visual foundation models [78].Source: §2 · pp. 2–3

Alignment of LLMs

  • RLHF learns rewards from human preferences and commonly optimizes with PPO [1] [28] [66], while RRHF ranks multiple responses and can extend to fine-grained tool agents [93].The paper notes that tool-agent feedback arrives after tool response and invocation, and its two-stage process combines visual and linguistic modalities.Source: §2 · p. 3

Image Restoration

  • Task-specific restoration addresses individual degradations, while unified methods target multiple degradation types; RestoreAgent [9] and AgenticIR [101] plan restoration tools for synthetic settings, unlike this work's real-world focus.The paper describes its approach as incorporating human feedback and large-scale unlabeled real-world data.Source: §2 · p. 3

Dataset construction workflow

  • CleanBench construction proceeds through degraded-image synthesis, assessment reasoning and optimal task-sequence generation, then instruction-response pair generation.Source: Figure 2 · p. 3

3. Methodology

The methodology pairs CleanBench data construction with JarvisIR, an agent that coordinates expert restoration models, then trains it through supervised fine-tuning and real-data feedback alignment.

3.1. CleanBench

  • CleanBench combines 150K synthetic instruction-response pairs with CleanBench-Real, 80K unlabeled real images across rainy, night, snowy, and foggy conditions.For evaluation, 500 images per real-world scenario form a 2K-image set; the rest are used for alignment tuning.Source: §3.1 · p. 4

3.2. JarvisIR

  • JarvisIR plans restoration tasks, selects expert models, executes them, and integrates their results into a final response.Its workflow assesses image degradation and parses user instructions before planning.Source: Figure 4 · p. 4, §3.2 · p. 4

3.2.1. JarvisIR-SFT

  • JarvisIR-SFT is initialized through standard supervised fine-tuning on instruction, degraded-image, and ground-truth-response triplets.The VLM predicts a response from the instruction and degraded image.Source: §3.2.1 · p. 4

3.2.2. JarvisIR-MRRHF

  • MRRHF aligns JarvisIR using offline and online response candidates, restoration outcomes scored by an IQA-based reward model, and entropy regularization for response diversity.The reward model standardizes and sums scores from multiple IQA models; the training objective also includes a ranking loss and cross-entropy for the highest-reward response.Source: §3.2.2 · p. 5
  • Vanilla RRHF uses a static offline preference dataset, which can lead to in-distribution reward over-optimization because the policy cannot query the preference oracle during training.Online sampling may mitigate this issue, but the passage says PPO requires additional GPU resources and significantly reduces training speed.Source: §3.2.2 · p. 6

4. Experiments

The experiments assess real-world decision-making and restoration on CleanBench-Real; JarvisIR surpasses expert-guided strategies and existing all-in-one methods.

4.1. Experimental Settings

  • Llava-Llama3-8b is fine-tuned for three epochs in both SFT and alignment, with CleanBench used for SFT and CleanBench-Real training data for alignment.Evaluation uses the CleanBench-Real validation set to measure decision-making and perception restoration in real-world scenarios; the implementation uses lightweight restoration models.Source: §4.1 · p. 6

4.2. Decision Making Capability

  • JarvisIR achieves the highest decision-making performance, surpassing expert-customized strategies; JarvisIR-MRRHF (4.8%) outperforms JarvisIR-SFT (14.3%) in score and ranking.Human expert strategies (IV) and (V) rank within the top 22.5% and 18.6% of possible strategies, respectively.Source: Table 1 · p. 6, §4.2 · p. 7

4.3. Perception Restoration Ability

  • JarvisIR outperforms existing all-in-one methods across all metrics; on Night scenes, JarvisIR-MRRHF scores 67.25 MUSIQ versus AutoDIR’s 47.30.JarvisIR-MRRHF also exceeds JarvisIR-SFT in all scenes, including Rain MUSIQ (70.38 vs. 65.03) and Fog MUSIQ (74.22 vs. 70.45).Source: Table 2 · p. 7, §4.3 · p. 7

5. Ablation Study

The ablations compare hybrid, offline, and online sample generation and test entropy regularization, reporting reward and response diversity during MRRHF training.

Sample generation strategy

  • Hybrid sampling reaches reward 0.67 and diversity 6.55, exceeding offline (0.43; 3.63) and online (-0.87; 1.27) strategies.Offline sampling is limited to finite SFT-generated data, while online sampling collapses during training; the hybrid combines both sources to maintain reward and diversity.Source: Table 3 · p. 8, §5 · p. 8, Figure 7 · p. 8

Entropy regularization

  • Without entropy regularization, reward decreases from 0.67 to 0.50 and diversity drops from 6.55 to 4.56.The authors report that entropy regularization fosters exploration and more diverse responses.Source: §5 · p. 8, Table 3 · p. 8, Figure 7 · p. 8

6. Conclusions

The conclusion presents JarvisIR as a VLM agent coordinating restoration experts, trained with SFT and human-feedback alignment, and reports improved perception metrics on CleanBench-Real.

Conclusion

  • JarvisIR uses Llava-Llama3 to connect restoration experts and autonomously schedule models for changing scenarios and coupled degradations.Source: §6 · p. 8
  • The two-stage framework combines supervised fine-tuning with human-feedback alignment, which tunes the VLM on large-scale unlabeled real-world data in an unsupervised manner.Source: §6 · p. 8
  • CleanBench contains 150K synthetic and 80K real instruction-response pairs; JarvisIR improves the average of all perception metrics on CleanBench-Real by 50%.Source: §6 · p. 8

Supplementary Material

The supplementary material covers implementation and dataset details, additional ablations, visual results, and limitations, broader impacts, and future work.

Supplementary Material

8. More implementation details

JarvisIR combines task-specific restoration tools with a Llava-Llama3-based VLM whose image and text inputs are fused for response generation. The implementation retrains a tool for missing task weights and fully fine-tunes the language model.

8.1. Restoration tool settings

  • Img2img-turbo lacks snow-to-day weights, so the authors retrain it on the ACDC snow subset using CycleGAN; they note their tools are not the latest state of the art.The authors identify considerable potential for enhancing the restoration models.Source: §8.1 · p. 14

8.2. Details of Model Setups

  • JarvisIR encodes images with frozen CLIP ViT-L/14, projects visual tokens into textual space, fuses them with instruction tokens, and uses Llama3-8B to generate responses.Input images are zero-padded to dimensions divisible by 14; visual tokens are 1024-dimensional and projected to the LLM's 4096-dimensional hidden space.Source: §8.2 · p. 14
  • Both SFT and MRRHF use Adam at a 1e-5 learning rate with cosine decay and train for three epochs, but their batch sizes are 128 and 2, respectively.Training used eight NVIDIA A100 80G GPUs; MRRHF also used diverse beam search with size 3, five groups, diversity penalty 2.0, and temperature 0.8.Source: §8.2 · p. 14

9. CleanBench dataset details

CleanBench combines synthetic degraded-clean pairs with real adverse-weather images, while its degradation library simulates nighttime, fog, rain, and snow using distinct image transformations.

9.1. Dataset statistics

  • CleanBench contains 150K synthetic degraded-clean image pairs and 80K real degraded images spanning night, fog, snow, and rain scenes.The real images come from autonomous-driving and natural-scene sources, including internet and public datasets.Source: §9.1 · pp. 14–15

9.2. Details of degradation library

  • The degradation library synthesizes rainy, nighttime, snowy, and foggy images with physical models and image transformations, with implementation principles and severity setups described for each simulator.The section provides examples of the four simulator outputs.Source: §9.2 · pp. 15–16, Figure 8 · p. 16
  • The night simulator converts daytime sRGB to RAW, attenuates it, adds shot/read noise, applies an ISP pipeline, and adds flare using Flare7K++ templates.Randomized parameters are applied online to represent diverse nighttime driving conditions.Source: §9.2 · p. 16
  • The fog simulator combines depth-based transmission and exponential attenuation with poor-lighting effects, atmospheric-light variation, Gaussian noise, and JPEG artifacts.Its online randomized process varies fog conditions; the stated ranges include β from 0.3 to 1.5 and brightness factor γ from 1.5 to 3.0.Source: §9.2 · pp. 16–17
  • The rain simulator adds physics-rendered raindrop occlusions, streaks, and wetness to a disentangled base image, with stochastic noise enabling varied effects.The rendering model uses parameters for differentiable and nondifferentiable rain properties.Source: §9.2 · p. 17
  • The snow simulator uses SD-Turbo with textual conditioning to transform daytime inputs, while LoRA adapters and skip connections help control scene characteristics and preserve structure.The conditioning text describes the snowy scene, and the resulting image retains consistency with the input scene.Source: §9.2 · p. 17

10. More ablation

The ablations assess JarvisIR on CleanBench-Real using reward, diversity, and non-reference perception metrics. They report benefits from MRRHF, its sampling and loss design, reasoning, and a unified reward model.

10.1. MRRHF vs. vanilla RRHF

  • Compared with vanilla RRHF, MRRHF improves average reward by 0.19 and diversity by 3.43.The authors attribute vanilla RRHF's degradation to offline sample generation restricting responses to the SFT model's finite sample space, whereas MRRHF combines hybrid generation with entropy regularization.Source: §10.1 · p. 17

10.2. Sample generation strategy and entropy regularization

  • Hybrid sampling and entropy regularization improve training stability, support high-quality exploration, and significantly improve CleanBench-Real testing performance.The ablation examines their effects on reward scores and response diversity.Source: §10.2 · p. 17

10.3. Effectiveness of differentiated contrast weights

  • Removing differentiated contrast weights lowers average reward by 0.14 and diversity by 3.93 relative to using the weighted ranking loss.The weights distinguish near-neutral negative responses from substantially worse ones, avoiding equal penalties that could confuse VLM training.Source: §10.3 · pp. 17–18

10.4. Impact of reasoning for decision-making

  • Providing degradation and reasoning insights before task sequences significantly improves all four non-reference metrics on the CleanBench-Real validation set.The authors say these descriptions help align selected tasks and restoration experts with image degradations and make decisions more interpretable and reliable.Source: §10.4 · p. 18, Table 6 · p. 18

10.5. Impact of reward model

  • Multiple VLM-based IQA models improve perception metrics but consume more training resources, while single or traditional-only reward models can yield metric-specific or asymmetric gains.The authors therefore combine VLM-based and traditional IQA models, including Q-Instruct, MUSIQ, and MANIQA, as a unified reward model.Source: §10.5 · pp. 18–19, Table 7 · p. 19

11. More visual results.

Visual comparisons show JarvisIR-MRRHF making more appropriate restoration decisions than JarvisIR-SFT in real adverse-weather examples. On CleanBench-Real scenes, JarvisIR also outperforms comparison methods in color enhancement, detail preservation, and degradation removal.

11.1. Perception restoration

  • In a coupled real-rain example, MRRHF selects denoising, deraining, and deblurring tools where SFT's sequence leaves raindrops and blurs the background.The authors report that MRRHF improves real-world decision-making, reduces hallucination errors, and improves generalization.Source: §11.1 · p. 19
  • Across four CleanBench-Real scenes, JarvisIR outperforms comparison methods in color enhancement, detail preservation, and degradation removal, while baselines struggle with coupled degradations.The reported baseline artifacts include overexposure, underexposure, and amplified noise.Source: §11.1 · p. 19

12. Limitations, broader impacts and future work

JarvisIR does not cover every real-world restoration scenario, and the discussion identifies tool capability and input resolution as areas for future improvement.

12. Limitations, broader impacts and future work

  • JarvisIR does not cover all real-world restoration scenarios, and its current tools remain limited in scope and capability.The paper names underwater restoration, old-photo enhancement, and blind-face restoration as examples outside its coverage; it proposes adding more capable tools.Source: §12 · p. 19
  • JarvisIR-MRRHF selects appropriate restoration models and task sequences in the comparison, while JarvisIR-SFT often makes unsuitable real-world decisions due to the domain gap.Figure 9 describes JarvisIR's process of planning subtasks, invoking expert models, and integrating their results.Source: Figure 10 · p. 20, Figure 9 · p. 20
  • The authors propose retaining original image resolution because fixed-size resizing may degrade performance as resolution variation affects degradation perception.One proposed approach adapts CLIP position embeddings with bicubic interpolation for varying image dimensions.Source: §12 · p. 20
  • The two-stage framework combines supervised fine-tuning with human feedback alignment, using human feedback and large-scale real unlabeled data for unsupervised VLM fine-tuning.The paper presents this as a way to enhance robustness, reduce hallucinations, and improve generalizability.Source: §12 · p. 20

Related papers

Loading 2504.04158v1…