PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessIR: Harnessing Multimodal Foundation Models for Universal Real-World Image Restoration

Xiangtao Kong, Shuaizheng Liu, Rongyuan Wu, Lingchen Sun, Zhengqiang Zhang, Jinxin Zhao, Yuhui Wu, Lei Zhang

arXiv:2610.10133v1cs.CV

TL;DR

Complex real-world degradations challenge agentic restoration systems that sequence task-specific models. HarnessIR guides an MFM with image-specific prompts and evidence, then verifies its output. The authors report improved restoration on synthetic and challenging real-world benchmarks, especially over direct MFM use.

  • Problem

    Real-world images contain mixed degradations that are difficult to model as single corruptions, while sequential task-specific restoration tools have limitations on interacting degradations and generalization.

    Source: §1 · p. 2

  • Method

    HarnessIR uses a single MFM as the restorer, guided by an image-specific prompt rather than a chain of task-specific restoration models.

    Source: §1 · p. 3, §3.1 · p. 3

  • Results

    HarnessIR consistently improved restoration over direct MFM use across synthetic and challenging real-world benchmarks, with particularly large gains on real-world images.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The authors conclude that a carefully designed harness can unlock restoration potential in suitable MFMs and make them more effective and controllable universal real-world restorers.

    Source: §5 · p. 13

    HarnessIR requires online VLM and MFM APIs, and its harness cannot raise the executor’s performance ceiling.

    Source: §5 · p. 13

Abstract

from arXiv · show

Real-world low-quality images suffer from complex mixed degradations, including but not limited to noise, blur, atmospheric effects, etc. Recent agentic methods usually model real-world image restoration (Real-IR) as a sequential tool calling problem over task-specific single-degradation restoration models. This paradigm, however, is fundamentally limited because complex real-world degradations cannot be cleanly undone degradation by degradation, and the tool used for task-specific models caps the capability of the agent system. In this work, we present HarnessIR, an agentic framework for Real-IR by harnessing a multimodal foundation model (MFM) as the executor. HarnessIR consists of five stages: perception and diagnosis, on-demand tool invocation, prompt composition, execution, and verification-driven refinement. Unlike prior agentic Real-IR methods that rely on tool chains assembled from task-specific models, HarnessIR feeds the restoration requirements, the perceptual diagnosis, and the evidence into an MFM that performs restoration in a single pass, followed by verification stages to determine whether the result warrants further processing. Under our harness, off-the-shelf MFMs handle restoration tasks remarkably well, achieving state-of-the-art results on the widely used MiO100 synthetic benchmark. More importantly, by exploiting the strong generalization ability of MFMs, HarnessIR delivers compelling restoration quality on challenging real-world scenes where previous agentic IR systems often struggle. Codes is available at https://github.com/PolyU-VCLab/HarnessIR.

1 Introduction

Real-world images combine degradations that challenge methods built around one identifiable corruption and agentic systems that sequence specialized restorers. HarnessIR instead guides an MFM with image-specific diagnosis and evidence, then checks whether its output needs refinement.

1 Introduction

  • Real-world images often contain complex, mixed degradations that single-degradation methods cannot model.Many all-in-one and multiple-in-one methods still assume one degradation type and explicitly or implicitly identify it before restoration.Source: §1 · p. 2
  • Prior agentic IR systems sequence task-specific restorers, but interacting degradations may resist clean sequential reversal and their tools generalize poorly.The pipeline can also lose useful perceptual detail between diagnosis and restoration, while evaluations often use synthetic degradations.Source: §1 · p. 2
  • Naively prompted MFMs can over-generate, embellishing details or repainting text and image content.The paper argues that appropriate instructions and additional tools can harness MFM capabilities for real-world restoration.Source: §1 · p. 2
  • HarnessIR outperforms prior methods on challenging real-world images and surpasses existing all-in-one and agentic methods on almost every MiO100 metric.The authors attribute the gains to the harness rather than the executor alone.Source: §1 · p. 2
  • HarnessIR uses one MFM as executor, with auxiliary tools providing evidence for prompts rather than serving as restoration tools.Its stages are perception and diagnosis, on-demand tool invocation, prompt composition, execution, and verification-driven refinement.Source: §1 · pp. 2–3

2 Related Work and Discussion

Prior restoration methods address isolated or identifiable degradations, while agentic systems plan sequences of specialized restoration networks. HarnessIR instead wraps an MFM with evidence gathering and verification to guide restoration.

All-in-One and Agent-based Image Restoration

  • All-in-one and multi-in-one methods unify restoration but generally assume a single identifiable degradation, limiting performance in complex real-world scenes [28].Agentic IR systems plan sequences of task-specific networks; their generalization is bounded by those tools under out-of-distribution degradations.Source: §2 · p. 3

Multimodal Foundation Models (MFMs)

  • MFM editing interfaces can express restoration as language instructions, but prior evaluations report over-synthesized details and difficulty preserving image fidelity [20] [21].The authors argue that an appropriate harness can improve perceptual quality while preserving image content.Source: §2 · p. 3

Harness: From Foundation Models to Agentic Systems

  • A harness frames an MFM with prompts, control flow, evidence-gathering tools, and verification; HarnessIR diagnoses first, gathers auxiliary evidence, constrains restoration, and may retry.The harness is presented as the component that turns a foundation model into an agentic system.Source: §2 · p. 3

3 HarnessIR

HarnessIR replaces sequential restoration-tool plans with a fixed MFM executor guided by an image-specific prompt. It gathers evidence on demand and re-executes from the original image only when verification calls for it.

3.1 Problem Definition and Overview

  • HarnessIR keeps a frozen MFM executor fixed across rounds, varies its prompt, and has the verifier select among complete candidates generated from the original input.Unlike prior agentic IR methods, it does not process successive outputs through different restoration networks.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 What is Planned: Evidence to Gather, Not Restorers to Run

  • The diagnoser chooses auxiliary tools according to image content, gathering evidence such as recognized text, face boxes, depth maps, or semantic labels for the composer.Simple images may not need tool calls; tool outputs go to the composer, not the executor.Source: §3.2 · pp. 4–5

3.3 What is Passed to Executor: Instruction Prompt, Not Execution Order

  • The composer interprets tool evidence alongside the image and folds relevant observations and restoration requirements into one image-specific textual instruction.The prompt specifies what to treat, what to preserve, and where those requirements apply.Source: §3.3 · p. 5

3.4 When the Loop Fires: On Failure, Not by Design

  • Verification uses quality scores and comparisons with the input as diagnostic evidence, then decides whether to accept a result or request another round.A retry starts from the original image, and the verifier retains the better candidate; at most three rounds are allowed.Source: §3.4 · p. 5

4 Experiments

Experiments compare HarnessIR with direct MFM use and prior restoration methods on MiO100 and two real-world test sets. Ablations examine executor inputs, refinement, VLM choice, cost, and human preference.

4.1 Experimental Setup / 4.2 Evaluation Protocol and Metrics

  • Tests cover synthetic MiO100 and paired and no-GT real-world sets; evaluation cautions that NR-IQA alone can reward altered content and adds joint VLM D/F and DF scoring.When GT exists, the study reports the absolute NR-IQA distance from GT; the per-image DF-Score is the geometric mean of D-Score and F-Score.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.3 Main Results

  • On MiO100, HarnessIR-NB2 achieves the best results on most metrics except SSIM and F-Score, while HarnessIR-GPT achieves the best ΔTOPIQ.Against direct-use baselines, HarnessIR-NB2 and HarnessIR-GPT gain 1.40 and 1.75 dB in PSNR, respectively; their DF-Score gains are 4.1 and 18.1 points.Source: §4.3 · p. 7, Table 1 · p. 7
  • On Real-Paired-200, HarnessIR-NB2 reaches DF-Score 62.0 and HarnessIR-GPT raises DF-Score from 40.1 to 60.2, both exceeding the prior best of 38.8.On Real-NoGT-200, HarnessIR reaches DF-Scores of 62.9 for NB2 and 49.3 for GPT-Image-2.5, above the prior best of 32.0. Visual comparisons show HarnessIR removing degradations while preserving scene content that direct MFM use sometimes changes.Source: Table 2 · p. 7, §4.3 · p. 8, Table 3 · p. 8

4.4.1 Ablation of Executor Inputs

  • Richer prompts and image-specific analysis improve restoration; text-composed auxiliary evidence avoids map artifacts and yields the strongest one-round result: PSNR 26.01 dB, DF-Score 59.6.The one-round ablation isolates executor inputs without verification or re-execution.Source: §4.4.1 · pp. 9–10, Table 4 · p. 10

4.4.2 Redo vs. In-place Repair During Re-execution

  • Redoing from the original LQ image raises DF-Score to 62.0, while in-place refinement lowers it from the single-pass score of 59.6 to 57.2.Only 37% of test images undergo re-execution; already satisfactory first-round outputs remain unchanged.Source: §4.4.2 · p. 10, Table 5 · p. 10

4.4.3 VLM Choice

  • GPT-5.6-Sol achieves the highest PSNR and DF-scores, but its gains over Gemini-3.7-Flash are modest: 0.17 dB in PSNR and 0.9 points in DF-Score.The final system uses Gemini-3.7-Flash because restoration performance is comparable and GPT-5.6-Sol incurs substantially higher API cost.Source: §4.4.3 · pp. 10–11, Table 6 · p. 11

4.5 Cost Analysis

  • The complete system reaches DF-Score 62.0 and PSNR 26.63 dB at an average cost of $0.129 per image; prior agentic methods cost $0.137 and $0.190.The reported costs are idealized estimates that exclude difficult-to-assign expenses such as deployment overhead.Source: §4.5 · pp. 11–12, Table 7 · p. 11

4.6 User Study

  • In a 5,000-vote user study, HarnessIR-NB2 receives 48.3% of selections and HarnessIR-GPT 28.1%, together accounting for 76.4%.Annotators chose the best of eight restorations for each image, judging degradation removal, scene-content preservation, and artifacts.Source: §4.6 · pp. 12–13, Figure 5 · p. 12

5 Conclusion

HarnessIR uses an MFM executor with a coordinated harness to improve restoration across synthetic and challenging real-world benchmarks. The conclusion attributes the gains to the harness design and notes that it cannot raise an executor’s performance ceiling.

5 Conclusion

  • HarnessIR consistently improves restoration over direct MFM use, with particularly large gains on real-world images.The conclusion says ablations attribute the improvements to the coordinated harness design at acceptable cost.Source: §5 · p. 13
  • Image-specific diagnosis, auxiliary tools, prompt composition, and verification-driven re-execution help remove degradations without altering image content.The harness is presented as unlocking the restoration potential of suitable MFMs and making them more effective and controllable universal real-world restorers.Source: §5 · p. 13
  • An agentic harness can elicit and regulate its executor’s capabilities but cannot raise the executor’s performance ceiling; HarnessIR also requires online VLM and MFM APIs.The conclusion identifies API access as a requirement shared with AgenticIR and 4KAgent.Source: §5 · p. 13

Appendix

  • The appendix covers a case study, test-set composition, the DF-Score protocol, detailed MiO100 results, and additional visual comparisons.Source: Appendix · p. 18

A.1 Input Request

The appendix uses short fixed requests to represent general-user intent, with mixed degradation as the default and additional removal instructions for specified degradations.

A.1 Input Request

  • Short, fixed requests represent the intent of a general user, and the mixed-degradation request serves as the default.Source: §A.1 · p. 18
  • When degradation types are specified, a removal instruction is prepended to the default request.Source: §A.1 · p. 18
  • For mixed degradations, applicable names are joined with “and” or placed at the end; haze and rain with a brightness request provide an example.Source: §A.1 · p. 18

A.2 A Complete Case Study

The case study follows HarnessIR from image diagnosis and evidence gathering through prompt composition, execution, and verification-driven re-execution. It shows how image-specific observations shape restoration instructions and how the output is checked against the input.

A.2 A Complete Case Study

  • Text, layered depth, and semantically distinct regions lead the diagnosis to request OCR, depth estimation, and segmentation for image-specific evidence.The example’s tool rationales include preserving watermark text, respecting shallow depth of field, and separating the bird from the background.Source: §A.2 · p. 19
  • Figure A.1 presents the input, auxiliary-tool outputs, and first- and second-round restorations as a complete HarnessIR workflow example.Source: §A.2 · p. 18, Figure A.1 · p. 20
  • The composer uses depth to target the bird and feeder while preserving naturally blurred foreground and background, and records the watermark for unchanged preservation.Source: §A.2 · p. 20
  • Verification detects an incorrectly generated visible claw and unsupported feather details; the revised instruction corrects the claw, and the next check requires no re-execution.Source: §A.2 · p. 21

B Real Test Set Composition

The appendix describes two real-world test sets, one with paired ground truth and one without, and specifies how their images are sampled and spatially standardized.

B Real Test Set Composition

  • Real-Paired-200 contains 200 paired low-quality and ground-truth images, while Real-NoGT-200 contains 200 real-world low-quality images without ground truth.Source: B Real Test Set Composition · p. 23
  • Images are randomly sampled from public source datasets, with each source’s contribution listed in Table B.1.Source: B Real Test Set Composition · p. 23
  • Images shorter than 1024 pixels on the longer side are isotropically resized; longer images are center-cropped along that dimension, with paired ground truth transformed to remain pixel-aligned.Source: B Real Test Set Composition · p. 23

C DF-Score Evaluation Protocol

The evaluation tests whether DF-Score aligns with human preferences and whether its scores are consistent across VLM evaluator families. DF-Score shows higher human alignment than the compared NR-IQA metrics, while score correlations across evaluators are high.

Alignment with Human Judgments

  • Annotators make one forced-choice judgment per pair, selecting the restoration that removes degradation while preserving content and structure and avoiding visible artifacts.Method identities and metric scores are hidden, and restoration order is randomized.Source: C DF-Score Evaluation Protocol · p. 23
  • DF-Score agrees with human preference in 87% of valid comparisons, versus 50.8%–67.1% alignment for four NR-IQA metrics.The study uses 300 restoration comparison pairs; pairs without a strict majority are discarded, and random ranking has an expected alignment of 50%.Source: C DF-Score Evaluation Protocol · p. 23, Figure C.2 · p. 24

Consistency Across Evaluator Families

  • D-Score, F-Score, and DF-Score correlate with Gemini-3.7-Flash scores at 0.85–0.94 across the alternative VLM evaluators.The evaluation compares scores from GPT-5.6-Sol and Claude-Opus-5 against Gemini-3.7-Flash on 400 restored images.Source: C DF-Score Evaluation Protocol · p. 23, Table C.2 · p. 24

D Detailed Results on MiO100

HarnessIR improves both executors across all three MiO100 degradation groups, with its advantage persisting as degradations become more complex.

D Detailed Results on MiO100

  • Group C combines three simulated degradations, compared with two in Groups A and B, and most methods have lower PSNR, SSIM, and DF-Score on Group C.For direct NB2, DF-Score falls from 61.75 in Group A and 61.00 in Group B to 44.85 in Group C.Source: D Detailed Results on MiO100 · p. 24
  • For NB2, HarnessIR raises DF-Score to 65.47, 65.22, and 49.12 for Groups A, B, and C, respectively.Both HarnessIR variants achieve higher DF-Scores than previous methods in all three groups.Source: D Detailed Results on MiO100 · p. 24
  • HarnessIR’s gains over direct GPT-Image-2.5 are 27.29 points for Group A, 20.85 for Group B, and 6.05 for Group C.Source: D Detailed Results on MiO100 · p. 24

E More Visual Results

Visual comparisons show HarnessIR restoring details while preserving image content, including facial features, text, and scene structure. The comparisons also describe failure modes in prior methods on real-world images.

E More Visual Results

  • HarnessIR restores facial details and text more faithfully than direct MFM use, while maintaining color appearance and avoiding observed text shifts or content changes.The described examples also show removal of rain or snow without changing the shape or appearance of other scene elements.Source: E More Visual Results · p. 24, Figure E.3 · p. 26
  • Sequential single-degradation methods can fail when processing one degradation changes the remaining degradations’ distribution, leaving the next model with an out-of-distribution input.The passage contrasts this error-propagation issue with HarnessIR’s unified MFM execution and redo refinement strategy.Source: E More Visual Results · p. 24
  • InstructIR may brighten scenes when low-light enhancement is not requested, while 4KAgent produces a substantially corrupted result in one example.Existing all-in-one and agentic methods often provide only limited visible improvement on the described real-world inputs.Source: E More Visual Results · p. 24
Loading 2610.10133v1…