PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Restore: Learning to Predict Before Acting for Agentic Image Restoration

Xu Zhang, Jun Wan, Lefei Zhang, Liangpei Zhang

arXiv:2610.20384v1cs.CV

TL;DR

Mixed-degradation restoration agents typically observe a tool’s effects only after it runs, so failed calls consume computation and harmful side effects are seen afterward. PreActIR predicts candidate-tool consequences before execution and verifies the selected output. In controlled replay, it uses fewer real and rejected calls than reactive trial and error, while the full system achieves the best overall performance across three compound-degradation groups.

  • Problem

    For mixed degradations, agents typically learn a tool’s effect only after execution, although failed calls cost computation and harmful side effects go unnoticed until afterward.

    Source: §1 · p. 1

  • Method

    PreActIR predicts separate candidate-tool consequences before execution, runs the top-ranked tool, verifies its output, and audits whether selection or the candidate space limits performance.

    Source: §1 · p. 2

  • Results

    42.5% fewer real calls and 63.4% fewer rejections than reactive trial and error in matched replay; the full system achieves the best overall performance on all three compound-degradation groups.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The candidate-frontier audit finds considerable headroom for better selection within the existing tool set, while targeted adaptation can push the frontier further.

    Source: §5 · p. 10

    Gains remain to be validated with estimated degradation states, and whether multi-step imagination adds value beyond single-step ranking is unresolved.

    Source: §5 · p. 10

Abstract

from arXiv · show

Agents for mixed-degradation image restoration typically learn what a tool does to an image only after running it: failed calls waste computation and harmful outputs go unnoticed until too late. We propose PreActIR, an agentic restoration framework that predicts tool consequences before execution. For each candidate tool, a tool-conditioned world model forecasts changes in residual degradation, image quality, and fidelity, together with the risk of harm and the uncertainty of the prediction. The agent executes the top-ranked candidate, and an independent verifier commits or rolls back its output before the agent replans from the observed state. PreActIR also audits whether performance is limited by the selector or by the candidate space, and studies targeted tool adaptation as a controlled way to expand that space. In a controlled evaluation that fixes the candidate pool, verifier, and budget and employs reference-annotated degradation states, PreActIR uses 42.5% fewer real tool calls and 63.4% fewer rejected calls than reactive, execute-then-evaluate trial and error, while improving Y-PSNR by 0.92 dB.

1 INTRODUCTION

Mixed degradations make restoration sequential: each tool changes the image state and the value of later actions, while reactive agents learn a tool’s effects only after execution. PreActIR predicts candidate-tool consequences before acting, verifies outputs, and audits whether selection or the tool space limits performance.

1 INTRODUCTION

  • An independent verifier commits or rolls back the top-ranked tool’s output before the agent replans from the observed state.The process is called the predict–act–verify loop.Source: §1 · p. 2
  • A tool-conditioned world model predicts changes in residual degradation, quality, and fidelity, plus benefit and harm probabilities and prediction uncertainty.The agent ranks candidates by these forecasts before execution.Source: §1 · p. 2
  • 42.5% fewer real calls and 63.4% fewer rejections than reactive trial and error in a matched replay with reference-annotated states.The comparison holds the candidate pool, verifier, and budget fixed.Source: §1 · p. 2
  • A candidate-frontier audit separates selector headroom from tool-space limits and measures how targeted tool adaptation shifts the attainable frontier.A poor result alone cannot distinguish a poor selection from the absence of a good reachable candidate.Source: §1 · pp. 1–2

2 RELATED WORK

The related work spans unified restoration models, agentic tool coordination, learned selection, and action prediction. PreActIR’s distinction is deployment-time prediction of separate tool consequences, paired with an audit of whether selection or the available tool set limits performance.

All-in-One and Agentic Image Restoration

  • All-in-one methods handle multiple degradations with unified models, while agentic systems coordinate specialized tools through planning and feedback.The described agentic systems observe tool effects only after execution; SEAR scores candidates using real rollouts.Source: §2 · pp. 2–3

Learned Tool Selection and Pre-Execution Estimation

  • Learned planners and stored-experience methods improve tool choice, while DiTTo and Causal-AgentIR score actions or plans before execution.PreActIR instead trains on real transitions and predicts separate outcomes for every candidate at deployment, while auditing selector versus tool-set limits.Source: §2 · p. 3

World Models and Predict-before-Act Agents

  • World models can let agents evaluate action-conditioned dynamics before acting, but restoration outcomes are continuous and multidimensional, with clean-image fidelity unobservable at test time.PreActIR supervises consequence heads with uncertainty estimates on real tool transitions and uses a post-execution verifier to catch prediction errors.Source: §2 · p. 3

3 METHOD

PreActIR frames restoration as a budgeted sequential decision problem and predicts tool consequences before execution. Its controller verifies executed actions, while an offline audit tests candidate-space limits and targeted adaptation expands the tool set.

3.1 PROBLEM FORMULATION

  • The agent chooses tools under a real-execution budget, but an action’s outcome is unknown until its tool runs.Rejected calls still consume budget; the objective balances final-image utility against relative execution cost.Source: §3.1 · p. 3

3.2 OVERVIEW

  • PreActIR combines a tool-conditioned world model, a predict–act–verify controller, and a tool-evolution loop for auditing and adapting the candidate set.The overview separates deployment-time action selection and verification from candidate-frontier audit and targeted adaptation.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 TOOL-CONDITIONED WORLD MODEL

  • For each candidate, the world model predicts changes in degradation, quality, fidelity, content damage, benefit or harm, and predictive uncertainty.The model is trained on observed before–after tool transitions with supervision for degradation, quality, fidelity, damage, and outcome categories.Source: §3.3 · pp. 4–5

3.4 PREDICT–ACT–VERIFY PLANNING

  • The planner ranks actions by predicted gains and costs, executes the top action, and has an independent verifier commit or roll back its output before replanning from the observed image.The one-step score includes target-degradation, quality, and fidelity gains, non-target side effects, content damage, uncertainty, and relative cost; optional harm-penalty and two-step variants did not improve consistently.Source: §3.4 · p. 5

3.5 CANDIDATE-FRONTIER AUDIT AND TARGETED TOOL ADAPTATION

  • The offline candidate-frontier audit measures the best outcomes reachable from a frozen tool pool, while targeted adaptation adds a tool trained on external task-matched data.References are used only for the post-hoc audit, never by the deployment controller; the expanded pool is re-audited and the adapted tool is incorporated into the deployed system.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate PreActIR on MiO100 and a paired real-world set, then test open-loop action ranking, controlled replay efficiency, and candidate-space limits. Results show strong but metric-dependent restoration performance, fewer calls in matched replay, and remaining headroom for better selection and tool adaptation.

4.1 DATASETS AND EXPERIMENTAL SETUP

  • Evaluation uses 160 training states and 1,440 test images across MiO100 Groups A–C, with five-fold cross-fitting and clean-image identities disjoint between fitting and outer validation.The base tool set contains 33 actions across eight degradation types; controlled planner comparisons use matched candidate pools and tool sets.Source: §4.1 · p. 6

4.2 MIO100 MAIN RESULTS

  • On MiO100, PreActIR leads SSIM across all groups and PSNR on Groups A and C; on the paired real-world set, it leads three perceptual metrics but trails in PSNR.On Group A it also leads LPIPS and MANIQA; on Group C, 4KAgent leads the three perceptual metrics. On the real-world set, PreActIR leads MANIQA, CLIP-IQA, and MUSIQ, while SEAR reaches 24.41 dB PSNR versus PreActIR’s 22.64 dB.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.3 OPEN-LOOP PRE-EXECUTION RANKING

  • Open-loop, the world model’s harm probability achieves ROC-AUC 0.729 versus 0.612 for Direct and 0.656 for WM-F, while its headroom recovery is similar to Direct.Its within-state rank correlation is 0.309 versus 0.247 for Direct; at matched harm rates of 5%, 10%, and 15%, harm-based abstention recovers 0.0, 3.5, and 10.7 more HR points than gain-based abstention, with all intervals including zero.Source: §4.3 · pp. 7–8, Figure 6 · p. 8, Table 3 · p. 8

4.4 PREDICT-BEFORE-ACT VERSUS REACTIVE TRIAL-AND-ERROR

  • With a shared candidate pool, verifier, and four-call budget, PreActIR uses 42.5% fewer real calls and 63.4% fewer rejected calls than reactive trial-and-error, while improving Y-PSNR by 0.92 dB.The replay uses reference-annotated degradation states; against fidelity-head ranking, structured utility reduces harm by 3.75 points, while the efficiency gain mainly comes from ranking before execution.Source: §4.4 · pp. 8–9

4.5 CANDIDATE-FRONTIER AUDIT AND TARGETED TOOL ADAPTATION

  • A post hoc audit finds 1.77 dB of selector headroom at the deployed fixed-pool system’s MUSIQ level, while HAT-S adaptation raises the attainable frontier by 0.50 dB at the final MUSIQ level.The final Group C system reaches 21.05/0.604 Y-PSNR/Y-SSIM at MUSIQ 54.78 but remains 0.7 dB below the fixed-pool frontier; the audit uses test references and is not deployable.Source: §4.5 · p. 9, Figure 7 · p. 9

5 CONCLUSION

The conclusion presents PreActIR as a predict–act–verify restoration framework and reports gains in controlled replays and MiO100 performance. It also identifies unresolved validation and candidate-selection questions.

5 CONCLUSION

  • PreActIR predicts candidate-tool consequences before execution, then uses an independent verifier to commit or roll back the selected tool’s output.In controlled replays with matched candidates, verifier, and budget, predicting before acting reduces real and rejected calls while improving restoration quality.Source: §5 · p. 9
  • On MiO100, the full system achieves the best overall performance on all three compound-degradation groups.The candidate-frontier audit finds considerable headroom for better selection within the existing tool set, and targeted adaptation can push the frontier further.Source: §5 · p. 10
  • The work leaves open validation under estimated degradation states and whether multi-step imagination adds value beyond single-step ranking.Source: §5 · p. 10

AI USE STATEMENT

The authors report using generative AI tools to edit the manuscript and improve readability and language.

AI USE STATEMENT

  • The authors reviewed AI-suggested edits against the original and take responsibility for the final content and AI-assisted artifacts.Source: AI USE STATEMENT · p. 10

A.1 WORLD-MODEL ARCHITECTURE

The appendix describes the world-model architecture, how action outcomes are labeled, and the training objective. The experiments use global actions at fixed strength, so regional control and strength generalization are not independently validated.

A.1 WORLD-MODEL ARCHITECTURE

  • The model takes a 256 × 256 image, degradation-state estimates for eight types, and one of 33 tool actions, then predicts degradation, quality, and fidelity changes, content damage, and benefit and harm outcomes.Its shared representation combines pooled visual features with degradation and tool embeddings and action attributes; separate heads predict the listed outcomes and uncertainty.Source: §A.1 · p. 15
  • The OOF experiments use global actions with fixed strength ω = 1, so regional control and strength generalization are not independently validated.Source: §A.1 · p. 15

A.2 SUPERVISION LABEL CONSTRUCTION

  • Training supervision comes from real before–after restoration interventions, with positive degradation-change values denoting improvement and the action represented by tool, compatible target, strength, and affected region.For global actions, the region outside the action is empty, so content damage is set to zero by convention.Source: §A.2 · p. 15
  • The proxy-quality target uses four RGB-based measures, while the separate fidelity target uses Y-channel PSNR and SSIM changes.Source: §A.2 · p. 15

A.3 WORLD-MODEL TRAINING OBJECTIVE

  • The training objective combines Gaussian negative log-likelihoods for continuous targets, binary cross-entropies for outcome labels, and within-state pairwise ranking losses for proxy utility and fidelity gain.The ranking utility combines target-degradation improvement, half the mean proxy-quality improvement, and minus twice content damage.Source: §A.3 · p. 16

B.1 PLANNING SCORE AND OPTIONAL TWO-STEP EXTENSION

The section specifies PreActIR’s planning score, a diagnostic for empirical harm thresholds, post-execution verification, and scene-aware candidate-value models. Its controlled ablation supports the one-step score, not a multi-step effectiveness claim.

B.1 PLANNING SCORE AND OPTIONAL TWO-STEP EXTENSION

  • The planning utility combines predicted target gain, proxy-quality and fidelity gains, side effects, content damage, uncertainty, and benefit probability; its cost prior is manually specified.The cost prior is only a weak ranking preference when predicted gains are similar.Source: §B.1 · pp. 16–17
  • 22.50 vs. 22.55 dB Y-PSNR: the two-step extension did not improve on the one-step score, while soft harm penalties raised harm to 18.8%.The authors therefore use the one-step score without penalties in Secs. 4.3–4.4.Source: §B.1 · p. 17

B.2 EMPIRICAL RISK-THRESHOLD DIAGNOSTIC

  • Empirical harm thresholds are diagnostic references, not distribution-free safety guarantees, and predicted harm probabilities should not be read as calibrated harm rates.The threshold procedure uses screening target ϵ = 0.25 and confidence parameter δ = 0.05.Source: §B.2 · pp. 17–18

B.3 POST-EXECUTION VERIFICATION

  • A separately trained verifier evaluates observed before–after images and the executed tool, then commits or rolls back the output before degradation is re-estimated.The verifier does not receive the degradation state or a reference image; its errors are partly correlated with the world model’s.Source: §B.3 · p. 18

C.1 SCENE-AWARE CANDIDATE VALUE MODELS

  • Scene-aware candidate-value models use deployment-visible features and identity-disjoint cross-fitting to predict candidate quality for final selection.The post-execution value model operates after an image is generated, unlike the world model’s pre-execution action predictions; ablation supports only one-step use.Source: §C.1 · pp. 18–19

D ADDITIONAL QUALITATIVE RESULTS

The appendix adds qualitative MiO100 examples and describes the open-loop ranking protocol. Methods score a shared candidate pool from pre-execution inputs, with outcomes evaluated using defined harm labels and identity-clustered intervals.

D ADDITIONAL QUALITATIVE RESULTS

  • Figure 8 adds MiO100 examples from Groups A–C in the same format as Figure 4, with degraded inputs and their degradation combinations in the first column.The passage identifies the figure’s organization but reports no comparative outcome.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 19, Figure 8 · p. 19

E OPEN-LOOP RANKING: PROTOCOL AND ADDITIONAL RESULTS

E.1 PROTOCOL

  • Deployable rankers score the same pool from pre-execution inputs and commit their top-ranked action; the Direct scorer predicts the same Y-channel fidelity target as the world model’s fidelity head.Intervals use clean-identity cluster-bootstrap draws, and rank correlation is computed within each state before averaging.Source: §E.1 · p. 19
  • A selection is labeled harmful when Y-PSNR, Y-SSIM, or mean RGB proxy-quality change crosses the stated negative thresholds.Severe harm has separate Y-PSNR and Y-SSIM thresholds.Source: §E.1 · p. 19

E.2 FULL RESULTS

The full results find better within-state ordering than Direct, but no corresponding improvement in top-1 choice. Additional diagnostics examine ranking depth, harm-probability calibration, and a secondary harm-penalty scan.

E.2 FULL RESULTS

  • Both world-model scores order candidates within a state better than Direct, but neither differs significantly from Direct in headroom recovery or harm.The improved rank correlations do not translate into a better top-1 choice.Source: §E.2 · p. 20
  • Removing fidelity gain v drops headroom recovery to −14.5%, while removing benefit probability pben raises harm from 42.5% to 46.2%.Removing the side-effect term slightly improves all top-1 metrics, but the frozen utility is reported unchanged.Source: §E.2 · p. 20

E.3 RANKING DEPTH, HARM CALIBRATION, AND SECONDARY SCAN

  • The smallest top-k reaching 90% of positive oracle headroom is 8 for the world model; its harm probability has ECE 0.106 and Brier 0.222.In the lowest-probability bins, observed harm is about 0.3, so probabilities are a ranking signal rather than calibrated risks.Source: §E.3 · p. 20, Figure 9 · p. 21
  • At matched harm rates of 5%, 10%, and 15%, WM-τ minus Direct-τ headroom-recovery differences are +0.0, +3.5, and +10.7 points, with all intervals including zero.The secondary WM-λ scan has higher headroom recovery above about 25% harm rates but does not reach the matched 5–15% rates.Source: §E.3 · p. 20

F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS

In controlled replay, PreActIR reduces calls and rejections relative to reactive trial and error while improving Y-PSNR. The additional results qualify these findings by budget, state inputs, and verifier behavior.

F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS

  • In paired PreActIR-minus-reactive comparisons, PreActIR used 1.26 fewer calls/image and 1.49 fewer rejected calls/image, improved Y-PSNR by 0.92 dB, and lowered no-commit rate by 23.1 points.The corresponding 95% intervals were −1.50 to −1.02 calls/image, −1.78 to −1.21 rejected calls/image, +0.49 to +1.34 dB Y-PSNR, and −30.6 to −15.6 points no-commit.Source: F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS · p. 21
  • At every budget, pre-execution ranking reaches a given Y-PSNR gain with fewer real calls than reactive trial and error and routing.For PreActIR, harm rises from 8.8% at B = 1 to 14.4% at B = 4 as no-commit falls from 31.2% to 15.6%.Source: F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS · p. 21, Figure 10 · p. 22
  • The larger closed-loop advantage may partly reflect accurate annotated states: estimated beliefs create an input shift because the world model is trained on annotated states.The replay uses an annotation-based pool, unlike the belief-based open-loop pool.Source: F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS · p. 21
  • With a ground-truth verifier at B = 4, routing needs 1.01 more calls per image and gains 1.58 dB less than PreActIR.The ordering advantage therefore is not an artifact of learned-verifier agreement, although the learned verifier catches fewer of PreActIR’s harmful choices than routing’s.Source: F CLOSED-LOOP REPLAY: ADDITIONAL RESULTS · p. 21, Figure 10 · p. 22

G.1 CANDIDATE-FRONTIER AUDIT

The candidate-frontier audit separates selector performance from fixed-pool limits using a post hoc, reference-assisted oracle; external replay adaptation then shifts the deployed Group C operating point beyond the controller trend without exceeding that oracle.

G.1 CANDIDATE-FRONTIER AUDIT

  • The audit exactly selects one frozen option per state under dataset-average constraints, using test references only to score options post hoc.It neither changes candidates or models nor serves as a deployable policy.Source: §G.1 · p. 22
  • The fixed-pool oracle jointly attains the pre-specified Group C targets, showing the pool is not the binding constraint in this audit.After raising the targets, the constrained selection reaches 21.79 PSNR, 0.617 SSIM, 0.325 MANIQA, 0.460 CLIP-IQA, and 54.58 MUSIQ; controller interpolation gives 20.05/0.571 Y-PSNR/Y-SSIM.Source: §G.1 · p. 23

G.2 EXTERNAL REPLAY CONSTRUCTION

  • Replay training uses 4,000 pairs from external images degraded with official AgenticIR operators, including 254 paired hazy images with added motion blur and ×4 degradation.No MiO100 HQ image is used for adaptation.Source: §G.2 · p. 23

G.3 REPLAY-ADAPTED SUPER-RESOLUTION TOOL

  • HAT-S is adapted for 4,000 iterations, improving fixed external-validation Y-PSNR/Y-SSIM from 24.84/0.7432 to 26.85/0.7692.The frozen adapted tool is used only on the 200 Group C states containing low resolution.Source: §G.3 · p. 23

G.4 FINAL GROUP C CANDIDATE FUSION

  • For Group C fusion, the adapted super-resolution candidate is combined with the original high-perceptual-quality candidate using Gaussian smoothing, with α = 0.62, κ = 0.12, and σ = 1.5.Other states retain the original high-perceptual-quality candidates.Source: §G.4 · p. 23, §G.3 · p. 23
  • The adapted candidate moves the deployed Group C operating point beyond the F1–F4 trend, but does not exceed the reference-assisted fixed-pool oracle.This demonstrates a practical intervention, not a necessary capacity expansion.Source: §G.4 · p. 23

Related papers

Loading 2610.20384v1…