PaperlayerView exact revision on arXiv

Source-linked AI summary

LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents

Bingchen Li, Xin Li, Yiting Lu, Zhibo Chen

arXiv:2412.04090v2cs.CV

TL;DR

Low-level image-processing models struggle to optimize customized objectives such as non-differentiable perceptual metrics, textual descriptions, and human feedback. LossAgent uses an LLM to interpret optimization feedback and adjust weights in a repository of differentiable losses, with experiments demonstrating flexibility and scalability across three tasks and multiple objectives.

  • Problem

    Customized perceptual, textual, and human-feedback objectives are often non-differentiable, limiting their direct use for end-to-end low-level image-processing optimization.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    LossAgent uses an LLM, historical optimization information, external feedback, and a compositional loss repository to adapt loss weights during training.

    Source: §1 · p. 1, §3.2 · p. 3, §3.4 · p. 5

  • Results

    Experiments across three representative low-level image-processing tasks and multiple customized objectives demonstrate LossAgent’s flexibility and scalability.

    Source: §5 · p. 8

  • Takeaways & Limitations

    LossAgent enables end-to-end optimization toward customized objectives by converting external objective feedback into adjustments of differentiable loss weights.

    Source: §1 · p. 2

    High-resolution testing images can produce insufficiently discriminative IQA feedback, causing unreasonable loss-weight adjustments.

    Source: §8.4 · p. 13

Abstract

from arXiv · show

We present the first loss agent, dubbed LossAgent, for low-level image processing tasks, e.g., image super-resolution and restoration, intending to achieve any customized optimization objectives of low-level image processing in different practical applications. Notably, not all optimization objectives, such as complex hand-crafted perceptual metrics, text description, and intricate human feedback, can be instantiated with existing low-level losses, e.g., MSE loss, which presents a crucial challenge in optimizing image processing networks in an end-to-end manner. To eliminate this, our LossAgent introduces the powerful large language model (LLM) as the loss agent, where the rich textual understanding of prior knowledge empowers the loss agent with the potential to understand complex optimization objectives, trajectory, and state feedback from external environments in the optimization process of the low-level image processing networks. In particular, we establish the loss repository by incorporating existing loss functions that support the end-to-end optimization for low-level image processing. Then, we design the optimization-oriented prompt engineering for the loss agent to actively and intelligently decide the compositional weights for each loss in the repository at each optimization interaction, thereby achieving the required optimization trajectory for any customized optimization objectives. Extensive experiments on three typical low-level image processing tasks and multiple optimization objectives have shown the effectiveness and applicability of our proposed LossAgent.

1. Introduction

LossAgent addresses the difficulty of optimizing low-level image-processing models for customized objectives that may be non-differentiable. It uses an LLM to interpret feedback and historical optimization information, then adaptively compose existing losses and update their weights.

Motivation

  • LossAgent targets customized optimization objectives, including complex perceptual metrics, textual descriptions, and human feedback that cannot directly support end-to-end optimization.The approach addresses the limitation that conventional loss combinations require differentiable objective-specific losses.Source: §1 · p. 1

Method

  • The method combines a repository of existing low-level losses with an LLM that adaptively assigns their weights during training.The weights are updated according to external feedback and the current optimization process.Source: §1 · pp. 1–2

Optimization loop

  • Historical model information and external feedback allow the loss agent to reason about optimization states and guide subsequent loss-weight decisions.The agent receives model weights from the beginning of training through the current stage.Source: §1 · p. 2, Figure 1 · p. 2

Capabilities

  • LossAgent supports automatic loss-weight updates and instruction-following for customized needs during training.Its stated flexibility includes receiving feedback from external environments rather than relying only on fixed objectives.Source: §1 · p. 2

Scope

  • The approach is presented as scalable across multiple low-level image-processing tasks and optimization objectives, including non-differentiable objectives.The introduction identifies this scalability as a core feature supported by the experimental evaluation.Source: §1 · p. 2

2. Related Works

Prior image-processing methods improve fidelity or perceptual quality through fixed losses and evolving architectures, while many advanced IQA objectives remain non-differentiable. LossAgent instead uses an LLM agent to convert external feedback into loss-weight adjustments for more open-ended optimization.

2.1. Image Processing

  • Conventional image-processing methods optimize fidelity or perceptual metrics, but fixed optimization trajectories and non-differentiable IQA metrics limit objective flexibility.Existing approaches include L1, MSE, perceptual, and GAN-based losses, while many advanced IQA metrics cannot be directly optimized.Source: §2.1 · pp. 2–3
  • LossAgent bridges customized objectives and end-to-end training by translating feedback into weighted combinations of differentiable loss functions.This distinguishes the method from directly using non-differentiable objectives as training losses.Source: §2.1 · p. 3, §2.2 · p. 3

2.2. LLM Agents

  • LLM-agent research shows that language models can use environmental feedback and instructions to coordinate tools or generate task-specific actions.These capabilities motivate using an LLM as a decision-making component for image-processing optimization.Source: §2.2 · p. 3
  • Unlike prior LLM applications in image restoration and compression, LossAgent is designed to handle customized optimization objectives for image-processing models.It adjusts loss weights using feedback from external models or metrics.Source: §2.2 · p. 3

3. Methods

The method is motivated by the instability and non-differentiability of advanced image-quality objectives. LossAgent addresses this by using an LLM to adjust the weights of a compositional repository of differentiable losses across staged training.

3.1. Motivation

  • No-reference IQA metrics and textual feedback can be unstable or non-differentiable, preventing their direct use as training objectives for image-processing models.Such metrics may fail to guide structural learning effectively, while textual objectives are inherently non-differentiable.Source: §3.1 · p. 3
  • LossAgent transfers feedback from customized objectives into an actionable weighted composition of commonly used losses instead of applying those objectives directly.This design allows external objective feedback to influence end-to-end model optimization.Source: §3.1 · p. 3

3.2. Weighted Compositional Loss Repository

  • The compositional loss repository defines the final training loss as L = w1L1 + w2L2 + · · · + wMLM, combining M differentiable losses with adjustable coefficients.Examples include L1 and LPIPS losses.Source: §3.2 · p. 3
  • Training is divided into N stages so the loss agent can update the weighting composition in response to the model’s current state and optimization feedback.The repository provides the adjustable direction, while the agent generates weighting coefficients over time.Source: §3.2 · p. 3

External Feedback

LossAgent uses external evaluation feedback and an LLM-based agent to adapt compositional loss weights across optimization stages. Its prompt design combines system, historical, and customized instructions so the agent can generate new weights for subsequent image-processing updates.

3.4. Loss Agent

  • LossAgent analyzes external feedback, historical weights, and customized instructions to generate new loss weights for the next optimization stage.The LLM-based agent uses system, historical, and customized-needs prompts to relate loss weights to optimization objectives.Source: Figure 2 · p. 4, §3.4 · p. 5

External Feedback

  • The loss repository represents the model objective as a weighted composition of existing low-level loss functions.The agent later produces a new combination of these loss weights for optimization at the next stage.Source: §3.2 · p. 4, §3.4 · p. 5
  • At each training stage, feedback is generated on randomly selected testing images from the current image-processing state.The workflow evaluates images produced at the current stage before updating the optimization trajectory.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. External Feedback

  • External evaluation expert O scores restored images or supplies textual descriptions, providing feedback F that represents the optimization objective.For a higher CLIPIQA score, CLIPIQA is selected as the expert; broader quality objectives can use multiple experts collaboratively.Source: §3.3 · p. 4

3.4. Loss Agent

  • The system prompt specifies the agent’s role, inputs, outputs, and objectives, including how to interpret whether higher or lower scores indicate better quality.This objective clarification is presented as important because incorrect score-direction assumptions can produce incorrect reasoning.Source: §3.4 · p. 5
  • Historical prompts provide the optimization trajectory, while customized-needs prompts impose rules and format regularization on the agent’s responses.These components are intended to reduce hallucination and undesirable responses while standardizing outputs for parsing.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate LossAgent across three optimization settings, multiple low-level image-processing tasks, and prompt-design ablations. Results support adaptive loss weighting for single, combined, and textual objectives, while historical context and output constraints improve decisions.

4.1. Settings

  • The experiments cover classical and real-world super-resolution plus all-in-one restoration, using SwinIR and PromptIR with three loss functions in the repository.Evaluation includes single, double, and textual optimization objectives with Meta-Llama-3-8B-Instruct as the loss agent.Source: §4.1 · p. 5

4.2.1. Single Optimization Objective

  • Across single-objective experiments, LossAgent outperforms Random and Fixed across almost all benchmarks and objectives, including real-world image super-resolution.The comparison spans different image-processing models and supports generalization beyond one task.Source: §4.2.1 · p. 6, Table 1 · p. 6, Table 3 · p. 7

4.2.2. Double Optimization Objectives

  • LossAgent combines PSNR and Q-Align feedback to improve PSNR across all benchmarks while maintaining comparable or better Q-Align performance.The result demonstrates flexibility when objective and subjective metrics are not positively correlated.Source: §4.2.2 · p. 6, Table 4 · p. 7

4.2.3. Textual Optimization Objectives

  • Co-Instruct-optimized models achieve comparable Q-Align results to Q-Align-optimized models, transferring textual objectives into loss-weight adjustments.No competing method is available for direct evaluation of textual guidance, so Q-Align is used as the comparison metric.Source: §4.2.3 · p. 7, Table 5 · p. 8

4.3. Evaluation on Effectiveness of Prompt Design

  • System prompts must explain metric direction, historical prompts must provide full trajectories, and customized prompts must constrain output formatting for reliable loss-weight decisions.Removing score-direction descriptions harms NIQE optimization; full history performs best, and formatting examples yield only one failure among 800 samples.Source: §4.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8

5. Conclusion

The conclusion presents LossAgent as an LLM-based loss agent for customized low-level image-processing objectives. Experiments across three tasks and varied objectives demonstrate its flexibility and scalability.

5. Conclusion

  • LossAgent uses an LLM to interpret optimization objectives, trajectories, and external stage feedback for low-level image processing.Optimization-oriented prompts provide instructions and historical information to reduce hallucinations and incorrect reasoning.Source: §5 · p. 8
  • Experiments on three representative low-level image-processing tasks with varied customized objectives demonstrate LossAgent’s flexibility and scalability.Source: §5 · p. 8

6. Training Details

Training uses staged dynamic loss-weight updates initialized from pretrained image-processing models. The implementation fixes optimization settings across tasks while tailoring the loss repository to super-resolution and restoration.

6. Training Details

  • Pretrained SwinIR and PromptIR weights initialize training to reduce early fluctuations that could misguide LossAgent’s loss-weight updates.SwinIR starts the two super-resolution tasks, while PromptIR initializes all-in-one restoration.Source: §6 · p. 12
  • Super-resolution uses GAN-based loss components, whereas all-in-one restoration uses L1, perceptual, and LPIPS losses.Source: §6 · p. 12
  • All three tasks use a constant learning rate of 1e-4 and Adam optimization, with eight NVIDIA TESLA V100 GPUs.Total batch size is 32 for image super-resolution and 16 for all-in-one restoration.Source: §6 · p. 12

7. Additional Results for All-in-One Image Restoration

On all-in-one image restoration, LossAgent is less robust than on the other evaluated tasks but still improves performance against competing methods across four optimization objectives.

7. Additional Results for All-in-One Image Restoration

  • LossAgent improves performance against other methods on all-in-one image restoration across four optimization objectives, despite lower robustness than on the other tasks.The results indicate adaptability across different image-processing models.Source: §7 · p. 12
  • Minimal differences between consecutive-stage images limit feedback information and hinder thorough loss-weight analysis and inference.Source: §7 · p. 12

8. More Ablation Studies

The ablations show that LossAgent depends on suitable stage intervals, stable loss compositions, informative testing feedback, and compatible backbones, while adding moderate training overhead. Across these settings, it adapts loss weights to different objectives and produces improved qualitative or benchmark outcomes.

8.1. Iterations for Each Stage

  • 5000 training iterations per stage achieve the best results, whereas 2500 complicates historical-weight reasoning and 10000 leaves too few update steps for adjustment.The same selection principle is applied to the other two tasks.Source: §8.1 · p. 12, Table 10 · p. 13, Table 13 · p. 14

8.2. Objective as Loss Function vs. LossAgent

  • Directly using CLIPIQA or NIQE as training losses gives suboptimal performance, while LossAgent reaches these objectives through combinations of stable reference-based losses.Using CLIPIQA or NIQE alone can cause training collapse because they do not directly guide structural restoration.Source: §8.2 · p. 12, Table 11 · p. 13

8.3. M in Loss Repository

  • Expanding the repository from three to five losses yields slightly better results, indicating scalability despite the current LLM's limited reasoning ability.The expanded repository contains L1, MSE, Perceptual, GAN, and LPIPS losses.Source: §8.3 · p. 13

8.4. Testing Image Set

  • Set14 provides better CLIPIQA feedback than DIV2K because resizing high-resolution images can make them harder for IQA models to distinguish, causing unreliable weight updates.LossAgent therefore uses Set14 for classical image super-resolution and applies the same principle to the other tasks.Source: §8.4 · p. 13

8.5. The Illustration of Loss Weight Curves

  • LossAgent's loss-weight curves share coarse trends across four perception-oriented objectives but retain fine-grained differences reflecting objective-specific preferences.The curves illustrate adaptive weight updates on classical image super-resolution.Source: §8.5 · p. 13, Table 12 · p. 14, Table 13 · p. 14

8.6. Flexibility of LossAgent across Different Image Processing Models

  • With HAT-S and MambaIRv2-light backbones, LossAgent outperforms other methods on almost all benchmarks across four metrics.The comparison covers classical image super-resolution results for both backbones.Source: §8.6 · p. 14

8.7. Time Analysis

  • LossAgent adds approximately 15% average training time across three tasks, with no testing-time overhead because the agent is used only during training.The authors consider this overhead acceptable relative to the performance improvement.Source: §8.7 · p. 14, Table 14 · p. 14

8.8. More Qualitative Results

  • On real-world image super-resolution, LossAgent restores more vivid textures aligned with the optimization objectives in qualitative comparisons.The comparison is shown across four optimization objectives.Source: §8.8 · p. 14, Figure 4 · p. 14

9. Case Study

The case study shows how LossAgent uses historical loss weights and score feedback to reason about new weights for NIQE optimization. It also exposes the formatting constraint that converts the agent's textual output into numeric loss weights.

9. Case Study

  • LossAgent analyzes historical relationships between loss weights and feedback scores, then considers each loss function's role when updating weights.The case study uses NIQE optimization on classical image super-resolution at 25000 iterations.Source: §9 · pp. 14, 17, Figure 6 · p. 17
  • The agent's output must follow a structured L1:Perceptual:GAN weight pattern so a Python parser can convert it into numeric weights.Rule-based formatting constraints help LLaMA3 produce the required pattern.Source: §9 · p. 15
  • The case study's NIQE prompt treats lower score feedback as better image quality and supplies weight-score history from initialization through 20000 iterations.The historical sequence includes weights and scores at 5000-iteration intervals.Source: §9 · p. 17

Related papers

Loading 2412.04090v2…