PaperlayerView exact revision on arXiv

Source-linked AI summary

PhotoAgent: Exploratory Visual Aesthetic Planning with Large Vision Models

Mingde Yao, Zhiyuan You, King-Man Tam, Menglu Wang, Tianfan Xue

arXiv:2602.22809v3cs.CV

TL;DR

Instruction-based image editing faces a critical bottleneck. PhotoAgent integrates visual aesthetic planning, tree-search exploration, closed-loop execution, memory, and visual feedback for autonomous multi-step editing. PhotoAgent achieves state-of-the-art results across quantitative metrics, qualitative assessment, and user studies, improving editing quality over baseline methods.

  • Problem

    Instruction-based image editing faces a critical bottleneck.

  • Method

    PhotoAgent integrates visual aesthetic planning, tree-search exploration, closed-loop execution, memory, and visual feedback for autonomous multi-step editing.

  • Results

    PhotoAgent achieves state-of-the-art results across quantitative metrics, qualitative assessment, and user studies, improving editing quality over baseline methods.

  • Takeaways & Limitations

    PhotoAgent provides an autonomous editing framework that produces semantically coherent and aesthetically consistent enhancements without requiring expert-level operation.

Abstract

from arXiv · show

With the recent fast development of generative models, instruction-based image editing has shown great potential in generating high-quality images. However, the quality of editing highly depends on carefully designed instructions, placing the burden of task decomposition and sequencing entirely on the user. To achieve autonomous image editing, we present PhotoAgent, a system that advances image editing through explicit aesthetic planning. Specifically, PhotoAgent formulates autonomous image editing as a long-horizon decision-making problem. It reasons over user aesthetic intent, plans multi-step editing actions via tree search, and iteratively refines results through closed-loop execution with memory and visual feedback, without requiring step-by-step user prompts. To support reliable evaluation in real-world scenarios, we introduce UGC-Edit, an aesthetic evaluation benchmark consisting of 7,000 photos and a learned aesthetic reward model. We also construct a test set containing 1,017 photos to systematically assess autonomous photo editing performance. Extensive experiments demonstrate that PhotoAgent consistently improves both instruction adherence and visual quality compared with baseline methods. The project page is https://mdyao.github.io/PhotoAgent/.

1. Introduction

PhotoAgent addresses the difficulty of autonomous photo editing by combining aesthetic planning with closed-loop execution, reducing reliance on precise sequential user instructions. It supports semantically meaningful edits and introduces evaluation resources for real-world assessment.

  • Motivation: PhotoAgent reframes photo editing as autonomous, long-horizon decision-making that reduces reliance on expert-level sequential instructions.The system integrates large vision and multimodal models with editing tools in a coherent framework.
  • Approach: Its closed-loop framework explores editing actions, executes selected edits, evaluates results, and revises decisions using visual feedback.This design is intended to avoid short-sighted decisions and irreversible mistakes.
  • Capabilities: PhotoAgent performs high-level, semantically meaningful edits beyond low-level color, contrast, and illumination adjustments.Examples include adding a sun to a dim sky, making a scene more vibrant, and modifying scene objects.
  • Architecture: The system combines a VLM-based perceiver, MCTS planner, tool-based executor, and evaluator to generate and select multi-step editing actions.The planner explores possible editing trajectories, while the evaluator allows positively contributing actions to pass.
  • Evaluation resources: UGC-Edit contains 7,000 real user photos with human aesthetic scores, and a separate test set contains 1,017 photographs for evaluation.The accompanying reward model supports aesthetic evaluation for multi-step editing.
  • Results: Experiments report state-of-the-art performance across quantitative metrics, qualitative assessment, and user studies, with significant improvements in editing quality.The complete system is presented as improving both instruction adherence and visual quality relative to baseline methods.

2. Related Work

Prior image-editing systems range from narrow task-specific models to tool-using agents, but they generally lack robust planning for complex autonomous multi-step editing. PhotoAgent combines search-based planning with user-preference-based evaluation to address this gap.

  • Image Editing: Early GAN and encoder-decoder approaches support style transfer and attribute manipulation but remain confined to narrow training distributions.Cycle-GAN enables unpaired image-to-image translation, while StarGAN [10] supports multi-attribute manipulation.
  • Image Editing: Diffusion-based instruction editing broadens natural-language control, yet existing methods do not adequately support complex autonomous multi-step editing.These limitations motivate a higher-level planning-based framework.
  • Planning with Autonomous Agents: Autonomous-agent research uses planning methods such as MCTS and language-model reasoning to decompose tasks into action sequences.Examples include AlphaGo, HuggingGPT [35], ReAct [46], and Voyager.
  • Planning with Autonomous Agents: Computer-vision agents such as JarvisArt [23], MonetGPT [14], and PhotoArtAgent [9] mainly apply low-level retouching through procedural tools.Their focus includes color, tone, and exposure adjustments using software such as Lightroom or GIMP.
  • PhotoAgent: PhotoAgent applies MCTS to instruction-based image editing with internal simulation, external execution, and a learned user-preference reward model.This combination supports planning with a diverse toolset and an editing-specific benchmark.
  • Image Evaluation: Generic full-reference and no-reference metrics provide insufficient guidance for open-world, fine-grained aesthetic editing of user-generated photos.PSNR and SSIM require ground-truth targets, while broader metrics such as FID [18] and CLIP-based scores [31] may miss subtle task-specific quality differences.

3. PhotoAgent

PhotoAgent implements autonomous editing as a closed-loop system in which a perceiver proposes actions, an MCTS planner explores trajectories, tools execute selected edits, and an evaluator accepts or rejects outcomes. Scene-aware prompting, top-K execution, and feedback-based state updates support iterative refinement.

  • System Overview: The architecture comprises a perceiver, MCTS-based planner, executor, evaluator, tool-selection module, and memory module.Together these components support multi-step editing through a structured closed-loop framework.
  • Perceiver: The perceiver uses a VLM to generate diverse atomic actions conditioned on the current visual scene and aesthetic attributes.It supports autonomous editing, user-guided intent, and scene-aware strategies for different image types.
  • Planner: MCTS explores candidate editing trajectories through selection, expansion, simulation, and backpropagation before choosing actions for execution.The planner balances exploration and exploitation and evaluates long-term consequences to avoid myopic decisions.
  • Executor: The executor runs top-K actions using traditional operators or generative models, then retains the highest-scoring result as the next state.Selecting multiple actions improves robustness against simulation inaccuracies.
  • Evaluator: The evaluator combines no-reference metrics, instruction-based assessments, and perceptual models to score candidate outputs.If the best candidate does not improve over the previous state, the system reverts and terminates when progress stops or the step limit is reached.

4. Evaluation for Editing Systems

PhotoAgent introduces an evaluation framework tailored to real user-generated photographs, combining a UGC-specific dataset, learned aesthetic reward model, and real-world editing benchmark. The framework supplies fine-grained feedback for autonomous editing and supports end-to-end comparisons.

  • Evaluation Framework: The evaluation framework combines a UGC-specific preference dataset, learned aesthetic reward model, and real-world photo-editing benchmark.These components are designed to align aesthetic assessment with human preferences and guide multi-step decisions.
  • UGC-Edit Dataset: UGC-Edit contains approximately 7,000 authentic user-generated photos sourced from LAION [33] and RealQA [20].A VLM performs image-type categorization before manual verification retains images with clear UGC characteristics.
  • UGC Reward Model: The UGC reward model is initialized from Qwen2.5-VL [3] and optimized with GRPO [34] to predict fine-grained aesthetic scores.Relative rankings within image groups improve robustness to annotation noise and capture subtle aesthetic cues.
  • Editing Benchmark: The final benchmark contains 1,017 real-world photographs spanning portraits, landscapes, urban scenes, food, objects, and low-light imagery.Each image is edited by multiple baseline methods for end-to-end evaluation.
  • Evaluation Protocol: Quantitative, qualitative, and user-study results are reported on the benchmark to assess real-world editing effectiveness.The benchmark therefore evaluates autonomous editing across multiple forms of evidence rather than a single metric.

5. Experiments

PhotoAgent is evaluated against non-agent and agent baselines using semantic-alignment, image-quality, qualitative, user-study, and ablation analyses. Across these evaluations, it produces more coherent edits, is favored by users, and benefits from its evaluator and long-horizon planning.

  • Quantitative Results: PhotoAgent achieves the best BRISQUE score and competitive ImageReward performance, while GPT-4o can produce perceptual gains alongside distorted, over-edited images.Table 1 reports the quantitative comparison across planning strategies and image-editing systems.
  • Qualitative Results: In qualitative comparisons, PhotoAgent improves color harmony, composition, and aesthetic expressiveness, whereas baseline outputs are often generic, incomplete, or less coherent.The comparisons use the vague instruction “make this image better” to reflect ambiguous real-world user intent.
  • User Study: PhotoAgent was favored by participants across 27 real-world scenarios, based on 540 votes assessing visual quality and willingness to share.The user study involved 20 participants and compared PhotoAgent with several baseline methods.
  • Ablation Studies: Removing the UGC evaluator significantly decreases aesthetic metrics such as Laion-Reward, indicating that the evaluator affects editing preferences.The UGC evaluator is trained on real user photos and aesthetic ratings rather than only generic aesthetic scores.
  • Ablation Studies: Reducing MCTS simulations to 10 or search depth to 1 causes suboptimal decisions and lower performance on multi-step edits, supporting strategic long-horizon planning.A search depth of 1 effectively implements greedy selection, while fewer simulations limit planning quality.
  • Comparison with Existing Editing Agents: PhotoAgent supports general photo editing by routing instruction types among heterogeneous executors and handling both low-level retouching and high-level semantic operations.Its scope includes exposure, color and tone adjustments, object addition or removal, composition changes, and background replacement.
  • Editing Process: Iterative editing first improves overall tone and then refines specific objects, while user-guided editing interprets abstract intents such as mood, atmosphere, or emotional tone.Early stopping uses a maximum iteration limit and a no-improvement criterion to avoid unnecessary edits on high-quality images.

6. Conclusion

PhotoAgent reframes photo editing as sequential decision-making through coordinated perception, planning, execution, and evaluation modules. The resulting closed-loop framework reduces reliance on precise instructions and improves semantic coherence and aesthetic consistency.

  • 6. Conclusion: PhotoAgent coordinates an LLM-based perceiver, MCTS exploration, tool-based execution, and VLM evaluation in a closed-loop framework supported by UGC-Edit.The paper emphasizes that the contribution lies in coordinating modules rather than changing the underlying editing models.
  • 6. Conclusion: Experiments show that PhotoAgent outperforms existing methods in producing semantically coherent and aesthetically consistent enhancements.

A. Sim-to-Real Gap in Low-Resolution Planner Simulation

PhotoAgent addresses the gap between reduced-resolution MCTS simulation and full-resolution execution through reward consistency checks, high-fidelity rescoring, and replanning after every edit.

  • A. Sim-to-Real Gap in Low-Resolution Planner Simulation: At one-quarter resolution, top-ranked decisions are largely preserved and reward-rank correlations with full-resolution evaluations remain high.Table 3 reports the alignment between simulated and real-environment reward rankings.
  • A. Sim-to-Real Gap in Low-Resolution Planner Simulation: PhotoAgent forwards the top-K actions from MCTS to full-resolution evaluation and selects the final action using those high-fidelity scores.This limits the effect of occasional low-resolution reward-estimation errors on executed decisions.
  • A. Sim-to-Real Gap in Low-Resolution Planner Simulation: After each full-resolution edit, MCTS restarts from the updated image so simulation–execution discrepancies do not accumulate across steps.The system applies only one action at a time, keeping each decision grounded in the real environment.

B. Generalization of UGC Reward Model

The UGC reward model generalizes beyond its training dataset to PARA, correlating more strongly with human aesthetic judgments than prior PIAA models.

  • B. Generalization of UGC Reward Model: SRCC scores around 0.75 on PARA surpass prior PIAA models, which attain roughly 0.70–0.72.PARA contains varied content, styles, and lighting conditions with human-annotated aesthetic scores.

C. Experimental details

PhotoAgent uses MCTS to explore multi-step editing trajectories, evaluates simulated outcomes, and selects actions through iterative search. Its visual comparisons show stronger preservation, atmosphere, texture, and aesthetic quality than the compared methods, while some cases still involve excessive edits.

  • MCTS planning: PhotoAgent’s MCTS planner uses selection, expansion, simulation, and backpropagation to explore editing trajectories and choose the most visited action.The planner uses UCT for exploration–exploitation, evaluates simulated terminal states, and updates visit counts and rewards before selecting the root action.
  • Evaluation: The evaluator combines semantic alignment, aesthetic quality, ImageReward, and UGC-specific assessment into a weighted overall score.CLIP receives weight 1.0, aesthetic and ImageReward models receive 2.0 each, and the UGC model receives 0.8, normalized by a total weight of 5.8.
  • Visual results: Against nano-banana2 and Grok-Imagine, PhotoAgent avoids unnecessary changes on already appealing images, whereas the baselines may alter subjects or over-enhance scenes.The comparison cites excessive detail and subject changes for nano-banana2 and exaggerated flares for Grok-Imagine.
  • Visual results: PhotoAgent preserves original content and creative intent while enhancing artistic mood, atmosphere, texture, and material quality in visual comparisons.The reported comparisons describe richer texture and material quality and state that PhotoAgent surpasses the other compared methods.

E. Dataset Diversity and Fairness

UGC-Edit and the end-to-end test set draw on diverse real-world photographs from multiple sources and cultural contexts. Their coverage spans varied locations, photographic genres, and capture conditions.

  • UGC-Edit construction: UGC-Edit combines LAION and RealQA photos, covering English- and Chinese-dominant sources and varied real-world locations and cultural contexts.Examples include tourist attractions, restaurants, hotels, leisure venues, and other user-active locations.
  • Test-set construction: The end-to-end evaluation uses 1,017 real user-captured images collected from Lofter, Flickr, consumer cameras, smartphones, public websites, and filtered LAION content.The sources are selected to provide authentic user-generated photographs for system evaluation.
  • Diversity coverage: The test set covers portraits, landscapes, nature, urban and architectural scenes, still life, food, night scenes, and casual snapshots under varied capture conditions.Images vary in lighting, camera settings, and composition.

F. Computational Cost

PhotoAgent’s main computational cost comes from MCTS simulations and in-loop execution, but reducing the search budget or replacing the editing backend can substantially lower runtime. The system also has image-dependent failure cases, especially for dark, low-quality, or already polished inputs.

  • Runtime profile: Most default-configuration latency arises from the MCTS planner, where simulation and in-loop execution dominate the cost.The evaluator contributes a smaller share of runtime.
  • Runtime profile: Under a 10-simulation configuration, total processing takes about 100s, with perceiver, executor, planner, and evaluator times of 10s, 60s, 20s, and 30s.Extremely simple images may require fewer simulations or no simulation at all.
  • MCTS search budget: Reducing MCTS simulations from 20 to 5 cuts runtime from roughly 250s to 60s while maintaining comparable BRISQUE, LAION-Reward, and UGC human scores.This indicates that the default configuration prioritizes quality over speed and that faster operating points are available without architectural changes.
  • Editing backend: Replacing Flux.1 Kontext-Dev with Step1x-Edit at 1080p reduces editing time from about 20s to about 10s through a one-line API change.PhotoAgent is tool-agnostic, so faster editing models can improve execution time without changing the framework.
  • Failure cases: PhotoAgent can produce unsatisfactory edits for dark or low-quality images and may add little value or refuse to edit already high-quality images.Some technically reasonable modifications also fail to match user expectations.
Loading 2602.22809v3…