Source-linked AI summary
From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing
Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee
TL;DR
Abstract, long-horizon image-editing tasks require coordinated planning and tool execution, while prior agent systems rely on handcrafted pipelines or teacher imitation and do not learn directly from editing outcomes. The paper introduces an experiential framework with checklist-guided planning, reward-driven orchestration, and refinement from judged edits. It reports more reliable, coherent, and instruction-faithful results than single-step and multi-step agent baselines.
Problem
Open-ended editing requires coordinated multi-step changes, but prior systems rely on fixed pipelines or teacher imitation and do not optimize tool selection from actual editing outcomes.
Method
A checklist-guided planner generates atomic subtasks, while an orchestrator selects tools and regions using outcome-based VLM rewards and feedback-driven refinement.
Results
The framework produces more reliable, coherent, and instruction-faithful results than single-step generation approaches and multi-step agent baselines.
Takeaways & Limitations
Coupling planning with experiential orchestration enables long-horizon, open-ended image editing beyond single-step generation.
Takeaways & Limitations
Training is constrained by the lack of large-scale abstract multi-step plan data, ambiguous tool selection, multiple valid outputs, and expensive tool invocation.
Abstract
from arXiv · showhide
Modern image editing models produce realistic results but struggle with abstract, multi step instructions (e.g., ``make this advertisement more vegetarian-friendly''). Prior agent based methods decompose such tasks but rely on handcrafted pipelines or teacher imitation, limiting flexibility and decoupling learning from actual editing outcomes. We propose an experiential framework for long-horizon image editing, where a planner generates structured atomic decompositions and an orchestrator selects tools and regions to execute each step. A vision language judge provides outcome-based rewards for instruction adherence and visual quality. The orchestrator is trained to maximize these rewards, and successful trajectories are used to refine the planner. By tightly coupling planning with reward driven execution, our approach yields more coherent and reliable edits than single-step or rule-based multistep baselines.
1 Introduction
Open-ended image editing requires coordinated changes across imagery, text, context, and layout, exceeding the capabilities of single-step editors. The framework addresses this with checklist-guided planning, outcome-driven orchestration, and refinement, achieving more reliable results than prior approaches.
- Motivation: Abstract, long-horizon edits require coordinated changes to imagery, slogans, audience messaging, and environmental context.Different subtasks may require different tools, such as object replacement or text modification.
- Limitations of prior work: Prior agent systems often rely on handcrafted pipelines or teacher imitation, fixing execution order and heuristics rather than learning from editing outcomes.These designs can produce distribution shift, limited generalization, and poor scalability for open-ended instructions.
- Motivation: Single-step editors struggle to jointly adapt visual themes, textual content, and audience-specific context while preserving advertisement layout and identity.The rural-audience example illustrates this limitation.
- Framework: The planner generates checklist-guided atomic subtasks, while the orchestrator selects tools and regions and receives VLM-judged rewards for adherence, identity preservation, and visual quality.A refinement stage prunes infeasible subtasks to align plans with executable actions.
- Results: The framework produces more reliable, coherent, and instruction-faithful results than single-step generation approaches and multi-step agent baselines.The authors describe this as state-of-the-art performance for open-ended image editing.
- Framework: The planner learns from its own sampled plans, reducing distribution shift relative to teacher imitation, while the orchestrator learns from judged edits rather than prompts or teacher traces.The design aligns training with inference and removes handcrafted rules.
2 Related Work
Prior image-editing and multimodal systems improve controllability or decompose complex requests, but many remain localized, fixed, training-free, or teacher-imitation based. This framework instead combines checklist-guided planning with experiential orchestration learned from judged editing outcomes.
- Controllable Image Editing with Diffusion Models: Diffusion-based editing methods improve fidelity and controllability, but training-free approaches are typically limited to localized changes and may over-edit or under-follow instructions.Training-based methods and added masks, boxes, or drag inputs improve robustness or spatial precision.
- Multimodal LLMs for Image Editing and Planning: Multimodal systems use language or vision-language models to decompose complex visual tasks into tool-executable subproblems.Some generate code for specialized modules, while others rewrite instructions before calling an editor.
- Multimodal LLMs for Image Editing and Planning: Many prior editing agents are training-free or imitate teacher plans, without training planners on their own distributions or policy-optimizing tool selection.The contrast motivates learning tool and region selection directly from judged editing outcomes.
- Experiential Learning for Long-Horizon Reasoning: Experiential-learning research improves long-horizon reasoning through decomposition, iterative refinement, and robustness, including in multimodal reasoning.This paper applies the idea to joint high-level planning and low-level image-tool execution.
3 Approach
The approach separates open-ended image editing into checklist-guided planning and reward-driven orchestration. A planner generates ordered atomic subtasks, while an orchestrator selects tools and regions and learns from judged editing outcomes.
- The framework learns long-horizon image editing through interaction with editing tools and feedback from a learned judge.
- Planner via Checklist-Guided Self-Training: A multimodal planner converts a high-level instruction into an ordered sequence of structured subtasks for modular, interpretable editing.
- Planner via Checklist-Guided Self-Training: Checklist-guided prompting enforces coverage and provides modular, human-interpretable supervision without requiring gold-standard plans.
- Planner via Checklist-Guided Self-Training: The planner is fine-tuned on checklist-guided plans sampled from its own generation distribution through autoregressive likelihood maximization.
- Reward-Driven Tool Selection: The orchestrator selects a tool and region for each subtask, sequentially composing edits into a final image.
- Reward-Driven Tool Selection: A VLM judge scores instruction adherence, identity preservation, and visual quality, while high-reward trajectories supervise orchestrator training toward successful tool-region decisions.
4 Experiments
Experiments evaluate the framework against single-step editors, multi-step orchestration variants, and alternative plan supervision. The results show stronger instruction execution, coherent long-horizon edits, and benefits from trained orchestration, multiple inference branches, and checklist-guided self-supervision.
- 4.1 Main Results: Comparison to End-to-End Editing Baselines: The framework is evaluated against FLUX.1-Kontext-dev and Qwen-Image-Edit-2511 using direct high-level instructions and decomposed multi-step inputs.The evaluation tests both single-pass reasoning and execution of plans generated by a general multimodal model.
- 4.1 Main Results: Comparison to End-to-End Editing Baselines: Our method achieves the highest instruction-following score while maintaining strong identity preservation and visual quality.FLUX.1-Kontext-dev has higher identity preservation and visual quality in one setting largely because it often leaves the image nearly unchanged.
- 4.1 Main Results: Comparison to End-to-End Editing Baselines: Randomized A/B user studies corroborate the judged results, with participants consistently preferring our approach.Each result pair was evaluated by three unique users while considering instruction following, identity preservation, and visual quality.
- 4.2 Ablation Study: Orchestrator Design Choices: The trained orchestrator significantly outperforms single-tool baselines and the untrained orchestrator, showing the value of learned tool selection through experience.The ablation fixes the planner output across variants and compares sequential editors, tool combinations, and orchestrator configurations.
- 4.2 Ablation Study: Orchestrator Design Choices: Instruction satisfaction rises from 63.9% with 1 branch to 71.8% with 3 branches and 74.0% with 5 branches, with corresponding visual-quality gains.The multi-branch variants achieve the highest instruction satisfaction and visual quality among the compared configurations.
- 4.3 Ablation Study: Plan Dataset: Self- vs. External Supervision: Teacher-generated plans have substantially higher perplexity than checklist-guided self-generated plans, while checklist self-distillation keeps supervision closer to the model’s native distribution.The comparison provides evidence of potential instability under off-policy imitation and supports on-policy supervision.
- 4.4 Ablation Study: Effect of Plan Refinement: Filtering infeasible subtasks increases average maximum reward, improving compatibility between plans and executable tools.The planner is retrained on revised plans after subtasks whose maximum achievable reward falls below threshold τ are removed.
- 4.5 Qualitative Results: On challenging advertisement transformations, the method produces globally coherent, instruction-faithful edits that jointly update themes, text, layout, and branding.Single-step editors often make partial stylistic changes or shallow modifications, whereas the proposed method handles coupled long-horizon transformations.
5 Conclusion
The paper presents experiential planning and orchestration for long-horizon open-ended image editing. Experiments and user studies report more coherent and instruction-faithful edits than single-step and rule-based multi-step baselines.
- 5 Conclusion: The framework combines checklist-guided on-policy planning with reward-driven orchestration to move beyond handcrafted pipelines and single-step generation.The planner generates structured decompositions, while the orchestrator selects tools and regions using outcome-based feedback and closed-loop refinement.
- 5 Conclusion: The planner learns structured decompositions, and the orchestrator selects tools and regions from outcome-based feedback to align plans with executable actions.This couples planning and execution through closed-loop refinement.
- 5 Conclusion: Extensive experiments and user studies show more coherent, instruction-faithful edits than strong single-step and rule-based multi-step baselines.The conclusion highlights this result for abstract, multi-step editing tasks.
A Qualitative Results
Step-wise visualizations show the system composing long sequences of heterogeneous edits while preserving visual coherence and consistency with the original content.
- A Qualitative Results: The visualizations include text changes, background modifications, and object-level alterations across long sequences of diverse edits.These examples illustrate the system’s ability to execute multiple heterogeneous editing operations.
- A Qualitative Results: The composed edits maintain visual coherence and consistency with the original content.The qualitative examples demonstrate reliability across the combined editing sequence.
B.1 Results on Image Editing Benchmarks
Across MagicBrush and GEdit, the method generalizes beyond advertisement editing and achieves strong instruction alignment while preserving visual quality. Its evaluation also highlights that single-reference similarity metrics can undervalue valid open-ended edits.
- Benchmark Setup: The orchestrator is evaluated on multi-turn MagicBrush and single-turn GEdit benchmarks alongside agentic and instruction-based editing models.MagicBrush uses direct edit sequences and evaluates the learned orchestrator without the planner.
- Qualitative Results: The method produces instruction-faithful edits while preserving scene structure, even when outputs differ from MagicBrush’s single ground-truth image.Examples include applying strawberry glaze, replacing an alcohol shelf, removing sunglasses, and adding specified objects.
- Metric Interpretation: Single-ground-truth similarity metrics may penalize visually valid edits that differ in appearance, motivating emphasis on textual alignment through CLIP-T.The benchmark’s open-ended instructions can have multiple visually valid solutions.
- MagicBrush: The method achieves the best MagicBrush CLIP-T performance among the listed agentic and instruction-based baselines.The comparison includes GenArtist, LayerCraft, Talk2Image, InstructPix2Pix, HIVE, and MagicBrush.
- GEdit: The method achieves the best GEdit performance across semantic consistency, perceptual quality, and the overall score.GEdit’s overall score is the geometric mean of SC and PQ, using GPT-4.1 evaluation.
B.2 Are Checklist-Based Plans Better?
The paper evaluates whether checklist-based supervision improves plan quality over direct plan generation by a prompted base model. A Gemini-3-Pro pairwise evaluation favors the checklist-trained planner.
- Comparison: The study compares checklist-supervised planner outputs with plans generated by a prompted base Qwen3-VL model.The base model is not required to satisfy explicit constraints.
- Evaluation: Plans are evaluated with Gemini-3-Pro using pairwise comparisons in both original and reversed orderings.The final preference score averages both evaluations to mitigate positional bias.
- Results: 70.25% of comparisons favor the checklist-based plan over the base-model plan.The authors state that this suggests more coherent and task-aware editing strategies.
B.3 GenArtist Performance on Advertisement Editing
The paper compares its advertisement-editing framework with GenArtist because other recent agentic approaches lack publicly available implementations. The reported comparison favors the proposed method on instruction following, identity preservation, and visual quality.
- Comparison Scope: The comparison focuses on GenArtist as a publicly implementable agentic baseline for the MadVerse advertisement-editing benchmark.The paper states that X-Planner and MIRA are excluded because implementations are unavailable.
- Results: The proposed method maintains substantially stronger identity preservation and overall visual quality than GenArtist while performing the requested edits.GenArtist often introduces excessive degradation to the original image, producing low scores across all evaluated metrics.
C Implementation Details
The implementation separates long-horizon editing into a Planner, editing tools, and an Orchestrator. The system evaluates instruction following, identity preservation, and visual quality during inference and execution.
- System Components: The Planner decomposes high-level editing requests into sequential atomic operations.This defines the structured plan consumed during execution.
- System Components: The Orchestrator selects the appropriate editing tool and spatial region for each planned operation.The tool set performs the underlying image transformations.
- Evaluation: The evaluation framework assesses instruction following, identity preservation, and visual quality.The paper also describes construction, training, inference, and the inference algorithm for each component.
C.1 Editing Tools
The system uses complementary whole-image, analysis, and region-level tools to support both global transformations and precise localized edits. An orchestrator selects tools and regions, while region discovery mechanisms provide editable masks or boxes for targeted inpainting.
- Toolset: Multiple specialized tools are used because no single model reliably supports global, object-level, text, and structural image transformations.The modular toolset includes analysis tools, whole-image editors, and region-level editors.
- Whole-image editing: Whole-image editors FLUX.1-Kontext-dev and Qwen-Image-Edit-2511 apply instruction-guided changes while preserving the image’s overall structure.Because they condition edits on the full image, they may unintentionally affect unrelated regions.
- Analysis tools: Analysis tools discover editable regions through semantic segmentation, text and layout detection, layer decomposition, or instruction-guided bounding boxes.SAM-2 plus Qwen3-VL produces indexed masks and descriptions; DeepSeek-OCR detects textual and layout regions; Qwen-Layered produces structural layers; Qwen-BBox predicts task-specific regions.
- Instruction-guided region discovery: Qwen-BBox conditions region prediction on the editing instruction and uses grid-annotated images to help localize regions that may not correspond to existing semantic entities.It predicts three candidate boxes that can be selected individually or combined.
- Region-level editing: For region-level editing, the orchestrator selects a candidate mask, converts boxes to binary masks, and applies FLUX-Kontext Inpaint within the selected area.The mask is dilated by 100 pixels to permit adjustments to the target’s size, shape, or surrounding context while leaving outside pixels unchanged.
- Orchestration: At each step, the orchestrator emits a structured tool call containing the selected tool and its arguments, choosing either analysis or direct whole-image editing.Tool calls can be executed directly without requiring intermediate code generation.
C.3 Evaluation
The evaluation framework measures whether edits satisfy the task, preserve identity, and maintain visual quality. It combines rubric-based judge scores with dense plan-derived checklists that quantify final-state constraint satisfaction.
- Evaluation axes: Evaluation covers Instruction Execution, Identity Preservation, and Visual Quality using the original image, edited image, and task instruction.The three axes capture requested changes, preservation of non-target content, and the quality of the final image.
- Evaluation protocol: Gemini-3-Pro is used as the evaluation judge after training with GPT-based rewards to reduce potential evaluator bias.The paper reports evaluation results in the main experiments and ablations, including comparisons under a common high-level instruction.
- Evaluation axes: Instruction Execution assesses both whether the intended changes were identified and whether those changes were correctly carried out.The judge therefore evaluates knowledge of what should change alongside execution success.
- Plan-conditioned evaluation: Plan-conditioned evaluation uses a dense checklist to test whether the final edit satisfies the specific modifications, preserved elements, and relationships required by the plan.The checklist is generated from the input image, high-level instruction, and multi-step plan.
- Checklist scoring: Each checklist item is marked satisfied or unsatisfied, and an image score is the fraction of satisfied items averaged across the dataset.Checklist entries use Preserve, Remove, Replace, Add, and Constraint categories with explicit verification rules.
- Checklist construction: Checklist generation produces atomic final-state constraints describing what must be preserved, removed, replaced, added, or spatially constrained.The specification excludes unstated aesthetic or realism requirements and allows multiple valid realizations when appropriate.