Source-linked AI summary
Meta-CoT: Enhancing Granularity and Generalization in Image Editing
Shiyi Zhang, Yiji Cheng, Tiankai Hang, Zijin Yin, Runze He, Yu Xu, Wenxun Dai, Yunlong Lin, Chunyu Wang, Qinglin Lu, Yansong Tang
TL;DR
Existing CoT-based image editing lacks a clear strategy for jointly improving understanding granularity and generalization across diverse tasks. Meta-CoT decomposes editing operations into task, target, and understanding ability, further reduces tasks to meta-tasks, and adds a consistency reward, achieving a 15.7% overall improvement across 21 editing tasks.
Problem
Existing CoT-based image editing needs paradigms that both stimulate understanding ability and generalize across diverse editing tasks.
Method
Meta-CoT decomposes editing into task, target, and required understanding ability, further represents tasks with five meta-tasks, and rewards CoT–editing consistency.
Results
15.7% overall performance improvement is achieved across 21 distinct editing tasks after supervised fine-tuning and consistency-reward optimization.
Takeaways & Limitations
Training on as few as five meta-tasks can support generalization to other editing tasks through compositional reasoning.
Takeaways & Limitations
Reasoning hinders text editing, likely because extensive textual reasoning interferes with identifying the correct text to modify.
Abstract
from arXiv · showhide
Unified multi-modal understanding/generative models have shown improved image editing performance by incorporating fine-grained understanding into their Chain-of-Thought (CoT) process. However, a critical question remains underexplored: what forms of CoT and training strategy can jointly enhance both the understanding granularity and generalization? To address this, we propose Meta-CoT, a paradigm that performs a two-level decomposition of any single-image editing operation with two key properties: (1) Decomposability. We observe that any editing intention can be represented as a triplet - (task, target, required understanding ability). Inspired by this, Meta-CoT decomposes both the editing task and the target, generating task-specific CoT and traversing editing operations on all targets. This decomposition enhances the model's understanding granularity of editing operations and guides it to learn each element of the triplet during training, substantially improving the editing capability. (2) Generalizability. In the second decomposition level, we further break down editing tasks into five fundamental meta-tasks. We find that training on these five meta-tasks, together with the other two elements of the triplet, is sufficient to achieve strong generalization across diverse, unseen editing tasks. To further align the model's editing behavior with its CoT reasoning, we introduce the CoT-Editing Consistency Reward, which encourages more accurate and effective utilization of CoT information during editing. Experiments demonstrate that our method achieves an overall 15.8% improvement across 21 editing tasks, and generalizes effectively to unseen editing tasks when trained on only a small set of meta-tasks. Our code, benchmark, and model are released at https://shiyi-zh0408.github.io/projectpages/Meta-CoT/
1 Shenzhen International Graduate School, Tsinghua University 2 Hunyuan, Tencent
Meta-CoT uses hierarchical triplet decomposition to support fine-grained reasoning over editing tasks and targets, alongside a CoT–Editing Consistency Reward. On the 21-task Editing Benchmark and ImgEdit, it improves instruction-based editing and supports generalization to unseen tasks.
- Experimental Results: 15.8% and 11.7% gains over the Train-Edit-Only baseline demonstrate stronger instruction-based editing on the 21-task Editing Benchmark and ImgEdit.The baseline uses the same parameters and editing data without Meta-CoT.
- Experimental Results: Meta-task training enables comparable performance to full-task training while training on unseen tasks.This generalization result is shown on the 21-task Editing Benchmark.
- Triplet Decomposition: Meta-CoT decomposes editing into task summary, task thinking, and target-wise editing traversal for fine-grained reasoning over task and target.Training jointly incorporates diverse visual understanding tasks to capture the (task, target, understanding capability) triplet.
1. Introduction
The introduction identifies the need for CoT that both stimulates fine-grained understanding and generalizes across diverse image-editing tasks. Meta-CoT addresses this through triplet and meta-task decomposition, consistency rewards, and broader evaluation.
- Motivation: Effective image-editing CoT must stimulate understanding ability and generalize across diverse editing tasks.Existing approaches often emphasize explicit understanding cues but can generalize poorly beyond tasks matching those cues.
- Triplet Decomposition: Triplet Decomposition represents each editing operation as task, target, and required understanding ability, enabling fine-grained reasoning over tasks and targets.Its three steps are Task Summary, Task Thinking, and Target-wise Editing Traversal.
- Meta-task Decomposition: Meta-task Decomposition reduces editing tasks to combinations of five primitive, universal meta-tasks, supporting compositional reasoning across the editing operation space.The framework treats these meta-tasks as analogous to bases in a vector space and uses them to redefine the Task element of the triplet.
- Meta-task Decomposition: Training on as few as five meta-tasks enables generalization to other editing tasks through compositional reasoning, without individually covering every task.For example, “Change” can correspond to the meta-tasks add and remove.
- Consistency Reward: The CoT–Editing Consistency Reward uses a VLM to assess task- and object-level consistency between reasoning and edited images, followed by Flow-GRPO optimization.The reward is designed to mitigate mismatches between CoT reasoning and editing outcomes.
- Evaluation: The evaluation benchmark covers 21 distinct editing tasks, combines existing and newly developed benchmarks, and is distributionally independent from training data.Experiments also evaluate the method on ImgEdit.
2. Related Work
Related work extends Chain-of-Thought reasoning from language models to multimodal vision-language settings and increasingly unifies multimodal understanding with generation in single models. GPT-4o exemplifies this unified direction by combining visual analysis and generation while outperforming earlier unified models.
- Chain-of-Thought Reasoning: CoT reasoning has expanded from step-by-step inference in large language models to multimodal reasoning in vision-language models.Early methods integrated visual information into textual CoT, while later approaches interleaved textual reasoning with visual tokens.
- Unified Multimodal Models: Recent work unifies multimodal understanding and generation within single models for joint understanding and synthesis.This research direction spans models designed for multimodal understanding, generation, and their integration.
- Unified Multimodal Models: GPT-4o fuses visual analysis and generation, outperforming earlier unified models.
3. Method
Meta-CoT decomposes image-editing operations into task, target, and required understanding capability, then further factors tasks into five reusable meta-tasks. It aligns reasoning with execution using a consistency reward and a two-stage SFT-to-RL training pipeline.
- Triplet Decomposition: Meta-CoT represents each editing operation as a triplet of task, target, and required understanding capability.The triplet space is defined as Striplet = T1 × T2 × T3, with lower editing complexity than the original CoT space.
- Triplet Decomposition: Triplet Decomposition infers the task, generates task-specific reasoning, and analyzes target-specific visual or logical attributes.Examples include analyzing style attributes for style transfer, object appearance or disappearance for camera motion, and implicit operations for logical reasoning-based editing.
- Meta-task Decomposition: Five meta-tasks form a basis for composing and generalizing diverse complex editing operations.Meta-CoT replaces Task Summary with Meta-task Summary and represents the triplet as meta-task, target, and required understanding capability.
- Training: Two-stage training first uses SFT for reasoning and editing, then applies Flow-GRPO to editing with consistency-reward optimization focused on early denoising timesteps.Later-timestep updates are omitted because semantic fidelity is most critical early, and this also alleviates potential Flow-GRPO noise artifacts.
- CoT–Editing Consistency Reward: The CoT–Editing Consistency Reward scores alignment between reasoning and generated edits across both operation and target.A VLM evaluates this consistency on a 0-to-10 scale because inferred operations may still fail to follow the stated reasoning.
4. Experiment
Meta-CoT is evaluated on diverse 21-task and ImgEdit benchmarks, where it improves overall editing scores over Bagel and a matched train-edit-only variant. Ablations show that meta-task decomposition, Task Thinking, and visual-understanding data support generalization, instruction following, and CoT quality, while text editing remains a limitation.
- Benchmark and Metrics: The evaluation uses a 21-task benchmark and ImgEdit’s nine-task suite, with GPT-4.1 assessing instruction following, consistency, quality, adherence, and detail preservation.The 21-task benchmark includes categories from GEdit-Bench, RiseBench, ComplexEdit, and five newly introduced categories; ImgEdit contains 734 real-world test cases.
- Overall Results: +13.1% and +19.7% are Meta-CoT’s gains over Bagel (no-think) on the 21-task benchmark and ImgEdit, respectively.Relative to Bagel (think), the gains are 20.1% and 13.0% on the same benchmarks.
- Overall Results: 15.8% is Meta-CoT’s improvement over a matched train-edit-only variant, isolating the contribution of the Meta-CoT paradigm.The RL stage further enhances alignment and stability.
- Overall Results: Meta-CoT consistently outperforms baselines across VIEScore’s four dimensions, with the largest improvement in Instruction Following.The result suggests that Meta-CoT reasoning improves semantic understanding of editing operations and targets, producing more instruction-faithful edits.
- Limitations: Text editing is the only task where Meta-CoT does not improve performance, likely because extensive textual reasoning interferes with identifying the correct text to modify.Preserving accurate text perception during reasoning is identified as a promising direction for future work.
- Ablation Studies: Training on a small set of universal meta-tasks with task decomposition generalizes to unseen editing tasks, while the defined five meta-tasks balance generalization and performance.Removing Task Thinking causes a significant drop in instruction-following performance, and reducing visual-understanding data lowers both editing performance and CoT quality.
5. Conclusion
Meta-CoT addresses the joint challenge of improving understanding granularity and generalization in Chain-of-Thought-guided image editing through Triplet Decomposition and Meta-task Decomposition.
- Conclusion: Meta-CoT uses Triplet Decomposition to stimulate model reasoning from both task and target perspectives.This decomposition is presented as a means to enhance understanding granularity in CoT-guided image editing.
- Conclusion: Meta-task Decomposition gives Meta-CoT strong generalization capability across diverse editing tasks.The conclusion identifies Meta-task Decomposition as the mechanism providing this generalization.