PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-CoT: Enhancing Granularity and Generalization in Image Editing

Shiyi Zhang, Yiji Cheng, Tiankai Hang, Zijin Yin, Runze He, Yu Xu, Wenxun Dai, Yunlong Lin, Chunyu Wang, Qinglin Lu, Yansong Tang

arXiv:2604.24625v1cs.CVcs.AIcs.LGcs.MM

TL;DR

Designing CoT for image editing that stimulates understanding while generalizing across tasks remains an open question. Meta-CoT decomposes edits into task, target, and understanding ability, further using five meta-tasks and a consistency reward; it reports 15.7% overall performance improvement in its 21-task evaluation.

  • Problem

    How to design CoT for image editing that both stimulates the model’s understanding ability and generalizes across diverse editing tasks remains underexplored.

    Source: §1 · p. 2

  • Method

    Meta-CoT decomposes editing into task, target, and understanding ability, further maps tasks to five meta-tasks, and uses a CoT–Editing Consistency Reward to align reasoning with edits.

    Source: §1 · pp. 2–3, §3.3 · p. 5

  • Results

    15.7% overall performance improvement is reported in evaluation spanning 21 distinct editing tasks, while training on five meta-tasks generalizes to other editing tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings indicate that training need not cover every editing task individually when remaining tasks can be expressed as combinations of five meta-tasks.

    Source: §1 · p. 3

    Text editing is an exception to the performance gains, as extensive textual reasoning can interfere with identifying the text to modify.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Unified multi-modal understanding/generative models have shown improved image editing performance by incorporating fine-grained understanding into their Chain-of-Thought (CoT) process. However, a critical question remains underexplored: what forms of CoT and training strategy can jointly enhance both the understanding granularity and generalization? To address this, we propose Meta-CoT, a paradigm that performs a two-level decomposition of any single-image editing operation with two key properties: (1) Decomposability. We observe that any editing intention can be represented as a triplet - (task, target, required understanding ability). Inspired by this, Meta-CoT decomposes both the editing task and the target, generating task-specific CoT and traversing editing operations on all targets. This decomposition enhances the model's understanding granularity of editing operations and guides it to learn each element of the triplet during training, substantially improving the editing capability. (2) Generalizability. In the second decomposition level, we further break down editing tasks into five fundamental meta-tasks. We find that training on these five meta-tasks, together with the other two elements of the triplet, is sufficient to achieve strong generalization across diverse, unseen editing tasks. To further align the model's editing behavior with its CoT reasoning, we introduce the CoT-Editing Consistency Reward, which encourages more accurate and effective utilization of CoT information during editing. Experiments demonstrate that our method achieves an overall 15.8% improvement across 21 editing tasks, and generalizes effectively to unseen editing tasks when trained on only a small set of meta-tasks. Our code, benchmark, and model are released at https://shiyi-zh0408.github.io/projectpages/Meta-CoT/

1. Introduction

Meta-CoT decomposes image-editing operations into task, target, and required understanding ability, then into compositional meta-tasks to support fine-grained reasoning and generalization. It also rewards consistency between CoT reasoning and edited results.

1. Introduction

  • Although CoT can improve editing performance [8] [11], approaches tailored to specific understanding forms can generalize poorly to tasks such as style transfer or viewpoint transformation.The paper frames its objective as improving understanding granularity and generalization together.Source: §1 · p. 2
  • Triplet Decomposition separates each editing operation into task, target, and required understanding ability, guiding task-specific reasoning and target-wise editing traversal.Training also incorporates diverse visual-understanding tasks to develop all three elements of the triplet.Source: §1 · p. 2
  • Meta-task Decomposition represents editing tasks as combinations of five universal operations, enabling compositional generalization from training on as few as five meta-tasks.It replaces Task Summary with Meta-task Summary; for example, “Quantity Change” maps to add and remove. Table 1 presents the five meta-tasks and possible combinations.Source: §1 · pp. 2–3, Table 1 · p. 3
  • The CoT–Editing Consistency Reward uses a VLM to assess reasoning-result consistency from task and object perspectives, then supplies rewards for Flow-GRPO optimization.The method uses Qwen2.5-VL [2] as an example VLM and aims to align CoT reasoning with editing outcomes.Source: §1 · p. 3
  • 15.7% overall performance improvement across 21 editing tasks over the base model is reported for Meta-CoT.The benchmark covers 21 distinct tasks and is distributionally independent from training data; experiments use Bagel with Meta-CoT applied via SFT and the consistency reward via GRPO optimization.Source: §1 · p. 3

2. Related Work

Prior work extends CoT to image editing but leaves a gap between generic reasoning and task-specific adaptability. Unified multimodal models provide architectural alternatives, with Bagel serving as this work’s foundation.

2.1. CoT in Understanding/Generation Models

  • Editing CoT methods improve consistency between visual understanding and generation, but are either too generic or too specialized for diverse tasks [8] [11].Earlier work moved from integrating visual information into textual CoT [40] [81] to native multimodal CoT interleaving text and visual tokens [12] [14] [18] [29] [50] [63].Source: §2.1 · p. 4

2.2. Unified Understanding and Generation Models

  • Unified models use autoregressive, additional-diffusion, or integrated-transformer architectures, and this work builds on Bagel [8].Bagel is a unified multimodal transformer with intrinsic CoT-based editing capabilities.Source: §2.2 · p. 4

3. Method

Meta-CoT represents image edits through task, target, and required understanding capability, then decomposes tasks into five meta-tasks. It also trains task-specific reasoning and target traversal and uses a consistency reward to align reasoning with edits.

3.1. Theoretical Definition

  • Theoretical analysis says triplet decomposition lowers editing complexity and yields higher understanding granularity than classical CoT.The triplet comprises task, target, and required understanding capability.Source: §3.1 · p. 4

3.2. Triplet Decomposition

  • Triplet Decomposition infers task type, generates task-specific reasoning, and traverses image targets to decide whether and how each should be edited.Training also incorporates diverse visual-understanding tasks to teach all three triplet elements.Source: §3.2 · p. 4

3.3. Meta-task Decomposition

  • Meta-task Decomposition replaces Task Summary with a summary of combinations of five meta-tasks, training these alongside targets and understanding capabilities.The meta-tasks are defined as bases that can combine to generalize to more complex editing operations; style transfer is given as a replacement operation on a target’s style attribute.Source: §3.3 · pp. 4–5

3.4. CoT-Editing Consistency Reward

  • The CoT–Editing Consistency Reward uses a VLM to score whether an edit matches the reasoning in both operation and target.Its prompt was calibrated against human annotations on 500 samples until Pearson correlation r ≥0.8 and mean absolute error ≤2.5; Flow-GRPO optimization focuses on early denoising timesteps, while omitting later updates alleviated potential noise artifacts.Source: §3.4 · p. 5

3.5. Training Pipeline

  • Training first tunes both understanding and generation experts plus the image-understanding encoder with SFT, then freezes the encoder and trains only the generation expert during RL.The authors report that preserving the SFT-learned CoTs and avoiding joint RL training prevents reasoning degradation from unstable optimization.Source: §3.5 · p. 5, Figure 4 · p. 5

3.6. Meta-CoT Data Creation Pipeline

  • The data pipeline classifies each instruction’s task, checks that classification, then generates and evaluates Meta-CoT reasoning using the source image, target image, instruction, and task type.Qwen2.5 performs task classification, Gemini-2.5-Flash checks consistency, and Qwen2.5-VL generates the summaries, task thinking, and target traversal.Source: §3.6 · p. 5, Figure 5 · p. 5

4. Experiment

Across the 21-task benchmark, Meta-CoT improves overall editing scores over comparison models and the matched training setting without Meta-CoT. Ablations examine generalization, task reasoning, and visual-understanding data, while qualitative examples cover diverse editing tasks.

4.1. Implementation Details

  • Evaluation uses a 21-task benchmark scored by VIEScore Overall Score (0–10) and ImgEdit’s nine tasks/734 real-world cases (scores 1–5); SFT uses 1.5M editing and 100K understanding examples, while RL adds 20K editing examples.Source: §4.1 · pp. 5–6

4.2. Quantitative Evaluation

  • On the 21-task benchmark, Meta-CoT outperforms Train Editing Only by 15.8%; that baseline uses the same parameters and editing data but excludes Meta-CoT training.Scores were evaluated using GPT-4.1.Source: Table 2 · p. 6, §4.2 · p. 7
  • Meta-CoT consistently outperforms baselines across VIEScore’s four dimensions, with the largest improvement in Instruction Following.At the per-task level, performance improves on all tasks except text editing, where extensive reasoning may interfere with identifying the text to modify.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. Abaltion Study

  • Training on five meta-tasks achieves performance comparable to the full-data model on the 21-task benchmark and generalizes to unseen tasks.Five meta-tasks balance generalization and performance: fewer cause a significant instruction-following drop, while more yield little additional improvement across tasks.Source: §4.3 · p. 7, Table 5 · p. 7
  • Removing Task Thinking causes a significant drop in instruction-following performance, supporting reasoning from task characteristics during editing.The ablation removes the second step of Meta-CoT.Source: §4.3 · p. 7, Table 5 · p. 7
  • Removing visual-understanding data or reducing it to 1K samples causes editing-performance drops, particularly in instruction following, and lowers CoT quality.The default setting mixes 100K understanding samples; GPT-4.1 evaluates CoT completeness and accuracy for Task Thinking and Target Editing Traversal.Source: §4.3 · pp. 7–8, Table 6 · p. 7

4.4. Qualitative Evaluation

  • Qualitative comparisons show improved instruction following, logical reasoning, and multi-instruction understanding across diverse editing tasks.Figure 6 includes conventional, reasoning-based, and multi-instruction editing, with partial visualizations of Meta-CoT reasoning and target traversal.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

Meta-CoT combines task-and-target Triplet Decomposition, Meta-task Decomposition, and a CoT-Editing Consistency Reward to improve understanding granularity and generalization in CoT-guided image editing. Experiments on a 21-task benchmark and ImgEdit show improved editing performance and generalization to unseen tasks.

  • Experiments on the 21-task benchmark and ImgEdit show improved editing performance and strong generalization to unseen editing tasks.Source: §5 · p. 8
  • Triplet Decomposition stimulates model reasoning from both task and target perspectives to enhance understanding granularity.Source: §5 · p. 8
  • Meta-task Decomposition supports generalization across diverse editing scenarios, while the CoT-Editing Consistency Reward aligns reasoning with editing behavior.Source: §5 · p. 8

Related papers

Loading 2604.24625v1…