PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Vision Co-Instructed Image Editing

Chenxi Xie, Yuhui Wu, Qiaosi Yi, Lei Zhang

arXiv:2606.16767v1cs.CV

TL;DR

Text-only image editing offers semantic expressiveness but limited spatial control, while visual prompts provide precise geometry but ambiguous semantic intent. TV-Edit jointly uses textual instructions and sparse visual prompts, and consistently outperforms instruction-based and drag-based competitors in semantic faithfulness and spatial precision.

  • Problem

    Text-only editing lacks fine-grained spatial control, whereas drag-based editing lacks sufficient semantic grounding for intent-faithful image manipulation.

  • Method

    TV-Edit jointly models textual semantic intent and sparse visual spatial constraints using paired training data and a unified editing framework.

  • Results

    TV-Edit consistently outperforms instruction-based and drag-based competitors, achieving stronger semantic faithfulness, spatial controllability, and visual consistency.

  • Takeaways & Limitations

    Joint textual-visual prompting provides more precise and intent-faithful image editing than single-modality control.

Abstract

from arXiv · show

Existing image editing methods can be generally categorized into textual instruction-based and visual prompt-based ones. Textual instructions are semantically expressive, but are limited by the coarse granularity of spatial control of the editing results. In contrast, visual prompts such as drag and point can provide precise spatial guidance, but are limited by the inherent ambiguity in semantic intent. To unify the strength of textual and visual prompts, we present Text-Vision Co-Instructed Image Editing, which jointly models textual instructions as semantic intent and sparse visual instructions as spatial guidance, aiming to achieve precise and intent-faithful image manipulation. To this end, we first construct a textual-visual instruction paired dataset with more than 23K samples derived from dynamic videos, enabling aligned supervision for cross-modal instruction. We then propose TV-Edit, a Textual-Visual instruction unified Editing framework to contextualize drag or point-based visual instructions with image-text semantics and lift them into semantic-aware control representations for pretrained editing backbones. By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives. Finally, we establish TV-Edit-Bench, a deliberately designed benchmark to evaluate semantic faithfulness, spatial alignment, and visual consistency with ground-truth references and controlled textual-visual variations for reliable assessment. Our experiments across multiple editing backbones demonstrate that TV-Edit consistently yields more precise and intent-faithful edits, significantly outperforming state-of-the-art instruction-based and drag-based baselines.

1 Introduction

The paper introduces text-vision co-instructed image editing, combining textual semantic intent with sparse visual spatial constraints to reduce single-modality ambiguity. It proposes TV-Edit and TV-Edit-Bench, and reports stronger semantic and spatial faithfulness than instruction- and drag-based competitors.

  • Introduction: Textual instructions specify semantic transformations, whereas drag-based prompts provide precise local motion and geometry but little semantic grounding.The introduction identifies complementary limitations in the two editing paradigms.
  • Introduction: Text-Vision Co-instructed Image Editing jointly uses textual semantic intent and sparse visual spatial constraints to achieve more precise, intent-faithful manipulation.The task treats text and visual prompts as complementary signals rather than relying on either modality alone.
  • Introduction: TV-Edit-Bench contains 120 curated evaluation pairs with aligned textual and visual instructions plus ground-truth editing targets.The pairs draw from real videos, image-to-video generated videos, and image pairs synthesized by advanced editing models.
  • Introduction: TV-Edit is applied to Qwen-Image-Edit and FLUX.1 Kontext as popular editing foundation models.These backbones are used to evaluate the proposed text-vision co-instructed editing approach.
  • Introduction: TV-Edit consistently outperforms state-of-the-art instruction-based and drag-based competitors in semantic intent and spatial-constraint faithfulness.The reported edits are more faithful to both aspects of user-specified editing requirements.

2 Related Work

Textual instruction-based image editing progressed from prompt-based inversion and regeneration toward direct-command instruction-based models. This shift reduced reliance on carefully aligned source–target prompt pairs and motivated paired-dataset training approaches.

  • Textual instruction-based editing evolved from early prompt-based methods toward instruction-based models.
  • Early methods used pretrained text-to-image models through inversion and regeneration, requiring carefully aligned source and target prompts.
  • Instruction-based editing replaced cumbersome prompt pairs with direct commands.
  • Pioneering instruction-based methods trained on paired datasets using diffusion or flow matching.

3 Text-Vision Co-Instructed Editing

TV-Edit jointly uses textual instructions for semantic intent and sparse visual prompts for spatial guidance, addressing the under-specification of single-modality editing. It introduces the TV-Edit-23K dataset and a control-branch architecture that grounds point correspondences in image content while preserving semantic and global coherence.

  • Motivation and formulation: TV-Edit jointly specifies edits with textual instructions and sparse visual prompts to combine semantic intent with spatial guidance.The formulation targets semantic faithfulness, spatial alignment with locally coherent transformations, and global coherence beyond edited regions.
  • Dataset construction: 23K high-quality sample groups form TV-Edit-23K, built from video-frame pairs through visual annotation, paired textual annotation, and filtering.Each raw group yields two bidirectional editing pairs, supporting diverse scenes, semantic motion transformations, and varying motion magnitudes.
  • Model architecture: TV-Edit combines a main editing branch with a sparse-point control branch that injects spatial intent into a pretrained editing backbone.The control branch uses point encoding and a content-aware controller to produce geometrically grounded guidance without training the full model from scratch.
  • Spatial control: The control branch renders indexed source and target point maps, encodes them, and fuses them with image and noised latents to ground correspondences in image content.Residual control features are injected into MM-DiT blocks in a ControlNet-like manner, enabling content-dependent rather than geometry-only interpretation.
  • Training strategy: Training freezes the main branch and optimizes the sparse point encoder and controller, using timestep sampling biased toward larger timesteps early in training.The strategy targets spatial layout in the high-noise regime and improves convergence speed and controllability.

4 Experiments

TV-Edit is evaluated on the newly constructed TV-Edit-Bench and established drag-based benchmarks, combining geometric accuracy, image fidelity, and semantic faithfulness. Across quantitative and qualitative results, it bridges drag-based spatial precision with instruction-based semantic control.

  • Evaluation Benchmarks: TV-Edit-Bench contains 120 curated samples pairing textual and visual prompts with ground-truth edited targets, plus auxiliary masks and descriptions.The benchmark evaluates text-vision co-instructed editing and includes strictly screened image, trajectory, and instruction annotations.
  • Evaluation Protocol: The benchmark measures image fidelity with LPIPS and DINOv3-based DStgt_global and DStgt_local, geometric accuracy with MD_s and MD_d, and semantic faithfulness with CP and PF.TV-Edit is also evaluated on established drag-based benchmarks to test generalization of its spatial control ability.
  • Quantitative Results: GoodDrag achieves an MD_d of 0.0648 but only 0.75 PF, showing that drag-based methods control editing location while struggling with semantic execution.Drag-based methods generally achieve strong geometric accuracy, but their image fidelity is often compromised.
  • Quantitative Results: NanoBanana Pro reaches 0.9432 on DStgt_global, nearly 1.0 on CP, and 0.89 on PF, but its MD_d consistently exceeds 0.10.Instruction-based models therefore show strong fidelity and semantic faithfulness but unpredictable geometric changes without explicit spatial guidance.
  • Quantitative and Qualitative Results: TV-Edit-Qwen reduces MD_d to 0.0462, a 28.7% improvement over the best drag-based method, and raises PF from 0.86 to 0.93 over Qwen-Image-Edit.Qualitative results further show control of motion magnitude under fixed text and distinct semantic actions under ambiguous visual prompts.

5 Conclusion · Appendix

The paper concludes by introducing a unified text-and-vision editing task, TV-Edit, a paired training dataset, and TV-Edit-Bench for reducing ambiguity through semantic and spatial control. The appendix supplements these contributions with dataset, training, benchmark, editing, ablation, drag-based comparison, and social-impact materials.

  • 5 Conclusion: The proposed task jointly leverages textual instructions and sparse visual prompts to reduce ambiguity in single-modality control.
  • 5 Conclusion: TV-Edit combines a decoupled Content-Aware Spatial Controller with a text-vision paired training dataset for unified semantic and spatial control.
  • 5 Conclusion: TV-Edit-Bench provides carefully designed cases and a comprehensive evaluation protocol for assessing the proposed editing approach.
  • Appendix: Appendix A provides more details of the TV-Edit-23K dataset, referring to Section 3.2 of the main paper.
  • Appendix: Appendix B analyzes the training strategy in greater detail, referring to Section 3.3 of the main paper.
  • Appendix: Appendix C details TV-Edit-Bench dataset construction and its evaluation protocol, while Appendix D presents additional TV-Edit results and visual comparisons.
  • Appendix: Appendix E reports ablation studies on TV-Edit, and Appendix F compares the method quantitatively and qualitatively with drag-based methods on Drag-Bench.
  • Appendix: Appendix G discusses the potential social impact of the work.

A More Details of TV-Edit-23K Dataset · A.1 Detailed Prompts for Paired Textual Annotation

The paired textual annotation stage uses Qwen-3-VL to describe the action transforming one image into another, with detailed prompts provided for the TV-Edit-23K construction pipeline.

  • A.1 Detailed Prompts for Paired Textual Annotation: Qwen-3-VL is instructed to provide the action that transforms one image into the other.This instruction is used during paired textual annotation.
  • A.1 Detailed Prompts for Paired Textual Annotation: The annotation target is an image-to-image transformation action.The prompt asks for the action capable of transforming one image into the other.
  • A.1 Detailed Prompts for Paired Textual Annotation: Detailed transformation prompts are included in Figure A.1.Figure A.1 presents the prompts used for paired textual annotation in the data construction pipeline.

A.2 Samples in TV-Edit-23K Dataset

The TV-Edit-23K training samples combine dense, accurate point pairs with semantic transformation instructions aligned to labeled-point motion, supporting geometric relationship learning.

  • Training Samples: TV-Edit-23K samples contain dense and accurate point pairs for learning corresponding geometric relationships.The data construction pipeline generates these point pairs as training data.
  • Training Samples: MLLM-generated semantic transformation instructions accurately match the motion of labeled point pairs.This alignment connects semantic instructions with the observed point-pair transformations.

B More Analysis of TV-Edit Training Strategy · C More Details of TV-Edit-Bench

TV-Edit’s training strategy prioritizes high-noise timesteps to improve global spatial layout and geometric accuracy through implicit loss weighting and explicit sampling. The section also introduces a comparison of drag-based editing datasets for TV-Edit-Bench.

  • B More Analysis of TV-Edit Training Strategy: TV-Edit prioritizes global spatial layout and geometric accuracy because large timesteps establish low-frequency spatial structures.Large timesteps correspond to the high-noise regime, where t→1.
  • B More Analysis of TV-Edit Training Strategy: The strategy focuses learning on high-noise timesteps through implicit loss weighting and explicit timestep sampling.Small timesteps instead emphasize high-frequency texture details.
  • B More Analysis of TV-Edit Training Strategy: TV-Edit replaces direct velocity supervision with Z0 supervision, estimating the clean latent from the intermediate state and predicted velocity.The forward process relates the intermediate latent to the clean latent through Z_t = Z0 + t v_t.
  • B More Analysis of TV-Edit Training Strategy: The training objective minimizes MSE between the estimated and ground-truth clean latents.This objective is defined as Lfm.
  • B More Analysis of TV-Edit Training Strategy: Z0 supervision is equivalent to v-supervision scaled by t^2, assigning larger penalties to errors at large t.The passage states that this implicit weighting strategy has been corroborated by recent controllable-generation studies.
  • B More Analysis of TV-Edit Training Strategy: Beta timestep sampling explicitly biases training toward large timesteps, with Beta(10, 2) concentrated around 0.9.This concentration corresponds to extremely high noise levels where spatial layout is determined.
  • C More Details of TV-Edit-Bench: The appendix includes Table C.1, comparing drag-based editing datasets.The supplied passage provides the table title but no dataset entries or comparison values.

C.1 Samples in TV-Edit-Bench … D.1 Visual Comparisons on TV-Edit-Bench

The appendix details TV-Edit-Bench’s paired, densely annotated samples and controlled variations, then defines feature- and MLLM-based evaluation protocols. Additional comparisons show TV-Edit handling diverse spatial transformations with strong fidelity, geometric accuracy, and robustness across edit magnitudes and affected areas.

  • C.1 Samples in TV-Edit-Bench: Each TV-Edit-Bench sample pairs a source image with a visually consistent reference target and annotated spatial correspondences.Larger annotated points serve as visual prompts and compute sparse MDs during evaluation.
  • C.1 Samples in TV-Edit-Bench: TV-Edit-Bench provides training and evaluation splits, motion-centric instructions, dense annotations, and test pairs with controlled variations.Dense annotations evaluate regional transformation consistency.
  • C.2 Evaluation Protocol: Patch-level similarity decouples content fidelity from spatial positioning and mitigates inaccuracies associated with strictly pixel-aligned metrics such as LPIPS.The protocol uses DINOv3 features and masked patch comparisons.
  • C.2 Evaluation Protocol: Global DINO Score measures overall semantic consistency between target and edited images using cosine similarity between their CLS-token features.Local DINO Score instead uses nearest-neighbour patch matching within masked edited regions to assess fine-grained fidelity.
  • C.2 Evaluation Protocol: The MLLM-based protocol uses Qwen-3-VL to score prompt following and concept preservation, incorporating benchmark reference ground truth into evaluation prompts.Concept preservation receives a score from 0 to 4, while prompt following evaluates the intended semantic transformation.
  • D.1 Visual Comparisons on TV-Edit-Bench: Additional results show TV-Edit performing rotation and translation with superior image fidelity and geometric accuracy compared with other methods.The visual comparisons are presented on TV-Edit-Bench.
  • D More Editing Results of TV-Edit: TV-Edit remains robust across edits with varied magnitudes and affected areas, including substantially moving a person to the image’s right side.This example appears in the last row of the visual comparisons.

D.2 Results on Simultaneous Spatial Control and Semantic Editing

The method combines precise spatial control with simultaneous semantic editing, enabling motion-constrained transformations alongside identity, object, and color changes.

  • The model rotates a dog’s head while transforming its identity into a tiger under motion constraints.This is demonstrated with the instruction “change it to a tiger” provided alongside the motion constraints.
  • The approach supports complex semantic modifications, including adding objects and altering colors, while preserving precise spatial control.

E Ablation Studies on TV-Edit · F Comparison on Drag-Bench

Ablations on Qwen-Image-Edit-based TV-Edit show that time-modulated control, timestep sampling, and controller depth govern spatial accuracy, semantic control, and efficiency. DragBench experiments further evaluate the method’s geometric-accuracy generalization against state-of-the-art baselines.

  • E Ablation Studies on TV-Edit: The ablation experiments use Qwen-Image-Edit-based TV-Edit.This establishes the editing backbone used for the ablation studies.
  • E Ablation Studies on TV-Edit: 0.1355 MDd results from injecting only sparse geometric conditions, showing insufficient spatial control.The baseline contains a sparse point encoder and linear layers that inject features into the editing backbone.
  • E Ablation Studies on TV-Edit: Time-modulated scaling substantially enhances control by dynamically adjusting control-branch influence across backbone blocks and denoising timesteps.Additional linear layers provide only marginal gains without explicit guidance for learning distinct representations.
  • E Ablation Studies on TV-Edit: Beta(5, 2) improves geometric accuracy by emphasizing larger timesteps, whereas overly concentrated high-noise sampling degrades geometric accuracy and prompt following.TV-Editing mainly involves low-frequency structural changes, motivating greater emphasis on larger timesteps.
  • F Comparison on Drag-Bench: DragBench experiments assess TV-Edit’s geometric-accuracy generalization through quantitative and qualitative comparisons with several state-of-the-art baselines.The experiments follow prior work [18] and use the drag-based benchmark DragBench [19].

F.1 Quantitative Comparison · F.2 Qualitative Comparison · G Broader Impacts

On DragBench, TV-Edit-Qwen is evaluated with mean point displacement distance and 1−LPIPS, while qualitative comparisons show stronger geometric and semantic editing than drag-based methods. The method may benefit creative industries but also risks enabling misleading or harmful content affecting privacy and security.

  • F.1 Quantitative Comparison: TV-Edit-Qwen is evaluated on DragBench using MD for mean point displacement distance and IF for 1−LPIPS.These metrics assess geometric accuracy and image fidelity, respectively.
  • F.1 Quantitative Comparison: 17.31 is the reported MD value for TV-Edit-Qwen on DragBench.The supplied passage identifies this value as notable but does not provide the corresponding IF value.
  • F.2 Qualitative Comparison: TV-Edit-Qwen is qualitatively compared with DragDiffusion, DragLora, GeoDrag, and GoodDrag.The comparison concerns geometric accuracy and fidelity in visual results.
  • F.2 Qualitative Comparison: For opening a lion’s mouth, only TV-Edit-Qwen successfully executes the intended semantic transformation.GoodDrag attempts to follow the prompt but introduces artifacts.
  • G Broader Impacts: The method could enhance creative industries by giving artists and designers powerful tools for content creation and modification.This is identified as a potential positive societal impact.
  • G Broader Impacts: The technology could be misused to create misleading or harmful content, with significant implications for privacy and security.The passage presents this as a potential negative societal impact.
Loading 2606.16767v1…