PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeEdit: Image Editing with Canvas Instructions

Jinjing Zhao, Fangyun Wei, Yitong Wang, Xiuyu Wu, Yunuo Chen, Yang Yue, Sirui Zhang, Wenbo Wang, Hongyang Zhang, Dong Chen, Yan Lu, Chang Xu

arXiv:2610.12229v1cs.CVcs.AI

TL;DR

Identifying the intended object in text-guided image editing can be cumbersome, especially when similar objects appear together. VibeEdit uses spatial marks and optional notes as canvas instructions for five edit operations, without a separate text prompt. On a human-curated 419-case benchmark, it outperforms evaluated baselines on rubric score and preservation outside the edited region.

  • Problem

    Text-guided editing requires users to describe both the desired change and its target, which can be cumbersome when several objects look alike.

    Source: §1 · p. 2

  • Method

    VibeEdit uses canvas instructions made of spatial marks and optional notes to guide five edit operations without a separate text prompt.

    Source: §1 · pp. 2–3

  • Results

    79.9 overall VLM rubric score and 32.8 dB outside-region PSNR outperform all evaluated baselines on both metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Canvas instructions provide direct spatial grounding for target selection in a benchmark emphasizing similar objects.

    Source: §1 · p. 3

Abstract

from arXiv · show

In text-guided image editing, describing the desired change is often straightforward, but identifying the intended object or region can be cumbersome, especially when several objects look alike. We introduce a new image editing interface that lets users place spatial marks and optional short notes directly on the image. Together, these annotations form a canvas instruction that specifies where to edit and what to change. Our editor, VibeEdit, follows these instructions to perform object addition, removal, replacement, attribute modification, and movement without a separate text prompt. We construct 1.55 million source-target edit pairs with object masks and structured edit descriptions, from which we render canvas instructions during training. We adapt Qwen-Image-Edit with layer-decoupled conditioning that separately encodes source images and canvas instructions for image editing. We train the model with region-weighted supervised fine-tuning, followed by rubric-guided reinforcement learning to improve edit completion, local edit quality, and preservation of unedited regions. We evaluate VibeEdit on an independently constructed, human-curated benchmark of 419 cases emphasizing target selection among similar objects. VibeEdit achieves a VLM rubric score of 79.9 and an outside-region PSNR of 32.8 dB, compared with 67.4 and 24.0 dB for FireRed, the highest-scoring text-instructed baseline in our evaluation.

1 INTRODUCTION

VibeEdit replaces separate text prompting with canvas instructions that pair spatial marks and optional notes to specify edit targets and changes across five operations. It is evaluated on a human-curated benchmark emphasizing similar-object target selection, where it outperforms evaluated baselines on overall rubric score and outside-region PSNR.

1 INTRODUCTION

  • Canvas marks provide direct spatial grounding when language alone makes identifying the intended object cumbersome, especially among similar-looking objects.The interface combines spatial cues with language for the requested change.Source: §1 · pp. 2–3
  • The training corpus contains 1.55 million source–target edit pairs with object masks and structured descriptions, rendered as varied canvas annotations during training.VibeEdit adapts Qwen-Image-Edit with layer-decoupled conditioning, region-weighted supervised fine-tuning, and rubric-guided reinforcement learning.Source: §1 · pp. 2–3
  • 79.9 overall VLM rubric score and 32.8 dB outside-region PSNR put VibeEdit ahead of all evaluated baselines on both metrics.The human-curated benchmark contains 419 cases emphasizing target selection among similar objects.Source: §1 · p. 2
  • Canvas instructions combine spatial marks and optional short notes to specify edit targets and desired changes for five operations without a separate text prompt.The operations are addition, removal, replacement, attribute modification, and movement.Source: §1 · pp. 2–3

2 RELATED WORK

Prior image editors use text instructions, spatial localization cues, or graphical controls, while VibeEdit places spatial and semantic guidance together on the canvas. Its localized-edit setting also motivates rubric-based reinforcement learning that balances edit completion with source preservation.

Instruction-based image editing

  • Text-only editors require prompts to describe both the requested change and its referent, while VibeEdit places notes beside intended targets.The related work includes diffusion editors such as InstructPix2Pix, MagicBrush, and UltraEdit.Source: §2 · p. 3

Spatial control and visual instructions

  • Spatial-control systems use geometric handles, region cues, or graphical controls; VibeEdit combines circles, scribbles, and movement arrows with open-vocabulary handwritten instructions.Unlike approaches using image-space cues only for localization, VibeEdit also places semantic guidance on the canvas across five edit types.Source: §2 · p. 3

Reinforcement learning for diffusion models

  • Rubric-based rewards assess prompt criteria independently, and VibeEdit adapts decomposed rewards to localized editing where completion must be balanced with source preservation.The related work describes RubricRL (Feng et al., 2025) as decomposing prompt requirements into criteria evaluated by a multimodal judge.Source: §2 · p. 3

3 DATA AND BENCHMARK CONSTRUCTION

VibeEdit’s data pipeline creates paired edits and renders canvas instructions during training, then uses a separate reward-selected set and an independent human-curated benchmark. The benchmark emphasizes whether an editor applies a change to the intended instance among similar objects.

3.1 SUPERVISED LEARNING DATA

  • Operation-specific pipelines generate pairs for addition, removal, attribute modification, replacement, and movement from selected objects and synthesized edits.Addition and removal use opposite directions of an original-to-inpainted pair; movement uses source and destination masks as annotation endpoints.Source: Figure 2 · p. 4, §3.1 · pp. 4–5
  • The supervised corpus stores source and edited images, masks, and structured descriptions, while rendering canvas instructions online so one edit pair supports varied annotation styles.The corpus contains 1,554,062 stored image pairs and retains information for both editing directions where applicable.Source: §3.1 · pp. 4–5
  • Quality screening filters low-quality synthesized pairs, and rubric-guided reinforcement learning complements filtering because residual errors may remain.The RL rewards favor successful edits, high local quality, and preservation of surrounding content.Source: §3.1 · p. 5
  • Training annotations use circles and notes for addition, attribute modification, and replacement; scribbles indicate removal, while connected circles and arrows indicate movement.Stroke shapes, widths, and typefaces are varied during rendering.Source: §3.1 · p. 5

3.2 REINFORCEMENT LEARNING DATA AND BENCHMARK

  • The RL set retains high-reward-variation conditions within each edit type, yielding 3,520 conditions with 704 per type.Selection favors differences among rollouts of the same condition rather than low mean rewards.Source: §3.2 · p. 5
  • The independent 419-case benchmark uses scenes with similar objects to test target selection, and supplies text-instructed baselines with detailed target-and-edit prompts.VibeEdit receives the source image and canvas instruction without a separate text prompt.Source: §3.2 · p. 5

4 METHOD

VibeEdit edits a source image from a canvas instruction without a separate text prompt, using separate semantic and generative conditioning followed by region-weighted supervised training and rubric-guided reinforcement learning.

4.1 LAYER-DECOUPLED CONDITIONING

  • The vision–language encoder interprets the annotated image, while the generative pathway separately encodes the source image and canvas instruction.The model uses semantic tokens from the annotated image alongside source-image and canvas tokens; rank-128 LoRA adapters are trained on DiT attention projections and modulation layers.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 REGION-WEIGHTED SUPERVISED FINE-TUNING

  • A dilated supervision mask covers the edited object and rendered annotations, upweighting these regions while retaining unit weight elsewhere.For movement, the object mask covers both source and destination; dilation accommodates approximate marks, nearby edit effects, and VAE downsampling. The mask weights training loss only and is not supplied at inference.Source: §4.2 · p. 6

4.3 RUBRIC-GUIDED DIFFUSION REINFORCEMENT LEARNING

  • For each condition, the sampling policy generates multiple edits that a VLM judge scores for edit success, outside preservation, and local edit quality.The judge receives source and edited images, aligned edit-region crops, edit type, and structured target description; that description is used only for reward computation, not by the editing policy.Source: §4.3 · p. 7
  • The rubric reward averages applicable binary judgments and penalizes insufficient preservation outside a padded edit region.For movement, the excluded region includes source and destination; the PSNR penalty addresses low-level changes that binary VLM judgments may miss.Source: §4.3 · p. 7
  • Rewards are centered within each condition and normalized by edit-type reward standard deviation; the resulting weights favor higher-reward rollouts in the positive branch.The normalization statistic is computed online across rollouts of the same edit type.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments compare VibeEdit and baselines under specified training, inference, and evaluation settings. They report the main benchmark results and ablations of conditioning, supervision, and reinforcement-learning choices.

5.1 EXPERIMENTAL SETUP

  • The setup trains LoRA adapters in supervised and RL stages, evaluates with rubric questions and outside-region PSNR, and compares baselines under different input and prompt conditions.The rubric covers edit success, outside preservation, and local edit quality.Source: §5.1 · pp. 7–8, §5.2 · p. 8

5.2 MAIN RESULTS

  • 79.9 VLM rubric score and 32.8 dB PSNR are the final VibeEdit configuration’s highest overall results.The configuration also achieves the highest rubric score on all five edit types.Source: Table 1 · p. 8, §5.2 · p. 9

5.3 ABLATION STUDIES

  • 24.7, 58.6, and 67.8 are rubric scores for source-only, annotated-image, and separate source/canvas VAE inputs; separate encoding raises PSNR from 24.2 to 27.0 dB.The VLM input remains fixed to the annotated image.Source: Table 2 · p. 8, §5.3 · p. 9
  • 64.2 versus 66.2 is the rubric score with region weighting but no dilation; adding dilation raises the score to 67.8.Dilation extends the upweighted region around object boundaries and annotations.Source: Table 3 · p. 8, §5.3 · p. 9

5.2 MAIN RESULTS

  • Without separate text prompts, Figure 4 spans all five edit types; baselines show retained marks, target confusion, unintended changes, and inaccurate relocation.VibeEdit follows canvas instructions more consistently than the baselines.Source: Figure 4 · p. 9, §5.2 · p. 9

5.3 ABLATION STUDIES

  • Adding local-quality rewards improves rubric score but lowers outside-region PSNR, while preservation rewards improve both; all three reward groups achieve 79.9 and 32.8 dB.The three groups address edit success, local quality, and outside preservation.Source: §5.3 · p. 9, Table 5 · p. 10
  • 79.9 versus 79.3 is the rubric score for high-variance versus random RL condition selection; low-variance selection scores 76.6.The comparison selects conditions based on rollout-reward variation.Source: §5.3 · p. 9, Table 4 · p. 10

6 CONCLUSION

The conclusion presents VibeEdit as an image-editing interface and model for five operations without a separate text prompt. It reports benchmark gains and describes the use of AI tools and reproducibility materials.

6 CONCLUSION

  • VibeEdit uses spatial marks and optional notes to specify where and what to edit, supporting five operations without a separate text prompt.It is trained on 1.55 million source–target edit pairs and evaluated on an independent, human-curated 419-case benchmark.Source: §6 · p. 10
  • On the 419-case benchmark emphasizing similar-object target selection, VibeEdit outperforms evaluated baselines in overall VLM rubric score and outside-region PSNR.The model uses layer-decoupled conditioning, region-weighted supervised learning, and rubric-guided RL.Source: §6 · p. 10
  • GPT models supported object selection, edit-description generation, RL rewards, and output evaluation; generative image models synthesized training pairs.The authors state they take responsibility for the work and artifacts produced with AI assistance.Source: AI USE STATEMENT · p. 10
  • The image data include publicly available datasets and AI-generated images; the appendix provides further data, RL, and evaluation details for reproducibility.Sections 3, 4, and 5.1 describe data construction, model and training objectives, and experimental settings.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A SUPPLEMENTARY EXPERIMENTS

Supplementary experiments examine task composition, annotation variation, removal instructions, judge agreement, and LoRA module choices. Results report both model comparisons and human preference or validation evidence.

Joint and single-task training

  • Joint training improves rubric scores over single-task models on addition, attribute modification, and movement, while removal-only and replacement-only models score higher on their respective tasks.All models are evaluated on the full benchmark.Source: Table 6 · p. 14, A SUPPLEMENTARY EXPERIMENTS · p. 14

Annotation-color generalization

  • Changing primary stroke color from red to blue, yellow, or per-case sampled colors changes rubric score by at most 0.6 points and PSNR by at most 0.2 dB.Movement destination circles remain green at evaluation.Source: Table 7 · p. 14, A SUPPLEMENTARY EXPERIMENTS · p. 14

Font diversity

  • Across 200 pairwise human preference comparisons, the model trained with multiple fonts achieves a higher Elo score than the single-font model.Participants wrote canvas instructions in their own handwriting.Source: Table 8 · p. 14, A SUPPLEMENTARY EXPERIMENTS · p. 14

Canvas instructions for removal

  • Text cues raise rubric scores and outside-region PSNR for both deletion marks; circled scribble with text performs best.The default removal instruction uses a circled scribble and includes “remove it” during supervised learning.Source: Table 9 · p. 14, A SUPPLEMENTARY EXPERIMENTS · p. 15

Human validation of the VLM judge

  • Human pass rates are 76.6% and GPT-5.6-sol pass rates are 77.0%; question-level agreement reaches 93.0% accuracy, 95.5% F1, and Cohen’s κ = 0.805.The comparison uses 947 binary rubric questions on 100 randomly sampled VibeEdit outputs.Source: Table 10 · p. 15, A SUPPLEMENTARY EXPERIMENTS · p. 15

LoRA module ablation

B DIFFUSIONNFT POLICY UPDATE

The policy update compares current and old velocity predictions on re-noised rollouts, then uses a reward-weighted objective to train a new LoRA adapter.

B DIFFUSIONNFT POLICY UPDATE

  • For each condition, the old policy generates K rollouts, which are re-noised before current and old policies predict velocities from the same latent, timestep, and editing condition.The current and old predictions are vθ and vold, respectively.Source: B DIFFUSIONNFT POLICY UPDATE · p. 16
  • β controls how far each implicit positive and negative prediction deviates from the old policy prediction.Source: B DIFFUSIONNFT POLICY UPDATE · p. 16
  • The reward-derived weight ρk balances positive- and negative-branch reconstruction losses, and training updates a new LoRA adapter while softly moving the old policy toward it.The setup uses K = 12 rollouts per condition, β = 1, and η = 10−4.Source: B DIFFUSIONNFT POLICY UPDATE · p. 16

C.1 CANVAS INSTRUCTION SYNTHESIS

Canvas instructions are rendered online from stored edit pairs and masks, with varied marks and notes expressing the selected operation and edit direction.

C.1 CANVAS INSTRUCTION SYNTHESIS

  • Stored records pair source and target images with object masks and structured edit fields; bidirectional sampling reverses images and updates the instruction fields when applicable.Addition and removal exchange object-present and object-absent images, while movement reverses source and destination masks and the arrow.Source: §C.1 · p. 16
  • The renderer applies shared crops and resizes, draws circles beyond mask boundaries, and combines operation-specific marks and nearby notes into canvas instructions.Annotations are rendered on both the source image and a gray background; the target remains unannotated.Source: §C.1 · pp. 16–17, Table 12 · p. 17
  • Movement uses a red source circle and green destination circle connected by an arrow, while removal uses a circled scribble and may include “remove it.”Notes remain part of the canvas annotations rather than a separate text prompt.Source: §C.1 · p. 17
  • Rendering varies circle spacing, shape, width, opacity, stroke overlap, and note typography and placement to create different instructions for the same edit pair.The annotation alpha mask is used to construct Mann for the region-weighted loss.Source: §C.1 · p. 17

C.2 DATASET STATISTICS

The dataset-statistics section describes the supervised corpus, reinforcement-learning set, and benchmark, including how their counts are defined.

C.2 DATASET STATISTICS

  • The RL set contains 3,520 conditions, with 704 per edit type, and the VibeEdit benchmark contains 419 cases across five edit types.The benchmark cases also include external text prompts averaging 21.26 words for text-instructed baselines.Source: §C.2 · p. 17, Table 13 · p. 17
  • Supervised-learning counts refer to stored image pairs, exclude online reversals and randomized canvas renderings, and count each bidirectional addition/removal record once.Source: §C.2 · p. 17, Table 13 · p. 17
  • The composition table reports the supervised corpus, RL set, and benchmark by edit type.Source: Table 13 · p. 17, §C.2 · p. 17

D QUALITATIVE RESULTS AND VISUALIZATIONS

The appendix presents qualitative edits and documents the selection and judging rules used in the data and evaluation pipelines. Its rubric checks edit completion, preservation, and local quality.

D QUALITATIVE RESULTS AND VISUALIZATIONS

E SYSTEM PROMPTS

  • The system-prompt appendix retains rules needed to understand and reproduce pipeline stages while omitting API syntax, exhaustive output schemas, and retry logic.Source: E SYSTEM PROMPTS · p. 19

E.1 OBJECT SELECTION RULES

  • Selection rules favor visible, well-bounded, localized, verifiable targets and reject ambiguous cases; attribute edits preserve identity and geometry, while movement requires a clear destination.The rules also specify operation-appropriate replacement objects and constrain attribute changes to one family.Source: §E.1 · p. 19

E.2 PER-CRITERION RUBRIC JUDGE PROMPT

  • Each rubric criterion is judged in a separate binary request using shared visual evidence, with full images for context and aligned crops for local comparison.The protocol distinguishes edit success, outside preservation, and local quality, and permits minor boundary spillover while rejecting unrelated changes.Source: §E.2 · pp. 19–20

E.3 COMPLETE RUBRIC QUESTIONS

  • The rubric covers edit success, outside preservation, and local edit quality, including checks for coherent geometry, natural scene integration, and local aesthetics.It also checks for unrelated edits and global style drift.Source: §E.3 · pp. 20–21
Loading 2610.12229v1…