PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-Aware Background-Controlled Editing via Weighted Velocity Guidance

Wuji Wang, Yue Wu, Chenhao Yi, Shuhui Wang

arXiv:2609.06288v1cs.CV

TL;DR

Prompt-induced velocity residuals can accumulate outside an edited object, causing background drift and unstable boundaries. OAVC adds object-level control to velocity integration, and experiments report improved preservation and localized editing across image and video benchmarks.

  • Problem

    Velocity-based editors often apply prompt-induced residuals globally, while object-centric edits need target changes without drift in non-target regions.

    Source: §1 · pp. 1–2

  • Method

    OAVC builds a source-prompt, background-anchored reference interface and applies object-localized semantic updates with constrained velocity injection.

    Source: §1 · p. 2

  • Results

    Across image and video benchmarks, OAVC improves preservation and localized editability; on PIE-Bench FLUX, Structure Distance falls from 18.87 to 4.07 versus DNAEdit.

    Source: §1 · p. 3, §4.2 · p. 9, §6 · p. 20

  • Takeaways & Limitations

    OAVC’s where/how velocity-control principle also transfers to another rectified-flow editor, according to the reported experiments.

    Source: §1 · p. 3

    OAVC depends on meaningful object support and is not designed for full-image transformations such as global style transfer or overall color grading.

    Source: §5.7 · p. 20

Abstract

from arXiv · show

Training-free image editing steers diffusion or flow-matching generative models at inference time by modifying prompt-conditioned denoising velocities. Existing velocity-based editors often apply prompt-induced residuals globally over the latent space and rely on the model to localize semantic changes implicitly. For object-centric edits, these residuals are rarely zero outside the target object, so small non-target components can accumulate during multi-step integration, causing background drift and unstable object boundaries. We propose Object-Aware Velocity Control (OAVC), a training-free framework that introduces object-level control into the velocity-integration process. OAVC decouples where semantic residuals are allowed to act from how they are injected into the dynamics. It constructs a background-anchored reference interface under the source prompt and then performs object-localized safe semantic injection under the target prompt. A constrained injection operator suppresses drift-inducing velocity components, while time-adaptive spatial weighting stabilizes the transition near object boundaries. OAVC requires no training or modification of pretrained model parameters. Experiments on object-centric image and video benchmarks with image and video rectified-flow backbones show improved background preservation, structural fidelity, boundary stability, and temporal consistency while retaining effective localized editability.

1 Introduction

Training-free velocity editors can let prompt-induced changes leak beyond the target object, undermining preservation during object-centric edits. OAVC introduces object-level control during velocity integration and reports improved preservation and editing outcomes on image and video benchmarks.

1 Introduction

  • Prompt-derived spatial cues can be imprecise, letting updates leak into backgrounds, miss object regions, or destabilize boundaries.Across solver steps, non-target residuals may accumulate into background drift and layout changes.Source: §1 · pp. 1–2
  • OAVC separates where prompt-induced residuals may accumulate from how they are injected, using an object support and suppressing source-flow-aligned components.It first builds a background-anchored reference interface under the source prompt, then injects target semantics through object-localized constrained updates.Source: §1 · p. 2
  • The object support is a velocity-level prior, with SAM3 as the default provider and user-provided or brush-refined supports also permitted.The authors distinguish localization from their contribution by using same-support and support-robustness studies to examine the trajectory controller.Source: §1 · p. 2
  • On PIE-Bench and DAVIS, OAVC improves background preservation, structural fidelity, boundary stability, and temporal consistency while retaining localized editability.The authors also report transfer of the same control principle to another rectified-flow editor.Source: §1 · p. 3

2 Related Work

Prior training-free editors modify attention, internal activations, or sampling trajectories, but often leave object localization implicit. OAVC instead brings object-level support and constrained injection directly to velocity integration.

2 Related Work

  • Training-free methods steer semantics by modifying cross-attention or internal activations [3] [7] [21], while trajectory-based editors inject prompt-induced velocity differences along multi-step integration in diffusion or flow-matching models [14] [28] [31].Source: §2 · p. 3
  • Global semantic velocity perturbations can accumulate outside the target, causing background drift and boundary instability in object-centric edits.Existing stabilization mechanisms do not explicitly constrain where and how velocity updates are spatially integrated.Source: §2 · p. 3
  • Unlike attention-level localization or background KV reuse [33], OAVC inserts an object-level prior at the editor/solver interface to constrain velocity integration.This approach builds on spatial-control work [20] and zero-shot object masks [2] [13].Source: §2 · pp. 3–4

3 Method

OAVC controls both where semantic residuals accumulate and how they are injected during rectified-flow velocity integration. It builds a source-anchored reference trajectory, then applies object-localized safe updates.

3 Method

  • OAVC introduces object-aware where/how control into velocity integration, first constructing a source-prompt reference interface and then injecting target semantics with localized safe updates.The controlled update separates the spatial weighting of residuals from their transformation before integration.Source: §3 · p. 4, §3.1 · p. 5

3.1 Preliminaries

  • Global steering can accumulate non-target residuals over solver steps; OAVC formulates editing as a controlled velocity update with separate spatial support and residual transformation.The reference interface provides aligned source trajectory information for comparing source and target velocities.Source: §3.1 · p. 5

3.2 Stage-1: Background-Anchored Reference Interface

  • Stage-1 builds a source-consistent reference without semantic editing, anchoring the background while retaining object-region freedom for later changes.The foreground coefficient λ_fg controls editability: larger values leave more freedom for semantic injection; background residual suppression removes source-flow-parallel components.Source: §3.2 · pp. 5–6

3.3 Stage-2: Object-Localized Safe Semantic Injection

  • Stage-2 computes target semantic residuals on aligned states, applies safe injection, and integrates updates only on the object support.Unlike masked blending after generation, this constrains the velocity field throughout the trajectory; projection, boundary smoothing, and confidence normalization stabilize injection.Source: §3.3 · pp. 6–7

4 Experiments

Experiments evaluate OAVC on object-centric image and video editing with rectified-flow backbones, comparing preservation and editability against training-free editors. Results show stronger image and video preservation, while ablations examine the where/how controls and hyperparameter robustness.

4.1 Setup

  • The evaluation uses PIE-Bench object-centric edit tuples and DAVIS mask-conditioned video edits, with SD3.5 and FLUX backbones and DNAEdit as the primary baseline.DAVIS uses ground-truth instance masks cached across solver steps; PIE-Bench background metrics use shared annotations for scoring.Source: §4.1 · pp. 8–9

4.2 Main Results

  • On PIE-Bench FLUX, OAVC lowers Struct. Dist. from 18.87 to 4.07 and raises background PSNR from 24.99 dB to 33.30 dB over DNAEdit.The paper reports the same trend on SD3.5: Struct. Dist. falls from 14.19 to 4.11 and PSNR rises from 26.66 dB to 32.64 dB over DNAEdit.Source: §4.2 · p. 9
  • Across diverse image edits, OAVC better localizes changes to target objects and preserves non-target regions and cleaner boundaries than global-steering baselines.For consecutive-frame DAVIS removal, OAVC constrains residuals within object support, yielding more stable removal and surrounding-scene preservation.Source: §4.2 · p. 10, Figure 5 · p. 10
  • On DAVIS versus UniEdit-Flow, OAVC improves BG-PSNR by 4.66 dB, reduces BG-L1 by 39.2%, improves TW-BG-PSNR by 1.68 dB, and reduces TW-BG-L1 by 6.7%.Evaluation uses four sequences and the first 41 frames; BG metrics measure per-frame fidelity and TW-BG metrics measure temporal background stability.Source: §4.2 · pp. 9–10, Table 3 · p. 10

4.3 Ablation Study

  • Stage-2 provides the major structure and background-preservation gains; support masking contributes most, the boundary ring adds a smaller consistent gain, and removing projection or gating degrades preservation.With identical support, full OAVC reduces Structure Distance by 22.0% versus mask-only gating and 34.6% versus final latent blending, while matching the latter’s LPIPS/MSE within 0.6%.Source: Table 2 · p. 9, §4.3 · p. 11
  • Across SD3.5 and FLUX, moderate foreground relaxation and small background suppression give stable LPIPS/MSE; defaults are 𝜆fg=0.6 and 𝜌bg=0.1.The study sweeps foreground relaxation and background suppression as Stage-1 hyperparameters.Source: §4.3 · p. 11
  • On FlowEdit-FLUX, OAVC reduces Struct. Dist. from 27.82 to 9.32 and raises PSNR from 21.96 dB to 30.77 dB; on SD3, PSNR rises but Struct. Dist. worsens.The SD3 changes are PSNR 22.22 dB to 27.65 dB and Struct. Dist. 27.12 to 35.72; the authors note that stable structure preservation also benefits from Stage-1’s reference interface.Source: Table 4 · p. 10, §4.3 · p. 11

5 Extended Evaluation and Analysis

Controlled tests attribute preservation gains both to explicit object support and to OAVC’s in-trajectory control, with human judgments favoring its background preservation. The analyses also characterize support sensitivity, localized editability, compositional scope, computational overhead, and limitations.

5.1 Controlled Same-Support Baselines and Human Evaluation

  • With identical support, OAVC reduces Structure Distance to 4.11 from 5.27 with mask-only velocity gating and 6.28 with final latent blending.It matches final blending’s BG LPIPS and BG MSE within 0.6% while improving Structure Distance by 34.6%.Source: §5.1 · pp. 11–12, Table 5 · p. 12
  • OAVC wins 74 of 94 decisive background-preservation comparisons (78.7%), while edit-realization preferences are not significant at 0.05 in this study.The absolute Yes-or-Partially task-success rate is 79.5% for OAVC versus 80.0% for DNAEdit.Source: Table 6 · p. 12, §5.1 · p. 12

5.2 Robustness to Input-Support Corruption

  • Erode-2 changes CLIPedit by only 0.86% (21.47 to 21.29) yet empties 6/100 supports; Dilate-4 raises BG LPIPS 24.7% (23.15 to 28.88) and BG MSE 28.4% (8.08 to 10.38).Erosion limits edit coverage, whereas dilation exposes more non-target regions to semantic residuals; aggregate metrics can understate incomplete-edit failures.Source: §5.2 · pp. 13–14, Figure 6 · p. 14

5.3 Qualitative Generalization Check on GEdit-Bench

  • On two GEdit-Bench examples, OAVC localizes object replacement and removal more closely than DNAEdit while preserving more non-target content.These examples are a qualitative generalization check, not benchmark-wide quantitative evaluation.Source: §5.3 · p. 14, Figure 7 · p. 15

5.4 Editability–Preservation Trade-off and CLIP Similarity

  • OAVC reduces the background leakage ratio from approximately 57.5% for DNAEdit to 6.3%, while foreground target gain falls from 0.0189 to 0.0130.The reported trade-off reflects target semantics being concentrated in the foreground rather than leaked into non-target regions.Source: Table 8 · p. 16, §5.4 · pp. 16–17

5.5 Multi-Object and Compositional Editing

  • A representative goat-and-cat to horse-and-dog edit changes both target objects while more closely preserving the original layout and non-target background.The shared velocity-control mechanism can use multiple object supports, but broader evaluation of correspondence and interactions remains future work.Source: §5.5 · pp. 18–19, Figure 9 · p. 19

5.6 Efficiency and Overhead

  • OAVC adds modest runtime and memory overhead over DNAEdit on SD3.5 and FLUX; SAM3 masks are computed once per image or video and cached across solver steps.The reported overhead primarily reflects OAVC’s sampling-time control logic rather than repeated segmentation.Source: §5.6 · pp. 19–20, Table 9 · p. 20

5.7 Limitations and Scope

  • OAVC requires a meaningful localized object support; near-global supports remove Stage-2 spatial selectivity, and support errors can cause incomplete edits or non-target drift.The current post-hoc control uses a single final latent hard blend; stronger pixel-space or multi-step compositing controls remain unevaluated.Source: §5.7 · p. 20

6 Conclusion

The conclusion presents OAVC as a training-free framework for object-centric image and video editing that controls where and how prompt-induced residuals accumulate during velocity integration. Evaluations report improved preservation and temporal consistency alongside effective localized editability, while future work targets support sensitivity and more complex compositional editing.

6 Conclusion

  • OAVC combines a background-anchored reference interface with object-localized safe injection to reduce non-target drift and improve structural fidelity and boundary stability.Experiments on PIE-Bench, GEdit-Bench, and DAVIS report improved background preservation and temporal consistency while maintaining effective localized editability.Source: §6 · p. 20

A Details of Algorithm

  • The algorithm is organized as Stage-1 background-anchored reference construction and Stage-2 object-localized safe semantic injection, which outputs an edited latent.The listed inputs include source latent, source prompt, initial noise state, and object support for Stage 1; Stage 2 takes source and target prompts, the reference interface, and a start step.Source: A Details of Algorithm · p. 23

B Implementation of the Safe Injection Operator

The safe injection implementation applies a shared where/how control principle to the reference anchor and the integrated velocity. Object supports gate velocity updates rather than overwriting latents, with boundary support buffering uncertainty and alternative supports available when automatic masks are imperfect.

B Implementation of the Safe Injection Operator

  • The same safe-injection principle is applied to the reference anchor and final integrated velocity; inside the object support, only a gated, flow-orthogonal semantic residual is injected.The flow-orthogonal projection removes the component parallel to the source-flow direction.Source: B Implementation of the Safe Injection… · p. 24

C Object Prior and Mask-Induced Velocity Constraints

  • OAVC uses object support to gate velocity updates during integration, not to overwrite latents or composite the final image.The default prior is SAM3, but the formulation also accepts user-provided, soft, attention-derived, or brush-refined supports.Source: C Object Prior and Mask-Induced… · pp. 25–26
  • The editable interior, boundary ring, and non-target background are constructed from the support; the ring absorbs moderate boundary uncertainty.Fixed boundary coupling is softer at earlier, noisier steps and decays later to avoid unnecessary leakage; parameters are not tuned per image.Source: C Object Prior and Mask-Induced… · p. 25

D Support Source Robustness

OAVC’s velocity-control update is not tied to SAM3: it can use different object-support sources while retaining the same control mechanism. Attention-derived support is coarser than SAM3 but still preserves the advantage over global steering.

D Support Source Robustness

  • Attention-derived support remains substantially better than global DNAEdit on structure and background preservation, despite moderate degradation relative to SAM3.The comparison supports using a usable support prior rather than requiring SAM3 specifically.Source: D Support Source Robustness · p. 27, Table 10 · p. 27

Continuation of Support Source Robustness

  • OAVC can use SAM3, attention-derived support, soft masks, user masks, or brush-refined masks with the same update rule.SAM3 is the default zero-shot support; localization itself is not the claimed contribution.Source: D Support Source Robustness · p. 26, Continuation of Support Source… · p. 27
  • When segmentation is ambiguous, users can refine the support with a brush; attention-derived support offers an internal alternative to an external segmenter.Both alternatives use the same OAVC update rule.Source: Continuation of Support Source… · p. 27
  • Attention-derived supports are coarser than SAM3 masks, yet still reduce background drift and preserve non-target regions better than global steering.Figure 11 shows qualitative examples using attention-derived support.Source: Figure 11 · p. 28

E Full Derivations for the Two-Stage Recursions

This appendix derives the two-stage OAVC recursions as solutions to alignment and constrained-control problems. It explains how Stage 1 produces a stable reference interface for Stage 2 semantic injection.

E.1 Setup: Noise-Parameterized Discrete Dynamics

  • The derivation separates scheduler noise levels from solver step magnitudes: the latter multiply applied velocity and need not follow an analytic relation to the former.The discrete reverse process is expressed as an Euler update using the per-step increment.Source: E Full Derivations for the Two-Stage… · p. 28, §E.1 · p. 28

E.2 DNA Discrepancy from Linear Transport

  • The DNA discrepancy measures the difference between the learned source-prompt velocity and the linear transport velocity implied by rectified-flow interpolation.This discrepancy supplies the alignment correction used in Stage 1.Source: §E.2 · p. 29

E.3 Additional Preliminaries: Control View and Spatial Support

  • Because injected controls accumulate across solver steps, OAVC separates where updates act from how they are transformed into safe injection directions.A spatial support gates the intended region, while a constrained operator removes drift-inducing components.Source: §E.3 · pp. 29–30

F Stage-1 Derivation: Regularized reference-trajectory Alignment

  • Stage 1 weakens the alignment correction inside the editable foreground, preserving freedom for Stage 2 to inject semantics without fighting a rigid foreground anchor.The appendix derives the correction from regularized reference-trajectory alignment.Source: F Stage-1 Derivation: Regularized… · p. 31, §F.1 · p. 31

F.1 Foreground Weakening via Diagonal Tikhonov Regularization

  • Foreground weakening follows diagonal Tikhonov regularization; the main-text linear shrinkage approximates its exact shrinkage and keeps a nonnegative foreground factor when λ_fg is in [0, 1].Naively weighting the same least-squares residual would not change its minimizer.Source: F Stage-1 Derivation: Regularized… · p. 31, §F.1 · p. 31

F.2 Background Anchoring as Soft Parallel-Component Removal

  • Stage 1 partially removes only the background residual component parallel to source flow, leaving the approximately orthogonal correction intact.ρ_bg = 0 retains the original residual, while ρ_bg = 1 gives hard orthogonal projection.Source: §F.2 · p. 32

G Stage-1 Recursion and Stored Offsets

  • The Stage-1 recursion stores the offset between its corrected latent and proxy point, which Stage 2 reuses to evaluate velocities without rerunning Stage 1.The stored offset is the interface between the two stages.Source: G Stage-1 Recursion and Stored Offsets · p. 33

H Stage-2 Derivation: Offset-Aligned and Constrained Semantic Injection

H.1 Offset-Aligned Evaluation and Stability

The appendix details how OAVC stabilizes offset-aligned semantic updates and controls their spatial support. Additional studies examine transfer to FlowEdit and support-geometry trade-offs.

H.1 Offset-Aligned Evaluation and Stability

  • Under a local-Lipschitz assumption on the prompt-conditioned model velocities, evaluating semantic differences at the offset-aligned state makes them stable with respect to offset magnitude.Stage 1’s suppression of spurious, especially background, offset components directly improves Stage-2 semantic-difference stability.Source: §H.1 · p. 33

H.2 First Projection: Transport-Orthogonal Semantic Direction

  • The first projection removes the source-flow-parallel component of the prompt difference, yielding a transport-orthogonal semantic direction for controlled accumulation.It is the constrained least-squares solution under the orthogonality condition.Source: §H.2 · p. 33

H.3 Boundary High-Pass Operator and Confidence Gate

  • An optional boundary-ring high-pass filter suppresses low-frequency drift that can accumulate as halos or contour wobble; an optional confidence gate downweights abnormally large residuals.The gate addresses unreliable projected residuals at large noise levels.Source: §H.3 · p. 34

I Stage-2 Recursion: Reference Constraint and Field-Level Projection

  • The final field is projected again because the intermediate reference-guided velocity is not guaranteed to be source-flow-orthogonal; only its source-orthogonal component is injected inside the object region.This second projection suppresses target-induced background residuals at each step.Source: I Stage-2 Recursion: Reference… · p. 34

J Additional Studies

  • The Stage-1 sweep varies foreground relaxation and background suppression, reporting background-preservation and semantic-alignment metrics around the highlighted default setting.The caption identifies the stable region around λ_fg = 0.6 and ρ_bg = 0.1.Source: Figure 12 · p. 35

J.1 Spatial Supports from the Object Prior

  • The object prior creates an editable interior, a boundary ring, and a time-adaptive reference support used for Stage-2 velocity control.The interior support determines where semantic injection is permitted.Source: §J.1 · pp. 35–36

J.2 Qualitative Transfer to FlowEdit

  • Applying OAVC-style Stage-2 control to FlowEdit qualitatively better localizes edits and reduces non-target changes, though full stability depends on the underlying editor’s alignment quality.The examples compare original, FlowEdit, and controlled FlowEdit outputs.Source: §J.2 · p. 36, Figure 13 · p. 37

J.3 Ablation on Interior Support Radius 𝑟fg

  • The default r_fg = 2 trades stronger aggregate preservation at r_fg = 1 for slightly higher CLIPedit and more spatial freedom in shape-changing edits; expansion to r_fg = 3 adds no semantic gain and weakens preservation.Table 11 reports the interior-support dilation-radius ablation.Source: Table 11 · p. 35, §J.3 · p. 37

J.4 Ablation on Maximum Ring Coupling

The ring-coupling ablation favors moderate boundary coupling, while the boundary-band thickness can vary across a broad range with little metric change.

J.4 Ablation on Maximum Ring Coupling

  • With β_max = 0, removing ring coupling worsens structure and background metrics; the default β_max = 0.45 gives the best overall balance and is robust to nearby values.The uncoupled hard spatial split is consistent with boundary artifacts accumulating during multi-step integration.Source: §J.4 · p. 37

J.5 Ablation on Boundary Band Thickness 𝑟bd

  • Varying r_bd yields nearly identical metrics, while removing the explicit ring slightly degrades stability-related measures, indicating the band matters more than its precise thickness.Table 13 studies the boundary-band dilation radius.Source: §J.5 · p. 37

K More Qualitative Results

Additional qualitative results show cleaner localized image edits and improved video consistency, alongside weak-benefit cases for OAVC. The video experiments use Wan2.1 on DAVIS.

K More Qualitative Results

K.1 More Qualitative Results for Image Editing

  • Across diverse PIE-Bench edits, OAVC better confines changes to the intended region and preserves non-target content, boundaries, structure, and human identity-related characteristics.Figs. 16, 17, and 19 provide additional image-editing examples.Source: K More Qualitative Results · p. 37, §K.1 · p. 38

K.2 More Qualitative Results for Video Editing on DAVIS

  • On challenging DAVIS sequences, OAVC reduces temporal drift and improves frame-to-frame consistency while maintaining original motion patterns and scene layout versus global-steering baselines.The video experiments use Wan2.1; all experiments ran on one 48 GB NVIDIA GPU.Source: K More Qualitative Results · p. 38, §K.2 · p. 39

L Additional Limitation Visualizations

OAVC depends on localized object support and is not designed to be optimal for full-image transformations. The section also identifies limits in compositing control, metric localization, and human-study analysis.

L Additional Limitation Visualizations

  • When the object support approaches a global mask, Stage-2 loses spatial selectivity and OAVC’s improvements rely on a non-trivial localized support.The stated example is M_fg ≈ 1.Source: L Additional Limitation Visualizations · p. 39
  • At t ≈ 1, controlled injection becomes global velocity steering, behaves similarly to DNAEdit, and offers limited advantage for inherently global edits.Source: L Additional Limitation Visualizations · p. 39
  • For near-global watercolor style transfer, object/background separation is ill-defined, making background anchoring less informative and boundary-aware control less effective.Figs. 14 and 15 illustrate representative weak-benefit cases.Source: L Additional Limitation Visualizations · p. 39
  • OAVC is not designed to be optimal for full-image transformations such as global style transfer, color grading, or domain-level appearance changes.The paper suggests alternative priors and revised constraints for these cases.Source: L Additional Limitation Visualizations · p. 39
  • The single final-latent hard blend, global Structure Distance metric, and underpowered per-category human-study analysis leave compositing strength, localization, and edit-category outcomes unresolved.The aggregate human-study results should not be interpreted as equivalence in edit strength.Source: L Additional Limitation Visualizations · p. 39
  • Supplemental figures compare image-editing methods and Stable Diffusion and FLUX, while the DAVIS video comparison reports reduced background drift and improved temporal consistency.Source: Figure 16 · p. 40, Figure 18 · p. 41, Figure 17 · p. 41
Loading 2609.06288v1…