PaperlayerView exact revision on arXiv

Source-linked AI summary

When Text-to-Image Helps Editing: The Effects of Conditioning During Denoising

Lidia Troeshestova, Alexander Ustyuzhanin, Sergey Kastryulin

arXiv:2610.01681v1cs.CV

TL;DR

Standard editing keeps source-image conditioning throughout denoising, raising the question of whether every edit needs it at every stage. The paper tests bounded switches to caption-only T2I and reports improved edit quality across models and benchmarks with judged preservation close to pure editing.

  • Problem

    Standard editing keeps source-image conditioning throughout denoising, though persistent anchoring can suppress intended changes when the target scene diverges from the input.

    Source: §1 · pp. 1–2

  • Method

    The authors temporarily replace source-image and instruction conditioning with a target caption for a bounded interval, then restore the editing inputs.

    Source: §1 · p. 2

  • Results

    Across three models and four benchmarks, bounded T2I intervals improve edit quality; [10, 16) raises all six quality scores on all three models while judged preservation stays close to pure editing.

    Source: §1 · p. 2, §4.1 · p. 5, §4.4 · p. 8

  • Takeaways & Limitations

    Unified editors can use both trained conditioning modes within an edit, with switch timing setting the balance between quality and preservation.

    Source: §5 · p. 9

    Whether image-similarity and GPT-judge preservation measures track human judgments remains untested.

    Source: §5 · p. 10

Abstract

from arXiv · show

Unified models are trained for both instruction-based image editing and text-to-image (T2I) generation, but standard editing pipelines keep source-image conditioning throughout denoising. We ask whether editing can benefit from T2I, and study how the effects of conditioning vary across edits and denoising stages. In pure editing, source attention declines for some edits over the sampling trajectory. This observation led us to task switching, which lets the model draw on its T2I capabilities. Across three unified editors and four benchmarks, switching to the T2I task for bounded intervals improves edit quality, while mean perceptual preservation remains close to pure editing across all three models. Unified editors therefore benefit from using both conditioning modes they are trained for, and the timing of the switch sets the balance between quality and preservation.

1 INTRODUCTION

Unified models can both generate images from captions and edit source images, but standard editing keeps source conditioning throughout denoising. The paper asks whether drawing on T2I during an edit can improve changes without sacrificing preservation.

1 INTRODUCTION

  • Standard editing uses source-image conditioning throughout denoising, leaving a unified model’s T2I capability unused within the same trajectory.The two tasks share a model but ordinarily use different inputs: a source image and instruction for editing, versus a caption for T2I generation.Source: §1 · pp. 1–2
  • Persistent source anchoring can suppress intended changes, causing under-execution or poorly integrated edits when the target scene diverges substantially from the input.The paper asks whether every edit benefits from source conditioning at every denoising stage.Source: §1 · p. 2
  • Source attention varies by edit and denoising stage: some edits attend progressively less to the source, while localized changes retain strong attention late.This observation motivates testing temporary removal of source conditioning.Source: §1 · p. 2
  • The proposed switch replaces source-image conditioning and the editing instruction with a target caption for a fixed interval, then restores the original editing inputs.The model weights and number of denoiser forward passes remain unchanged; the illustrated switch helps integrate added objects, while returning to editing preserves more of the room.Source: Figure 1 · p. 1, §1 · p. 2
  • Switching to T2I for bounded intervals improves edit quality across three models and four benchmarks while keeping preservation virtually the same.The study also varies interval placement and duration, conditioning inputs, and guidance formulation.Source: §1 · p. 2

2 RELATED WORK

Related work studies attention control, temporal changes to conditioning, and handoffs between generation and editing. This paper changes conditioning inputs within one unified model’s denoising trajectory and measures quality and preservation.

Attention and source conditioning

  • Prompt-to-Prompt limits reuse of source-generation cross-attention maps [Hertz et al., 2022], SaaS scales instruction attention, and GRAG modulates source-image and instruction-token influence [Zhang et al., 2025].The paper instead changes conditioning inputs and measures outcomes, using attention maps as diagnostics.Source: §2 · p. 2

Temporal conditioning

  • Prior temporal-conditioning work varies denoisers, guidance, or prompts across sampling stages [Balaji et al., 2022]; this paper switches between editing and caption-only generation.The cited work includes concept-specific temporal profiles that carry over to editing.Source: §2 · p. 2

Combining generation and editing

  • HiDream-E1 hands an editing latent to a generator, while RvR regenerates images without source VAE tokens [Guo et al., 2026].This study examines source conditioning, paired text, and interval timing within unified editors.Source: §2 · pp. 2–3

3 SOURCE CONDITIONING ALONG EDITING TRAJECTORIES

The section examines how source-image use varies across edits and denoising, compares editing and T2I predictions, and describes interventions that switch conditioning within one trajectory.

3.1 SOURCE ATTENTION VARIES ACROSS EDITS AND TIMESTEPS

  • Source attention depends on edit type: in the examined examples, background replacement attends to the kept foreground, lighting change attends strongly throughout, and makeup attends weakly throughout.The measurements use 50-step pure-editing trajectories on FLUX2-Klein 4B; per-patch attention is summed over source-image tokens and averaged across heads and transformer blocks.Source: §3.1 · p. 3, Figure 2 · p. 3
  • Source-attention mass generally decreases during denoising, but some local edits retain high attention through the final steps.Where attention has faded, source conditioning constrains without contributing much; the paper asks where a T2I phase would disturb the edit least.Source: §3.1 · p. 3

3.2 COMPARING EDITING AND T2I PREDICTIONS

  • At each latent state of a pure-editing trajectory, the study compares editing and target-caption predictions; their agreement and norm ratio vary across denoising stages.The diagnostic does not alter the sampled trajectory; the authors test switches in the comparatively stable middle region.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 CONDITIONING INTERVENTIONS AND CONTROLS

  • The intervention runs T2I with a VLM-generated target caption either from step k onward or on a bounded interval [k1, k2), then resumes editing for the bounded schedule.The same latent continues denoising, with only one mode active per step; the controls vary whether the source image or text changes, and include a VLM-improved instruction.Source: §3.3 · p. 4, Table 1 · p. 4

4 EFFECTS OF CONDITIONING AND INTERVAL TIMING

Across three unified editors and four benchmarks, bounded T2I intervals improve edit quality while judged preservation remains close to pure editing. The effects depend on what conditioning is changed and when the interval occurs; selective routing shifts the balance toward preservation.

4.1 TASK SWITCHING IMPROVES EDITING QUALITY

  • The six-step [10, 16) interval raises all six quality scores on all three models while GPT-4.1 judged preservation stays within a fraction of a point of pure editing.The evaluation covers GEdit-Bench, ImgEdit, MagicBrush, and GIE-Bench.Source: §4.1 · p. 5, §4 · p. 5, Table 2 · p. 6

4.2 WHAT ACCOUNTS FOR THE IMPROVEMENTS?

  • Removing the source image while using a target caption permits larger gains in fidelity, naturalness, and artifact scores; neither change alone accounts for the improvement.Removing the image while retaining only the editing instruction, or pairing the caption with the source image, loses semantic consistency.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Figure 6 · p. 8
  • The best T2I interval beats improved-instruction-only editing on every model in functional correctness, MagicBrush alignment, and perceptual quality, while continuous blending can improve some scores at four denoiser passes per step.The improved instruction is a strong GEdit baseline on the two FLUX models, but its quality advantage does not extend to BAGEL.Source: §4.2 · pp. 6–7

4.3 WHEN DOES SWITCHING HELP?

  • Earlier or longer T2I intervals trade semantic consistency for perceptual quality, and only short early-middle intervals improve both scores.Every duration sweep returns to the pure-editing point as the interval slides toward the trajectory's end.Source: §4.3 · pp. 7–8, Figure 7 · p. 8

4.4 QUALITY AND SOURCE PRESERVATION

  • Across the three models, fixed switching shifts mean GPT-4.1 judged preservation by at most 0.16 points on a ten-point scale, with no significant shifts.The featured intervals gain correctness at judged preservation indistinguishable from pure editing, although early one-way switches that gain the most correctness pay for it.Source: §4.4 · p. 8, Figure 8 · p. 9
  • Across the 4B sweep, SSIM varies while judged preservation stays at the pure-editing level until SSIM falls below it, because the metrics track different changes.SSIM tracks pixel displacement; the instruction-aware judge is presented as closer to what a viewer would notice.Source: §4.4 · pp. 8–9

4.5 SELECTIVE APPLICATION AS AN ILLUSTRATION

  • The selective rule keeps edits with strong or uneven source attention in pure editing and switches the rest, using statistics from the ordinary edit pass at no extra denoiser cost.Thresholds are tuned on GEdit and then shared across benchmarks.Source: §4.5 · p. 9
  • Selective switching retains quality gains on every score and model, improves SSIM significantly on all three models, and moves judged preservation closer to pure editing than fixed switching.It does so at the price of somewhat smaller quality gains; against random routing at equal mean preservation, the attention rule wins on 4B and 9B.Source: §4.5 · p. 9

5 DISCUSSION AND CONCLUSION

Task switching uses a unified editor’s T2I mode within an edit, improving quality while keeping judged preservation essentially unchanged. The discussion notes unresolved limits in the switching schedule and preservation evaluation, and provides reproducibility pointers.

5 DISCUSSION AND CONCLUSION

  • Switching to T2I within an edit improves quality across tested models and benchmarks while judged preservation remains essentially unchanged.The switch uses the model’s T2I mode during part of the trajectory and returns to editing.Source: §5 · p. 9
  • The interval sweeps are coarse, and preservation metrics’ alignment with human judgments has not been tested.The authors also note that captions prepared before sampling cannot react to the trajectory and that caption errors can cause unintended changes.Source: §5 · p. 10
  • The authors report AI-assistant contributions and provide code, notebooks, and pointers to sections describing schedules, evaluation, and extended results.They state that they verified citations, experimental code, and numerical outputs and take responsibility for the final paper.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL QUALITATIVE EXAMPLES

The appendix adds 15 GEdit examples organized around failures of pure editing, illustrating missed, incorrect, unnatural, and incomplete changes.

A ADDITIONAL QUALITATIVE EXAMPLES

  • The appendix adds 15 GEdit examples grouped by failure type under pure editing.The groups are missed edits, incorrect edits, unnatural results, and incomplete edits.Source: A ADDITIONAL QUALITATIVE EXAMPLES · p. 12
  • The examples compare the source and pure-editing outputs with a target caption, a step-10 T2I switch, and a bounded return to editing.All use FLUX2-Klein 4B with 50 denoising steps.Source: Figure 5 · p. 12
  • Pure editing largely fails to execute the requested change in the missed-edit example.The comparison includes a one-way T2I switch from step 10 and a return to editing over [10, 16).Source: Figure 9 · p. 12
  • In the incorrect-edit example, pure editing changes the image but does not produce the requested result.The columns follow the comparison layout used for the missed-edit example.Source: Figure 9 · p. 12, Figure 10 · p. 13
  • Other examples show unnatural appearance or artifacts, and an incomplete transformation, under pure editing.The first four rows show unnatural appearance or artifacts; the final row shows an incomplete transformation.Source: Figure 11 · p. 14

B EXTENDED EDITING RESULTS

The extended results show that removing image conditioning yields the highest perceptual quality in interval and single-switch sweeps, while text schedules and interval choices affect other scores. The appendix documents the conditioning sweeps, benchmark coverage, and how to interpret their tables.

B EXTENDED EDITING RESULTS

C CONTINUOUS GUIDANCE COMPARISONS

The section compares reducing image guidance with continuously blending editing and target-caption generation. On FLUX2-Klein 4B, their effects differ across intervals, and intermediate multimodal guidance requires more denoiser passes than either endpoint.

C CONTINUOUS GUIDANCE COMPARISONS

  • Multimodal guidance interpolates between editing and target-caption-conditioned generation, with α = 0 recovering E →T2I →E and α = 1 recovering pure editing.It applies the interpolation on a bounded interval and retains edit conditioning elsewhere.Source: C CONTINUOUS GUIDANCE COMPARISONS · p. 20
  • Lowering image guidance changes GEdit overall by at most +0.06 and reduces ImgEdit on either interval; applying it throughout sampling lowers both further.Even at zero image guidance, image-conditioned predictions still enter the update, so this control is distinct from caption-only T2I.Source: C CONTINUOUS GUIDANCE COMPARISONS · p. 20
  • Intermediate multimodal guidance requires four denoiser passes per step, compared with two at the editing and T2I endpoints; image guidance with α < 1 requires three.Figure 12 shows multimodal guidance along the diagonal of the weight plane and image guidance along its rightmost column.Source: C CONTINUOUS GUIDANCE COMPARISONS · p. 20, Table 11 · p. 20, Figure 12 · p. 21
  • At [25, 35), the pure T2I endpoint has the highest GEdit overall, ImgEdit, CLIP-T, and functional-correctness scores among multimodal settings, while intermediate weights retain more source structure.At [10, 20), α = 0.25 improves GEdit semantic consistency and overall score, GIE-Bench functional correctness, and preservation over the pure T2I interval, but reduces perceptual quality, ImgEdit, and CLIP-T.Source: C CONTINUOUS GUIDANCE COMPARISONS · p. 20, Table 12 · p. 22

D END-TO-END TIMING AND CAPTIONER CHOICE

Task switching reduces image-generation time on FLUX models by omitting source-image tokens without changing the denoiser pass count. Total latency depends on the captioner, while evaluated Qwen alternatives retain quality gains over pure editing with a similar preservation cost.

D END-TO-END TIMING AND CAPTIONER CHOICE

  • Switching reduces image-generation time on FLUX models because T2I omits source-image tokens while keeping the same denoiser pass count as editing.With Qwen captioners, total time stays close to pure editing; the Gemini API call dominates measured runtime.Source: D END-TO-END TIMING AND CAPTIONER CHOICE · p. 23, Table 13 · p. 23
  • At the same [10, 20) interval, evaluated Qwen alternatives retain improvements over pure editing in GEdit overall, ImgEdit, MagicBrush alignment, and GIE-Bench correctness, with a similar preservation cost.A faster and cheaper captioner can retain the task-switching benefit while avoiding the large captioning overhead.Source: D END-TO-END TIMING AND CAPTIONER CHOICE · p. 23, Table 14 · p. 23

E SELECTIVE-RULE SETTINGS

The selective rule uses a GEdit-derived operating point and transfers it to other models using probe distributions. On FLUX2-Klein 4B and 9B, it outperforms preservation-matched random routing on overall, perceptual-quality, and fidelity scores.

E SELECTIVE-RULE SETTINGS

  • The shared rule probes at k = 9 before the [10, 16) T2I interval, averaging source attention across heads and transformer layers before computing spatial CV and the 75th percentile π.Attention patterns grow more differentiated over steps 0–9.Source: E SELECTIVE-RULE SETTINGS · p. 23
  • For 4B, 8 of 111 GEdit threshold rules met the target fidelity gain of at least 0.25 and preservation loss of at most 0.05; (0.50, 0.22) was selected.This was one feasible operating point.Source: E SELECTIVE-RULE SETTINGS · p. 23
  • For 9B, the CV threshold was retained and π adjusted to approximately match 4B’s gate and overall switching rates; for BAGEL, thresholds matched each gate’s exceedance rate separately to 4B.The BAGEL exceedance rates were 13.5% for CV and 18.5% for π; ImgEdit, MagicBrush, and GIE-Bench were not used to select thresholds.Source: E SELECTIVE-RULE SETTINGS · pp. 23–24
  • On FLUX2-Klein 4B and 9B, the attention rule exceeds preservation-matched random routing on overall, perceptual-quality, and fidelity scores; the margin is larger on 4B than 9B.BAGEL was not included in this replay.Source: E SELECTIVE-RULE SETTINGS · p. 24, Table 15 · p. 24

F PRESERVATION ANALYSIS

Preservation is assessed with a judge score and SSIM, using paired comparisons on GIE-Bench. Selective switching recovers some SSIM loss relative to fixed switching, while its GPT-preservation score is not significantly lower than pure editing.

F PRESERVATION ANALYSIS

  • The judge scores unnecessary changes to unrelated content from 0 to 10, without penalizing changes needed to follow the instruction; 0 indicates substantial unrelated changes and 10 indicates all unrelated content is preserved.The protocol uses GPT-4.1 and adapts GEdit’s semantic-consistency evaluation by retaining preservation assessment and removing fidelity assessment.Source: F PRESERVATION ANALYSIS · p. 24
  • GIE-Bench preservation comparisons pair edits by identifier and equally weight per-edit score differences; confidence intervals and significance tests cluster standard errors by source image.The evaluation uses GPT scores and SSIM outside the edit mask.Source: F PRESERVATION ANALYSIS · p. 24
  • Selective switching significantly improves SSIM over fixed [10, 16) on all three models and has no significant GPT-preservation reduction relative to pure editing; every fixed and selective configuration significantly lowers SSIM versus pure editing.Only the 4B [25, 35) interval shows a significant mean GPT-preservation reduction after correction.Source: Table 16 · p. 25, F PRESERVATION ANALYSIS · p. 25
  • Figure 8 plots GPT-4.1 preservation and Gemini-3-Flash correctness by configuration, while Table 16 supplies paired comparisons against pure editing and between fixed and selective switching.Figure error bars are ordinary 95% confidence intervals for individual preservation means; Table 16 uses paired differences and simultaneous intervals.Source: F PRESERVATION ANALYSIS · pp. 24–25, Table 16 · p. 25

Related papers

Loading 2610.01681v1…