Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing

1 INTRODUCTION

• Prior inference-time interventions use fixed, empirically chosen coefficients that may need retuning and do not explicitly couple edit-region enhancement with keep-region suppression.Source: §1 · p. 1
• Reference-attention profiles vary by approach: some show a localized edit-versus-keep gap, while LoomVideo assigns less than 1% of attention mass to the reference for edit queries.These patterns motivate selecting layers for intervention based on the measured attention profile.Source: §1 · pp. 1–2
• RefGAP applies positive reference-logit offsets to edit queries and negative offsets to keep queries, deriving per-layer strength online from measured attention mass.The correction uses attention statistics already produced during a forward pass, reducing the need for model- or task-specific tuning.Source: §1 · pp. 1–2
• RefGAP improves identity similarity across seven approaches in head swapping and transfers without retuning to face swapping and virtual try-on, while background-replacement results are mixed.The shared configuration is calibrated on four development approaches using 40 head-swap validation clips.Source: §1 · p. 2

2 RELATED WORK

• In-context editors place reference and content tokens in joint attention, whereas decoupled methods inject reference features through separate branches; RefGAP targets the in-context setting.In that setting, relative reference-token mass can be modified directly within the shared softmax.Source: §2 · p. 2
• Prior training-free steering includes reference-attention enhancement, multi-reference attention with concept masks, and reference/self-attention mixing controlled by a user-specified scalar.Examples include Diptych Prompting, FreeCustom, and RefDrop [Fan et al., 2024].Source: §2 · p. 2
• Existing controls commonly use fixed or externally selected coefficients; RefGAP instead measures edit- and keep-query reference attention to set positive and negative logit offsets online.The same coefficients transfer after a single calibration without further manual tuning.Source: §2 · p. 2

3.1 OVERVIEW

• The inference-time pipeline measures reference attention in the masked edit region and its keep-region complement, then applies reference-logit shifts at selected layers without weight updates.The edit mask is mapped to the latent token grid; regional mass is measured at each attention layer and denoising step.Source: §3.1 · p. 3

3.2 REFERENCE ATTENTION MASS AND DIAGNOSIS

• Across five approaches, edit-region reference attention falls below the token-share baseline, VACE is the exception, and Qwen-Image-Edit and OmniGen2 attend more to references in keep than edit queries.JoyAI-Video-Edit is excluded because its reference-token share varies with the streaming KV cache.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 TWO-SIDED REFERENCE CORRECTION

• RefGAP adds a shared offset to reference-key logits, positive for edit queries and negative for keep queries, with magnitude adapted to measured regional reference mass.For keep queries, the negative-offset magnitude decreases as reference mass increases; the heuristic does not distinguish necessary changes beyond the mask from unwanted leakage.Source: §3.3 · p. 4

3.4 LAYER SELECTION

• A fixed rule uses layer-wise edit/keep selectivity and average inverse edit mass to choose correction layers from validation profiles, then reuses those layers unchanged across tasks.Approach-specific profiling is required, but approach-specific strength or threshold tuning is not.Source: §3.4 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation spans seven diffusion editors, head and face swapping, virtual try-on, and background replacement, using task-specific datasets and metrics.Base and RefGAP share inputs, random seeds, and input-derived masks; metrics are macroaveraged over clips.Source: §4.1 · pp. 5–6

4.2 PARAMETER CALIBRATION AND TRANSFER

• An 18-setting joint sweep on 40 head-swap validation clips selected fixed coefficients and thresholds, with each approach’s selected layers reused across tasks without further tuning.The selected values were γedit = 0.6, γkeep = 0.8, τB = 2.7, and τb = 3.7.Source: §4.2 · p. 6

4.3 RESULTS ACROSS APPROACHES AND TASKS

• Across seven approaches, RefGAP improves identity similarity in head and face swapping and virtual try-on fidelity, while background-replacement results vary by approach.Head-swap replacement rates reach at least 86% on every applicable approach; face-swap replacement and correct retrieval rates reach at least 82% and 75%, respectively.Source: Table 2 · p. 7, §4.3 · pp. 7–8

4.4 HUMAN EVALUATION

• Raters preferred RefGAP for reference fidelity in head swap, face swap, and virtual try-on, while pose/expression preservation judgments were mixed.Comparisons were selected for visible differences; ties pooled both-success and both-failure judgments.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 ABLATION STUDIES

• RefGAP matches approach-specific constant edit-side biases without per-approach strength sweeps, while keep-side correction and layer coverage affect preservation outcomes.Omitting keep-side correction raises FLUX.2-klein-base LPIPSne from 0.1206 to 0.2115; the layer rule uses B and ¯bedit to select correction coverage.Source: Table 4 · p. 8, §4.5 · p. 9, Table 5 · p. 9

5 LIMITATIONS

• Delaying correction can reduce keypoint error while retaining identity gains, but the required delay varies by sampling schedule and stronger geometry preservation reduces identity gains.On JoyAI-Video-Edit and FLUX.2-klein, skipping the first correction step substantially reduces keypoint error versus the all-step default.Source: §5 · p. 9
• On Qwen-Image-Edit, RefGAP reduces background fidelity, and restricting layer coverage does not resolve the degradation.The authors suggest this approach-task pair may require a different correction strategy.Source: §5 · p. 9
• Applying RefGAP to a new approach requires a validation profiling run to select layers, adding setup cost despite avoiding repeated correction-strength sweeps.The selected layers are reused across tasks, while coefficients and thresholds remain fixed.Source: §5 · p. 9

6 CONCLUSION

• A single configuration calibrated on four development approaches improves identity fidelity across seven approaches in head and face swapping, with positive virtual try-on gains but mixed background-replacement results.The fidelity-preservation trade-off is comparable to approach-specific constant edit-side biases without per-approach strength sweeps.Source: §6 · p. 10
• The authors state that attribute preservation and reliable cross-task transfer remain directions for further work.The paper reports code and per-clip result files will be released.Source: §6 · p. 10, REPRODUCIBILITY STATEMENT · p. 10
• The work studies editing tasks including head and face swapping, which can be misused to fabricate media of real individuals.Evaluation uses public benchmarks; the authors release no new identity data or model, and RefGAP modifies editors at inference time.Source: ETHICS STATEMENT · p. 10

A IMPLEMENTATION AND STATISTICAL DETAILS

• The default implementation uses (γedit, γkeep) = (0.6, 0.8), with approach-specific corrected layers and the edit mask applied only to content queries.JoyAI-Video-Edit uses an unfused hook that materializes per-head fp32 attention matrices, incurring 54% end-to-end overhead and 1.4 GB additional peak memory.Source: A IMPLEMENTATION AND STATISTICAL DETAILS · p. 13
• Fused denoising overhead is 0.5–7.9% on VACE, LoomVideo, Qwen-Image-Edit, and OmniGen2, versus 24% on FLUX.2-klein-base and 68% on FLUX.2-klein.The two FLUX variants have short image sequences, making attention inexpensive relative to the added kernel call and output combination.Source: A IMPLEMENTATION AND STATISTICAL DETAILS · p. 13

B REFERENCE-MASS DIAGNOSIS ACROSS APPROACHES

• The mean log-inverse edit mass ranges from 2.47 to 5.91 across approaches, indicating substantial variation in reference-attention allocation.This statistic summarizes potential correction strength for layer selection; it is not the mean deployed offset.Source: B REFERENCE-MASS DIAGNOSIS ACROSS… · p. 14, Table 7 · p. 14

C.1 CALIBRATION GRIDS

• The selected shared coefficients are (γedit, γkeep) = (0.6, 0.8), a fidelity-preservation compromise among four feasible pairs on the development approaches.The calibration jointly compares layer coverage and coefficients; removing keep-side correction violates preservation in several settings.Source: §C.1 · p. 14, Table 8 · p. 15

C.2 ONLINE VERSUS FIXED EDIT-SIDE STRENGTH

• The fixed-bias comparison selects per-approach edit-side constants on a 40-clip validation split while holding masks, keep-side correction, and layer sets unchanged.The selected constants are 1.5 for JoyAI-Video-Edit, 2.5 for LoomVideo, and 2.0 for both FLUX.2 variants.Source: §C.2 · p. 14

D.1 HEAD SWAPPING

• On DirectSwap, identity similarity rises from 0.7019 to 0.7450, alongside increased keypoint error and little change in whole-frame LPIPS.This task-fine-tuned model uses its released inference pipeline, including attention-based background compositing.Source: §D.1 · pp. 15–16, Table 10 · p. 16
• Whole-head similarity increases on all seven general-purpose approaches with RefGAP on HeadSwapBench.The table reports paired-clip win percentages and includes DirectSwap as a task-fine-tuned reference.Source: §D.1 · p. 16, Table 11 · p. 16

D.2 FACE SWAPPING

• Face-centered crops raise retrieval and identity similarity and lower keypoint error versus full-frame inputs on the evaluated JoyAI-Video-Edit pairs.The crop places the face at roughly a quarter of the image area; all reported face-swap results use crops.Source: §D.2 · p. 17, Table 12 · p. 17

D.3 VIRTUAL TRY-ON

• On 100 VITON-HD images, DINO improves by 0.009–0.046 across four image approaches, with LPIPS increases of at most 0.0056.The evaluation reuses the same coefficients and layer sets, supporting transfer without retuning.Source: §D.3 · p. 17, Table 13 · p. 17

D.4 BACKGROUND REPLACEMENT

• The background-replacement evaluation pairs 150 static-camera HeadSwapBench clips with person-free SUN397 scene references and uses SAM3 masks to define subject and background regions.Video approaches edit original clips; image approaches edit their middle frames.Source: §D.4 · p. 17

• The analysis pairs Base and RefGAP under identical inputs, seeds, and masks, then uses paired bootstrap intervals that account for shared identities and reference content.Frames within videos are aggregated before resampling, and the reported intervals are pointwise rather than simultaneous.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · pp. 17–18
• Resampling units reflect task dependencies: subjects for head swap, shared target/source identities for face swap, clips and garments for try-on, and identities and scenes for background replacement.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · p. 18
• All 27 intervals for head-swap Repl. and ID and face-swap ID and Retr. exclude zero in the improving direction.LoomVideo’s face-swap ID gain is +.017 [+.003, +.034]. Face-swap Repl. also improves with an interval excluding zero on all six approaches where it is defined.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · p. 18
• Every listed preservation change has an interval excluding zero, including keypoint-error increases and LPIPS improvements or increases across different approaches and tasks.Head-swap LPIPS improves on Qwen-Image-Edit, FLUX.2-klein-base, VACE, and OmniGen2; other preservation changes include increases in keypoint error and LPIPS.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · p. 18
• 1.85 was the median factor increase in interval width under dependence-aware resampling, while intervals excluding zero fell from 76 to 62.The intervals are pointwise 95% bootstrap intervals for paired differences.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · p. 18, Table 14 · p. 19
• Transfer gains are supported on several approaches, but background-replacement DINO decreases by −.257 [−.374, −.153] on Qwen-Image-Edit and improvements are not uniform.Try-on DINO gains have intervals excluding zero on four approaches; background-replacement DINO improves on JoyAI-Video-Edit, LoomVideo, and OmniGen2.Source: E UNCERTAINTY OF THE MAIN QUANTITATIVE… · p. 19

F HUMAN EVALUATION

• The evaluation sampled items with large visible differences between methods, then randomly selected from that pool while rotating over subject identities.It used five items per approach for head swapping and ten for other tasks, with stated exceptions for FLUX.2-klein try-on and background replacement and VACE background replacement.Source: F HUMAN EVALUATION · p. 19
• 27.9% of unscreened reference-fidelity judgments favored RefGAP, 3.6% favored the base, and 68.6% were ties; among decided judgments, RefGAP had an 88.6% share.The audit used five raters and 56 randomly sampled items without screening for visible differences.Source: F HUMAN EVALUATION · pp. 19–20, Table 15 · p. 20
• Visual quality was tied in 92.1% of unscreened judgments, while all 22 decided quality judgments favored the base.The base-favoring judgments were concentrated in five items; the screened LoomVideo face-swap evaluation also showed a base-favoring quality preference.Source: F HUMAN EVALUATION · p. 20
• Ties do not necessarily indicate successful preservation: table 16 reports 46 both-fail judgments for head swapping and 138 for face swapping.Source: F HUMAN EVALUATION · p. 20, Table 16 · p. 20
• Reference-fidelity judgments favored RefGAP in both face-swap subgroups, while pose/expression preferences shifted with whether the base output was closer to the input or reference identity.When the base was closer to the input identity, pose/expression preferences were 25 for the base versus 3 for RefGAP; when closer to the reference, they were 16 versus 15, with 234 ties.Source: F HUMAN EVALUATION · p. 20

G ATTRIBUTE TRADE-OFFS AND FAILURE ANALYSIS

• Face-swap keypoint error is measured against the driving input, so identity-specific face shape can raise the metric even when input pose is preserved.The analysis identifies identity-geometry confounding and examines residual keypoint and expression costs.Source: G ATTRIBUTE TRADE-OFFS AND FAILURE… · p. 21

G.1 IDENTITY-ATTRIBUTE TRADE-OFFS IN FACE SWAPPING

• Identity-related geometry changes can contribute to increased face-swap keypoint error because evaluation compares landmarks with the input identity, not the reference identity.For Qwen-Image-Edit, LoomVideo, and OmniGen2, keypoint increases concentrate among clips with large identity gains; residual increases remain for JoyAI-Video-Edit and both FLUX.2 variants.Source: §G.1 · p. 21

G.2 DENOISING-STEP INTERVENTIONS

• Skipping the first denoising step reduced keypoint error from 0.085 to 0.052 on JoyAI-Video-Edit and from 0.153 to 0.101 on FLUX.2-klein, while identity similarity also fell.For FLUX.2-klein-base, skipping the first ten steps reduced keypoint error from 0.236 to 0.183 and identity similarity from 0.799 to 0.735.Source: §G.2 · p. 22, Table 19 · p. 22
• Delaying correction can partially preserve input geometry at the cost of weaker identity transfer, so the study retains correction at all steps to prioritize recognizable reference identity.The trade-off depends on the approach and sampling schedule.Source: §G.2 · p. 22

G.3 BACKGROUND REPLACEMENT FAILURE

• On Qwen-Image-Edit, background-replacement DINO fell from 0.299 to 0.042; restricting correction to the second half or using a fixed edit-side bias of 2.0 did not avoid similar degradation.The diagnostic clips were the first 30 of the 150-clip list and were not selected on outcome.Source: §G.3 · p. 22, Table 20 · p. 22
• The background-replacement failure appears approach-dependent, and its underlying cause remains unresolved; other approaches improve on the task.Here the reference specifies the surrounding scene, while the person must be preserved.Source: §G.3 · pp. 22–23

G.4 RARE HEAD-SWAP COLLAPSES

• OmniGen2 had missing or heavily smeared faces in 3/1040 base outputs and 8/1040 RefGAP outputs; changing the seed can recover plausible faces, while repeating it reproduces failures.Source: §G.4 · p. 23, Figure 4 · p. 23

• Figure 5 compares the input, reference, and base and RefGAP outputs across all seven approaches, with reference-fidelity scores beneath each example.Green borders mark gains of at least 0.05.Source: H QUALITATIVE RESULTS · p. 23, Figure 5 · p. 24
• The main-paper examples’ reference-fidelity scores are shown alongside outputs, providing a comparison of base and RefGAP results for each approach.Source: Figure 5 · p. 24
• Figure 6 adds qualitative examples across seven approaches and four tasks, following Figure 5’s layout and annotations.Source: Figure 6 · p. 25