Group Relative Attention Guidance for Image Editing

1. Introduction

• Instruction-based editing must balance responsiveness to the edit with fidelity to the source image, a trade-off that can require prompt engineering or repeated inference.The introduction frames continuous editing-strength control as a response to this challenge.Source: §1 · p. 2
• Query and Key embeddings cluster around a shared bias vector, and the authors hypothesize that token deviations from it encode contextual understanding.The paper’s Figure 2 visualizes the bias across tokens.Source: §1 · p. 2, Figure 2 · p. 2
• GRAG weights token deviations from the group bias to adjust editing strength; experiments report a better responsiveness–consistency trade-off at fixed guidance and smooth progressive outputs as its coefficient changes.The paper reports validation on state-of-the-art DiT-based editors and describes contributions across multiple baselines and editing tasks.Source: §1 · p. 2

2.1. Text-driven Image Editing

• Text-driven editing progressed from synthetic-data fine-tuning and training-free methods to specialized editors, yet balancing editing effectiveness with source consistency remains a challenge.The section also identifies precise, continuous control over editing effects as an unresolved challenge.Source: §2.1 · pp. 2–3
• Proprietary multimodal systems and creative platforms have also integrated advances in instruction-driven image editing workflows.Examples named include GPT-4V, Gemini, Midjourney, and RunwayML.Source: §2.1 · p. 2

2.2. Editing Strength Control for Image Editing

• Text-to-image controls reweight textual concepts or attention, while real-image editing methods regulate edited regions or instruction strength; direct attention-map manipulation can cause artifacts and limit fast-attention compatibility.GRAG instead analyzes DiT attention representations and modulates embedding features for continuous editing-strength control with minimal overhead.Source: §2.2 · p. 3

3. Preliminaries

• MM-DiT merges text and visual modalities to generate images aligned with textual semantics.The described framework is also called a multi-modal diffusion transformer.Source: §3 · p. 3
• The layer maps text and image tokens into a shared space, then computes joint attention by combining queries and keys from both modalities.The passage identifies projection matrices and a shared dimension for the mappings.Source: §3 · p. 3
• During image editing, the visual tokens comprise both the editing target and original image, whose query, key, and value embeddings are concatenated.The text specifies Q_i = [Q_e ⊕ Q_s], K_i = [K_e ⊕ K_s], and V_i = [V_e ⊕ V_s].Source: §3 · p. 3

4. Bias Vector In The Embedding Vectors

• Visual embeddings concentrate in high RoPE-frequency components, while text embeddings concentrate in low-frequency components associated with semantics.The analysis describes visual high-frequency components as capturing spatial relations, suggesting the modalities are not fully aligned in the shared space.Source: §4 · p. 4, Figure 4 · p. 4
• Across attention heads, the embedding distributions reveal a significant bias vector, and the paper describes Query and Key embeddings as decomposable into bias plus token variation.The attention layer fuses editing instructions and conditional image information; Query and Key embeddings influence how much content is sampled from each token.Source: §4 · pp. 3–4, Figure 5 · p. 4
• The authors hypothesize that the layer’s bias is a fixed editing action and that individual token variations relative to it encode the content being edited.They report that same-layer feature distributions remain highly similar across time steps and input samples.Source: §4 · p. 4
• A strong shared bias can dilute the influence of key-token deviations and reduce attention sensitivity to semantic differences; modulating those deviations can control conditioning-signal influence.The passage gives edit instructions as an example of conditioning signals.Source: §4 · p. 4

• GRAG modifies the cross-attention component of MM-Attention corresponding to Qe, applying group-relative modulation to a selected token group Ks.Source: §5 · p. 5
• GRAG computes a group-level bias as the mean of selected conditional keys, defines each token’s deviation from that bias, and uses the updated keys to recompute attention.Source: §5 · p. 5
• The positive factors λ and δ balance the shared bias and token-specific variations: λ changes selected-token influence, while δ adjusts editing focus.λ > 1 enhances selected-token influence and λ < 1 reduces it; δ > 1 concentrates editing, whereas δ < 1 diffuses it.Source: §5 · p. 5
• The guidance algorithm consists of four lines and can be integrated into existing methods.Source: §5 · p. 5

6.1. Experiment Setting

• The evaluation applies fixed GRAG parameters across six editing models, uses PIE and preservation and preference metrics, and compares CFG with two attention-control methods.The methods are evaluated with a fixed seed, batch size 1, and 24 inference steps.Source: §6.1 · pp. 5–6

6.2. Qualitative Analysis

• Across training-based editors, GRAG preserves source-image details while applying requested changes, enables editing in original Kontext, and adjusts results in training-free editing.Examples include changing a bird’s texture while retaining tree-trunk details and changing an apple’s color while preserving surface details.Source: §6.2 · pp. 6–7, Figure 7 · p. 6, Figure 8 · p. 6

6.3. Quantitative Analysis

• On PIE, overall EditScore increases for Step1X-Edit, Qwen-Edit, and Kontext; training-free methods also gain, though GRAG adapts less well than with training-based methods.Step1X-Edit and Qwen-Edit improve consistency despite a slight PF decline, while Kontext improves PF and EditScore.Source: §6.3 · p. 7, Table 1 · p. 7

6.4. Ablation Study

• Against CFG and attention controls, GRAG provides smoother, more consistent strength adjustments, while CFG has limited control and Attn-gate and Attn-weight can produce discontinuities or incorrect edits.Attn-weight can replace the cat with a horse; CFG can degrade image quality at a small guidance scale.Source: §6.4 · p. 8, Figure 10 · p. 8
• Adjusting δ alone yields the smoothest metric variation and most continuous editing transitions; λ alone has no significant impact, while jointly adjusting λ and δ is less precise and can degrade fidelity.The comparison also evaluates CFG and attention-based controls across their effective parameter ranges.Source: §6.4 · p. 8, Figure 10 · p. 8, Table 2 · p. 8

7. Conclusion

• GRAG modulates token deviations from a shared bias to provide fine-grained, continuous editing-strength control and improve controllability and fidelity in existing DiT-based editors.The paper frames the shared bias as governing editing behavior and the method as a practical direction for future DiT architectures.Source: §7 · p. 8

Supplementary Material

• The supplementary contents list “Qwen-Edit Embedding Visualization” as Section E.2 on page 17.Source: Supplementary Material · p. 11

A. Pytorch Implementation of GRAG

• GRAG is presented as requiring only a few lines of modification to a typical Diffusers MM-Attention block.Source: A. Pytorch Implementation of GRAG · p. 12
• The example applies rotary embeddings to image and text queries and keys before scaling and joint attention.Source: A. Pytorch Implementation of GRAG · p. 12
• The example replaces image keys with scaled group bias plus scaled token deviation, using image-token indices 4096–8192, bias scale 1.0, and delta scale 1.05.Source: A. Pytorch Implementation of GRAG · p. 12

B.1. Toy Experiment of the GRAG Theoretical Analysis

• GRAG-δ preserves stable attention on editing-text tokens while smoothly adjusting source-image emphasis, giving the most continuous and reliable editing-strength control among tested strategies.By contrast, increasing γ or λ, alone or together, rapidly suppresses editing-text attention and can cause instruction collapse, artifacts, or editing failure.Source: §B.1 · p. 13, Figure S1 · p. 14

B.2. Relationship Between Attention Entropy and GRAG

• As GRAG increases, source-image attention entropy decreases monotonically across layers, while query attention concentrates progressively on original cat contours.The authors interpret this as stronger focus on conditioning information and greater consistency with the reference image.Source: §B.2 · p. 15, Figure S2 · p. 15

B.3. GRAG of Source Image Tokens

• Modulating image-token variation changes edited semantic content while image quality remains largely consistent; bias-only modulation leaves editing relatively stable but changes image quality.These observations support interpreting bias as intrinsic editing behavior and variation as content-specific editing signals.Source: §B.3 · p. 16, Figure S3 · p. 16

B.4. GRAG of Editing Text Tokens

• For text tokens, bias governs inherent editing behavior and variation drives content-specific semantic changes; text-embedding modulation offers stronger, more direct control of editing intensity.The analysis reports a stronger degree of semantic-level editing control for text tokens than for image tokens.Source: §B.4 · p. 17, Figure S4 · p. 17

C. Limitation & Discussion

• GRAG’s effectiveness is limited for training-free editing: inversion and attention injection approximate feature interaction, and structural mismatch can degrade image quality or yield ineffective edits.Training-based TI2I models instead integrate editing instructions and source-image information within the same MM-Attention layers.Source: C. Limitation & Discussion · p. 18, Figure S5 · p. 18, Figure S6 · p. 18

D. Additional Qualitative Results

• Additional qualitative results and comparisons with existing editing-strength control methods are presented in Figures S8 and S9.The supplied captions identify the material as additional results but do not state comparative outcomes.Source: Figure S7 · p. 19, Figure S8 · p. 20

E. Additional Feature Visualization

• Additional feature-distribution statistics cover different samples, denoising steps, model layers, token aggregation, and token-wise distributions across attention heads.The visualizations include Kontext and Qwen-Edit; token-number dimensions are downsampled by four and embedding dimensions by two.Source: E. Additional Feature Visualization · p. 21

E.1. Kontext Embedding Visualization

• Kontext visualizations indicate that features within a layer have similar distributions, with limited correlation to model inputs or denoising steps.Figures S9–S14 show feature distributions, token aggregation, and embedding statistics across attention heads.Source: Figure S9 · p. 21, Figure S10 · p. 22, Figure S11 · p. 23

E.2. Qwen-Edit Embedding Visualization

• Figure S15's additional visualizations of text and image embedding features show that features within the same layer share similar distributions, indicating limited correlation with model inputs or denoising steps.Other captions describe token aggregation and embedding mean magnitudes and standard deviations across attention heads (Figures S16–S20).Source: Figure S15 · p. 27, Figure S16 · p. 28, Figure S17 · p. 29