QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing

Figure 1

• QK-Edit manipulates query-key components within the vision parts of MM-SA for text-guided image and video editing.The caption describes the approach as training-free and focused on preserving structural consistency and visual details.Source: Figure 1 · p. 1
• The editing examples include object addition, removal, and replacement, as well as changes to backgrounds, materials, colors, and style.Source: Figure 1 · p. 1
• Supplementary video results include 1K resolution, landscape and portrait formats, and lengths up to 129 frames.Source: Figure 1 · p. 1

1. Introduction

• MM-SA concatenates text and vision embeddings in a joint attention operation, intertwining their interactions and complicating independent control.Unlike U-Net architectures, which separate text-vision cross-attention from vision-vision self-attention, MM-DiTs process both modalities together.Source: §1 · p. 2
• Applying U-Net cross-attention or self-attention editing techniques directly to MM-DiTs can produce undesirable outputs.A quadrant-based decomposition of MM-SA does not make applying prior methods to the corresponding maps reliably effective.Source: Figure 2 · p. 2, §1 · p. 2
• QK-Edit's analysis finds text-to-vision attention has negligible impact, vision-to-text links regions with text tokens, and vision-to-vision supports spatial and temporal consistency.Source: §1 · p. 2
• QK-Edit replaces source vision components in the target query and key embeddings during denoising, targeting precise edits that follow the prompt and preserve structure.The authors describe the strategy as training-free and requiring minimal code modifications.Source: §1 · p. 2
• The method is reported to work across image and video tasks, including high-resolution, long-frame editing with spatiotemporal consistency.The paper positions QK-Edit as bridging generative power and editable flexibility in Transformer-based diffusion models.Source: §1 · pp. 2–3

2. Related Work

• Image editing includes training-free inversion and latent-manipulation methods, such as Null-Text Inversion [45] and StyleDiffusion [65], alongside fine-tuned methods such as InstructPix2Pix [9].Training-based approaches generally add computational overhead, while training-free methods use pretrained models directly.Source: §2 · p. 3
• QK-Edit is described as a training-free technique for MM-DiT-based models, intended to support high prompt fidelity and structural consistency.Source: §2 · p. 3
• Video editors include training-free approaches that use attention injection or latent inversion, and training-based approaches that fine-tune models for video editing.The passage notes increased computation for training-based approaches.Source: §2 · p. 3

3. Method

• Given a source image or video and source and target prompts, the goal is to make the requested changes while preserving unedited regions.The method targets MM-DiT models.Source: §3 · p. 3

3.1. Preliminaries

• MM-DiTs concatenate text and vision embeddings and apply one self-attention operation, intertwining interactions that U-Net methods manipulate separately.The passage notes that direct cross-attention and self-attention manipulation performs less satisfactorily on FLUX than on U-Net diffusion.Source: §3.1 · p. 4, §3.2 · p. 4

3.2. Analysis of MM-DiT

• On HunyuanVideo, skipping blocks has uneven effects; MM-SA determines spatial arrangement, whereas bypassing FFN has negligible effect.Skipping the 22nd single block produces no visible change, unlike bypassing certain other blocks.Source: §3.2 · p. 4, Figure 4 · p. 5
• Quadrant analysis finds first-quadrant text-token relations dominate attention values; removing the second has negligible effect, while zeroing the third prevents proper FLUX generation.The vision-related attention also defines spatial layout.Source: §3.2 · p. 5

3.3. Editing using Multimodal Self-Attention

• QK-Edit manipulates the third and fourth quadrants during denoising to preserve unchanged source content and make edits according to the target prompt.These manipulations correspond to changing vision query-key components.Source: §3.3 · p. 5
• Rather than directly modifying encapsulated attention maps, QK-Edit substitutes saved source vision queries and keys into the target generation.The method describes saving source components and replacing target components during generation.Source: §3.3 · p. 5

4.1. Implementation details

• Implementation uses FLUX.1-dev for images and HunyuanVideo for videos, applying QK manipulation during the first 30% and 20% of inference steps, respectively.Video resolutions range from 320×320 with 65 frames to 1280×720 with up to 129 frames.Source: §4.1 · p. 5

4.2. Quantitative Evaluation Metrics

• Image metrics assess text alignment, structural similarity, and editing quality, while video metrics assess frame consistency, human preference, and video-text alignment.The reported metrics include CLIPtext, DINO, and PickScore for images, and CLIPframes, PickScore, ViCLIPdir, and ViCLIPout for videos.Source: §4.2 · pp. 5–6
• Pilot figures visualize MM-SA attention quadrants and Q/K vision-component substitution as a strategy for balancing prompt fidelity with source-structure preservation.Source: Figure 5 · p. 6, Figure 6 · p. 6

4.3. Quantitative Results

• QK-Edit leads all PIE-Bench image metrics, outperforms DiTCtrl on generative TGVE+, and outperforms all baselines on real TGVE+ video editing.The real-video setting requires video inversion.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 6

4.4. Qualitative Comparisons

• Qualitative comparisons show fine-grained image detail and structure preservation, along with video frame-to-frame consistency and temporal continuity.The examples include object addition and removal, replacement, background and material changes, color changes, and style transformation.Source: §4.4 · pp. 6, 8

4.5. Ablation Study

• QKreplace performs best on HiLo-Benchmark, with higher editing fidelity, structural consistency, and semantic alignment across images and videos.HiLo-Benchmark contains 43 challenging videos with resolutions exceeding 1K and sequences longer than 100 frames.Source: Table 4 · p. 8, §4.5 · p. 8

5. Conclusion

• QK-Edit is a training-free framework that uses MM-DiT attention dynamics for text-guided image and video editing.Source: §5 · p. 8
• The framework covers object addition, removal, and replacement, background, material, and color changes, and style transformation, with implementation requiring only a few lines of code.Source: §5 · p. 8
• Complex-scene video inversion can compromise edits; unintended changes may affect regions meant to remain unchanged, and base-model limitations include text rendering and complex-motion deficiencies.The authors suggest masks and more advanced generative models as future directions.Source: §5 · p. 8

Supplementary Material

• Figure 9 presents the pseudocode of QK-Edit.Source: Figure 9 · p. 12

6. Pseudo Code

• The section provides PyTorch-style pseudocode to illustrate QK-Edit’s straightforward implementation.Source: §6 · p. 12

• The additional image-editing examples demonstrate preservation of fine-grained details.Source: §7 · p. 12
• The examples demonstrate structural consistency across diverse editing scenarios.Source: §7 · p. 12
• The examples demonstrate precise semantic alignment across diverse editing scenarios.Source: §7 · p. 12

• The MP4 results include a 1280×720 video with 129 frames.Source: §8 · p. 12
• The MP4 results include a 720×1280 video with 65 frames.Source: §8 · p. 12
• The examples showcase the method in both landscape and portrait formats.Source: §8 · p. 12

• QK-Edit’s performance is inherently bounded by the capabilities of its base model.Source: §9 · p. 12
• The authors discuss some failure cases in the Limitations section.Source: §9 · p. 12
• Figure 10 illustrates failure cases and artifacts.Source: §9 · p. 12