PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

Niange Yu, Ye Tian, Biaolong Chen, Miao Lu, Aixi Zhang, Hao Jiang, Yunhai Tong, Pipei Huang

arXiv:2609.11507v1cs.CV

TL;DR

Multi-subject subject-to-video generation lacks controllable fidelity and often exhibits semantic drift. DIAL identifies intrinsic spatial grounding signals in Diffusion Transformers and uses them for inference guidance and preference-based alignment, achieving state-of-the-art OpenS2V-Eval performance while improving identity consistency and reducing drift.

  • Problem

    Existing subject-to-video models provide inflexible fidelity strength and suffer semantic drift, including subject disappearance, attribute mismatch, and multi-reference entanglement.

  • Method

    DIAL identifies Intrinsic Spatial Grounding Maps in selected DiT attention blocks, using them for training-free fidelity control and zero-cost preference-pair construction for reinforcement learning.

  • Results

    DIAL achieves state-of-the-art performance on OpenS2V-Eval, improving identity consistency and enabling controllable fidelity strength without compromising visual quality.

  • Takeaways & Limitations

    Intrinsic attention signals support both fine-grained inference-time fidelity control and stronger reference-semantic anchoring in multi-subject video generation.

  • Takeaways & Limitations

    Inference-time guidance alone is insufficient during the initial high-noise regime, where stochastic and diffuse attention signals can cause unstable layouts and irreversible semantic drift.

Abstract

from arXiv · show

Multi-subject video generation faces two key challenges: uncontrollable fidelity strength and potential semantic drift. We address these by analyzing the internal mechanisms of Diffusion Transformers (DiTs). We found that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects. Building on this insight, we propose Dual-phase Intrinsic Attention Leveraging (DIAL), a framework that uses these internal signals for both training and inference. In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining. In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning (RL). This RL procedure effectively anchors the model's attention to reference subjects and mitigates semantic drift. Extensive experiments show that DIAL significantly outperforms baseline models on the OpenS2V-Eval benchmark, consistently improving identity consistency and enabling controllable fidelity strength.

1 Introduction

Subject-to-video generation must preserve one or more reference identities while following motion instructions, yet existing systems offer inflexible fidelity and suffer semantic drift. DIAL addresses these issues by exploiting intrinsic attention signals for controllable inference guidance and training-time alignment.

  • Motivation: Existing subject-to-video systems struggle with inflexible fidelity strength and semantic drift, including subject disappearance, attribute mismatch, and multi-reference entanglement.These limitations are linked to training-data-dependent consistency strength and misalignment between reference subjects and denoising attention.
  • Intrinsic Spatial Grounding Maps: Layer-wise analysis identifies attention blocks whose concentrated patterns form Intrinsic Spatial Grounding Maps that precisely localize reference subjects.The analysis compares attention weights with segmentation-derived subject masks and finds stronger grounding in selected blocks.
  • DIAL: DIAL uses ISGM for training-free, inference-time attention guidance with independently adjustable fidelity strength for each reference subject.The method reuses ISGM during denoising to construct subject-specific attention biases without model updates.
  • DIAL: DIAL also constructs preference pairs automatically from ISGM alignment and uses reinforcement learning to anchor reference grounding during noisy denoising stages.The preference construction requires no manual annotation and targets semantic-drift reduction.
  • Results: DIAL consistently improves quantitative and qualitative evaluations and achieves state-of-the-art performance on OpenS2V-Eval.The reported gains include improved identity consistency, controllable fidelity, and reduced semantic drift.

2 Related Work

Prior work advances subject-to-video identity preservation, attention control, and reinforcement-learning alignment, but DIAL applies intrinsic attention guidance to controllable multi-subject fidelity.

  • Subject-to-Video Generation: Subject-to-video research has progressed from per-subject optimization toward efficient end-to-end adapters for zero-shot identity preservation.Representative systems use joint injection, unified context adapters, and region-aware identity injection.
  • Attention Control: Attention maps have been used for grounding-aware editing, subject emphasis, structural guidance, and video or image control in diffusion models.Related methods extend attention control from U-Net architectures to Diffusion Transformers.
  • Reinforcement Learning: Reinforcement-learning methods increasingly use preference-based alignment for diffusion and temporal video generation.The cited line of work includes policy-gradient, Direct Preference Optimization, and video-specific variants.
  • DIAL Positioning: DIAL combines intrinsic spatial grounding with training-free attention guidance to provide independent, continuous fidelity control for multiple subjects.Its Phase I mechanism reuses ISGM and adds a lightweight log-scale bias without model updates.

3 Method

DIAL analyzes DiT attention to identify intrinsic subject-grounding signals, then uses them for training-free fidelity control in later denoising and preference optimization in noisier stages. Its two phases modulate reference attention and train the model to maintain subject grounding.

  • 3 Method: DIAL combines attention-based inference guidance and preference optimization to improve controllable subject fidelity while anchoring reference semantics.The framework operates on DiT self-attention over concatenated reference and video tokens.
  • 3.2 Identification of Intrinsic Spatial Grounding Maps (ISGM): DIAL identifies attention blocks whose concentrated patterns form ISGMs that align with reference-subject regions.The analysis computes video-to-reference attention and compares it with subject masks to find the block with highest grounding accuracy.
  • 3.3 DIAL Phase I: Low-noise Inference Guidance: During the final 80% of denoising, DIAL reuses ISGMs as spatial guides to bias attention toward each corresponding reference subject.The guide is normalized or thresholded, then injected into attention logits through a subject-specific bias.
  • 3.3 DIAL Phase I: Low-noise Inference Guidance: User-specified scales γ_k provide independent, continuous fidelity control for individual subjects without model updates.The bias targets video queries and the key range of subject k, while leaving other attention interactions unchanged.
  • 3.4 DIAL Phase II: High-noise Anchoring via Preference Optimization: Because ISGMs are unreliable in the initial high-noise regime, DIAL scores dual-seed trajectories by mask alignment and uses the resulting pairs for DPO.The higher-scoring trajectory is selected as the winner, and the procedure is intended to stabilize grounding and mitigate semantic drift.

4 Experiments

DIAL achieves state-of-the-art OpenS2V-Eval performance while enabling controllable fidelity and reducing semantic drift through ISGM guidance and preference optimization.

  • 4 Experiments: DIAL achieves state-of-the-art performance on OpenS2V-Eval across evaluated DiT-based backbones and fidelity settings.The main-results table evaluates aesthetics, motion, identity, consistency, and naturalness metrics across different γ values.
  • 4 Experiments: Increasing γ from 2 to 8 with Kaleido raises FaceSim from 43.80% to 57.46%, a 31.2% relative gain without significant naturalness degradation.This demonstrates controllable identity-strength enhancement during inference.
  • 4 Experiments: With γ = 0, DIAL improves NexusScore from 38.62% to 41.22% for Kaleido and from 41.10% to 42.76% for Phantom-14B.At γ = 0, Phase I guidance is disabled, isolating the contribution of Phase II preference optimization.
  • Ablation Study: ISGM-guided enhancement preserves naturalness better than uniform guidance, achieving FaceSim comparable to Uni at γ = 2 while reaching 75.23% NaturalScore at γ = 16.Uniform enhancement substantially degrades NaturalScore even at γ = 2.
  • Ablation Study: Two DPO iterations provide the best trade-off, while three yield marginal consistency gains but slightly lower realism.The authors therefore use Tdpo=2 as the default setting.
  • Ablation Study: Phase II increases early-denoising grounding scores and smooths their timestep curve after preference optimization.This supports earlier and more stable reference attention during denoising.

5 Conclusion

The paper concludes that DIAL combines intrinsic subject localization with dual-phase guidance to improve controllable S2V generation while preserving visual quality.

  • 5 Conclusion: DIAL combines ISGM-based inference-time fidelity control with zero-cost preference optimization for high-noise semantic anchoring.Experiments on OpenS2V-Eval show improved identity consistency and reduced semantic drift without compromising visual quality.
  • 5 Conclusion: DIAL achieves state-of-the-art OpenS2V-Eval performance while significantly improving identity consistency and mitigating semantic drift.The conclusion attributes these results to the unified use of intrinsic subject-localization signals.

A Additional Qualitative Results for Low-noise Inference Guidance

Additional results show that ISGM provides localized, low-noise fidelity enhancement and that γ offers smooth identity control without requiring uniform global guidance.

  • A Additional Qualitative Results for Low-noise Inference Guidance: ISGM guidance strengthens subject fidelity while preserving the background, whereas uniform guidance introduces global artifacts.This supports localized rather than spatially uniform enhancement.
  • A Additional Qualitative Results for Low-noise Inference Guidance: ISGM maps are scattered during early high-noise steps but become sharply localized after sufficient denoising, motivating delayed Phase I guidance.The low-noise regime is therefore used for spatially conditioned control.
  • A Additional Qualitative Results for Low-noise Inference Guidance: Phase II improves subject presence across 320 evaluated case–subject instances when Phase I is disabled.The video-level detector counts a match when the queried subject appears in at least one generated frame.
  • A Additional Qualitative Results for Low-noise Inference Guidance: Increasing γ consistently improves identity similarity on Kaleido, including under prompt–reference ambiguity and cross-domain references.The results support smooth and robust fidelity controllability.

B Analysis of Semantic Drift Mitigation

Phase II addresses semantic drift by optimizing denoising trajectories toward stronger reference grounding, improving subject presence and preserving reference identities.

  • B Analysis of Semantic Drift Mitigation: Phase II uses preference optimization over denoising trajectories because deterministic ISGM bias injection is less reliable in high-noise stages.This phase targets early reference grounding rather than direct attention-bias injection.
  • B Analysis of Semantic Drift Mitigation: With Phase I disabled, Phase II increases the video-level match ratio, meaning fewer reference subjects disappear entirely from generated videos.Subject presence is measured with an image-prompt open-vocabulary detector using each reference image as the query.
  • B Analysis of Semantic Drift Mitigation: Qualitative examples show that Phase II better preserves reference identities and key attributes than the Kaleido backbone.This qualitative evidence is consistent with the reduced subject disappearance measured quantitatively.

C.1 Identification of Intrinsic Spatial Grounding Maps (ISGM)

The method measures attention grounding by comparing attention mass inside versus outside each predicted subject region, then averages the ratio across references and prompts.

  • C.1 Identification of Intrinsic Spatial Grounding Maps (ISGM): Grounding strength is computed as the ratio of average attention inside each subject region to average attention outside it.The score uses per-reference attention scores over foreground and background video-query sets.
  • C.1 Identification of Intrinsic Spatial Grounding Maps (ISGM): Subject regions come from detected bounding boxes mapped onto the latent token grid, with the complement defining background queries.The resulting foreground/background partitions support the attention-ratio calculation.
  • C.1 Identification of Intrinsic Spatial Grounding Maps (ISGM): The reported grounding score averages the aggregated ratio across all 180 OpenS2V-Eval prompts.This provides the evaluation scope for the grounding measurement.

C.2 DIAL Phase I: Low-noise Inference Guidance

DIAL extracts ISGM from a privileged attention block at each denoising step, adding minimal overhead, while evaluations use 50 denoising steps unless specified otherwise.

  • C.2 DIAL Phase I: Low-noise Inference Guidance: ISGM is extracted independently at every denoising step from one pre-identified privileged attention block, keeping additional overhead minimal.The same extraction procedure supports the inference-time guidance mechanism.
  • C.2 DIAL Phase I: Low-noise Inference Guidance: Unless specified otherwise, evaluations use T = 50 denoising steps and each backbone’s default resolution.Kaleido uses original prompts, whereas Phantom follows its official prompt-rephrasing recommendation.

C.3 DIAL Phase II: High-noise Anchoring via Preference Optimization

Phase II uses pseudo segmentation masks and ISGM-based preference optimization during high-noise denoising to improve reference consistency, including for background-style references.

  • C.3 DIAL Phase II: High-noise Anchoring via Preference Optimization: Phase II training improves consistency for background references despite limited background-style references in Phantom-Data.This observation suggests improved reference utilization and generalization for that setting.
  • C.3 DIAL Phase II: High-noise Anchoring via Preference Optimization: ISGM is diffuse and unreliable during roughly the first 20% of denoising steps, so Phase I guidance is disabled in this high-noise regime.As denoising proceeds, ISGM sharpens and expands across the subject before concentrating on discriminative regions near the end.
  • C.3 DIAL Phase II: High-noise Anchoring via Preference Optimization: Figure 10 evaluates fidelity-scale control by varying γ from 0 to 16, while Figure 11 compares Phase II anchoring with Phase I guidance disabled.The supplied captions identify the evaluation configurations but do not provide Figure 11’s outcome values.
Loading 2609.11507v1…