PaperlayerView exact revision on arXiv

Source-linked AI summary

Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models

Yuanyang Yin, Yufan Deng, Shenghai Yuan, Kaipeng Zhang, Xiao Yang, Feng Zhao

arXiv:2601.07287v1cs.CV

TL;DR

I2V models must reconcile reference-image fidelity with text-directed changes, yet prompt adherence remains under-explored. The paper diagnoses Semantic-Weak Layers and Condition Isolation, proposes Focal Guidance with FSG and Attention Cache, and reports gains on two I2V models.

  • Problem

    I2V models struggle to preserve visual details while faithfully executing text-directed motion and semantic transformations.

  • Method

    Focal Guidance combines Fine-grained Semantic Guidance, which grounds keywords in reference-frame regions, with Attention Cache, which transfers semantic attention from responsive to weak layers.

  • Results

    3.97% raises Wan2.1-I2V’s Total Score from 0.6973 to 0.7250, while 7.44% raises HunyuanVideo-I2V’s from 0.5185 to 0.5571.

  • Takeaways & Limitations

    The benchmark and Focal Guidance provide an evaluation and inference-time framework for improving instruction following in DiT-based I2V models.

Abstract

from arXiv · show

The task of Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt. This requires diffusion models to reconcile high-frequency visual constraints and low-frequency textual guidance during the denoising process. However, while existing I2V models prioritize visual consistency, how to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored. In this work, we observe that in Diffusion Transformer (DiT)-based I2V models, certain intermediate layers exhibit weak semantic responses (termed Semantic-Weak Layers), as indicated by a measurable drop in text-visual similarity. We attribute this to a phenomenon called Condition Isolation, where attention to visual features becomes partially detached from text guidance and overly relies on learned visual priors. To address this, we propose Focal Guidance (FG), which enhances the controllability from Semantic-Weak Layers. FG comprises two mechanisms: (1) Fine-grained Semantic Guidance (FSG) leverages CLIP to identify key regions in the reference frame and uses them as anchors to guide Semantic-Weak Layers. (2) Attention Cache transfers attention maps from semantically responsive layers to Semantic-Weak Layers, injecting explicit semantic signals and alleviating their over-reliance on the model's learned visual priors, thereby enhancing adherence to textual instructions. To further validate our approach and address the lack of evaluation in this direction, we introduce a benchmark for assessing instruction following in I2V models. On this benchmark, Focal Guidance proves its effectiveness and generalizability, raising the total score on Wan2.1-I2V to 0.7250 (+3.97\%) and boosting the MMDiT-based HunyuanVideo-I2V to 0.5571 (+7.44\%).

1. Introduction

I2V models must preserve reference-image details while following text-directed dynamics, but prompt adherence remains under-explored and often gives way to visual conditions and learned priors. The paper identifies weak intermediate-layer semantics and proposes Focal Guidance plus an instruction-following benchmark.

  • I2V combines a starting image as a visual anchor with a text prompt guiding the video’s dynamic evolution.
  • Current models often prioritize visual conditions and internal priors over textual directives during denoising.
  • Prompt adherence has received less attention than temporal consistency and aesthetic quality, with existing remedies often limited to training-time interventions.
  • Focal Guidance combines Fine-grained Semantic Guidance, which aligns keywords with reference regions, and Attention Cache, which transfers structured attention to weaker layers.
  • +7.44% improvement is reported for MMDiT-based HunyuanVideo-I2V, while Wan2.1-I2V improves by +3.97% on the introduced instruction-following benchmark.

2. Related Work

Related work spans I2V conditioning, explicit controllable-video signals, and interpretability studies of non-uniform semantic responsiveness. These strands motivate intrinsic methods for improving text–visual controllability without requiring precise external signals or labeled data.

  • Image-to-Video Generation Models: Early I2V methods used motion priors, reference videos, or physical-phenomenon modeling, while later diffusion methods improved first-frame conditioning and video fidelity.
  • Controllable Video Generation: Explicit controllable-video methods use bounding boxes, trajectories, or 3D camera parameters, but require precise external signals and labeled data.
  • Interpretability and Conditioning in Generative Models: Interpretability studies report non-uniform semantic responsiveness across Transformers, ViTs, U-Nets, and DiT-based diffusion models.
  • Interpretability and Conditioning in Generative Models: Prior attention interventions provide general correctives but do not address a specific root cause of conditioning failures.

3. Background and Problem Formulation

This section formulates video generation with Rectified Flow and diagnoses controllability failures in DiT-based I2V models. Semantic-Weak Layers and Conditioning Isolation weaken text influence by making heterogeneous visual and textual conditions difficult to align.

  • 3.1. Rectified Flow for Video Generation: Rectified Flow linearly interpolates between a VAE-encoded data latent and Gaussian noise, then trains a model to predict the path’s constant velocity.
  • 3.1. Rectified Flow for Video Generation: During inference, a numerical ODE solver integrates the learned velocity field from t = 1 to t = 0 to recover the final data latent.
  • 3.2. Issues in Current DiT-based I2V Models: Conditioning Isolation arises because the VAE reference latent, image features, and text features are injected relatively independently from heterogeneous representation spaces.
  • 3.2. Issues in Current DiT-based I2V Models: Without explicit pre-alignment, generic attention must learn spatial–semantic correspondences, making textual entities difficult to align with visual counterparts.
  • 3.2. Issues in Current DiT-based I2V Models: Weak semantic responsiveness can increase reliance on learned internal priors, including generic motion patterns and stylistic biases.
  • 3.2. Issues in Current DiT-based I2V Models: Moran’s I measures spatial autocorrelation in visual–textual similarity maps to assess whether layer responses are clear and spatially coherent.
  • 3.2. Issues in Current DiT-based I2V Models: Semantic-Weak Layers are layers with lower Moran’s I values and weaker responses to text prompts.
  • 3.2. Issues in Current DiT-based I2V Models: A higher standard deviation of normalized similarity indicates more focused, less uniform, and more semantically significant text-conditioned attention.

4. Method: Focal Guidance Framework

Focal Guidance improves controllability in DiT-based I2V models through Fine-grained Semantic Guidance, which binds textual concepts to reference-image regions, and Attention Cache, which transfers semantic attention to weak layers.

  • Focal Guidance Framework: Focal Guidance combines Fine-grained Semantic Guidance and Attention Cache to enhance semantic controllability in DiT-based I2V models.FSG couples multimodal conditions, while Attention Cache transfers structured attention from responsive layers to weak ones.
  • Fine-grained Semantic Guidance: FSG injects visual anchors into text and visual features, establishing fine-grained correspondence before attention computation.The anchors enrich text-token values and are also injected into localized latent regions, while query and key vectors remain unchanged.
  • Fine-grained Semantic Guidance: FSG selects text keywords by text–image similarity and uses corresponding visual regions as anchors for cross-modal conditioning.Keywords enter the set K when their maximum similarity with spatial visual tokens exceeds τsel.
  • Qualitative Evaluation: FG strengthens text–reference alignment so motions and dynamic attributes follow instructions more faithfully in qualitative comparisons.The framework is compared across mainstream open-source I2V models and ablated on Wan2.1-I2V.
  • Attention Cache: Attention Cache aggregates keyword–visual similarity maps from semantic-responsive layers and uses them to guide Semantic-Weak Layers.Only positions above τcache are retained, and semantic-weak layers receive guidance from the cached attention patterns.
  • Attention Cache: Attention Cache assigns zero weight to semantic-weak layers and aggregates the remaining layer maps to guide condition injection.For other layers, α_l = 1/(L−m), where m is the number of semantic-weak layers.

5. Experiments

Experiments evaluate Focal Guidance on two open-source I2V architectures using a benchmark that measures instruction following alongside subject and background consistency. FG improves semantic control, while ablations show gains from explicit guidance and complementarity with post-training.

  • Evaluation Setup: The evaluation covers Wan2.1-I2V and HunyuanVideo-I2V, using a benchmark that measures dynamic attributes, human motion, and human interaction.The benchmark also retains I2V Subject and I2V Background metrics for reference-frame consistency.
  • Ablation Study: Additional post-training data has little impact on conventional metrics, and standard post-training alone yields only marginal gains such as Human Motion from 0.6920 to 0.6980.
  • Main Results: FG-Wan2.1 shows higher textual responsiveness than Wan2.1, although traditional metrics focused on fidelity do not reflect this improvement.
  • Evaluation Setup: The Total Score averages I2V Subject, I2V Background, Dynamic Attributes, Human Motion, and Human Interaction.
  • Main Results: 3.97% (0.6973→0.7250) improves Wan2.1-I2V’s Total Score with FG, while HunyuanVideo-I2V improves by 7.44% (0.5185→0.5571).
  • Ablation Study: FG without post-training significantly boosts Human Motion to 0.7020 and Human Interaction to 0.5220, while combining FG with post-training achieves the best performance.

6. Conclusion, Limitation and Future Work

The conclusion identifies Semantic-Weak Layers and Condition Isolation as barriers to text controllability in DiT-based I2V models, and presents Focal Guidance and a dedicated instruction-following benchmark. It also notes that FSG depends on the image encoder and base-model capabilities.

  • Conclusion: Semantic-Weak Layers respond weakly to text prompts, limiting text-aligned video generation and increasing reliance on visual priors.
  • Conclusion: Focal Guidance addresses this issue, while the proposed benchmark automatically assesses alignment between generated videos and their prompts.
  • Limitation and Future Work: FSG effectiveness is constrained by the underlying image encoder and the base model’s capabilities.

Appendix

The appendix provides qualitative examples spanning human motion, human interaction, and dynamic attribute changes. These examples include object actions, interactions, and gradual attribute transformations.

  • Human Motion: Examples of human motion include a man taking off his hat while singing and a man taking off his earphones.
  • Human Interaction: Interaction examples include an old man and woman shaking hands and a woman handing a book to a man.
  • Human Interaction: Additional interaction examples depict two women or two people shaking hands.
  • Dynamic Attributes: Other dynamic examples depict a butterfly changing from brown to green and the moon changing from silver to yellow.
  • Dynamic Attributes: Dynamic-attribute examples show leaves changing color, fish changing from blue to red, and flowers changing from white to red.
  • Qualitative Examples: The qualitative examples cover human motion, human interaction, and dynamic attribute changes.The figure describes the outputs as realistic, temporally consistent, and semantically coherent.

A. Experimental Setup

Focal Guidance is trained and evaluated as a model-agnostic controllability method using limited data and model-specific configurations. The setup includes adaptive reference-image crops derived from annotated subject boxes.

  • Dataset: FG uses an internally curated 12K-sample dataset with accurate captions for limited post-training and aims to preserve the model’s original capabilities.
  • Dataset: The method is described as model-agnostic and applicable across underlying models or datasets.
  • Implementation Details: Evaluation uses CrossDiT-based Wan2.1-I2V and MMDiT-based HunyuanVideo-I2V configurations with FG applied to designated Semantic-Weak Layers.
  • Reference Images: For the benchmark, subject bounding boxes produce canonical 16:9 and 1:1 crops, enabling adaptive-resolution reference images for I2V generation.

B. Controllability Evaluation and Dataset Annotation

The benchmark evaluates I2V instruction following across dynamic attributes, human motion, and human interaction using video-based multi-question answering and annotated image–prompt datasets.

  • Metric Design: The evaluation targets dynamic attributes, human motion, and human interaction to assess adherence to textual and visual conditions.These dimensions address controllability beyond visual quality and subject consistency.
  • Metric Design: Video-based multi-question answering checks whether actions or attributes in the reference image are generated according to the text.The framework reduces evaluation noise because I2V has lower content freedom than text-to-video generation.
  • Metric Design: The evaluation uses questions Q, generated video V, and the prompt’s semantic structure S to determine instruction-following scores.The formulation is presented as VQA(Q_i, V | S).
  • Dataset Annotation and Cropping: The datasets contain 86, 100, and 100 image–prompt pairs for dynamic attributes, human motion, and human interaction, respectively.They yield 258, 278, and 303 corresponding questions, respectively.
  • Dataset Annotation and Cropping: Reference images are annotated with subject bounding boxes and cropped using an aspect-ratio–aware protocol to keep subjects visible.This protocol follows VBench-I2V.

C. Qualitative Results

Qualitative results for FG-Wan2.1-I2V 14B cover human motion, human interaction, and dynamic attributes, illustrating realistic and temporally expressive generation across scenarios.

  • Qualitative Results: FG-Wan2.1-I2V 14B qualitative examples span human motion, human interaction, and dynamic attributes.The examples are presented as additional results for the best-performing model.
  • Qualitative Results: The examples illustrate realistic, coherent, and temporally expressive videos under varied scenarios.These observations are reported for the three instruction-following dimensions.
Loading 2601.07287v1…