Source-linked AI summary
When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models
Zhengyang Sun, Yu Chen, Xin Zhou, Xiaofan Li, Xiwu Chen, Dingkang Liang, Xiang Bai
TL;DR
Text-to-video models struggle to represent precise object counts because numeral attention is diffuse and latent instances are ambiguous. NUMINA identifies these inconsistencies with attention-derived layouts, refines them under count constraints, and guides regeneration; on CountBench, it improves counting accuracy while preserving alignment and temporal coherence.
Problem
Text-to-video models often fail to align prompt numerals with accurate object counts because numeral attention is diffuse and DiT latents poorly separate instances.
Method
NUMINA is a training-free identify-then-guide framework that extracts countable layouts from discriminative attention heads, refines them, and guides regeneration.
Results
NUMINA improves counting accuracy across CountBench models, increases CLIP alignment, and maintains temporal coherence during numerically guided generation.
Takeaways & Limitations
Structural guidance complements existing seed-search and prompt-enhancement approaches, providing a practical route toward count-accurate text-to-video generation.
Takeaways & Limitations
Instance-separable attention can focus on salient object parts, over-segmenting one instance into multiple objects and propagating an irrecoverable layout error.
Abstract
from arXiv · showhide
Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.
1 Huazhong University of Science and Technology, 2 Zhejiang University, 3 Afari Intelligent Drive
The listed affiliations identify Huazhong University of Science and Technology, Zhejiang University, and Afari Intelligent Drive, with contribution and correspondence markers supplied separately.
- The author affiliations include Huazhong University of Science and Technology, Zhejiang University, and Afari Intelligent Drive.
1. Introduction
Text-to-video models often fail to align prompt numerals with object counts, motivating NUMINA, a training-free identify-then-guide framework evaluated on CountBench.
- Text-to-video models often prioritize fidelity, motion smoothness, and semantic alignment but struggle to represent requested object counts accurately.This limits reliability in precision-sensitive applications such as instructional visualization.
- Numeral tokens show diffuse cross-attention, while downsampled DiT latents limit separability of individual object instances.These factors weaken count control and make stable numerical alignment difficult.
- NUMINA is a training-free framework that identifies prompt-layout inconsistencies and guides generation while preserving visual fidelity and temporal coherence.Its identify-then-guide paradigm supports natural instance-level addition and removal.
- NUMINA detects misalignment early, selects discriminative attention heads, segments layouts, refines them under count constraints, and regenerates guided videos.The two phases separately establish an explicit layout and use it for numerically aligned generation.
- 7.4% counting-accuracy improvement is reported on Wan2.1-1.3B, with additional gains of 5.5% on a 14B model in CountBench evaluation.CountBench contains 210 prompts spanning counts from 1-8 and scenes with 1-3 object categories.
- The contributions include exposing instance information in attention, introducing training-free count guidance, and demonstrating practical count-accurate text-to-video generation.
2. Related Work
Prior work advances T2V generation and editing but leaves precise numerical control challenging; NUMINA provides training-free global guidance without videos, masks, or auxiliary re-layout networks.
- DiT-based T2V models synthesize coherent, high-fidelity videos by injecting textual semantics through attention in compressed latent spaces.
- Existing T2V editing methods primarily address motion, style, appearance, and video inpainting, often using object masks from segmentation models.
- Precise numerical constraints remain difficult in generative models, while Count-Gen targets static text-to-image generation with SDXL-specific observations and additional training.
- NUMINA offers training-free global guidance for T2V without input videos, spatial masks, or auxiliary re-layout networks, while preserving strict temporal consistency.
3. Preliminary
T2V models inject text through cross-attention, but numeral activations are diffuse because standard attention handles localized attributes better than global count constraints; NUMINA extracts explicit layouts from attention.
- DiT-based T2V pipelines evolve Gaussian noise into video latents conditioned on text prompts, with prompt information injected mainly through multi-head cross-attention.
- Cross-attention maps encode relevance between visual latent features and text tokens, but numeral tokens often produce diffuse, low-contrast activations.
- Standard cross-attention is effective for localized attributes but struggles with global numerical constraints, so larger models or more training data may not suffice.
- NUMINA extracts an instance-aware layout from attention and enforces the desired count during regeneration to provide global numerical guidance.
- Self-attention heads exhibit diverse spatial patterns, motivating selection of an instance-separable head for countable layout construction.
4. Method
NUMINA uses an identify-then-guide pipeline that converts attention into a countable layout, corrects instance counts conservatively, and guides regeneration toward the corrected structure.
- Numerical Misalignment Identification: NUMINA first selects discriminative self- and cross-attention heads, then fuses their maps into an explicitly countable instance-level layout.Self-attention provides spatial structure, while cross-attention provides focused semantic alignment for each noun token.
- Numerical Misalignment Identification: Dynamic head selection targets self-attention maps with instance-separating structure and cross-attention maps with concentrated responses for noun tokens.Self-attention heads are scored using foreground-background separation, structural richness, and edge clarity; cross-attention heads are selected by peak activation concentration.
- Countable Layout Construction: Spatial proposals are clustered from self-attention, semantic focus regions are clustered from thresholded cross-attention, and proposals are retained when their overlap exceeds τ.The resulting semantic map contains disjoint foreground regions whose number provides an explicit object count.
- Numerically Aligned Video Generation: Layout refinement adds or removes object instances until each frame’s region count matches the prompt target, guided by minimal structural change.This conservative correction preserves the intrinsic coupling between sampled noise and prompt semantics rather than aggressively manipulating the latent space.
- Numerically Aligned Video Generation: Layout-guided generation modifies selected attention scores or biases with stronger early guidance and weaker late guidance to preserve fine-grained details.New template instances use bias modification, while reference-region instances overwrite pre-softmax scores to promote consistency.
5. Experiments
NUMINA is evaluated on CountBench against practical training-free baselines across Wan models, with analyses of numerical alignment, generation quality, and design choices. It consistently improves counting accuracy while preserving or improving semantic and temporal quality, and ablations examine efficient layout construction and inference overhead.
- Experiment Setup: CountBench evaluates numerical fidelity across 210 prompts, counts from 1 to 8, and scenes containing 1 to 3 object categories.The benchmark targets complex prompts with multiple numerical constraints.
- Experiment Setup: The evaluation reports Counting Accuracy, Temporal Consistency, and CLIP Score to measure numerical alignment, count stability, and generation quality.Counting Accuracy averages class-level matches across frames, while Temporal Consistency compares counts between adjacent frames.
- Main Results: 49.7% CountAcc is achieved by NUMINA on Wan2.1-1.3B, compared with 42.3% for a single trial, 45.5% for Seed search, and 47.2% for Prompt enhancement.On larger models, NUMINA exceeds the 5B and 14B baselines by 4.9% and 5.5%, respectively.
- Main Results: CLIP Score rises from 33.9 to 35.6 for the 1.3B model, while Temporal Consistency reaches 84.0% for the 14B model.The authors report that instance-level guidance preserves temporal coherence without introducing flickering or temporal artifacts.
- Main Results: For eight-object prompts, NUMINA raises accuracy from the 1.3B baseline’s 11.3% to 20.7%, while three-object accuracy improves by 16.2%.The gains become more pronounced as the requested count increases.
- Analysis and Ablation Study: Attention-derived layouts outperform GroundingDINO-derived layouts by 2.2%, and timestep 20 reaches 49.7% CountAcc as a favorable accuracy-efficiency trade-off.Using timestep 40 provides only a 3.2% relative gain while doubling pre-generation cost; later timesteps reduce accuracy.
6. Conclusion
NUMINA is a training-free framework for count alignment that improves numerical accuracy while preserving video quality. Perfect accuracy remains challenging, especially for very dense instance counts.
- NUMINA uses explicit layout construction, conservative refinement, and layout-guided generation to correct prompt-layout inconsistencies.The framework leverages instance-separable attention heads in DiTs.
- NUMINA significantly improves counting accuracy, particularly at higher counts where baselines falter, without sacrificing video quality.
- Perfect accuracy across all scenarios remains challenging, and generating very dense instances such as tens or hundreds remains unexplored.The paper identifies fully numerically precise video generation for any number as future work.
S1. Additional Results
Additional experiments show that NUMINA generalizes across architectures and enhancement strategies, improves count accuracy, and preserves visual and temporal quality. Ablations further identify effective guidance operations and stable layout-refinement choices.
- Cross-architecture evaluation: 44.4% accuracy is achieved by NUMINA on CogVideoX-5B, compared with 40.2% under minimal settings.NUMINA also raises TC and CLIP scores to 80.2% and 35.4%, respectively, using simple prompts and one generation pass.
- Enhancement integration: 54.2% counting accuracy is achieved by combining NUMINA with prompt enhancement and seed search, exceeding standalone NUMINA at 49.7%.The combined approach also surpasses prompt enhancement at 47.2% and seed search at 45.5%.
- Quality and temporal consistency: NUMINA maintains competitive VBench Subject-Consistency performance, indicating that edited instances remain temporally stable and visually coherent.The metric averages DINO-feature cosine similarities with the first and preceding frames across instances and non-initial frames.
- Ablations: The circular layout-refinement prior reaches 49.7% accuracy without reference instances, compared with 49.5% for the rectangular prior and 48.8% without intervention.The circular prior is used because it minimizes structural assumptions.
- Ablations: Performance remains stable across a wide range of hyperparameter values, with t⋆= 20 and ℓ⋆= 15 used for the ablation study.Layer and timestep selections vary with intrinsic model differences rather than exhaustive heuristic tuning.
- Ablations: Combining object addition and removal yields the highest accuracy, with addition alone providing a 5.4% gain and removal a 1.5% gain.The results suggest object omission is the baseline model’s primary difficulty and indicate synergy between the two operations.
- Quality and temporal consistency: 61% of blind-study participants preferred NUMINA to the baseline, versus 39%, across 100 randomly sampled video pairs.The 10-participant study evaluated visual quality and instruction following.
S2. More Visualization
Qualitative visualizations show successful numerical alignment across Wan2.1 and CogVideoX, alongside a characteristic failure mode in which salient object parts are mistaken for separate instances.
- Successful cases: Additional demonstrations show successful numerical alignment across Wan2.1 and CogVideoX model architectures.The examples cover prompts specifying varied object counts and support the method’s cross-architecture generalizability.
- Failure cases: A failure case occurs when attention heads focus on salient object parts, such as a parrot’s head, rather than the whole instance.This produces an over-segmented layout in which parts of one object are mistaken for multiple objects.
- Failure cases: The resulting segmentation error propagates into an irrecoverable error in the final video output.The paper suggests incorporating more holistic perceptual grouping cues in future work.
- Successful cases: The qualitative examples include scenes with specified counts of tourists, suitcases, luggage trolleys, skiers, robots, and humans.