Source-linked AI summary
When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models
Zhengyang Sun, Yu Chen, Xin Zhou, Xiaofan Li, Xiwu Chen, Dingkang Liang, Xiang Bai
TL;DR
Text-to-video diffusion models often fail to generate the object counts specified in prompts, limiting reliability in precision-sensitive applications. NUMINA uses a training-free identify-then-guide framework, improving counting accuracy by 7.4% on Wan2.1-1.3B and 5.5% on 14B.
Problem
Text-to-video models often struggle to align generated object counts with prompts, limiting their reliability in precision-sensitive applications such as instructional visualization.
Source: §1 · p. 1Method
Results
NUMINA improves counting accuracy by 7.4% on Wan2.1-1.3B and 5.5% on 14B, while increasing CLIP scores and maintaining temporal coherence.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
Structural guidance offers a practical complement to seed search and prompt enhancement for count-accurate text-to-video generation.
Source: §6 · p. 8NUMINA can over-segment objects when attention heads focus on salient parts rather than the whole, causing a single instance to be mistaken for multiple objects.
Source: S2. More Visualization · p. 14
Abstract
from arXiv · showhide
Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.
1. Introduction
NUMINA is a training-free identify-then-guide framework for improving numerical alignment in text-to-video generation while preserving visual fidelity and temporal coherence. It uses attention-derived latent layouts to guide count-constrained regeneration.
1. Introduction
- Numerical alignment is limited by diffuse numeral cross-attention and by downsampled latent representations that make individual object instances difficult to separate.Retraining is computationally prohibitive and requires difficult-to-construct numerical annotations, while increasing numeral attention risks visual quality and motion coherence.Source: §1 · pp. 1–2
- The training-free framework explores latent instance separation while allowing natural instance addition and removal, aiming to retain appearance, motion, and semantics across T2V models.The paper presents NUMINA as a lightweight alternative to retraining that enhances alignment between object counts and prompt instructions.Source: §1 · p. 2
- NUMINA detects numeral-layout misalignment early in denoising, selects discriminative attention heads, segments the latent layout, then adjusts it under count constraints to guide regeneration.The adjustment heuristically considers spatial relationships between instances; attention exposes visual information related to instance counts.Source: §1 · p. 2
- On CountBench, NUMINA improves counting accuracy by 7.4% on Wan2.1-1.3B [59] and 5.5% on a 14B model, with consistent CLIP-score increases across baselines.CountBench contains 210 prompts with counts from 1–8 and 1–3 object categories; NUMINA also integrates with EasyCache [73] to reduce inference overhead.Source: §1 · p. 2
2. Related Work
Despite advances in text-to-video generation and editing, models still struggle to ground textual numerals and add the specified number of instances. NUMINA is positioned against prior image-counting methods as training-free global guidance for T2V generation without input videos, spatial masks, or auxiliary re-layout networks.
2.1. Diffusion Transformer for Video Generation
- T2V models have advanced from early 3D U-Nets to scalable DiT frameworks, yet weak numeral grounding and instance separability still cause numerical misalignment.Leading models synthesize coherent, high-fidelity videos and inject textual semantics through attention in compressed latent spaces.Source: §2.1 · p. 2
2.2. Video Editing for T2V Models
- Video editing methods focus on motion, style, appearance, or object removal, but overlook instance-level addition needed to align textual numerals with visual content.Most methods operate video-to-video and rely on object masks from segmentation models.Source: §2.2 · pp. 2–3
2.3. Counting in Vision and Generation
- Count-Gen [4] targets static-image counting with SDXL-specific observations and added networks and masks, whereas NUMINA provides training-free global T2V guidance while preserving temporal consistency.NUMINA requires no input videos, spatial masks, or auxiliary re-layout networks.Source: §2.3 · p. 3
3. Preliminary
Text-to-video DiTs condition video latents on text through cross-attention, but this mechanism can represent localized attributes more effectively than global numerical constraints. NUMINA addresses this gap with training-free layout guidance during denoising.
3. Preliminary
- Recent T2V models [7] [37] [59] use DiT [53] with flow-matching [43] [46] or diffusion sampling [51] [53] to evolve Gaussian noise into text-conditioned video latents.In vanilla DiT blocks, prompts are injected mainly through multi-head cross-attention.Source: §3 · p. 3
- Cross-attention matches visual latent queries with text keys, producing head-wise maps that encode relevance between each visual and text token.The visual latent provides Q, text embeddings provide K, and each attention map has dimensions N × L.Source: §3 · p. 3
- Cross-attention handles localized attributes but struggles with global constraints, leaving numeral-token activations diffuse and low-contrast.Thus, simply increasing training data or model size may not suffice to encode numerical constraints faithfully.Source: §3 · p. 3
- NUMINA introduces training-free global guidance by extracting an instance-aware layout from attention and enforcing the desired count during regeneration.The framework exposes prompt-layout inconsistencies early in denoising and then corrects them.Source: §3 · p. 3
4. Method
NUMINA uses a training-free identify-then-guide pipeline: it extracts an explicit countable layout from attention, then corrects that layout and uses it to guide video regeneration.
4.1. Numerical Misalignment Identification
- Selected separately for each noun, an instance-discriminative self-attention head and a text-concentrated cross-attention head provide spatial structure and focused semantic alignment.Head patterns vary substantially: naive averaging or random selection can blur instance boundaries, while cross-attention heads are selected by their token map’s peak activation.Source: §4.1 · p. 4
- Fusing instance-discriminative self-attention with noun-aligned cross-attention yields disjoint foreground regions whose number explicitly estimates the object count.Self-attention regions are filtered by overlap with a focus mask derived from cross-attention; cross-attention responses below a 0.1 peak-ratio threshold are suppressed before clustering.Source: §4.1 · pp. 4–5
4.2. Numerically Aligned Video Generation
- Layout refinement minimally adjusts each frame to the prompt’s target count, removing the smallest instance or adding a copied or circular template at a location chosen to limit overlap and temporal disruption.Placement balances collision avoidance, proximity to the existing layout’s center, and consistency with the instance’s previous-frame position.Source: §4.2 · p. 5
- The refined layout guides training-free cross-attention modulation, with stronger guidance early and weaker guidance later; removal suppresses attention, while addition boosts new regions using a circle bias or scores transferred from an existing region.The guidance intensity decreases over denoising steps to preserve fine-grained detail.Source: §4.2 · pp. 5–6
5. Experiments
The experiments introduce CountBench for numerical alignment and evaluate NUMINA across Wan model scales against practical training-free baselines. NUMINA improves counting accuracy while maintaining video-text alignment and temporal consistency, with ablations examining its layout guidance and computational overhead.
5.1. Experiment Setup
- CountBench evaluates 210 prompts with counts from 1–8 and 1–3 object categories, addressing numerical scenarios underrepresented by existing T2V benchmarks.CountAcc and temporal consistency are measured using GroundingDINO-detected object counts; CLIP score assesses semantic alignment with prompts.Source: §5.1 · p. 6
5.2. Main Results
- 49.7% CountAcc on Wan2.1-1.3B exceeds single-trial, Seed search, and Prompt enhancement results of 42.3%, 45.5%, and 47.2%; NUMINA also beats 5B and 14B baselines by 4.9% and 5.5%.The 1.3B model’s 49.7% also exceeds Wan2.2-5B’s 47.8%.Source: Table 1 · p. 6, §5.2 · p. 6
- CLIP score rises from 33.9 to 35.6 for the 1.3B model, while NUMINA reaches 84.0% TC on the 14B model.Despite adding or removing objects, the method maintains temporal coherence; qualitative examples also show exact counts and natural layouts where commercial models often fail.Source: Table 1 · p. 6, §5.2 · p. 7, Figure 5 · p. 7
5.3. Analysis and Ablation Study
- At t⋆=20, CountAcc reaches 49.7%; increasing t⋆ to 40 yields only a 3.2% relative gain while doubling pre-generation cost, and later timesteps reduce accuracy.The authors select t⋆=20 as an accuracy-efficiency trade-off because early denoising forms instance-separable attention, while later attention may be fragmented or over-fused.Source: Figure 7 · p. 7, §5.3 · pp. 7–8
5.2. Main Results
- For eight-object prompts, NUMINA reaches 20.7% accuracy versus the baseline’s 11.3%; its advantage also includes a 16.2% improvement for three-object prompts.Baseline accuracy is 68.7% for two objects but falls to 44.5% for three, with performance declining as requested counts rise.Source: §5.2 · p. 7
5.3. Analysis and Ablation Study
- Attention-derived layouts outperform detector-derived layouts by 2.2%, and combining overlap, center, and temporal refinement costs achieves 49.7% accuracy.Overlap alone improves accuracy by 2.8%; adding center cost reaches 46.9%, while temporal cost yields 48.9%, indicating complementary contributions.Source: Table 2 · p. 8, Table 3 · p. 8, §5.3 · p. 8
- Top-1 self-attention head selection by S(SAh) reaches 49.7% accuracy, compared with 44.1% for a random head and 43.0% for averaging all heads.Performance degrades when averaging fewer discriminative heads, supporting the finding that instance-separable information is concentrated in a few heads.Source: Table 4 · p. 8, §5.3 · p. 8
- NUMINA integrates with EasyCache to reduce preprocessing overhead with minimal VRAM use and acceptable runtime, offering a deterministic alternative to exhaustive seed search.The preprocessing stage constructs a coarse latent layout and avoids high-precision computation.Source: Table 5 · p. 8, §5.3 · p. 8
6. Conclusion
NUMINA is a training-free framework that uses instance-separable attention heads to identify and correct prompt-layout inconsistencies for count alignment in text-to-video diffusion. It improves counting accuracy, especially at higher counts, without sacrificing video quality, though perfect accuracy remains challenging and very dense instances remain unexplored.
6. Conclusion
- NUMINA boosts counting accuracy, particularly at higher counts where baselines falter, without sacrificing video quality.The paper presents structural guidance as a complement to existing methods for count-accurate text-to-video generation.Source: §6 · p. 8
- NUMINA uses instance-separable attention heads in DiTs to construct and conservatively refine layouts, then guide generation to correct prompt-layout inconsistencies.The framework is training-free.Source: §6 · p. 8
- Perfect numerical accuracy across all scenarios remains challenging, and generating very dense instances, such as tens or hundreds, remains unexplored.The paper identifies fully numerically precise video generation for any number as an important direction for future research.Source: §6 · p. 8
Supplementary Material · S1. Additional Results
Supplementary results show that NUMINA improves counting when combined with prompt enhancement and seed search and when applied to CogVideoX-5B. Additional analyses examine layout refinement, temporal consistency, hyperparameter stability, and visual quality.
S1. Additional Results
- 44.4% accuracy on CogVideoX-5B with NUMINA exceeds its 40.2% minimal-setting accuracy; seed search and prompt enhancement add only 2.5% and 2.3%, respectively.NUMINA also raises the reported TC and CLIP scores to 80.2% and 35.4%, respectively. For MMDiT, the method decomposes unified attention into video-to-video self-attention and text-to-video cross-attention.Source: Table 6 · p. 12, S1. Additional Results · p. 12
- 54.2% counting accuracy is achieved by combining NUMINA with prompt enhancement and seed search, exceeding standalone NUMINA (49.7%), prompt enhancement (47.2%), and seed search (45.5%).The combined approach also exceeds the reported Wan2.2-5B (47.8%) and Wan2.1-14B (53.6%) baseline accuracies.Source: Table 7 · p. 12, S1. Additional Results · p. 12
- NUMINA achieves competitive VBench Subject-Consistency performance, indicating that edited instances remain temporally stable and visually coherent.The score averages DINO-feature cosine similarities to both the first and preceding frames across non-initial frames and instances; this metric captures a complementary aspect of temporal coherence to TC.Source: Table 8 · p. 12, S1. Additional Results · p. 12
- 49.7% accuracy with a circular prior and 49.5% with a rectangular prior improve on the 48.8% no-intervention baseline for adding instances without references.The circular prior is used in practice to minimize structural assumptions and leave models flexibility to form contextually plausible objects.Source: S1. Additional Results · pp. 12–13, Table 9 · p. 13
- Instance-separable attention layouts emerge most clearly at middle denoising steps, with intermediate layers providing the sharpest spatial separation.The analysis uses t⋆=20 and ℓ⋆=15 for localization and early stopping, reducing pre-generation denoising steps by approximately 60% without significantly sacrificing accuracy; performance remains stable across a wide range of hyperparameter values.Source: S1. Additional Results · p. 13, Figure 8 · p. 13, Table 10 · p. 13
- Object addition boosts accuracy by 5.4%, removal yields a 1.5% gain, and combining both operations achieves the highest accuracy, slightly exceeding the sum of their individual gains.The results indicate that the baseline more often omits objects than includes extraneous ones.Source: Table 11 · p. 13, S1. Additional Results · p. 13
- 61% of participants preferred NUMINA to the baseline; VBench Aesthetic & Imaging Quality scores remained comparable or superior without degrading video quality.The blind study included 10 participants and 100 randomly sampled video pairs.Source: Table 12 · p. 13, S1. Additional Results · p. 13
S2. More Visualization
NUMINA’s additional visualizations show successful count alignment across Wan2.1 and CogVideoX, while a failure case reveals that attention can split object parts into separate instances.
Failure cases
- A failure occurs when instance-separable attention heads focus on salient parts, such as an animal’s head, and mistake them for separate objects.This over-segments the layout and propagates an irrecoverable error into the video; the paper suggests holistic perceptual grouping cues as future work.Source: S2. More Visualization · p. 14, Figure 9 · p. 14
Additional demos
- Figure 10 shows successful object-count alignment examples across Wan2.1 and CogVideoX, demonstrating improvement across model architectures.The examples are described as representative cases where the method faithfully generates the specified number of objects.Source: S2. More Visualization · p. 13, Figure 10 · p. 15