PaperlayerView exact revision on arXiv

Source-linked AI summary

MATRIX: Mask Track Alignment for Interaction-aware Video Generation

Siyoon Jin, Seongchan Kim, Dahyun Chung, Jaeho Lee, Hyunwook Choi, Jisu Nam, Jiyoung Kim, Seungryong Kim

arXiv:2510.07310v2cs.CV

TL;DR

Video DiTs still struggle to represent multi-instance interactions and maintain subject–object bindings over time. The paper analyzes attention using MATRIX-11K mask tracks and introduces MATRIX to align attention in interaction-dominant layers. MATRIX improves interaction fidelity and grounding while reducing drift and duplication.

  • Problem

    Video DiTs struggle to localize prompted subjects and objects, bind their interactions, and preserve those bindings across frames.

    Source: §1 · p. 2

  • Method

    The paper curates MATRIX-11K, analyzes grounding and propagation in video DiT attention, and aligns attention in interaction-dominant layers with mask tracks using MATRIX.

    Source: §1 · p. 2

  • Results

    MATRIX improves interaction fidelity, strengthens noun and verb grounding, and reduces drift and duplication.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Ablations confirm the importance of layer selection and the complementary contributions of SGA and SPA.

    Source: §7 · p. 10

    Very small instance masks can provide weak or ambiguous grounding signals, potentially degrading accurate motion generation.

    Source: I LIMITATIONS · p. 37

Abstract

from arXiv · show

Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.

1 INTRODUCTION

Video DiTs struggle to ground subject–object interactions and maintain those bindings over time. The paper analyzes grounding and propagation with mask tracks, then uses the findings to motivate MATRIX and an interaction-aware evaluation protocol.

1 INTRODUCTION

  • Video DiTs struggle to localize prompted subjects and objects, bind their verb-specified interaction, and preserve that grounding over time.These failures can produce text-video mismatch, drift, duplication, or hallucination.Source: §1 · p. 2
  • Video-to-text attention maps nouns to subject or object regions and verbs to their union, while video-to-video attention links those regions across frames.The analysis uses multi-instance mask tracks for spatial precision, temporal continuity, and instance separability.Source: §1 · p. 2, Figure 3 · p. 3, Figure 4 · p. 3
  • Grounding and propagation concentrate in a small subset of interaction-dominant layers, with stronger alignment in successful generations than failures.The analysis uses MATRIX-11K, a dataset of 11K videos with interaction-rich captions and instance mask tracks.Source: §1 · p. 2
  • MATRIX aligns attention in interaction-dominant layers with multi-instance mask tracks through SGA and SPA losses and a lightweight causal decoder.SGA aligns noun tokens with subject/object regions and verb tokens with their union; SPA promotes consistent instance tracks across frames.Source: §1 · pp. 2–3
  • InterGenEval assesses interaction-aware generation with KISA, SGI, and SPI, reporting interaction fidelity as the mean of KISA and SGI.KISA checks key interaction stages, SGI evaluates subject, object, and verb grounding, and SPI measures temporal persistence.Source: §1 · p. 3

2 RELATED WORK

Prior work studies image and video diffusion representations, human-object interaction synthesis, relation customization, and controllable video generation. The cited approaches do not formalize or directly control open-vocabulary, multi-instance interaction semantics in video diffusion.

2 RELATED WORK

  • Prior work on image and video diffusion representations does not formalize interaction representations, leaving interaction analysis in video DiTs unaddressed.The paper defines interactions through token-level semantic grounding and temporal semantic propagation, analyzed through attention.Source: §2 · pp. 3–4
  • Human-object interaction synthesis ranges from 2D recognition and localization to 3D motion generation, but remains restricted to motion-level synthesis.The cited methods have not been integrated into video diffusion, where interaction modeling must govern pixel generation.Source: §2 · p. 4
  • Relation-specific adapters and motion priors rely on closed verb sets, while controllable video methods remain agnostic to interaction semantics.Multi-instance controls such as boxes or mask sequences operate independently of text, leaving subject-action-object relations under-specified.Source: §2 · p. 4

3 MATRIX-11K DATASET

MATRIX-11K pairs interaction-aware captions with per-instance mask tracks through an interaction-aware curation pipeline. Caption processing extracts and filters interaction triplets, and mask construction verifies instance boxes before propagating tracks and manually filtering residual errors.

3 MATRIX-11K DATASET

  • MATRIX-11K pairs videos and interaction-aware captions with instance mask tracks, using a curation pipeline to align captions and masks.The dataset is intended for analyzing and training semantic binding in video DiTs.Source: §3 · p. 4

3.1 INTERACTION-AWARE CAPTIONING

  • Caption processing extracts subject–verb–object triplets and instance IDs, filters interactions by dynamism and contactness, and records appearance descriptions.IDs not linked to retained interactions are removed; appearance descriptions help distinguish same-class instances during verification.Source: §3.1 · pp. 4–5

3.2 MULTI-INSTANCE & INTERACTION MASK TRACKS

  • For each instance, the pipeline verifies candidate boxes with a VLM, initializes SAM2 from the first verified anchor, propagates masks, and manually filters residual errors.Instances without a verified candidate are removed, along with linked interactions; videos with no remaining valid interactions are discarded.Source: §3.2 · p. 5

4 INTERACTION-AWARENESS ANALYSIS IN VIDEO DITS

The analysis treats semantic grounding as token-to-region alignment and semantic propagation as persistence of those bindings across frames. It finds that alignment concentrates in a small set of layers, with some influential layers separating successful from failed generations.

4 INTERACTION-AWARENESS ANALYSIS IN VIDEO DITS

  • The analysis asks whether textual tokens ground to visual regions and whether those bindings remain spatially coherent over time.It examines video-to-text attention for grounding and video-to-video attention for propagation.Source: §4 · p. 5, §4.1 · p. 5

4.1 PRELIMINARIES- VIDEO DIFFUSION TRANSFORMERS

  • Video DiTs use a unified sequence of video latents and text embeddings, whose full attention contains video-to-video and video-to-text regions among four attention directions.The analysis focuses on video-to-video and video-to-text attention.Source: §4.1 · p. 5

4.2 SEMANTIC GROUNDING

  • For semantic grounding, noun-token attention is evaluated against subject and object regions, while verb-token attention represents their interaction through joint attention to both.The verb interaction region is the per-frame union of subject and object masks.Source: §4.2 · p. 6

4.3 SEMANTIC PROPAGATION

  • For semantic propagation, attention from first-frame subject and object regions is tracked across spatiotemporal tokens to assess identity-consistent regions and a stable interaction region.The resulting per-frame heatmaps shift the analysis from token alignment to temporal consistency.Source: §4.3 · p. 6

4.4 EVALUATION METRIC: ATTENTION ALIGNMENT SCORE (AAS)

  • The Attention Alignment Score sums attention-map values at locations marked by the corresponding downsampled subject, object, or verb masks.The verb mask is the element-wise union of the subject and object masks.Source: §4.4 · p. 7

4.5 ANALYSIS

  • Alignment repeatedly concentrates in a few influential layers, while interaction-dominant layers separate successful from failed generations by their mean AAS.Influential layers rank highly across videos; dominant layers have relatively high AAS on successes and low AAS on failures.Source: Figure 7 · p. 6, §4.5 · p. 7
  • When attention concentrates on the corresponding instance or union regions, generations are correct and preferred; diffuse or mislocalized attention is associated with failures.Perturbation guidance sharpens attention toward instance regions and slightly improves interaction fidelity.Source: §4.5 · p. 7

5 MATRIX FRAMEWORK

MATRIX aligns attention with instance mask tracks using grounding and propagation losses. It applies those losses to interaction-dominant layers while retaining the denoising objective and freezing the backbone.

5 MATRIX FRAMEWORK

  • Semantic Grounding Alignment and Semantic Propagation Alignment losses align attention with instance mask tracks using a shared composite loss of BCE, soft Dice, and L2 regression.The losses target grounding and propagation, respectively.Source: §5 · p. 8
  • MATRIX conditions generation on noise, the first RGB frame, a first-frame multi-instance ID map, and a prompt with subject, verb, and object tokens.The ID map gives each instance a stable color across the clip and provides users control over targets at inference.Source: §5 · p. 8
  • A lightweight causal decoder maps attention to pixel-scale mask tracks so attention can be supervised against the ground-truth tracks.Its upsampling schedule follows the 3D VAE, with causal alignment at the first frame.Source: §5 · p. 8
  • MATRIX applies the alignment losses only to the interaction-dominant layer and adds them to the denoising loss while updating LoRA and selected components with the backbone frozen.The updated components include the input projection and lightweight decoder.Source: §5 · p. 8

6 EXPERIMENTS

MATRIX achieves the strongest reported interaction fidelity and leads on human fidelity and video quality, while ablations show complementary gains from grounding and propagation losses. Applying the method to Wan2.1-14B-I2V also improves interaction fidelity while preserving visual quality.

6.1 EXPERIMENTAL SETUP

  • InterGenEval reports KISA and SGI reweighted by SPI, with IF defined as the mean of KISA and SGI.The experimental setup uses 60 synthetic and 58 real-domain image-prompt pairs; HA, MS, and IQ provide additional measures.Source: §6.1 · p. 9

6.2 COMPARISON AND ANALYSIS

  • MATRIX achieves the strongest KISA, SGI, and IF, and the highest HA, IQ, and MS among the compared models.The comparison attributes the result to maintaining correct bindings and tracks with SGA and SPA on interaction-dominant layers.Source: §6.2 · p. 9, Figure 10 · p. 10

6.3 ABLATION STUDIES

  • Combining SGA and SPA yields the best overall balance, with strongest KISA, SGI, and IF, best HA, and improved video quality over the ablation baselines.SGA alone boosts grounding while retaining propagation; SPA alone enhances propagation but trades off noun/verb alignment.Source: §6.3 · p. 10

6.4 RESULTS WITH OTHER DIT BACKBONES

  • On Wan2.1-14B-I2V, MATRIX consistently improves interaction fidelity while preserving overall visual quality with minimal architectural modifications.The model is fine-tuned solely with SGA and SPA objectives.Source: §6.4 · p. 10

7 CONCLUSION

The paper studies interaction binding in video DiTs, finding grounding and propagation concentrated in a few interaction-dominant layers and using those findings to guide MATRIX. Its appendices detail dataset curation, interaction annotation, and instance-track verification.

7 CONCLUSION

  • Grounding and propagation concentrate in a small set of interaction-dominant layers, motivating MATRIX's SGA and SPA alignment of attention to mask tracks.The conclusion reports improved interaction fidelity and stronger noun and verb grounding, with reduced drift and duplication; ablations support the roles of layer selection and both losses.Source: §7 · p. 10

APPENDIX

  • The appendix describes dataset curation, expanded analysis, model and guidance details, evaluation, additional results, limitations, and future directions.It assigns curation to Appendix A, analysis and additional models to B and C, and evaluation details to F and G.Source: APPENDIX · p. 17

A DATASET CURATION DETAILS

  • The curation pipeline uses interaction-aware prompts and VLM/LLM processing to produce filtered data for the dataset.The passage describes detailed prompt design and examples of filtered data.Source: A DATASET CURATION DETAILS · p. 17

A.1 DETAILS FOR INTERACTION-AWARE CAPTION PROCESSING DETAILS

  • Caption processing identifies valid active subject-object actions, assigns stable instance IDs and roles, scores contact and dynamism, and extracts per-instance descriptors.Descriptors include noun, appearance, and spatial cues; interactions are retained based on sufficient contact and motion, and unused instances are pruned.Source: §A.1 · p. 17

A.2 DETAILS FOR INTERACTION-AWARE MULTI-INSTANCE MASK TRACKS WITH VERIFICATION

  • Instance mask tracks provide instance-level precision, temporal continuity, and same-class disambiguation needed to supervise interaction structures.GroundingDINO supplies class-consistent proposals, while VLM verification checks candidates against appearance descriptors and selects one box per noun; unverified instances are dropped.Source: §A.2 · pp. 18–19
  • Human reviewers inspect ten frames per clip and accept tracks only when both subject and object are clean and consistent after any minor fixes.Tracks with identity drift, swaps, duplication, hallucination, or clear temporal gaps are dropped; about 5% of accepted clips receive a second-annotator spot check.Source: §A.2 · p. 18

A.3 DATASET EXAMPLES AND STATISTICS

MATRIX-11K is interaction-enriched and linguistically broad, with pairwise settings common in the corpus. The analysis evaluation set uses controlled, verified prompt-video pairs with pseudo-ground-truth mask tracks, alongside a real-domain set reconstructed and checked for interaction fidelity.

A.3 DATASET EXAMPLES AND STATISTICS

  • MATRIX-11K combines interaction-aware video-text sources and is enriched for dynamic, contact-rich interactions while spanning a broad range of dynamism at each contact level.Pairwise settings dominate, and the dataset shows broad distributions of interaction verbs and identity nouns beyond frequent human-related terms.Source: §A.3 · pp. 19–20

B.1 ANALYSIS EVALUATION DATASET

  • The controlled analysis set retains 50 validated prompt-video pairs with per-instance pseudo-ground-truth mask tracks for evaluating grounding and propagation.Prompts vary interaction direction, participant count, interaction type, and class; videos are fixed-seed, standardized, and human-reviewed.Source: §B.1 · pp. 20–22
  • A real-domain analysis set reconstructs selected captioned clips with image-to-video generation and retains human-verified interactions and checked pseudo mask tracks.The authors motivate reconstruction because real-video inversion can introduce errors and prompt-video alignment and distribution differences complicate analysis.Source: §B.1 · p. 22

B.2 ADDITIONAL ANALYSIS

The analysis uses attention-to-mask alignment to identify where grounding and propagation occur, then tests this framework across architectures and applies it to selective training and training-free guidance. Results show that interaction signals concentrate in a small number of layers, with layer placement varying by architecture.

B.2 ADDITIONAL ANALYSIS

  • AAS measures spatiotemporal attention mass on instance mask tracks, avoiding thresholded overlap scores that discard magnitude or whole-scene aggregation that ignores target trajectories.The authors motivate preserving raw magnitude because joint attention is normalized across text and video tokens, whose counts differ substantially.Source: §B.2 · pp. 22–23
  • Attention alignment tracks generation quality: successful examples keep instance attention compact across frames, while diffuse or jumping attention accompanies drift and hallucinated parts.The analysis also finds strong noun- and verb-alignment in a subset of CogVideoX layers, while many others show grid-like positional responses.Source: §B.2 · p. 23

C MATRIX ON OTHER VIDEO DITS

  • HunyuanVideo-I2V concentrates dominant video-to-text layers at 6, 7, 9, and 10, and video-to-video layers at 11, 26, 27, and 29.Wan2.1-14B-I2V instead places video-to-video dominance at layers 6–8 and video-to-text dominance at 20–24 after filtering prompt-insensitive dummy layers.Source: C MATRIX ON OTHER VIDEO DITS · pp. 23–25, Figure 16 · p. 25

D.1 ARCHITECTURAL DETAILS

  • The CogVideoX implementation adapts selected query/key/value/output projections and reads targeted-layer attention through lightweight decoders trained against mask tracks.The causal decoder upsamples attention from 13 latent steps to 49 mask frames, preserving temporal order and sharper instance boundaries than four-frame OR-compressed supervision.Source: §D.1 · pp. 25–27

D.2 IMPLEMENTATION DETAILS

  • Training confines SGA to video-to-text blocks 7 and 11 and SPA to video-to-video block 12, leaving most pretrained weights frozen.The trainable set also includes the input projection and lightweight decoder heads.Source: §D.2 · p. 27

E TRAINING-FREE CROSS-MODAL GUIDANCE DETAILS

  • Training-free guidance targets interaction-dominant layers, perturbing token-to-entity or cross-frame attention and steering predictions away from degraded grounding or propagation.CMG handles grounding and CAG handles propagation without additional training.Source: E TRAINING-FREE CROSS-MODAL GUIDANCE… · p. 27

E.1 ARCHITECTURAL DETAILS

  • CAG suppresses cross-frame attention after softmax, while CMG suppresses noun-to-instance and verb-to-union alignments, preserving other attention interactions.The final prediction is guided away from the perturbed prediction using the perturbation guidance scale.Source: §E.1 · p. 28

E.2 IMPLEMENTATION DETAILS

  • CAG guides one video-to-video layer and CMG guides two video-to-text layers at every sampling timestep, with no added parameters or training.In CogVideoX-5B-I2V, the selected examples are layer 12 for CAG and layers 7 and 11 for CMG.Source: §E.2 · p. 28

E.3 EXPERIMENTAL RESULTS

The results illustrate how mislocalized or weak attention can coincide with interaction failures, and how perturbation guidance can improve selected cases. The section also motivates InterGenEval as an interaction-focused protocol, with metrics for action execution, instance grounding, and temporal consistency.

E.3 EXPERIMENTAL RESULTS

  • Guidance on interaction-dominant layers sharpens video-to-text attention and stabilizes video-to-video carry-over, turning many borderline failures into plausible interactions in the illustrated cases.The examples include maintained contact during a cut and a man whose motion starts and proceeds without drift.Source: §E.3 · pp. 28–29

F.1 RELATED WORKS

  • Existing protocols assess global semantics or attributes, while InterGenEval targets role- and time-sensitive alignment, including whether the correct subject acts on the intended object and contact and causal unfolding match the prompt.The supplied comparison-table caption describes InterGenEval as the only listed protocol targeting interaction-level semantic alignment.Source: Table 3 · p. 29, §F.1 · p. 30

F.2 OVERVIEW

  • InterGenEval averages per-interaction KISA and SGI scores, then uses SPI to account for instances that appear or disappear over time.KISA checks action execution across interaction stages; SGI checks subject and object grounding, and SPI penalizes inconsistent propagation.Source: §F.2 · p. 31
  • The evaluation uses GPT-5 question-answering for KISA and SGI and instruction-based assessment of instance emergence and disappearance for SPI.Annotated frames include the first and last video frames, with intermediate frames sampled at a fixed stride of five.Source: §F.2 · p. 31

F.3 EVALUATION METRICS

InterGenEval scores interaction execution, instance grounding, and temporal consistency, then combines these dimensions into Interaction Fidelity. The accompanying experiments describe the comparison setup, evaluation data, and score distributions.

F.3 EVALUATION METRICS

  • KISA assesses pre-, during-, and post-interaction stages, while SGI checks whether the subject and object are correctly grounded in the specified action.The protocol generates ten yes/no questions per key interaction: six for KISAraw and four for SGIraw.Source: §F.3 · pp. 31–32
  • SPI penalizes instance emergence and disappearance relative to the first frame, and the final IF score averages the reweighted KISA and SGI scores.The paper sets the SPI penalty weight λ to 5.Source: §F.3 · p. 32

G.1 COMPARISON MODELS

  • The comparison includes CogVideoX-2B-I2V, CogVideoX-5B-I2V, TaVid, and Open-Sora, alongside additional VBench and VBench-2.0 quality metrics.The additional metrics cover temporal and frame-wise quality, including subject consistency, motion smoothness, aesthetics, imaging quality, and human anatomy.Source: §G.1 · p. 32, §G.2 · p. 33

G.3 EVALUATION DATASET

  • The interaction-aware evaluation set contains 118 image-prompt pairs varying instance count and motion complexity, with prompts specifying participants, their interactions, and scene appearance.The prompts were generated with an LLM following the dataset-curation guidelines.Source: §G.3 · p. 33

G.4 ADDITIONAL ANALYSIS

  • MATRIX score distributions skew toward higher normalized KISA, SGA, and IF values, with less low-score mass than comparison models, indicating fewer severe failures.The authors report that this distributional pattern accompanies improved mean performance and reduced variance.Source: §G.4 · p. 33

G.5 HUMAN EVALUATION

In human comparisons, MATRIX outperforms its CogVideoX-5B-I2V backbone across interaction-related criteria and overall quality. The section also describes the pairwise evaluation protocol and reports a separate subset for non-human and non-contact interactions.

G.5 HUMAN EVALUATION

  • Raters compared videos side by side across six model pairs, judging interaction accuracy, grounding, propagation, semantic alignment, and overall quality.Thirty-one participants each completed 12 comparisons, with randomized left/right presentation and no ties.Source: §G.5 · pp. 33–34
  • MATRIX exceeds 0.9 win rate across human-evaluation criteria, while CogVideoX-5B-I2V scores around 0.36–0.44.The criteria include interaction accuracy, semantic grounding and propagation, semantic alignment, and overall quality.Source: Figure 23 · p. 34, §G.5 · pp. 34–35

H.1 ADDITIONAL QUALITATIVE RESULTS

  • A separate evaluation subset addresses non-human and non-contact interactions, extending the reported qualitative comparisons beyond human-object contact scenarios.The supplied figure caption identifies this subset as covering generalization to non-contact and non-human interactions.Source: §H.1 · p. 35, Figure 24 · p. 35

H.2 ADDITIONAL QUANTITATIVE RESULTS AND ANALYSIS

The additional analyses show that interaction-aware gains come with a modest training-time increase, while common quality and consistency metrics can mislead when outputs have little motion.

H.2 ADDITIONAL QUANTITATIVE RESULTS AND ANALYSIS

  • Additional VBench comparisons span CogVideoX-2B-I2V, CogVideoX-5B-I2V, Open-Sora-I2V, TaVid, and MATRIX.The comparison uses standard VBench metrics.Source: §H.2 · p. 35
  • SC, BC, AQ, and IQ can mislead when near-static outputs inflate consistency and quality; assess them alongside motion- and interaction-aware measures.The authors report that comparable motion sharply lowers SC, BC, and AQ, even as human raters prefer results with intended motion and correct bindings.Source: §H.2 · pp. 35–36, Figure 25 · p. 36

H.3 ADDITIONAL ABLATION RESULTS

  • KISA, SGI, and IF improve 22.7%, 21.9%, and 22.0% over LoRA-only, with 6.7 additional training hours per epoch.Compared with the unfinetuned model, MATRIX improves KISA from 0.406 to 0.546, SGI from 0.491 to 0.641, and IF from 0.449 to 0.593.Source: Table 5 · p. 36, §H.3 · p. 36

I LIMITATIONS

MATRIX supports up to five instance tracks, with very small masks posing a grounding challenge; the discussion identifies complementary motion and depth cues as future directions.

I LIMITATIONS

  • The framework supports up to five instance mask tracks per scene, matching a dataset where scenes with more than five interacting instances are rare.Most examples contain two to four distinct interacting objects.Source: I LIMITATIONS · pp. 36–37
  • Very small instance masks can provide weak or ambiguous grounding signals, potentially degrading accurate motion generation.Hierarchical mask encoding, spatially adaptive attention, and resolution-aware learning are proposed as future strategies.Source: I LIMITATIONS · p. 37

J DISCUSSION AND FUTURE WORK

  • Optical flow could add fine-grained motion-continuity supervision, while depth could help disambiguate occlusions and 3D proximity.The paper proposes integrating these cues with mask tracks in future work.Source: J DISCUSSION AND FUTURE WORK · p. 37
  • Mask tracks encode instance roles and interaction structure; complementary point tracks could refine local motion and detail propagation.Point tracks provide fine local motion but lack clear instance grouping or subject-object role semantics.Source: J DISCUSSION AND FUTURE WORK · p. 37

Related papers

Loading 2510.07310v2…