PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Subject Consistency over Dynamic Subject Sets in Video Generation

Tongcheng Zhang, Jun Zhu, Jianfei Chen

arXiv:2610.01052v1cs.CV

TL;DR

Long-video generation lets subjects enter and leave, making consistency across changing subject sets an important evaluation question. DynSC-Eval tracks eligible subjects and measures local and global consistency, and its rewards improve consistency after post-training. Gains are reported across 5s models and extended to 10s and 30s generation.

  • Problem

    As videos lengthen, visible subjects and their relationships can change, while existing consistency metrics may miss later entrants or lack sensitivity to individual-subject failures.

    Source: §1 · p. 2

  • Method

    DynSC-Eval dynamically tracks eligible subjects across their visible lifespans and measures local continuity, global identity preservation, and inconsistency events with six metrics.

    Source: §1 · p. 2

  • Results

    13.82% and 5.66% average relative reductions across six inconsistency metrics were reported for Wan-2.1-1.3B and SANA-2B, respectively, with consistency improvements also reported for ReSim and at 10s and 30s.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DynSC-Eval reveals consistency differences across models and durations that conventional metrics can obscure, and its rewards improve subject consistency while largely preserving other capabilities.

    Source: §6 · p. 10

    The study leaves multi-view evaluation and rewards for broader physical constraints to future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

We argue that as video generation extends to longer durations, subject consistency should be evaluated over \textit{dynamic subject sets}. We therefore introduce \textbf{DynSC-Eval}, an evaluation framework that dynamically tracks eligible subjects throughout their visible lifespans and measures local continuity and global identity preservation using six complementary object-level metrics, with explicit detection of inconsistency events. To validate its effectiveness, we design synthetic experiments that actively inject inconsistency events, demonstrating both the sensitivity of DynSC-Eval and the limitations of existing metrics. Evaluations of diverse models on 5s, 15s, and 60s video generation further reveal substantial subject consistency differences that are obscured by conventional metrics. Beyond evaluation, we construct rewards from DynSC-Eval and apply DiffusionNFT post-training in an autonomous-driving testbed. On 5s generation, our approach reduces the six inconsistency metrics by an average of 13.82\% for Wan-2.1-1.3B and 5.66\% for SANA-2B, with improvements also observed on the I2V model ReSim. Qualitative comparisons further demonstrate the effectiveness of our method. We then extend generation to 10s and 30s through curriculum learning and show that consistency optimization remains effective while largely preserving other capabilities.

1 INTRODUCTION

The introduction argues that long-video subject consistency requires evaluation over changing subject sets and presents DynSC-Eval to track them. It validates the metrics across synthetic inconsistencies and varied video models, then uses them to guide post-training with gains from 5s to 30s generation.

1 INTRODUCTION

  • As videos lengthen, subjects can enter, leave, and interact, so both the visible subject set and relations among its members may evolve over time.The introduction argues that consistency remains challenging even within a continuous single shot.Source: §1 · p. 2
  • DynSC-Eval tracks eligible subjects throughout their visible lifespans and measures local continuity, global identity preservation, and inconsistency events with six metrics.The framework updates the subject set dynamically as eligible subjects appear and disappear.Source: §1 · p. 2
  • Synthetic experiments show greater sensitivity to injected inconsistencies than holistic VBench evaluation, while evaluations across 5s, 15s, and 60s videos distinguish models and reveal complementary local and global failure patterns.Evaluation of only subjects visible in initial frames misses inconsistencies involving later entrants.Source: §1 · p. 2
  • 13.82% and 5.66% average relative reductions across six inconsistency metrics were achieved for Wan-2.1-1.3B and SANA-2B, respectively.The held-out evaluation used 100 videos; improvements were also reported for I2V model ReSim, with other capabilities largely preserved.Source: §1 · p. 3
  • Curriculum learning extends generation from 5s to 10s and then 30s, where consistency rewards improve DynSC while largely preserving other capabilities.The extension uses an additional 100 and 200 hours of OpenDV data for 10s and 30s generation, respectively.Source: §1 · p. 3

2 RELATED WORK

Prior temporal-consistency evaluations use holistic frame comparisons or subject-specific protocols, but may miss localized events and subjects entering later. DynSC-Eval addresses this coverage gap, while its reward design builds on prior consistency-oriented video optimization approaches.

2.1 TEMPORAL CONSISTENCY EVALUATION

  • Holistic evaluations offer broad perceptual and semantic coverage but can struggle to localize individual-subject inconsistencies and detect discrete events.Subject-level protocols also include designated targets or first-frame-based assessments that may omit later entrants.Source: §2.1 · p. 3
  • DynSC-Eval covers eligible subjects that first appear after the initial frame and separately measures local disruptions, global drift, and inconsistency events.The related-work discussion contrasts this dynamic coverage with prior designated-target or first-frame protocols.Source: §2.1 · p. 3

2.2 POST-TRAINING FOR TEMPORAL CONSISTENCY

  • Prior video consistency methods use frequency-domain, identity-based, or pixel-motion rewards; this work combines dynamic local and global subject-consistency signals within DiffusionNFT.The cited approaches target I2V frame-feature discrepancy, facial identity, or instantaneous and long-term motion consistency.Source: §2.2 · p. 4

3 DYNSC-EVAL

DynSC-Eval tracks subjects throughout their visible trajectories and separates local event and short-term variation from long-term identity preservation. Synthetic tests and evaluations across models and durations assess its sensitivity and discriminative power.

3.1.1 TRAJECTORY EXTRACTION

  • Trajectories cover valid visible observations for each detected subject, with new objects assigned new trajectories as subjects enter, leave, or become occluded.Each trajectory contains the valid frames and segmented crops for one object.Source: §3.1.1 · p. 4

3.1.2 LOCAL CONSISTENCY

  • Local consistency covers sudden appearance, sudden disappearance, and severe appearance mutation, using SR, ER, and MAD for failure prevalence, event frequency, and short-term drift.A severe mutation is counted when appearance distance exceeds threshold τ.Source: §3.1.2 · p. 5

3.1.3 GLOBAL CONSISTENCY

  • Global metrics apply only to trajectories with at least 16 frames and measure endpoint drift, mean reference drift, and worst-frame drift from an initial identity reference.Lower values indicate stronger temporal consistency.Source: §3.1.3 · pp. 5–6

3.2 CONTROLLED SYNTHETIC EXPERIMENTS

  • In the synthetic experiment, 30-second videos contain 20 objects entering sequentially; injected local and global inconsistencies test whether metrics respond to known corruptions.Restricting evaluation to subjects visible in initial frames can miss inconsistencies among later arrivals.Source: §3.2 · p. 6
  • SR tracks injected inconsistency proportion ρ with R2 = 0.995, ED rises with ρ, and VBench Subject Consistency remains nearly unchanged.Isolated perturbations also show complementary metric roles: local-only injections produce little change in ED and MRD, while global-only injections have limited effects on local metrics.Source: §3.2 · pp. 6–7

3.3 EVALUATION

  • For the same commercial models, increasing duration from 5s to 15s raises mean ED, MRD, and WFD by 25.7%, 22.1%, and 18.5%, respectively, versus 3.0% for MAD.At 60s, streaming-model SR ranges from 0.2870 to 0.4647.Source: §3.3 · p. 7
  • At 5s, SANA-2B’s VBench SC 94.10 is comparable to commercial models (93.92–94.65), yet it performs worse on all six DynSC-Eval metrics.The passage reports a similar discrepancy for 60s streaming models versus 15s commercial models.Source: §3.3 · p. 7
  • Seedance 2.0 has the lowest SR among commercial models at both durations, while Kling 3.0 leads on the other five metrics at 15s; streaming-model strengths also differ by metric.LongLive has the lowest ER (8.356) and MAD (0.1236), while Helios performs best on SR (0.2870) and all three global metrics.Source: §3.3 · p. 7

4 FROM CONSISTENCY EVALUATION TO MODEL OPTIMIZATION

The section uses autonomous-driving videos to optimize subject consistency with DynSC-Eval-derived rewards, then tests the approach on T2V, I2V, and longer videos. Across these settings, consistency improves while other capabilities are largely preserved.

4.1 TESTBED

  • The testbed uses real-world OpenDV clips to fine-tune Wan-2.1-1.3B and SANA-2B baselines on approximately 100 hours of text–video pairs.Driving videos provide multiple persistent vehicles in continuous motion and frequent interactions, yielding abundant object trajectories.Source: §4.1 · pp. 7–8

4.2 METHOD

  • DiffusionNFT optimizes online rewards from beta-smoothed SR, MAD, WFD, and MRD, covering local and global consistency.The reward combines normalized inconsistency costs; rollouts without qualified object trajectories receive −10, and a frozen reference policy regularizes training.Source: §4.2 · p. 8

4.3 EXPERIMENTS

  • Both T2V models improve across all six metrics; for Wan-2.1-1.3B, SR, ER, and MAD decrease by 13.3%, 22.2%, and 16.6%.SANA-2B's local SR, ER, and MAD decrease by 5.11%, 5.99%, and 6.65%; its global ED, MRD, and WFD decrease by 6.08%, 5.96%, and 4.19%.Source: Table 3 · p. 9, §4.3 · p. 9

4.4 GENERALIZATION TO IMAGE-TO-VIDEO GENERATION

  • For ReSim I2V generation, local and global inconsistency metrics decrease by averages of 6.61% and 3.14%, respectively.The results extend the feedback-reward approach beyond T2V generation.Source: §4.4 · p. 9

4.5 SCALING CONSISTENCY OPTIMIZATION TO LONG-VIDEO GENERATION

  • After curriculum learning to 10s and 30s, DiffusionNFT improves local and global consistency at both durations while broadly preserving VBench scores.At 10s, ER, MAD, and SR decrease by 22.09%, 15.72%, and 12.81%; at 30s, SR, ER, and MAD decrease by 4.50%, 9.54%, and 2.47%.Source: §4.5 · p. 9, Table 4 · p. 10

5 ABLATION STUDY

The ablation finds that neither local-only nor global-only rewards balance consistency and other capabilities as well as the combined reward.

5 ABLATION STUDY

  • The combined reward achieves the lowest SR and ER, at 0.2220 and 7.139, while maintaining other capabilities comparable to or above baseline.Local-only optimization lowers MAD and global drift metrics but reduces Dyn from 92.00 to 85.00; Global-only preserves capabilities but worsens SR.Source: Table 5 · p. 10, §5 · p. 10

6 CONCLUSION AND FUTURE WORK

The conclusion presents DynSC-Eval as an evaluation for dynamic subject sets and reports consistency gains from reward-guided optimization across T2V, I2V, and longer generation. Future work will extend evaluation to multi-view generation and add rewards for broader physical constraints.

6 CONCLUSION AND FUTURE WORK

  • DynSC-Eval tracks eligible subjects and evaluates local and global consistency, including inconsistency-event detection; synthetic experiments and diverse model evaluations validate its sensitivity and discriminative power.The conclusion reports limitations of existing methods and optimization gains across T2V and I2V models, extending to 10s and 30s while largely preserving other capabilities.Source: §6 · p. 10
  • Future work will extend evaluation to multi-view generation and add rewards for broader physical constraints on generated dynamics and interactions.The planned multi-view evaluation would jointly assess temporal continuity and consistency across viewpoints.Source: §6 · p. 10

A IMPLEMENTATION DETAILS OF DYNSC

The appendix distinguishes short-term subject continuity from long-term identity preservation. It argues that measuring these dimensions separately provides clearer signals for optimization than a single aggregate score.

A.1 LOCAL AND GLOBAL TEMPORAL CONSISTENCY

  • Local consistency measures smooth state evolution over short intervals and targets transient failures such as sudden appearance or disappearance and severe deformation.Source: §A.1 · p. 16
  • Global consistency measures whether a subject's established state and identity remain compatible throughout its trajectory despite motion and viewpoint changes.Source: §A.1 · p. 16
  • Separately evaluating local discontinuities and global trajectory drift yields complementary subject-trajectory signals for optimization, unlike a single aggregate score that can obscure distinct failure mechanisms.Source: §A.1 · p. 16

A.2 ROBUSTNESS TECHNIQUES

The robustness techniques stabilize short-term appearance estimates and avoid ambiguous track boundaries in feature-based measurements. Local consistency captures brief events and adjacent appearance variation, while global consistency measures identity drift over sufficiently long trajectories.

A.2 ROBUSTNESS TECHNIQUES

  • Temporal smoothing averages every two consecutive subject embeddings, then measures adjacent appearance variation between the pooled representations.Trajectory Boundary Truncation excludes four boundary observations from appearance-distance calculations when the track is not near a video boundary, while retaining neighboring frames for appearance or disappearance event checks.Source: §A.2 · p. 16

A.3.1 LOCAL CONSISTENCY

  • Local consistency counts sudden appearances, sudden disappearances, and severe appearance mutations, alongside continuous short-term variation measured by MAD.SR counts trajectories with at least one event, while ER counts events per 100 track-seconds; MAD captures variation below the event threshold.Source: §A.3.1 · pp. 16–17

A.3.2 GLOBAL CONSISTENCY

  • Global metrics apply only to trajectories lasting at least 16 frames and compare identity appearance at trajectory endpoints and across the full trajectory.ED measures start-to-end difference, MRD tracks average deviation from the initial reference, and WFD captures the most severe identity deviation.Source: §A.3.2 · pp. 17–18

A.4 DETAILS OF THE SYNTHETIC EXPERIMENT

The synthetic experiment pairs videos with matched scene properties to test metric responses to controlled local and global shape perturbations. It varies perturbation type and rate while holding the eligible object set consistent across conditions.

A.4 DETAILS OF THE SYNTHETIC EXPERIMENT

  • Each paired synthetic video contains 480 frames at 16 FPS and 20 sequentially introduced geometric objects, with shared colors, motion, birth times, and radii across conditions.Objects have scheduled 10-second lifetimes, including entry and exit; 15 objects per seed meet the eligibility constraints for perturbation.Source: §A.4 · p. 18
  • Global perturbations smoothly transition selected objects between polygon shapes over six seconds while holding their center, radius, and color fixed.Source and target shapes are circle, triangle, square, and pentagon; each object retains the source shape before the transition and the target shape afterward.Source: §A.4 · p. 18
  • Local perturbations assign selected objects one brief severe shape mutation, delayed appearance, or premature disappearance.The mutation lasts eight frames, while appearance and disappearance events are introduced inside the image at prescribed object ages.Source: §A.4 · p. 18
  • Local-only injections raise SR and ER while ED stays nearly unchanged, whereas global-only shape transitions raise ED and MRD while SR and ER change little.The figures plot metric responses as the fraction of perturbed objects increases; each point averages three seeds with one-standard-deviation error bars.Source: Figure 6 · p. 19, Figure 7 · p. 19

A.5 ABLATION OF LOCAL-ONLY AND GLOBAL-ONLY INJECTIONS

The ablation separates the responses of local and global metrics to injected inconsistencies, and tests whether SR tracks injected event rates. The evaluation prompts cover varied subject-rich scenes and timed interactions.

A.5 ABLATION OF LOCAL-ONLY AND GLOBAL-ONLY INJECTIONS

  • At ρ = 0.5, Local-only raises SR from 0 to 0.5000 and ER from 0 to 6.6687, while Global-only raises ED from 0.0963 to 0.2502 and MRD from 0.0261 to 0.1601.Under Local-only, ED changes from 0.0963 to 0.1071; under Global-only, mean SR stays 0.0167 at every nonzero rate and ER reaches 0.2173 at ρ = 0.5.Source: §A.5 · p. 19
  • Under joint local and global inconsistency injection, as the perturbed-object fraction increases, SR closely follows ideal SR = ρ, while ER, ED, and MRD increase; ED and MRD reach 0.2535 and 0.1735.Source: Figure 8 · p. 20, §A.5 · p. 20

A.6 PROMPT CONSTRUCTION

  • Evaluation prompts cover animals, pedestrians, football matches, and autonomous driving, with 25 prompts per scenario..Source: §A.6 · p. 20

A.7 VIDEO GENERATION PROMPTS

  • Video examples script timed subject interactions, including changing positions and brief overlaps, while specifying anchor identities to track.Examples include football players, road vehicles, and pedestrians in 15-second prompts, and pedestrian scenes in 60-second prompts.Source: §A.7 · pp. 21–23

B CURRICULUM TRAINING

The paper establishes 5-second fine-tuning baselines and extends the selected model to longer videos. Validation losses decrease across durations, but the 5-second loss curves alone do not reliably indicate generated video quality.

B.1 SUPERVISED FINE-TUNING OF 5S MODELS

  • The 5-second baselines use 72,000 nominal five-second OpenDV clips, split into 68,400 training and 3,600 validation clips, with full transformer-parameter updates.Both models train for one epoch, or 8,550 updates, on eight A100 GPUs.Source: §B.1 · p. 25
  • Fine-tuned Wan-2.1-1.3B is favored over SANA-2B by final consistency evaluation and visual inspection, despite reduced validation loss for both models.The authors use Wan-2.1-1.3B exclusively for subsequent 10-second and 30-second fine-tuning, citing its stronger empirical performance and limited computational budget.Source: §B.1 · p. 26

B.2 EXTENDING TO LONGER VIDEOS

  • During fine-tuning on 10s and 30s clips with dense spatiotemporal attention, validation loss decreases steadily for both durations and gradually stabilizes toward training’s end, indicating stable optimization.Source: §B.2 · p. 27

C.1 IMPLEMENTATION DETAILS OF CONSISTENCY POST-TRAINING

Post-training optimizes LoRA adapters through current, old, and reference policies. Reference regularization further improves consistency metrics while preserving motion dynamics.

C.1 IMPLEMENTATION DETAILS OF CONSISTENCY POST-TRAINING

  • The base diffusion transformer is frozen while rank-32 LoRA adapters with scaling factor α = 64 are optimized in four attention projections.Adapters are inserted into to_q, to_k, to_v, and to_out to target temporal consistency while maintaining efficient training and preserving generative capabilities.Source: §C.1 · p. 27
  • Three policies divide optimization and rollout roles: the current policy is trained, the EMA-updated old policy generates rollouts, and a frozen reference constrains deviation.For each prompt, the old policy generates eight videos whose rewards are standardized within the group; four prompt groups produce 32 rollouts per update in the 5s setting.Source: §C.1 · p. 27
  • Regularization lowers SR from 0.2397 to 0.2220 and ER from 7.853 to 7.139, while VBench motion is 98.36 vs. 98.49 and dynamic degree rises 91.00 to 92.00.Both optimized variants use the same combined consistency reward and are evaluated at Step 75; without regularization, all six consistency metrics also improve over the 5s baseline.Source: §C.1 · p. 27, Table 7 · p. 28
Loading 2610.01052v1…