Source-linked AI summary
Temporal State Transport in Video Generation: Diagnosing and Correcting Spectral Imbalance
Luyao Tang, Bingjun Luo, Dong Yi, Jialin Guo, Haoning Xi, Cheng Chen, Yizhou Yu, Chaoqi Chen
TL;DR
Reliable video generation requires temporal attention to preserve visual state across frames, but existing interaction-strength and local-entropy views do not diagnose balanced transport. The paper introduces Spectral Tension and a training-free regulator to identify and correct temporal imbalance. Experiments report improved quantitative and human evaluations, stronger visual consistency, and larger corrections at the worst hotspots.
Problem
Temporal attention may fail to preserve identity, scene stability, natural motion, and fine details, while existing interaction-strength and local-entropy views do not reveal whether transport is balanced.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Balanced temporal transport is presented as a useful principle for understanding and improving video generation.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Reliable video generation requires more than high-quality frames to form a coherent story: a model must maintain a persistent state, transporting visual attributes such as identity, scene layout, motion, and fine details across time. Existing training-free methods mainly strengthen cross-frame attention or analyze local attention entropy, but these views do not reveal whether temporal interactions stay in a healthy transport regime. In this work, we study video generation through the perspective of Temporal State Transport. We introduce Spectral Tension, a signed diagnostic that compares local attention diffuseness with global spectral diversity, and use it to identify two opposite temporal failures: fragmented transport and over-mixing hotspots. Based on this diagnosis, we propose Spectral Transport Homeostasis, a training-free regulator that softly corrects pathological temporal states while largely preserving balanced ones. Experiments on pretrained video generation models show that the original model often occupies imbalanced temporal regimes, whereas our method selectively applies larger corrections to the worst temporal hotspots and improves temporal consistency and visual quality without finetuning. Code: https://github.com/lytang63/temporal-state-transport
1. Introduction
Reliable video generation requires temporal attention to preserve identity, scene stability, natural motion, and fine details across time. The paper frames this as balanced Temporal State Transport and introduces a signed diagnostic and a training-free regulator for temporal imbalance.
1. Introduction
- Temporal attention can coexist with identity drift, background flicker, unnatural motion, and unstable details, so frame-level fidelity alone is insufficient.The target is persistent visual state across time, not merely high-quality individual frames.Source: §1 · p. 1
- Existing methods strengthen cross-frame interaction or measure local attention diffuseness, but neither reveals whether temporal attention remains balanced.The paper frames video coherence as organizing frame interactions into stable temporal structure.Source: §1 · p. 1
- Spectral Tension compares local attention diffuseness with global spectral diversity, assigning negative values to fragmentation and positive values to over-mixing.Values near zero indicate a more balanced state.Source: Figure 1 · p. 1, §1 · p. 2
- Spectral Transport Homeostasis is a training-free query-temperature regulator that selectively corrects pathological temporal states while largely preserving balanced ones.The paper presents it as a way to improve video quality without simply increasing cross-frame interaction.Source: §1 · p. 2
2. Preliminaries with Diagnostic View
The diagnostic view treats frame-level temporal attention as a transport operator that propagates visual state. It measures imbalance by comparing local diffuseness with global spectral diversity, then uses the diagnosis to interpret hotspot corrections.
2. Preliminaries with Diagnostic View
- Frame-level attention A represents visual-state transfer between frames, supporting propagation of identity, motion, scene layout, and fine details over time.The paper treats attention as a transport operator rather than a simple interaction map.Source: §2 · p. 2
- Fragmented transport reflects insufficient transport, while over-mixing reflects lost structural diversity; both can occur in different layers and denoising steps.The two failure modes need not be mutually exclusive within a model.Source: §2 · p. 2
- Spectral Tension compares normalized row-wise entropy, which measures local attention diffuseness, with normalized spectral entropy, which measures global spectral diversity.Negative values indicate fragmentation, positive values indicate over-mixing, and values near zero indicate a more balanced state.Source: §2 · pp. 2–3
- The original model retains deep-layer over-mixing hotspots, while the method selectively pulls back the worst layer-step hotspots and applies larger corrections to more pathological states.These observations are presented as evidence that video quality is tied to maintaining balanced temporal attention.Source: §2 · p. 3, Figure 3 · p. 3
3. Methodology
The methodology treats temporal attention as a transport operator and regulates it according to whether its signed Spectral Tension indicates fragmentation, over-mixing, or balance. A layer-step schedule applies stronger query-temperature adjustments in deeper layers and earlier denoising steps, while aiming to leave balanced states largely unchanged.
3. Methodology
- Spectral Transport Homeostasis selectively adjusts pathological temporal states and minimally perturbs balanced ones, without learnable parameters or finetuning.It evaluates native temporal attention, computes Spectral Tension and a homeostatic factor, rescales queries, then produces regulated attention outputs.Source: §3 · p. 3, §3.3 · pp. 3–4
3.1. Temporal transport operator
- Positive Spectral Tension marks overly diffuse transport for sharpening; negative tension marks fragmented transport for softening, while near-zero tension calls for minimal change.The method uses Spectral Tension as a signed state variable for deciding how to regulate temporal attention.Source: §3.1 · p. 3
3.2. Homeostatic query-temperature regulation
- The regulator adjusts the temporal transport operator through query-temperature scaling rather than amplifying attention outputs.The intervention changes how temporal information is aggregated rather than directly scaling hidden features.Source: §3.2 · p. 3
3.3. Layer-step scheduling
- Intervention is stronger in deeper layers and early denoising steps, then decays during late refinement; τ is the only user-facing hyperparameter.Both layer and time weights use cosine schedules without extra hyperparameters.Source: §3.3 · p. 3
4. Experiments and Analysis
The experiments report improved VBench overall performance and better human-evaluation scores across the assessed dimensions, alongside qualitative gains in detail and physical structure. The τ ablation finds a stable default while noting that larger values can reduce prompt faithfulness.
Quantitative evaluation
- The method improves the overall VBench score while remaining competitive on each of seven selected dimensions.The evaluation uses three prompts per dimension and averages over three random seeds; the text identifies motion smoothness, aesthetic quality, and imaging quality as strongest dimensions.Source: §4 · p. 4
Human evaluation
- Human raters score the method above the original model across all five dimensions, with prominent gains in motion naturalness and imaging quality and clear improvement in prompt adherence.The Wan2.2-14B study uses 30 prompts and 50 raters, scoring subject consistency, background stability, motion naturalness, imaging quality, and prompt adherence on a 10-point scale.Source: §4 · p. 4, Figure 4 · p. 4
Visual analysis and ablation
- The method adds realistic fine details and corrects implausible physical structures, while pulling pathological temporal hotspots toward a more balanced regime.These qualitative changes are reported in representative samples and are consistent with the teaser analysis.Source: §4 · p. 4
- The τ ablation supports 0.2 as a stable default; larger values can move outputs away from the original prompt content.Smaller values can be conservative, while the method remains stable over a range of values.Source: §4 · p. 4, Figure 5 · p. 4
5. Conclusion
The conclusion argues that video quality depends on balanced temporal transport, not only cross-frame interaction strength, and presents Spectral Tension and Spectral Transport Homeostasis as an interpretable, effective approach. The appendix analyzes inference-time attention calls on Wan2.2-T2V-14B and reports that modulation reduces tension, especially in deeper layers and high-tension calls.
5. Conclusion
- Video quality depends on whether temporal attention remains in a balanced transport regime, and the method improves quantitative and human evaluations while producing stronger visual consistency.The conclusion also reports that its largest corrections target the worst temporal hotspots.Source: §5 · p. 4
Appendix
- The appendix analyzes 38,400 temporal self-attention calls from 30 Wan2.2-T2V-14B videos, with records indexed by sample, layer, denoising step, and processor/head.The prompts emphasize challenging dynamic scenes, and the analysis avoids aggregating inactive calls.Source: Appendix · p. 7
- Among 17,045 active calls, mean absolute Spectral Tension falls from 0.4032 before modulation to 0.3774 after modulation, a 6.4% relative reduction.The report defines an active call as one where |γ −1| > 0.03; 44.4% of all recorded calls are active and the rest are near-identity passes.Source: Figure A Core Mechanism: Spectral… · pp. 7–8, Appendix · p. 7
- Calls with |∆| > 0.3 fall from 75.5% before modulation to 68.4% after, while the post-modulation distribution has reduced density in the high-tension tail.For |∆| > 0.6, the peak shifts left by approximately 0.07–0.10; the report interprets the distributional change as clipping its right tail.Source: Figure A Core Mechanism: Spectral… · p. 8
- All active calls show non-negative tension reduction, with larger corrections associated with larger |∆| and a small number of deep-layer, early-step calls contributing the largest reductions.The reduction distribution has a mode near 0.006 and a mean of 0.0258; some reductions exceed 0.15.Source: Figure A Core Mechanism: Spectral… · p. 8
- Mean reduction is below 1% in the shallowest layers, rises to 8.9% at L70%–L80%, and remains 7.4% in the deepest bucket.The analysis divides layer depth into ten equally sized buckets and attributes the weaker shallow-layer corrections to τeff being suppressed below 0.05.Source: §A.4 · p. 8
A.4 Per-layer stratification (right panel).
The analysis reports that the scheduled correction is directionally aligned with Spectral Tension and reduces it in both fragmented and over-mixed calls. At the video level, most samples show lower mean tension, and a separate sign test finds positive reductions in every analyzed sample.
A.4 Per-layer stratification (right panel)
- Every one of 30 videos shows a net positive reduction on active calls (two-sided binomial sign test, p < 10^-8).The prompts span cooking, fire and smoke, fluid dynamics, fast human actions, FPV camera motion, and static macro shots.Source: §A.4 · p. 8
- Across 17,045 active calls, positive ∆ maps to γ > 1 and negative ∆ to γ < 1, with no exceptions.The report identifies the two occupied quadrants as the directionally correct ones.Source: Figure B Directional Correctness of… · p. 8
- The scatter plot’s γ-versus-∆ comparison tests whether the adjustment direction matches the sign of the tension; the caption says it is subsampled to 3,000 points.The accompanying account reports all active calls in the correct quadrants, with no exceptions.Source: Figure B Directional Correctness of… · p. 8, Figure 7 · p. 9
- Fragmented calls outnumber over-mixing calls by approximately 4.9:1, so softening is applied more often than sharpening.The reported mean γ is approximately 0.952 for fragmented calls and 1.065 for over-mixing calls.Source: Figure B Directional Correctness of… · p. 9
B.3 Pre/post |∆| by correction direction (right panel)
- For calls with |∆| > 0.05, post-modulation |∆| falls by approximately 5%–7% for over-mixing and 6%–8% for fragmented transport.The report describes these reductions as similar in relative magnitude across the two correction directions.Source: §B.3 · p. 9
- OURS yields smaller mean |∆| per video in 24 of 30 cases (two-sided sign-test p = 0.0014; Cohen’s d = −0.508).The analysis aggregates mean values over deep-layer calls for each video.Source: §C.1 · p. 9, §B.3 · p. 9
C.1 Spectral tension ∆(top-left).
Across paired samples, the method shifts spectral tension and several attention diagnostics while concentrating its modulation in deep-layer, early-step regions. Its layer-step modulation remains selective rather than universal.
C.1 Spectral tension ∆(top-left).
- The spectral-tension shift has Cohen’s d = −0.508, with approximately 80% power at α = 0.05 in the paired n = 30 study.The paired scatter plots compare ORIGINAL and OURS for six attention-level diagnostic metrics.Source: §C.1 · pp. 9–10, Figure 8 · p. 10
- Self-retention increased in 26/30 samples (p < 0.001, d = +0.402), effective rank in 24/30 (p < 0.01, d = +0.447), and Gini in 23/30 (p = 0.0052, d = +0.286).The paper interprets the simultaneous increases in self-retention and effective rank as greater decisiveness without loss of temporal diversity.Source: §C.1 · pp. 10–11
C.5 Cross-frame mass (middle-centre).
- Cross-frame mass decreased in the same 26/30 samples where self-retention increased (p < 0.001, d = −0.402).The two measures sum to 1, so the paired pattern reflects reduced off-diagonal attention.Source: §C.5 · p. 11
C.6 Per-call |∆| reduction (middle-right).
- A positive mean per-call tension reduction appeared in 22/30 samples (p = 0.0161, d = +0.343).The paper presents this as per-video corroboration of the aggregate within-call result.Source: §C.6 · p. 11
D.1 Effective temperature τeff (top-left).
- The effective temperature τeff rises from near zero in shallow layers and late steps to 0.200 in deep-layer, early-step regions.The reported maximum is 0.200; the cosine schedule is near zero in layers below 20% depth and steps beyond 80%.Source: §D.1 · p. 11
D.2 Modulation magnitude |γ −1| (top-centre).
- Layer-step modulation magnitude |γ −1| averages 0.029, with 19.3% of cells exceeding 0.05.The heatmap shows the strongest reductions in mid-to-deep layers and early steps, where effective temperature and pre-modulation tension are high.Source: §D.2 · p. 11, Figure 9 · p. 12, §D.3 · p. 12
D.4 Pre-modulation |∆| (middle-left).
Pooling all calls makes the ORIGINAL and OURS distributions appear nearly unchanged, while filtering to deep layers reveals a modest shift toward zero. Baseline tension is concentrated in early steps and mid-depth layers, and global averaging dilutes the observed effect.
D.4 Pre-modulation |∆| (middle-left).
- Baseline spectral tension is highest in early steps and mid-depth layers, aligning the strongest scheduled intervention with high-|∆| regions.Because modulation is concentrated in about 44% of calls, averaging over all calls dilutes the signal by roughly 2.3×; active-call or deep-layer analyses recover more of it.Source: §D.4 · p. 12
E.2 Deep layers ≥50% (centre column).
- For deep layers, the ∆ distribution shifts toward zero, with the mean moving from approximately 0.31 to 0.29.The deep-layer row-entropy distribution shifts slightly right; including late-step calls with low τeff makes the separation modest.Source: Figure 10 · p. 13, §E.2 · p. 13
E.1 All calls (left column).
- Across all 38,400 calls, the ORIGINAL and OURS KDE curves nearly overlap for all three metrics, with a mean ∆ difference of −0.002.The paper notes that pooled similarity does not mean the method has no effect, because inactive calls dilute the shift.Source: Figure 10 · p. 13, §E.1 · p. 13
E.3 Active calls: deep + |γ −1| > 0.03 (right column).
Active-call analysis shows that spectral correction redistributes attention away from dominant temporal modes, with benefits concentrated in targeted calls and stronger for higher-tension states. The analysis also finds mostly positive per-call changes while documenting content-specific and measurement caveats.
E.3 Active calls: deep + |γ −1| > 0.03 (right column).
- Across all deep-layer calls, including calls with near-zero τeff, OURS slightly reduced λ1/Σλ versus ORIGINAL from 0.349 to 0.347 (∆ = −0.0017), increasing mass on λ2–λ5.This small reduction suggests a mild weakening of leading-mode dominance, consistent with a reduced tendency toward rank-1-like temporal degeneracy.Source: Figure F Eigenvalue Spectrum of the… · p. 14
F.3 Per-sample normalized HvN scatter (right panel).
- Normalized HvN did not rise uniformly with reduced λ1 because entropy also depends on how the remaining eigenvalues are distributed.The paper characterizes the combined spectral effect as a mild inference-time regularizer without retraining or per-video hyperparameter search.Source: §F.3 · p. 15
Summary of Mechanistic Evidence
- All 17,045 active calls showed the prescribed γ direction and reduced |∆|; five of six attention diagnostics reached p < 0.01 across 30 challenging prompts.The active subset is 44% of calls, and global averages dilute the signal by approximately 2.3×.Source: Summary of Mechanistic Evidence · p. 15
- Corrections were content-adaptive: high-motion prompts received larger adjustments, while simpler prompts were nearly undisturbed.This adaptivity arises from the instantaneous ∆-dependent schedule rather than a content-classification module.Source: Summary of Mechanistic Evidence · p. 15
F.2 Per-sample λ1/ P λ scatter (centre).
- OURS lowered the leading eigenvalue fraction on 22 of 30 samples (p = 0.016), with a mean absolute reduction of −0.0017.The shift is small and supports a mild regularising effect rather than a hard constraint on spectral shape.Source: §F.2 · p. 14, §F.3 · p. 15, Summary of Mechanistic Evidence · p. 16
Summary of Mechanistic Evidence
- Two texture-heavy samples showed small negative effects on self-retention and Gini, where broad attention may suit water, reflections, or smoke.These were 2/30 samples, with negative effects below 1% relative; the spectral-tension metric still improved on them.Source: Summary of Mechanistic Evidence · p. 16
Supplementary High-Order Analysis (Figures G–M)
- Across six metrics, 68%–73% of active calls improved, and the positive-rate curves indicate that the largest gains are concentrated in a smaller subset.The all-call marginal plots can obscure this shift because 55.6% of calls are near-identity passes; active aligned pairs reveal it more clearly.Source: Figure G Per-call Improvement… · p. 17, §G.2 · p. 17
G.2 Cumulative positive-rate curve (bottom row).
- Active-call heatmaps localize improvement to deep-layer, early-step regions where τeff peaks, with the footprint tapering as τeff approaches zero.For spectral tension, improvement is measured as |∆|orig − |∆|ours, so movement toward zero is counted in either correction direction.Source: Figure H Active-gated Layer×Step… · p. 17, §H.2 · p. 17
H.2 Spectral tension direction.
Active-gated heatmaps reveal where corrections occur across layer–step space, while baseline tension predicts their size. The method has little effect near homeostasis and increasingly large effects at higher tension.
H.2 Spectral tension direction.
- Active calls form a band expanding outward from deep-layer, early-step regions; neighboring late-step cells at the same depth may remain inactive.Collapsing across either layer or step would hide this operational footprint.Source: §H.2 · p. 17
- Across six metrics, effect sizes rise with baseline |∆|: the lowest bin (< 0.1) is near zero or slightly negative, while the highest (> 0.6) exceeds dx > 0.5 for most metrics.Calls with |∆| < 0.2 are already near homeostasis and receive little modification.Source: §H.2 · p. 18
- A threshold near |∆| ≈ 0.25 separates essentially no effect (dx ≈ 0) from an effect that grows approximately linearly with tension.The threshold reflects the interaction of the τeff schedule and γ = exp(τeff · ∆).Source: §I.2 · p. 18
I.3 Metric-specific divergence.
Metric responses differ in how strongly they track baseline tension, while per-sample results are broadly positive rather than driven by one prompt. A secondary analysis finds slightly greater dispersion among attention heads on active calls.
I.3 Metric-specific divergence.
- Self-retention and cross-frame mass show the steepest increase with |∆|, reaching dx > 1.0 in the highest bins, while Gini and effective rank rise more moderately.The authors interpret attention-mass reallocation as the primary improvement channel, with rank and entropy benefits as secondary consequences.Source: §I.3 · p. 18, Figure 14 · p. 19
J.1 Majority positive.
- Most per-sample bars are positive across all six metrics, with mean Cohen’s d ranging from approximately +0.004 to +0.020.Positive values span the baseline-tension ranking rather than concentrating in a single outlier; two texture-heavy samples are identified as small negative-effect cases.Source: §J.1 · p. 19, Figure 15 · p. 20
J.3 Content-adaptive ranking.
- On active calls (n ≈17,000 active pairs), head-∆ standard deviation rose by 0.75% of mean spread (p ≈ 10^-270), opposite global convergence: the method amplified, rather than reduced, head differences.Source: §K.1 · p. 20
K.2 Explanation: per-head baseline diversity.
The method slightly increases dispersion across heads while leaving future–past attention balance unchanged. Its improvement signal strengthens with modulation magnitude, though near-threshold calls can show a small reversal.
K.2 Explanation: per-head baseline diversity.
- Active-call head-∆ spread rises because one γ is computed from the mean ∆ across 40 heads, so heads with different baseline tension receive unequal corrections.The mean |∆| across heads nevertheless decreases; per-head γ scheduling is proposed as a future improvement.Source: §K.2 · p. 20, Figure 16 · p. 21
- Future–past balance is unchanged: mean b is approximately −0.007 in both conditions, with no systematic per-sample deviation (p > 0.2).The method acts on attention concentration within past and future pools, not their directional allocation.Source: §K.2 · pp. 21–22, Figure 17 · p. 21
M.1 Dose–response relationship.
- Five of six metrics exhibit strong monotonically increasing trends with |γ −1|: HvN norm (ρ = +0.74, p < 0.05), H̄row norm (ρ = +0.88, p < 0.01), Gini (ρ = +0.88, p < 0.01), self-retention and cross-frame mass (both ρ = +0.93, p < 0.001).For self-retention and cross-frame mass, |γ −1| increases from 0.032 to 0.109; the two metrics are complements.Source: §M.1 · p. 22
M.2 Near-threshold behaviour.
- Self-retention and cross-frame mass show a small negative effect near |γ −1| ≈ 0.032, but later bins improve; reliable improvement is suggested above ≳0.036.The authors describe this as an operational lower-bound calibration guideline, approximately 20% above the active threshold of 0.03.Source: §M.2 · pp. 22–23
M.3 Mechanistic consistency summary.
- The mechanistic evidence combines within-call direction, per-video changes, spatial localization, dose response, and targeting by baseline need.Together, these analyses support direct modulation in the intended direction rather than effects only from incidental side effects.Source: §M.3 · p. 23