Source-linked AI summary
Learning the Autoregressive: Direct Score-Difference Estimation with Amortized Future Credit for Video Distillation
Zixuan Duan, Yabo Chen, Xunzhi Xiang, Xin Zhang, Haibin Huang, Yin Chen, Chi Zhang, Qi Fan
TL;DR
Few-step autoregressive video distillation must estimate the teacher–student correction and account for how a current chunk affects later generation. Leta combines direct residual estimation with learned next-chunk credit, and improves visual and motion quality over DMD in the reported controlled experiments. Its approximation advantage for ScoreDelta is established conditionally in a fixed-feature linear model.
Problem
Few-step AR refinement must estimate the teacher–student correction and account for how generated chunks influence subsequent generation.
Source: §1 · pp. 1–2Method
Leta combines ScoreDelta’s direct teacher–student residual estimation with FutureCredit’s learned auxiliary signal from next-chunk teacher-residual VJPs.
Source: §1 · p. 2Results
Takeaways & Limitations
ScoreDelta is population-equivalent to DMD’s score-difference estimation, with a conditional approximation advantage under limited capacity in a fixed-feature linear model.
Source: §1 · p. 2, §4 · p. 5The approximation advantage is conditional on a fixed-feature linear model and is not a general guarantee for arbitrary neural estimators.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Few-step distillation for autoregressive (AR) video generation commonly first initializes an AR student and then refines it with distribution matching distillation (DMD). This refinement process faces two challenges. First, although initialization already yields a reasonably capable student, DMD obtains the refinement direction indirectly by estimating the full student score before forming the teacher-student correction. Second, detached-history training prevents the generator from assigning credit to a current chunk for its influence on subsequent chunks. We introduce Leta, a few-step distillation framework that addresses these challenges with ScoreDelta and FutureCredit. ScoreDelta directly estimates the teacher-student correction from denoising residuals, while FutureCredit learns an auxiliary cross-chunk update signal from vector-Jacobian products (VJPs) of next-chunk teacher residuals. Leta combines these signals into a joint teacher-guided update field in a single forward pass during generator updates. Teacher evaluations and credit VJPs are used only for estimator training. For ScoreDelta, we establish population equivalence to score-difference estimation and a conditional approximation advantage under limited capacity in a fixed-feature linear model. In our controlled experiments, Leta improves over DMD by 0.9 points on VBench and 0.66 on VideoAlign. This advantage persists at a higher generator update frequency (a generator-to-estimator update ratio of G:E=1:1), where Leta improves over DMD by 1.6 on VBench and 0.35 on VideoAlign. Project page: https://scdzx.github.io/Leta/.
1. Introduction
Leta targets two refinement gaps in few-step autoregressive video generation: estimating the teacher–student correction and crediting current chunks for their influence on later generation. Its controlled comparisons report gains over DMD, while its two learned signals address these gaps through complementary updates.
1. Introduction
- CD-initialized samples change less under teacher refinement than ODE-initialized samples, indicating a stronger starting point and smaller residual gap.DMD still estimates the full student score, so estimation errors may mask the subtle correction needed after initialization.Source: §1 · p. 2
- ScoreDelta directly estimates the teacher–student residual, which is population-equivalent to DMD’s score difference up to timestep scaling.In a fixed-feature linear setting, direct residual estimation can have lower approximation error when the residual is simpler to represent under the same capacity constraint.Source: §1 · p. 2
- FutureCredit uses next-chunk teacher-residual VJPs to provide auxiliary teacher-guided updates for a current chunk under detached-history training.An estimator predicts the processed VJP target from the noisy rollout, making the guidance available through a forward pass during generator updates.Source: §1 · p. 2
- Leta improves VBench Overall and VideoAlign visual and motion scores over DMD in the controlled setting.Released RAVEN and DistillAlign comparisons are references because those checkpoints were not retrained under the controlled protocol.Source: §1 · p. 2
- Leta combines the local correction and cross-chunk credit fields, with controlled ablations showing gains from both components.The method is evaluated for few-step text-to-video distillation with Wan.Source: §1 · p. 2
2. Related Works
Prior work reduces diffusion sampling steps and improves autoregressive video-student initialization, while Leta focuses on the refinement signal and cross-chunk credit. Its components build on these lines by estimating a local correction and amortizing next-chunk guidance.
Few-Step Distillation
- Progressive distillation and consistency-based methods reduce sampling steps, while DMD trains generators through distribution matching with teacher–student score differences.For AR video, Causal Forcing and Causal Forcing++ improve initialization through causal ODE and consistency distillation.Source: §2 · pp. 2–3
- Leta’s ScoreDelta focuses on refinement by directly learning the teacher–student correction from noisy student rollouts.Its conceptual overview contrasts direct correction estimation with estimating the full student score first.Source: §2 · p. 3, Figure 3 · p. 3
Autoregressive Video Generation
- Self Forcing trains on student-generated histories, while RAVEN and Self Gradient Forcing enable future losses to supervise historical context representations.Synthetic gradients more broadly learn update signals that decouple neural-network training.Source: §2 · p. 3
- Leta’s FutureCredit learns guidance from next-chunk teacher-residual VJPs, making cross-chunk information available through an estimator forward pass during generator updates with detached history.Source: §2 · p. 3
3. Background
The background frames autoregressive generation as sequential chunk production and DMD as on-policy distribution matching over complete student rollouts. It also identifies detached history as the point where ordinary generator updates omit credit for effects on future chunks.
Autoregressive generation and self-rollouts
- An autoregressive student factorizes a video into sequential chunks, each conditioned on previously generated content.In a self-rollout, each chunk is conditioned on the student’s own generated history, and the complete rollout is treated as one video sample.Source: §3 · p. 3
On-policy distribution matching
- At each timestep, distribution matching minimizes reverse KL from the student marginal to the teacher marginal using noisy complete student rollouts.The generator update is governed by the difference between teacher and student scores.Source: §3 · pp. 3–4
DMD training
- DMD alternates between updating an online fake-score model on on-policy rollouts and updating the generator using teacher and fake-score evaluations.The fake-score model is continually updated because the student distribution changes during training.Source: §3 · p. 4
- Caching prior chunks as fixed key-value pairs restricts gradients to the current chunk and omits credit for its influence on future generation.Source: §3 · p. 4
4. Learning the Autoregressive
Leta combines SCOREDELTA, a local teacher–student correction, with FUTURECREDIT, a cross-chunk credit signal, to guide autoregressive refinement through a joint update field. The section connects SCOREDELTA to score-difference estimation and gives a conditional finite-capacity result, while framing FUTURECREDIT as an approximate local credit signal.
Learning the Autoregressive
- Leta’s shared estimator predicts SCOREDELTA and FUTURECREDIT, whose fused direction guides generator updates in one forward pass.Teacher evaluations and credit VJPs are used only during estimator training.Source: §4 · pp. 4, 6, Figure 4 · p. 5
SCOREDELTA
- SCOREDELTA directly regresses the teacher denoising residual from noisy student rollouts rather than estimating the full student score first.For a fixed student distribution and optimal teacher, the population-optimal residual estimator recovers the same teacher–student score difference.Source: §4 · p. 4
Finite-capacity analysis
- Under the stated rank condition, direct residual estimation has zero approximation error while indirect estimation has positive error in the fixed-feature linear model.The condition is rank(A∆) ≤ κ < rank(AG).Source: §4 · p. 5
Scope
- The finite-capacity advantage is conditional on a fixed-feature linear model and is not a general guarantee for arbitrary neural estimators.The spiral illustration is a fixed-density score-regression example, not a test of the rank condition or the full denoising procedure.Source: §4 · p. 5
FUTURECREDIT
- FUTURECREDIT replays the next chunk’s final denoising step and propagates its teacher residual through the replay Jacobian to form a local credit target.The VJP is a biased approximation to cross-chunk distribution-matching credit, not exact KL-gradient recovery; its effectiveness depends on the local replay approximation and mixing weight λ.Source: §4 · pp. 5–6
5. Experiment
Under controlled few-step distillation, Leta improves over DMD, and ablations show gains from both components, including under a balanced update schedule. A moderate FutureCredit weight performs best.
Experimental setup
- The controlled variants share the same causal initialization, training data, rollout procedure, and sampling-step configuration.Standard runs use G:E = 1:4; the 1:1 ablation uses 600 total optimizer updates.Source: §5 · p. 6
Evaluation protocol
- Evaluation uses five shared fixed seeds for VBench and VideoAlign, making controlled-variant values paired averages less affected by sampled-noise variation.VideoAlign evaluates visual quality, motion quality, and text alignment.Source: §5 · p. 6
Main results
- Leta improves VBench Overall and VideoAlign visual and motion quality over DMD; released-checkpoint comparisons are reference comparisons.Against released RAVEN and DistillAlign checkpoints, Leta has higher VideoAlign visual and motion scores and competitive VBench performance, with gains concentrated in appearance and motion rather than text alignment.Source: §5 · p. 7
- Matched-timestamp Figure 6 samples show better preservation of object structure, scene composition, and subject consistency for Leta than DMD.Each prompt is shown above its corresponding pair.Source: §5 · p. 7, Figure 6 · p. 7
Ablations
- Adding FUTURECREDIT to SCOREDELTA yields further gains over DMD in VBench and VideoAlign, with the same ordering at G:E = 1:1.Table 2 compares component ablations under standard G:E = 1:4 and balanced G:E = 1:1 schedules.Source: §5 · p. 7, Table 2 · p. 7
Effect of future-credit weight
- A future-credit weight of λ = 0.25 performs best; removing future credit gives performance close to SCOREDELTA, while larger weights can degrade scores.Table 3 studies the effect of λ.Source: §5 · p. 7, Table 3 · p. 7
6. Conclusion
Leta combines direct local correction with cross-chunk credit, and experiments show improved visual and motion quality over DMD. The appendix frames residual estimation under Gaussian corruption and establishes its population connection to score estimation.
6. Conclusion
- Leta combines SCOREDELTA’s direct local correction with FUTURECREDIT’s cross-chunk credit in a shared estimator, improving visual and motion quality over DMD across evaluated settings.Current AR video generation lacks explicit planning for future motion and scene evolution, which may lead to unnatural dynamics in complex scenes and error accumulation over long horizons.Source: §6 · p. 7
A. Population Equivalence of Residual and Score Estimation
- The population-equivalence analysis fixes the student distribution and conditions on corruption level t and prompt c, with x = az + sϵ and square-integrable regression targets.Here t is the Gaussian forward-corruption noise level for the complete rollout, distinct from the chunk index and the generator’s internal solver-step index.Source: A. Population Equivalence of Residual… · p. 10
A.1. Proof of Proposition 1
The proof identifies the population-optimal residual estimator as the conditional mean of noisy teacher denoising residual targets. It connects this residual to the teacher–student score difference and shows local gradient equivalence under specified conditions.
A.1. Proof of Proposition 1
- The residual target’s conditional mean is the unique population minimizer, up to sets of zero student probability, despite noise in individual training targets.The decomposition’s second term is independent of the predictor.Source: §A.1 · p. 10
- For an optimal teacher denoiser, the Gaussian denoiser–score identity cancels the shared −x/s^2 term, proving the residual corresponds to the teacher–student score difference.The derivation uses Gaussian noising and the teacher’s denoiser prediction.Source: §A.1 · p. 10
- At the population optimum, SCOREDELTA and DMD using the same fixed teacher recover the same correction, and both inherit that teacher’s error relative to the true score.For an approximate teacher, the same teacher score field is used in both methods.Source: §A.1 · p. 10
A.2. Equivalence of the local ScoreDelta gradient
- With λ = 0 and wt = α_t^2, the SCOREDELTA surrogate gradient equals the DMD stop-gradient expression when both methods use the same retained generator graph.The equivalence holds when sampling and weights agree and gradients are integrable; it concerns the local update before RMS matching and excludes FUTURECREDIT.Source: §A.2 · p. 10
B.1. Proof of Proposition 2
The proof compares indirect and direct residual estimation under the same fixed features and finite rank budget. Direct estimation can have lower approximation error when the residual is simpler than the full student denoiser.
B.1. Proof of Proposition 2
- The indirect residual’s squared error equals the denoiser approximation error, so both approaches can be compared on the same residual target after removing common noise.Subtraction does not amplify absolute error.Source: §B.1 · p. 10
- With a shared fixed-feature linear bottleneck of rank at most κ, the two estimators approximate the full student denoiser and the teacher–student residual, respectively.The bottleneck has at most κ(d+m) trainable parameters.Source: §B.1 · pp. 10–11
- If the residual’s rank is at most κ while the full denoiser’s rank exceeds κ, direct residual estimation fits exactly while the indirect denoiser must discard a nonzero singular value.The rank condition is sufficient, not necessary; a smaller singular-value tail for the residual also gives an advantage.Source: §B.1 · p. 11
B.2. A Gaussian example
A Gaussian construction gives a case where the teacher and student share all but one coordinate, making their residual simpler than the full denoiser. The capacity advantage depends on the chosen function classes and target complexity.
B.2. A Gaussian example
- The construction uses positive-definite Gaussian covariances and Gaussian corruption, yielding a valid teacher–student case for the rank condition.The Gaussian conditioning uses a common normalized input with standard normal distribution.Source: §B.2 · p. 11
- The teacher and student distributions share their first d−1 coordinates and differ only in the variance of the last coordinate, producing cancellation in the shared denoiser coefficients.The residual’s last-coordinate coefficient is the remaining difference.Source: §B.2 · p. 11
- The complete denoiser has rank d while the residual has rank 1, so for rank budgets 1 ≤ κ < d the residual can be fit exactly while the full denoiser cannot.The example’s zero is approximation error; both optimal training losses retain the same positive denoising noise.Source: §B.2 · p. 11
B.3. What the capacity comparison assumes
- The capacity advantage is conditional on the chosen function classes and target complexity; changing only the target’s name does not create an advantage.With the same teacher skip parameterization, the losses and their parameter gradients are identical for the same samples and weights.Source: §B.3 · p. 11
B.4. Nonlinear spiral illustration
In fixed-density spiral regression, direct prediction of the score difference has lower reported NMSE than indirect score estimation at ρ = 0.6. The illustration is conditional and does not test the rank condition or the full denoising training procedure.
B.4. Nonlinear spiral illustration
- The teacher’s log-density ratio defines a bounded spiral score-difference field, while small transverse variation gives the output distribution thickness.The figure plots held-out difference vectors with matching colors and common axis limits.Source: §B.4 · p. 12
- The comparison regresses exact score differences or student scores using the same 51 fixed features and 102-parameter readouts on 8,192 training inputs.Ridge strength is selected independently for each route on 4,096 validation inputs; test NMSE uses 32,768 independent inputs.Source: §B.4 · p. 12
- 0.00923 ± 0.00938 versus 0.1264 ± 0.1774 NMSE across five seeds at ρ = 0.6, for direct prediction and sT −bsG, respectively.On the full test set, NMSEs are 0.00209 and 0.0720, respectively.Source: §B.4 · p. 12
- At ρ = 0, the full score is affine and the indirect route attains numerical precision.The paper frames the spiral example as a conditional representation effect in fixed-density regression.Source: §B.4 · p. 12
C. FutureCredit Implementation
FutureCredit trains an estimator from replay-based VJPs of next-chunk teacher residuals, making cross-chunk guidance available through a forward pass during generator updates. Its implementation uses projected, scale-matched targets and shares estimator features across two readouts.
C. FutureCredit Implementation
- A replayed next chunk supplies a VJP of its teacher residual through the generator’s final denoising step, and the detached target trains an estimator to predict the processed credit signal.The replay uses the saved noisy input, solver timestep, and detached history as a differentiable input; generator parameters and teacher residual are held fixed for the VJP target.Source: C. FutureCredit Implementation · p. 12
- The credit target applies a projection that subtracts each channel’s frame mean, followed by RMS-based scale matching for the chunk target and generator direction.The projection is specified for a chunk with m latent frames.Source: C. FutureCredit Implementation · p. 12
- Two zero-initialized 1 × 1 × 1 readouts share a bidirectional representation and are optimized together under complementary estimator losses.The default update weight is λ = 0.25; the λ = 0 ablation retains the auxiliary loss and its effect on shared features.Source: C. FutureCredit Implementation · p. 12