Source-linked AI summary
TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation
Ariel Shaulov, Eitan Shaar, Amit Edenzon, Lior Wolf
TL;DR
Long-horizon autoregressive video generation is vulnerable to temporal drift because corrupted latent tokens are repeatedly reused in the conditioning context. TokenTrim detects high-drift tokens and hard-prunes them at inference time, improving long-video coherence without retraining or architectural changes. The method reports consistent gains across autoregressive video-generation settings, including a 5.91% Final Score improvement under Self Forcing.
Problem
Autoregressive video generation suffers temporal drift when corrupted latent conditioning tokens are repeatedly reused, causing errors to accumulate over long horizons.
Method
TokenTrim compares latent representations across adjacent batches, identifies unstable high-drift tokens, and hard-prunes them from the cached context before regeneration.
Results
5.91% Final Score improvement under Self Forcing raises TokenTrim to 81.84%, while it consistently improves aggregate quality and semantic metrics.
Takeaways & Limitations
Controlling the conditioning context at inference time can suppress error amplification and improve long-horizon temporal consistency without changing the underlying model.
Takeaways & Limitations
TokenTrim cannot fully correct persistent backbone limitations, and its fixed global pruning budget may be suboptimal across prompts, content types, and rollout lengths.
Abstract
from arXiv · showhide
Auto-regressive video generation enables long video synthesis by iteratively conditioning each new batch of frames on previously generated content. However, recent work has shown that such pipelines suffer from severe temporal drift, where errors accumulate and amplify over long horizons. We hypothesize that this drift does not primarily stem from insufficient model capacity, but rather from inference-time error propagation. Specifically, we contend that drift arises from the uncontrolled reuse of corrupted latent conditioning tokens during auto-regressive inference. To correct this accumulation of errors, we propose a simple, inference-time method that mitigates temporal drift by identifying and removing unstable latent tokens before they are reused for conditioning. For this purpose, we define unstable tokens as latent tokens whose representations deviate significantly from those of the previously generated batch, indicating potential corruption or semantic drift. By explicitly removing corrupted latent tokens from the auto-regressive context, rather than modifying entire spatial regions or model parameters, our method prevents unreliable latent information from influencing future generation steps. As a result, it significantly improves long-horizon temporal consistency without modifying the model architecture, training procedure, or leaving latent space.
1 Introduction
Long-horizon autoregressive video generation suffers temporal drift because corrupted latent conditioning tokens are repeatedly reused. TokenTrim detects and prunes unstable tokens at inference time, improving coherence without retraining or architectural changes.
- Autoregressive chunk generation extends video duration but accumulates artifacts and inconsistencies over time.
- Corrupted cached latent regions can be repeatedly reused and dominate attention, propagating errors through later generation steps.
- TokenTrim compares latent summaries across adjacent chunks to identify high-drift tokens before they are reused.
- The method hard-prunes unstable tokens from the cached conditioning context while operating entirely at inference time with negligible overhead.
- TokenTrim substantially reduces temporal drift across compatible autoregressive frameworks and improves long-horizon temporal consistency without modifying the model.
2 Related Work
Related work addresses long-video drift through training, architecture, caching, memory, and inference-time guidance. TokenTrim instead focuses on selectively controlling unreliable cached tokens during inference, with qualitative examples showing reduced degradation.
- Long-video systems have shifted toward causal autoregressive architectures that generate sequential frame chunks.
- Training-based approaches stabilize generation through self-generated rollouts, block-causal distillation, relaxed causality, attention sinks, or KV recaching.
- Inference-time guidance methods refine noise distributions or optimize latent motion variance to improve structure and motion coherence.
- Temporal drift reflects accumulated early errors and is commonly discussed alongside exposure bias in autoregressive generation.
- TokenTrim qualitative comparisons report fewer color shifts, artifacts, and unnatural motion than Rolling Forcing and Self Forcing.
3 Preliminaries: Self-Attention in Autoregressive Text-to-Video Models
Autoregressive text-to-video models generate frame chunks while conditioning on earlier latent tokens through self-attention and temporal KV caching. This efficient long-range context can also propagate imperfections and amplify temporal drift.
- Modern text-to-video generators synthesize videos in chunks, conditioning each new chunk on the prompt and earlier generated representations.
- Each chunk is represented by latent spatiotemporal tokens that provide long-range temporal dependencies for subsequent generation.
- Self-attention lets current tokens aggregate information across spatial and temporal context during repeated diffusion denoising steps.
- Temporal KV caches store key/value projections from earlier chunks so current queries can reuse past context without recomputing it.
- The attention inputs concatenate current and cached keys and values along the token dimension.
- Cached imperfections may be repeatedly attended to and propagated, producing identity changes, structural inconsistency, and degraded motion coherence.
4 Method
TokenTrim stabilizes autoregressive video generation by detecting drift in latent summaries and selectively pruning unstable cached tokens before reuse. It uses adaptive drift-triggered regeneration while operating entirely at inference time without retraining or architectural changes.
- Motion-Stabilized Initialization: FlowMo is applied only to the first batch to provide a motion-stabilized latent anchor, while later batches use the base autoregressive model.This initialization reduces early-stage corruption, whose effects can propagate through subsequent generation steps.
- Per-Token Latent Drift Estimation: TokenTrim compares latent summaries of consecutive batches to identify unstable tokens whose representations exhibit excessive drift.The summaries share the same spatial token structure, enabling direct token-wise comparison; drift captures patch-level semantic and structural deviations.
- Drift Severity and Trigger Criterion: At each step, TokenTrim ranks per-token drift values, selects the top p·N tokens, and computes their mean drift as the batch drift severity.The method maintains running statistics over previously accepted batches to support adaptive triggering.
- Drift Severity and Trigger Criterion: Pruning is triggered only when current drift severity exceeds an adaptive threshold; pruning is disabled during warm-up while drift statistics accumulate.The sensitivity hyperparameter controls trigger strictness, with λ = 2.0 used in the method.
- Hard Pruning and Regeneration: When triggered, TokenTrim removes high-drift token positions from the temporal KV cache and regenerates the current batch using the pruned context.The procedure allows at most one regeneration attempt; if the criterion remains violated, the regenerated batch is accepted to avoid unbounded loops.
- Method Scope and Integration: The method supports Self Forcing and Rolling Forcing, operates in latent space, requires no additional training, and adds almost no computational overhead.In Rolling Forcing, pruning is restricted to recent temporal-context tokens rather than the global anchor cache.
5 Experiments
TokenTrim is evaluated as an inference-time addition to Rolling Forcing and Self Forcing, using VBench, qualitative comparisons, and human preference studies. It improves long-horizon quality and consistency with modest overhead, while outperforming FlowMo in reported aggregate comparisons.
- Evaluation setup: TokenTrim is evaluated on Rolling Forcing and Self Forcing with the Wan2.1-1.3B model, using identical generation settings and no model-weight changes.Experiments generate 30-second videos at 16 FPS and 832 × 480 resolution.
- Automatic metrics: 79.67% Final Score under Rolling Forcing, up from 75.12% (+4.55%), with gains in Quality Score (+5.58%) and Semantic Score (+3.53%).These are aggregated VBench results comparing TokenTrim with the Rolling Forcing baseline.
- Automatic metrics: 81.84% Final Score under Self Forcing, up from 75.93% (+5.91%), with Quality Score increasing by +6.90% and Semantic Score by +4.91%.The improvement magnitude exceeds the reported Rolling Forcing improvement.
- Comparison with FlowMo: Under Self Forcing, TokenTrim reaches 81.84% Final Score (+5.91%), compared with FlowMo’s 76.05% (+0.12%).TokenTrim also gains +4.91% in Semantic Score and +6.90% in Quality Score, versus FlowMo’s -0.73% and +0.96%.
- Inference-time overhead: TokenTrim adds ×1.08 wall-clock runtime relative to Rolling Forcing, compared with ×2.18 for the FlowMo-adapted setup.The comparison averages runtime over 128 generated 30-second videos.
- Qualitative results: TokenTrim maintains object identities, colors, and structure over extended rollouts, while baselines and FlowMo show color shifts, structural distortions, background corruption, and identity drift.Examples include missing or duplicated limbs, texture degradation, lens-flare accumulation, and background warping.
6 Limitations & Future Work
TokenTrim’s gains remain bounded by the underlying video diffusion backbone and training data, and its fixed pruning budget may not suit every prompt or rollout.
- Scope boundary: TokenTrim can attenuate error propagation but cannot fully correct persistent backbone failures in object representation, identity preservation, or stable motion.The method leaves model parameters unchanged, so its gains depend on the base model’s capabilities and biases.
- Future work: A fixed global pruning setting may be suboptimal across prompts, content types, and rollout lengths.The paper proposes adaptive pruning based on drift statistics or uncertainty estimates as future work.
7 Conclusions
TokenTrim addresses temporal drift by pruning corrupted latent tokens during inference rather than adding capacity or retraining. This turns the KV cache into an active memory that selectively discards unreliable context.
- Conclusion: TokenTrim computes latent drift scores to distinguish stable context from hallucinated artifacts and prunes unstable tokens during inference.The stated goal is to break the feedback loop caused by accumulated corrupted tokens.
- Conclusion: Selective pruning helps the model maintain reliable anchors while reducing the repeated influence of corrupted tokens across long rollouts.The conclusion frames future progress as selective forgetting in addition to retaining context.
A VBench Metrics Breakdown
VBench results show broad gains across motion, semantic, structural, and aggregate dimensions for both autoregressive baselines. TokenTrim generally outperforms FlowMo while preserving motion complexity rather than merely suppressing dynamics.
- Overall pattern: TokenTrim improves nearly all motion- and stability-related VBench metrics for both autoregressive baselines.The reported pattern is attributed to mitigating error accumulation across generation steps rather than localized within-chunk refinement.
- Rolling Forcing: Under Rolling Forcing, Temporal Flickering improves by +2.12% to 98.89%, Motion Smoothness by +1.81%, and Dynamic Degree by +2.54%.The gains are reported alongside higher Human Action (+4.19%), Scene (+2.98%), and Overall Consistency (+3.45%).
- Self Forcing: Under Self Forcing, TokenTrim improves Temporal Flickering by +0.03%, Motion Smoothness by +1.01%, Dynamic Degree by +1.22%, and Appearance Style by +2.07%.Overall Consistency also increases by +0.64%, despite Self Forcing’s stronger temporal anchoring.
- Aggregated metrics: TokenTrim increases aggregate Final Score by +4.55% under Rolling Forcing and +5.91% under Self Forcing.The corresponding Semantic Score gains are +3.53% and +4.91%, while Quality Score gains are +5.58% and +6.90%.
- Comparison with FlowMo: TokenTrim achieves the best score in most VBench dimensions, whereas FlowMo leads only in a small subset and often trails both TokenTrim and the baseline.FlowMo’s gains are described as stronger in only selected categories.
- Comparison with FlowMo: TokenTrim improves Dynamic Degree by +2.54%, while FlowMo reduces it by -3.50%, indicating different effects on motion complexity.TokenTrim’s motion gains are described as preserving realistic dynamics while reducing drift.
- User study: The user study evaluates text alignment, quality, motion coherence, and drift using paired videos generated with and without FlowMo.Participants compare videos on the VideoJAM benchmark through randomized Google Forms responses.
C Ablation Study - VBench Metrics Breakdown
The ablation study shows that TokenTrim’s strongest results depend on drift-triggered, adaptive pruning combined with stable initialization; excessive pruning substantially harms quality.
- Full TokenTrim: TokenTrim improves Temporal Flickering by +2.12%, Motion Smoothness by +1.81%, Dynamic Degree by +2.54%, and Overall Consistency by +3.45% over the baseline.The full configuration achieves the strongest performance across most motion- and stability-related dimensions.
- Pruning Strength: A conservative 5% pruning rate improves Final Score by +3.23%, but trails full TokenTrim on Human Action (+1.48% vs. +4.19%) and Overall Consistency (+0.73% vs. +3.45%).These results favor adaptive pruning over a fixed conservative rate.
- Pruning Strength: Aggressive 20% pruning decreases Motion Smoothness by -11.69%, Dynamic Degree by -3.60%, Quality Score by -14.43%, and Final Score by -11.87%.Excessive token removal harms semantic and visual fidelity.
- Initialization: Removing FlowMo from initialization reduces Final Score by 2.22% relative to full TokenTrim, although the variant still outperforms the baseline.The ablation identifies stable initial latent anchoring as an important component.
- Conclusion: Overall effectiveness depends on drift-triggered pruning, moderate adaptive pruning ratios, and stable initialization; weakening any component causes measurable degradation.The study evaluates variants on Rolling Forcing to isolate inference-time context management.
D Additional Qualitative Experiments: TokenTrim vs Self Forcing
Against Self Forcing, TokenTrim preserves appearance, structure, motion, and texture more consistently during long-horizon generation as errors accumulate.
- Self Forcing: TokenTrim preserves stable body structure and consistent appearance for a translucent jellyfish, while Self Forcing introduces shape distortions and loss of translucency over time.The comparison uses identical long-horizon generation settings.
- Self Forcing: TokenTrim maintains coherent global motion, uniform texture, synchronized bird motion, stable silhouettes, and consistent robotic-spider limb geometry despite Self Forcing’s accumulated drift.Self Forcing exhibits spatial warping, uneven density, loss of collective structure, and structural drift across examples.
E Additional Qualitative Experiments: TokenTrim vs Rolling Forcing
Against Rolling Forcing, TokenTrim maintains stable geometry and background appearance during long-horizon rollouts that otherwise progressively degrade.
- Rolling Forcing: TokenTrim preserves helicopter geometry and consistent forest background appearance, whereas Rolling Forcing progressively introduces background distortion and structural inconsistencies.Rolling Forcing remains plausible in early frames but degrades as the rollout advances.
F Additional Qualitative Experiments: TokenTrim vs FlowMo
This section compares TokenTrim qualitatively with Rolling Forcing and FlowMo in additional long-horizon generation results. Against FlowMo, TokenTrim preserves more stable visual structure as the rollout proceeds.
- In identical long-horizon settings, FlowMo shows progressive structural and semantic drift, while TokenTrim preserves a stable silhouette in the humanoid book-stack example.FlowMo initially produces plausible motion but gradually introduces book-stack misalignment and deformation, causing inconsistent body proportions.
- Figure 9 presents an additional qualitative comparison between TokenTrim and Rolling Forcing.
- Figure 10 presents an additional qualitative comparison between TokenTrim and FlowMo.