PaperlayerView exact revision on arXiv

Source-linked AI summary

GRACE: Generation-aware latent compression for efficient video generation

Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam, Donghoon Lee, Hyunsung Go, Yeonkyeong Lee, Hansaem Kim, Seungryong Kim

arXiv:2610.10524v1cs.CV

TL;DR

Highly compressed autoencoders reduce the tokens processed by video diffusion models, but can lower reconstruction quality and produce latents mismatched with a pretrained DiT. GRACE retains a pretrained base latent, adds residual information, aligns the compressed latent in the frozen DiT’s feature space, and adapts the DiT with asymmetric denoising. It matches the pretrained pipeline’s VBench generation quality while reducing latency by 11.1× at 480×832×81.

  • Problem

    Higher compression reduces video-diffusion token counts but can degrade reconstruction, while changed latent distributions create a compatibility challenge for pretrained DiTs.

    Source: §1 · p. 2

  • Method

    GRACE preserves a downsampled pretrained base latent, learns a residual for lost information, aligns the compressed latent in frozen DiT feature space, and denoises the base ahead of the residual.

    Source: §1 · pp. 2–3, §4.3 · p. 8

  • Results

    11.1× lower latency at 480×832×81 while matching the pretrained pipeline’s VBench generation quality, with nearly 8× fewer tokens.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across the compared autoencoders, better reconstruction does not necessarily mean better generation.

    Source: §6 · p. 12

    Applying GRACE to a new pretrained pipeline requires a new autoencoder and one Stage 1 run, costing 8.5 H200 GPU days in the reported setting.

    Source: J LIMITATIONS · p. 43

Abstract

from arXiv · show

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose Generation-Aware Latent Compression for Efficient Video Generation (GRACE), a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained latent in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by 8x and its latency by 11.1x at 480x832x81, while matching the generation quality of the pretrained pipeline before compression on VBench.

1 INTRODUCTION

Video autoencoder compression reduces the tokens processed by the DiT, but can harm reconstruction and generation or require costly adaptation. GRACE preserves the pretrained representation, complements it with residual information, and aligns the compressed latent with the pretrained DiT.

1 INTRODUCTION

  • Higher compression reduces tokens but limits latent capacity and reconstruction quality, while wider latents can impair generation and slow convergence.Changing the autoencoder can also require costly retraining, and reconstruction-only compression can shift the latent away from the distribution learned by the pretrained DiT.Source: §1 · p. 2
  • GRACE keeps a downsampled pretrained-encoder base latent and learns a residual latent for spatial detail and inter-frame motion lost through compression.The base preserves the pretrained representation; the residual complements what the downsampled input leaves out.Source: §1 · pp. 2–3
  • During generation, GRACE denoises the base before the residual, which uses the denoised base as an anchor without adding sampling cost.Both latents are denoised in a single pass.Source: §1 · p. 3
  • GRACE aligns the compressed latent to the pretrained latent in the frozen DiT’s feature space to preserve compatibility with its learned distribution.Intermediate DiT features measure the distance between the compressed and pretrained latents.Source: §1 · pp. 2–3
  • 11.1× lower latency at 480×832×81 while matching the pretrained pipeline’s VBench generation quality; the token count falls nearly 8×.The method compresses both axes to 16× spatial and 8× temporal; latency reduction reaches 15.5× at 736×1280×81.Source: §1 · p. 3

2 RELATED WORK

Prior work accelerates video generation through fewer denoising steps, cheaper computation per step, or more compressed latents. Autoencoder adaptation methods vary in whether they train from scratch or preserve compatibility with a pretrained DiT.

2 RELATED WORK

  • Latent-compression approaches increase spatial compression through autoencoder changes, increase temporal compression progressively, or widen tokens to preserve reconstruction, though wider latents can hurt generation.These methods extend compression beyond the common f8t4 setting.Source: §2 · p. 3
  • One approach regularizes latents with a pretrained vision foundation model to ease diffusion learning, but trains the autoencoder and generator from scratch (Yao et al., 2025).Changing the latent distribution can require retraining the DiT to restore compatibility.Source: §2 · pp. 3–4
  • Other methods reuse a pretrained DiT by constraining decoder reconstructability or aligning patch embeddings after latent divergence; GRACE instead uses intermediate features from the frozen DiT.This keeps the optimized autoencoder latent close to the representation space in which the pretrained generator operates.Source: §2 · p. 4

3 PRELIMINARIES

Latent video diffusion combines a video autoencoder, which compresses and reconstructs videos, with a diffusion transformer that denoises latents. The preliminaries describe their representations and the flow-matching setup used by the transformer.

3 PRELIMINARIES

  • A video autoencoder encodes a video into a latent and decodes that latent back into a video, while the diffusion transformer learns to denoise latents.The video latent has C channels, with f and t denoting spatial and temporal compression factors.Source: §3 · p. 4
  • The DiT patchifies the latent, so token count depends on spatial and temporal compression and patch size; a shift function maps schedule time u to flow-matching timestep τ.Larger shift values place more of the schedule at higher noise.Source: §3 · p. 4
  • The noisy latent interpolates between the clean latent and Gaussian noise, and the DiT predicts their difference conditioned on timestep modulation and text embeddings.The timestep embedding conditions transformer blocks through adaptive layer normalization.Source: §3 · p. 4

4 METHOD

GRACE compresses a pretrained video autoencoder in two stages, using the pretrained pipeline to guide its own compression. Stage 1 learns a base-and-residual latent with generation-aware alignment; Stage 2 adapts the DiT with the base denoised ahead of the residual.

4.1 OVERVIEW

  • GRACE fine-tunes the pretrained autoencoder and then adapts the pretrained DiT to the resulting compressed latent, using the pipeline before compression to guide both stages.The DiT adaptation uses LoRA and denoises the base at a lower noise level than the residual.Source: §4.1 · p. 4, §4.2 · p. 5, §4.3 · p. 8

4.2 STAGE 1: VIDEO AUTOENCODER TRAINING

  • Stage 1 combines a frozen pretrained base latent from downsampled input with a learned residual latent that carries information lost under compression.The residual encoder processes full-resolution video; the two latents are concatenated for decoding.Source: §4.2 · p. 5
  • Generation-aware alignment matches compressed and pretrained latents through intermediate features of the frozen DiT, optimizing the compressed representation without changing the autoencoder architecture.The pretrained features remain fixed, and gradient-norm weighting balances alignment and reconstruction losses.Source: §4.2 · pp. 6–7
  • The dual latent distorts the body less than a single latent, while alignment improves all three latent-uniformity metrics and leaves fewer sample artifacts.With alignment, latent components follow object regions and retain structure across frames.Source: §4.2 · pp. 6–7, Figure 3 · p. 6

4.3 STAGE 2: VIDEO DIFFUSION TRANSFORMER ADAPTATION

  • The base is denoised ahead of the residual during DiT adaptation, letting the base anchor residual generation while both are denoised in the same forward pass.The asymmetric schedule recovers subject-face detail where motion is large without increasing the number of function evaluations.Source: §4.3 · pp. 7–8, Figure 4 · p. 7

5 EXPERIMENTS

Experiments evaluate GRACE’s reconstruction and generation quality, speed, training cost, and design choices. GRACE preserves pretrained-pipeline quality while reducing latency, and ablations examine how alignment and denoising order affect results.

5.1 SETUP

  • The evaluation uses Wan2.1-I2V-14B and GRACE-VAE fine-tuned from the pretrained autoencoder, with the same 38.5-H200-GPU-day budget for DiT adaptation comparisons.GRACE allocates 8.5 days to the autoencoder and 30 to the DiT; generation latency is measured on one A100 GPU.Source: §5.2 · p. 8, §5.1 · p. 8

5.2 VIDEO RECONSTRUCTION RESULTS

  • GRACE-VAE achieves the highest generation quality among compressed autoencoders at the smallest token count, within 0.02 of the pretrained pipeline, despite a single-latent baseline reconstructing 1.13 dB better.Step-Video-VAE reconstructs 1.91 dB above LTX-VAE but scores 3.01 lower on VBench-I2V, even with 4× the tokens.Source: §5.2 · p. 9

5.3 VIDEO GENERATION RESULTS

  • At 480×832×81, GRACE stays within 0.02 of the pretrained pipeline on I2V and exceeds it on T2V while running 11.1× faster; at 736×1280×81, it runs 15.5× faster and stays within 0.04 on I2V.At the higher resolution, GRACE also reaches the highest T2V total at the lowest latency.Source: §5.3 · p. 9, Table 2 · p. 9, Table 3 · p. 10

5.4 TRAINING COST AND CONVERGENCE

  • With generation-aware alignment, the residual converges faster to lower flow-matching loss while base-channel behavior remains similar; GRACE matches the pretrained pipeline with 8.5 H200 GPU days for the autoencoder and 30 for the DiT.The reported training-cost comparison attributes most cost to DiT adaptation.Source: §5.4 · p. 11, Figure 7 · p. 11

5.5 ABLATIONS AND DISCUSSION

  • The dual latent and base-ahead offset raise T2V totals by 1.43 and 1.13, respectively, while alignment raises I2V and both T2V scores, recovering most of the I2V gap and exceeding the pretrained pipeline on T2V.The dual latent alone keeps the semantic score at the pretrained level; alignment raises that score as well as both T2V scores.Source: §5.5 · p. 11, Table 4 · p. 11
  • Aligning inside the pretrained DiT raises the semantic score to 82.55, compared with 80.04 for V-JEPA 2.1 features and 78.65 without that target.Asymmetric denoising is disabled in all rows of this alignment-target comparison.Source: §5.5 · p. 11, Table 5 · p. 11
  • Denoising the base first improves the total, whereas reversing the order drops below synchronous denoising, and the gain adds no inference cost because both latents share a forward pass.The schedule comparison covers both DiT adaptation and sampling.Source: §5.5 · p. 12

6 CONCLUSION

GRACE compresses a pretrained video diffusion pipeline while retaining a frozen base latent, learning a residual, and adapting the DiT. The appendix documents the autoencoder design and supplementary evaluations, ablations, analyses, and costs.

6 CONCLUSION

  • GRACE retains a frozen base latent, learns a residual for information lost under compression, aligns latents in the frozen DiT’s feature space, and denoises the base ahead of the residual.On Wan2.1-I2V-14B, it matches the pretrained pipeline’s VBench generation quality while running 11.1× faster at 480×832×81 and 15.5× faster at 736×1280×81.Source: §6 · p. 12

APPENDIX

  • The appendix covers autoencoder and DiT implementation, evaluation protocols, additional evaluations and ablations, latent and convergence analyses, computational costs, related work, qualitative results, and limitations.It also describes GRACE’s autoencoder architecture and training setup.Source: REPRODUCIBILITY STATEMENT · p. 12, APPENDIX · p. 17

A GRACE Autoencoder Details

  • The frozen encoder produces the base latent from spatially and temporally reduced video, while the residual encoder processes full-resolution video to capture detail and motion absent from the base.The stated reduction strategy uses bilinear interpolation spatially and temporal subsampling; among tested strategies, that setting reconstructs best.Source: A GRACE Autoencoder Details · p. 18
  • The autoencoder reuses Wan2.1 encoder and decoder backbones, adding a downsample block with a non-parametric shortcut to encode the residual.The shortcut folds space and time into channels; the decoder adds a symmetric upsample block and can use first-frame features through gated cross-attention.Source: A GRACE Autoencoder Details · p. 18, Figure A.1 · p. 18

A.2 IMPLEMENTATION DETAILS

The autoencoder is trained in two phases, with the frozen pretrained encoder retained throughout. The DiT is extended to accept compressed latents and denoise the base and residual separately.

A.2 IMPLEMENTATION DETAILS

  • Training first updates the residual encoder and decoder at 256×256×81, then raises resolution to 512×512×81 and trains only the decoder while keeping both encoders frozen.The autoencoder is trained on Panda-70M with 81-frame clips because reconstruction degrades quickly on shorter clips under high temporal compression.Source: §A.2 · p. 19
  • The residual encoder and decoder start from pretrained weights, while added downsample and upsample blocks are trained from scratch and residual decoder channels are zero-initialized.First-frame cross-attention gates are also zero-initialized, so the first frame initially has no effect.Source: §A.2 · p. 19
  • The alignment loss is applied only in phase one, when the latent changes; the single-latent baseline instead trains with reconstruction loss alone.Each alignment projection has two layers, a 16-channel bottleneck, and zero-initialized output.Source: §A.2 · p. 19

B.1 ARCHITECTURAL DETAILS

  • The DiT’s input and output projections are widened for the compressed latent, with pretrained weights copied to existing channels and added channels zero-initialized.The mask width doubles with the latent’s temporal compression factor.Source: §B.1 · p. 20
  • RoPE indices are scaled by the spatial and temporal compression factors to restore the token spacing expected by the pretrained DiT.Source: §B.1 · p. 20
  • Because the base and residual use different noise levels, the DiT receives two timesteps, uses summed block modulations, and conditions each output head on its corresponding timestep.The single-latent baseline uses one timestep and omits the residual modulation projection.Source: §B.1 · p. 20

B.2 IMPLEMENTATION DETAILS

The pretrained DiT is adapted while the autoencoder remains frozen, using selective fine-tuning and LoRA. Training samples denoising offsets so the model can use an asymmetric base-first schedule; comparison autoencoders receive a separate three-stage adaptation.

B.2 IMPLEMENTATION DETAILS

  • Adaptation standardizes each latent part and fine-tunes input and output projections, residual modulation, block modulation and normalization, with LoRA on attention and feed-forward linear layers.For image-to-video, LoRA also covers key and value projections of image cross-attention.Source: §B.2 · p. 20
  • The DiT is trained on Panda-70M at 480×832×81, with clips filtered for scene cuts or excessive motion and captions rewritten to match Wan2.1-14B prompt length and detail.Pretrained base-channel weights are retained, while residual-channel weights and the residual modulation projection are zero-initialized.Source: §B.2 · p. 20
  • Training samples the denoising offset and shift per video; inference uses 50 steps, denoising the base first and the residual last, with δ = 0 recovering the synchronous schedule.A lower shift was found to recover detail better on the compressed latent.Source: §B.2 · p. 20, Table B.1 · p. 20

C.1 ADAPTATION OF OTHER AUTOENCODERS

  • Compared autoencoders are adapted with DC-Gen’s three stages—patch embedding, input and output layers, then the full transformer with LoRA—using the same 38.5 H200 GPU-day budget as GRACE.The compared autoencoders lack GRACE’s base latent from the frozen Wan2.1 encoder because their latent spaces differ from the DiT’s training space.Source: §C.1 · p. 21

C.2 EVALUATION PROTOCOL

The evaluation compares adapted autoencoders and generators under specified sampling protocols, and reports per-dimension VBench outcomes at two resolutions. It also describes the V-JEPA 2.1 feature alignment evaluation.

C.2 EVALUATION PROTOCOL

  • The evaluation compares Step-Video-VAE v2, Video DC-AE, and LTX-VAE, and samples adapted models on VBench-I2V with shared guidance and flow-shift settings.Latency includes conditioning encode, denoising, and decode on one A100, but excludes model loading, text encoding, and MP4 writing; the compared released generators use differing prompt and guidance procedures.Source: §C.2 · p. 21
  • Latent uniformity is measured by embedding one randomly positioned token from each of 10,000 distinct Panda-70M clips, then reporting three distribution statistics averaged over two t-SNE seeds.Source: §C.2 · p. 21

C.3 ALIGNMENT TO V-JEPA 2.1

  • For V-JEPA 2.1 alignment, the evaluation uses its final encoder block and aligns the first 65 clip frames, temporally upsampling latent frames to match the encoder tokens.The frozen ViT-L encoder operates at 256×256; the alignment uses per-token cosine loss and adaptive weighting.Source: §C.3 · p. 22

D.1 VIDEO GENERATION

  • In text-to-video, DC-Gen scores above GRACE on aesthetic quality while scoring below on every total; the cited aesthetic predictor tracks photographic taste rather than prompt or conditioning-image fidelity.The passage attributes the score difference to more saturated DC-Gen samples.Source: §D.1 · p. 22, Table D.1 · p. 23
  • In text-to-video, GRACE leads on background consistency and temporal flickering at both resolutions, while its dynamic degree is 62.50 versus 56.94 at 480×832.At 480×832 image-to-video, LTX-Video leads on background consistency, motion smoothness, and temporal flickering while moving less than GRACE: 24.80 versus 38.62.Source: §D.1 · p. 22, Table D.1 · p. 23
  • In text-to-video, GRACE stays close to the pretrained pipeline’s color and style, while DC-Gen often departs; this difference is weaker in image-to-video, where the conditioning image fixes appearance.Source: §D.1 · p. 22
  • At 736×1280×81, DC-Gen again leads on text-to-video aesthetic quality while trailing on both totals, and LTX-Video has the highest image-to-video subject consistency.Source: §D.1 · p. 22, Table D.2 · p. 24

D.2 HUMAN EVALUATION

The blind study compares GRACE with Wan2.1-14B and DC-Gen across text-to-video and image-to-video. The section also reports autoencoder conditioning and DiT-alignment design findings.

D.2 HUMAN EVALUATION

  • Against DC-Gen, GRACE received 60.3–71.2% of votes for visual quality and temporal consistency, versus 14.1–24.4% for DC-Gen.Against Wan2.1-14B, GRACE was preferred in T2V on all three criteria; Wan2.1-14B was preferred in I2V, with many I2V votes rating the models about equal.Source: Table D.3 · p. 25, §D.2 · p. 25

E.1 AUTOENCODER DESIGN

  • Dropping the first frame with probability 0.5 during I2V training teaches the decoder to reconstruct without relying on that frame.Reconstruction without the first frame at inference stays close to reconstruction with it.Source: §E.1 · p. 27, Table E.1 · p. 27

E.2 ALIGNMENT DESIGN

  • Supervising the first 10 DiT blocks preserves generation quality while avoiding the reconstruction and convergence costs observed when supervising all 40.With all 40 blocks, PSNR falls by 0.42 dB, LPIPS rises from 0.032 to 0.038, and convergence is slower; generation is within 0.04 on I2V total between settings.Source: §E.2 · p. 27, Table E.2 · p. 27

F.1 LATENT ANALYSIS

The latent analyses distinguish the roles of GRACE’s base and residual latents and examine how alignment affects their visual structure and DiT adaptation. The discussion also situates GRACE’s computational costs and approach relative to other work.

F.1 LATENT ANALYSIS

  • Noise degrades reconstruction faster in zbase than zres; corrupting zbase breaks scene layout, while corrupting zres blurs frames and makes motion appear slower.The study sweeps noise level τ from 0 to 1 on Panda-70M videos at 480×832×81, measuring PSNR and LPIPS.Source: Figure F.1 · p. 28, §F.1 · p. 28
  • With Lalign, PCA of zres suppresses noise and reveals objects, whereas without alignment its components show little of the scene.The zbase latent is unchanged because the encoder is frozen; PCA colors are computed per clip and are not comparable across panels.Source: Figure F.2 · p. 29, §F.1 · p. 29

F.2 CONVERGENCE BEHAVIOR

  • Lalign accelerates convergence and lowers flow-matching loss on residual channels, while base-channel behavior remains almost identical with or without alignment.The base space is already modeled by the DiT and has little to adapt.Source: Figure F.3 · p. 29, §F.2 · p. 29

G COMPUTATIONAL COST

  • Table G.1 reports training cost in H200 GPU days and inference latency at 480×832×81 on one A100 with 50 sampling steps.The supplied passages state the table’s scope but not its cost values.Source: Table G.1 · p. 30, G COMPUTATIONAL COST · p. 30

H MORE DISCUSSION ON RELATED WORK

  • Unlike latent-space work that mostly trains generation models from scratch, GRACE shapes latents for reuse of an already trained model; wavelet designs reduce autoencoder cost, not DiT token count.Source: H MORE DISCUSSION ON RELATED WORK · p. 30

I ADDITIONAL QUALITATIVE RESULTS

The appendix presents additional text-to-video and image-to-video samples across resolutions, alongside style variations and latent visualizations. The captions specify the models and shared prompts or conditioning used for the comparisons.

I ADDITIONAL QUALITATIVE RESULTS

J LIMITATIONS

GRACE’s compression trades some reconstruction detail for generation quality, and applying it to another pretrained pipeline requires a separate autoencoder training run.

J LIMITATIONS

  • Small objects, including distant faces, sign text, and thin structures, are often lost or broken; this effect is weaker at 736×1280, where the same compression ratio leaves more tokens per frame.At the same token count, GRACE-VAE’s PSNR is 1.13 dB below the single-latent baseline; adding residual channels could recover detail, but adapting a pretrained DiT to a higher-dimensional latent is difficult.Source: J LIMITATIONS · p. 43
  • Each new pretrained pipeline requires a new autoencoder and one additional Stage 1 run, costing 8.5 H200 GPU days in the reported setting.The base latent comes from that pipeline’s frozen encoder, and Lalign is supervised by its DiT.Source: J LIMITATIONS · p. 43

Related papers

Loading 2610.10524v1…