PaperlayerView exact revision on arXiv

Source-linked AI summary

LightVAE: Towards Compact and Efficient Video Autoencoders

Shiqiao Gu, Yang Yong, Fengjun Zhong, Ruihao Gong, Chengtao Lv, Xiangyu Fan, Lei Yang, Xianglong Liu

arXiv:2610.14726v1cs.CV

TL;DR

Video VAE encoding and decoding remain computationally expensive, while existing Conv3D compression methods do not separately account for spatial and temporal compression sensitivities. LightVAE combines temporal low-rank compression, width-preserving block pruning, and probe-guided recovery. It accelerates decoding across evaluated backbones and GPUs, with reported throughput and quality gains against efficient decoders and the original MiniMax-H3 model.

  • Problem

    Video VAE inference remains expensive, while existing methods constrain spatial and temporal computation jointly or remove temporal computation without distinguishing their compression sensitivities in pretrained Conv3D.

    Source: §1 · pp. 1–2

  • Method

    LightVAE exactly separates causal Conv3D into a dense aggregate spatial response and a temporal correction, then combines low-rank correction compression, width-preserving block pruning, and probe-guided feature alignment.

    Source: §1 · p. 2

  • Results

    LightVAE consistently accelerates decoding across three backbones and two GPUs; on MiniMax-H3 it reaches up to 2.97× original decoding throughput while retaining 99.51% of the VBench-2.0 overall score.

    Source: §6 · p. 10

  • Takeaways & Limitations

    LightVAE’s decoding acceleration extends across evaluated video VAE backbones, and its framework also extends to reference encoding.

    Source: §6 · p. 10

    Convolutional TLR is evaluated only on the Wan VAE family, while MiniMax-H3 experiments evaluate block pruning rather than TLR.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D. To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth. LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction. Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction. An algebraically equivalent folded implementation converts the arithmetic savings into practical acceleration. LightVAE further uses width-preserving block pruning to reduce depth and probe-guided feature alignment, guided by feedback from a frozen teacher suffix, to recover quality after compression. The framework also extends to video encoders. Across three video VAE backbones and two GPU platforms, LightVAE consistently accelerates decoding. For the Wan VAE family, it achieves up to $2.24\times$ and $1.11\times$ the decoding throughput of state-of-the-art efficient VAE decoders on NVIDIA H100 and RTX~5090, respectively, with higher DAVIS PSNR and lower LPIPS. On MiniMax-H3, it achieves up to $2.97\times$ the original decoding throughput while retaining 99.51\% of the original model's overall VBench-2.0 score. Our code is available at https://github.com/ModelTC/LightVAE.

1 INTRODUCTION

Video VAE inference remains costly even as denoising accelerates, and existing Conv3D compression methods constrain spatial and temporal modeling jointly or remove temporal interactions. LightVAE separates these compression targets and combines temporal compression, depth pruning, and probe-guided recovery; it reports faster decoding across evaluated backbones and GPUs.

1 INTRODUCTION

  • Existing approaches either constrain spatial filtering and channel mixing jointly or remove local temporal interactions while narrowing spatial representations, motivating separate treatment of Conv3D’s spatial and temporal components.VAE inference can account for a larger share of end-to-end latency as denoising becomes faster.Source: §1 · pp. 1–2
  • LightVAE combines temporal low-rank compression within eligible convolutions with width-preserving block pruning across network blocks, reducing temporal computation and network depth, respectively.LightVAE-Pro applies TLR at the original decoder depth; LightVAE-Lite additionally prunes blocks, while Transformer autoencoders use block pruning without TLR.Source: §1 · p. 2
  • TLR exactly reparameterizes causal Conv3D as an aggregate spatial response plus a difference-driven temporal correction, retaining a dense spatial kernel while applying layer-specific low-rank approximation only to the correction.An algebraically equivalent folded implementation reduces convolution calls and intermediate tensors for practical acceleration.Source: §1 · p. 2
  • Probe-guided recovery uses gradients from a frozen teacher suffix and signed student–teacher feature discrepancies to weight feature alignment according to their effects on reconstruction.The probe loss itself is excluded from the recovery objective.Source: §1 · p. 2
  • Across three backbones and two GPUs, LightVAE consistently accelerates decoding; LightVAE-Lite reaches 2.24× Turbo-VAED throughput on Wan2.2 and 1.11× Flash-VAED throughput on RTX 5090 for Wan2.1, with higher DAVIS PSNR and lower LPIPS.On MiniMax-H3, it reaches up to 2.97× the original decoding throughput while retaining 99.51% of the VBench-2.0 overall score.Source: §1 · p. 3

2 RELATED WORK

Prior video-autoencoder acceleration combines lightweight operators and pretrained-decoder compression, while separate lines of work study structural pruning and feature distillation. LightVAE distinguishes itself by preserving dense spatial modeling while selectively compressing temporal computation and using a teacher-suffix probe to weight feature alignment.

Efficient video autoencoders and operator design

  • Turbo-VAED uses depthwise-separable Conv3D in low-resolution stages and dense Conv3D at higher resolutions, while Flash-VAED adds channel-pruned Conv2D and finds later stages tolerate reduced temporal computation.These operators constrain joint spatial filtering and channel mixing or remove local temporal interactions while narrowing spatial representations.Source: §2 · p. 3
  • Prior spatiotemporal-convolution factorization and low-rank kernel approximation provide alternatives, whereas TLR exactly separates an aggregate spatial response from a difference-driven temporal correction.TLR preserves the dense spatial kernel and controls temporal capacity through layer-specific ranks.Source: §2 · p. 3

Structured pruning and knowledge distillation

  • Channel pruning reduces width and block pruning reduces depth; LightVAE adapts recoverability-aware structural selection to video-autoencoder blocks and weights feature alignment using a frozen teacher-suffix probe.Unlike direct cross-head prediction distillation, the probe loss is excluded from the recovery objective.Source: §2 · p. 3

3 PRELIMINARIES

A video VAE encodes video into a latent representation and reconstructs it through a decoder; LightVAE focuses on decoder compression but also extends to causal-Conv3D encoders. The preliminaries frame causal Conv3D in terms of spatial input patches and show how LightVAE combines operator conversion with optional block removal.

3 PRELIMINARIES

  • A video VAE maps video V to latent representation Z through encoder E and reconstructs it as Y through decoder D; the framework focuses on decoder compression and also extends to causal-Conv3D encoders.Reference-conditioned encoding experiments are reported in Appendix G.2.Source: §3 · p. 3
  • For causal Conv3D, spatial input patches are arranged as columns of a matrix, with the source convolution’s spatial stride, dilation, and padding retained; patch matrices are used only for derivation, not inference.The output feature matrix contains Yt, and the bias is broadcast over spatial positions.Source: §3 · pp. 3–4
  • Figure 1 depicts TLRConv2D preserving the dense aggregate spatial response while approximating the temporal correction at low rank, with optional width-preserving block pruning reducing network depth.These are complementary forms of compression: per-layer temporal computation and network depth.Source: Figure 1 · p. 4

4 LIGHTVAE

LightVAE combines temporal compression within retained convolutions with width-preserving block pruning, then uses probe-guided recovery and a folded execution path. Its sections develop temporal low-rank approximation, efficient execution, and block selection and recovery.

4.1 TEMPORAL LOW-RANK COMPRESSION

  • TLR preserves the aggregate spatial response and approximates only the temporal correction, which is driven by historical–current feature differences.The temporal correction is factorized with a truncated SVD; the two historical kernels share an output projection.Source: §4.1 · pp. 4–5
  • Layer-specific ranks retain a thresholded fraction of temporal singular values, and TLR produces smaller x–t slice errors than Conv2D in the reported example.For fixed truncation, the stated layer-output error bound decreases with smaller historical–current feature differences.Source: §4.1 · p. 5
  • TLR reduces MACs per output feature frame when r < 2Coutd/(2d + Cout).Source: §4.1 · p. 5

4.2 WEIGHT-FOLDED EXECUTION WITH PROJECTED-HISTORY CACHING

  • Weight folding combines the current-frame correction with the spatial kernel, while projected-history caching reduces convolution calls and intermediate tensors without changing the MAC count.The joint convolution computes the current response and projections; cached lagged projections supply the history term.Source: §4.2 · pp. 5–6

4.3 BLOCK SELECTION AND RECOVERY

  • Block selection adapts recoverability-aware search to convolutional and compatible Transformer blocks, retaining required shortcuts, projections, and resampling operations.Shared LoRA parameters are merged into retained weights before TLRConv2D initialization.Source: §4.3 · p. 6
  • Probe-guided recovery uses a frozen teacher suffix to derive feature-alignment weights, alongside reconstruction, perceptual, and adversarial losses.Teacher and probe computations are used only during recovery.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

LightVAE is evaluated across Wan and MiniMax-H3 backbones on two GPUs, with reconstruction, generation-quality, and throughput comparisons. The ablations examine temporal approximation, operator efficiency, joint compression, and recovery choices.

5.1 EXPERIMENTAL SETUP

  • Evaluation uses 81-frame Wan clips and 124-frame MiniMax-H3 clips for throughput, with reconstruction quality assessed on UCF-101 and DAVIS.Within each backbone, only the decoder varies in generation-quality evaluation; prompts, seeds, and sampling settings remain fixed.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • Across Wan VAEs, Lite reaches up to 4.06×/4.20× original throughput on H100/RTX 5090; MiniMax-H3 reaches 2.95×/2.97× while retaining 99.51% of the VBench-2.0 overall score.The reported Wan throughput gains are across the Wan VAEs; the MiniMax-H3 values are for H100/RTX 5090, respectively.Source: §5.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • LightVAE-Lite reaches 2.24× Turbo-VAED throughput on Wan2.2 and 1.11× Flash-VAED throughput on Wan2.1, with higher DAVIS PSNR and lower LPIPS in both comparisons.On Wan2.1, Lite reaches 1.12×/1.11× Flash-VAED throughput on H100/RTX 5090; Wan2.2 Pro exceeds Turbo-VAED on reconstruction metrics and reaches 38.84 versus 29.24 FPS on H100.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 ABLATION STUDIES

  • At a shared retention threshold, temporal-only approximation yields the smallest output errors, while joint-factorization errors closely track spatial-only errors.At matched MAC budgets, TLR has the lowest error among tested allocations; near the 70% MAC floor joint compression reduces error, while at 80%–95% spatial-only compression performs better.Source: §5.3 · pp. 7–8, Figure 4 · p. 9, Figure 5 · p. 9
  • Higher η improves fidelity at reduced throughput, and η = 0.6 is used as the default balance.At the tested five-of-fourteen-branch architecture, uniform feature MSE underperforms reconstruction-only recovery, whereas probe-guided alignment improves recovery.Source: §5.3 · pp. 8, 10, Figure 7 · p. 9
  • TLR retains higher PSNR than the tested lighter operators before and after matched recovery, and folding improves throughput on both GPUs.TLR costs more than the lighter operators that it outperforms in PSNR.Source: §5.3 · p. 8, Figure 6 · p. 9
  • Applying folded TLR after pruning adds 1.17×–1.40× throughput over pruning alone, while the selected stage-specific configuration improves throughput by 1.27×–1.30× over using η = 0.6 throughout.The ablation uses pruning that retains 5 of 14 residual main branches and η = 0.6 for TLR.Source: §5.3 · pp. 8–9, Table 3 · p. 10
  • At comparable inference cost, block pruning reaches higher PSNR than channel pruning under matched recovery, and probe-guided alignment improves recovery over reconstruction-only loss.Block pruning retains 6 of 14 residual branches, while channel pruning removes approximately 41.7% of channels; uniform feature MSE underperforms reconstruction-only recovery.Source: §5.3 · pp. 9–10, Figure 8 · p. 9, Figure 9 · p. 10

6 CONCLUSION

LightVAE combines temporal low-rank compression, folded execution, width-preserving block pruning, and probe-guided recovery. It reports faster decoding with quality retention across Wan and MiniMax-H3 evaluations.

6 CONCLUSION

  • LightVAE reaches up to 2.24×/1.11× the throughput of evaluated efficient Wan decoders on H100/RTX 5090 with higher DAVIS PSNR and lower LPIPS; MiniMax-H3 retains 99.51% of VBench-2.0 while reaching up to 2.97× decoding throughput.The framework also extends to reference encoding.Source: §6 · p. 10
  • The authors report using generative AI tools solely to improve manuscript language, readability, and presentation, with all revisions reviewed by the authors.They state that the tools were not used for tasks requiring disclosure under the ICLR 2027 policy and that the authors take responsibility for the final manuscript.Source: AI USE STATEMENT · p. 10

A LIMITATIONS

The reported convolutional TLR evaluation is limited to Wan VAEs, while MiniMax-H3 evaluates block pruning rather than TLR. The accompanying analysis assumes a specific three-tap causal-convolution setting.

A LIMITATIONS

  • TLR’s convolutional evaluation is currently limited to the Wan VAE family; MiniMax-H3 evaluates block pruning, not TLR.Temporal ranks are selected from weight spectra, retention follows predefined group-wise quotas, and measured latency depends on hardware and implementation.Source: A LIMITATIONS · p. 14

B ANALYSIS OF TEMPORAL LOW-RANK COMPRESSION

  • The analysis assumes three-tap dense causal convolutions with unit temporal stride and dilation, identical layer inputs, and identical bias and boundary handling.SVD error results describe local conversion at initialization, whereas the folding identity remains exact after recovery.Source: B ANALYSIS OF TEMPORAL LOW-RANK… · p. 14

B.1 LOCAL ERROR AND COMPARISON WITH CONV2D

The appendix analyzes TLR conversion error and rank selection, then specifies its arithmetic cost and an equivalent folded execution with history indexing. It relates approximation error to discarded temporal components and historical–current feature differences.

B.1 LOCAL ERROR AND COMPARISON WITH CONV2D

  • Preserving the aggregate spatial response removes its approximation error term, leaving conversion error only in the temporal correction.The source bias is applied once; the temporal correction acts on historical–current feature differences.Source: §B.1 · p. 15
  • The actual output error depends on both discarded singular values and the directions of the feature differences.For a fixed truncation, the bound scales with the historical–current feature differences.Source: §B.1 · p. 15
  • Adding singular components can only reduce local conversion error, and rank q recovers the source layer exactly.At r = 0, the operator reduces to the summed-slice Conv2D initialized with S.Source: §B.1 · p. 15

B.2 RANK SELECTION AND INPUT DEPENDENCE

  • The spectral rule retains a fraction η of each layer’s temporal singular-value sum, producing layer-specific ranks that remain fixed after conversion.Unless otherwise specified, η = 0.6; zero temporal-correction matrices use no temporal factors.Source: §B.2 · p. 16

C.1 COMPUTATIONAL COST AND RANK CONDITION

  • TLRConv2D keeps dense spatial computation and uses fewer convolutional MACs when its temporal rank satisfies r < 2Coutd/(2d + Cout).The spatial cost remains fixed while temporal cost grows with rank; measured latency also depends on execution overhead and memory traffic.Source: §C.1 · pp. 16–17

C.2 FOLDING EQUIVALENCE AND HISTORY INDEXING

  • Folding combines shared-input spatial convolutions into one joint call while retaining pointwise history projection and TLR MAC count, with equivalent results up to floating-point order.The cache persists across chunks of a video, resets between videos, and uses zero virtual entries at sequence start under zero padding.Source: §C.2 · p. 17

D BLOCK-PRUNING IMPLEMENTATION

LightVAE adapts recoverability-aware search to video autoencoder blocks, selecting pruning structures while retaining required shortcuts and compatible dimensions. The appendix also visualizes high-similarity blocks, while clarifying that pruning decisions use aggregate search scores.

D BLOCK-PRUNING IMPLEMENTATION

  • Grouped block selection adapts TinyFusion’s recoverability-aware search to convolutional and Transformer blocks while retaining required shortcuts, projections, and resampling operations.Selection logits and shared LoRA parameters are optimized with frozen base weights.Source: D BLOCK-PRUNING IMPLEMENTATION · p. 18, §D.1 · p. 18

D.1 SELECTION, ADAPTATION, AND EXPORT

  • After search, the highest-scoring mask per group is retained, LoRA updates are merged, and eligible weights proceed through TLR conversion, recovery, and folded export.Block selection preserves retained representation widths; depth selection and temporal-rank allocation are sequential.Source: §D.1 · p. 18
  • Block search applies to compatible convolutional and Transformer blocks in encoders and decoders, while experiments use TLR for Wan VAEs and pruning alone for MiniMax-H3.Decoder reconstruction uses teacher-encoder latents; encoder compression is evaluated in downstream reference-conditioned generation.Source: §D.1 · p. 18

D.2 FEATURE VISUALIZATION

  • The two illustrated high-similarity blocks have small residual responses, but pruning decisions come from aggregate calibration-data search scores.The figure shows channel-wise RMS for shortcut, block output, and residual at feature-frame index 8, with a shared scale within each row.Source: §D.2 · p. 18, Figure D.1 · p. 19

E TEACHER-SUFFIX PROBE AND FEATURE ALIGNMENT

Probe-guided alignment uses a frozen teacher suffix to assess how student-feature discrepancies affect reconstruction, then weights feature matching accordingly. In a controlled comparison, it improves recovery over reconstruction alone, whereas uniform feature MSE underperforms it.

E TEACHER-SUFFIX PROBE AND FEATURE ALIGNMENT

  • At a compatible intermediate boundary, a frozen teacher suffix probes the student feature, and probe gradients set detached weights for feature matching.The probe loss constructs weights but is excluded from the recovery objective.Source: E TEACHER-SUFFIX PROBE AND FEATURE… · p. 19
  • Probe-guided weighting suppresses coordinate-wise feature-matching directions that locally increase probe loss and weights the remaining feature errors.Damping attenuates updates for small discrepancies, and clipping limits their magnitude without changing the sign relation.Source: E TEACHER-SUFFIX PROBE AND FEATURE… · pp. 19–20
  • Under the controlled comparison, probe-guided alignment improves recovery, while uniform feature MSE underperforms reconstruction alone.All objectives use the same architecture, initialization, data, recovery budget, feature boundaries, and feature-loss coefficient.Source: E TEACHER-SUFFIX PROBE AND FEATURE… · p. 20
  • The nonpositive first-order change is guaranteed for direct feature-space updates, but not necessarily for parameter updates because the prefix Jacobian couples coordinates.The paper evaluates the resulting training effect in Figure 8.Source: E TEACHER-SUFFIX PROBE AND FEATURE… · p. 20

F.1 TRAINING AND RECOVERY

  • Recovery uses fixed coefficients λperc = 0.8, λadv = 0.05, and λF = 0.01, with uniform and probe-guided feature objectives sharing the same feature coefficient and boundaries.Search and recovery use AdamW with a per-GPU batch size of one and no gradient accumulation.Source: §F.1 · p. 20

F.2 COMPRESSION AND ABLATION SETTINGS

The appendix evaluates spatial-only, temporal-low-rank, and joint compression under specified rank-allocation protocols, alongside model configurations and inference settings. Under the tested budgets, TLR has the lowest error at every evaluated budget.

F.2 COMPRESSION AND ABLATION SETTINGS

  • Wan Lite retains one residual main branch per stage (14→5) and uses r = 0 in up2/up3, while MiniMax-H3 uses block pruning only.MiniMax-H3 reduces decoder Transformer blocks from 36 to 12 and encoder residual main branches from 12 to 6.Source: §F.2 · p. 21
  • The approximation comparisons use original Wan2.1 decoder depth and width, identical encoder latents, and stage-wise or matched-MAC protocols without recovery.The stage-wise comparison uses η = 0.6; matched-MAC rank allocation greedily adds ranks by retained squared singular-value energy per added MAC.Source: §F.2 · p. 21
  • Under the stated allocation, TLR yields the lowest error at every evaluated budget; at 70% total MAC, spatial-only and joint errors are 1.1585 and 0.6965.At 80%, spatial-only error drops below joint compression after its spatial budget rises to 34.17%.Source: §F.2 · p. 21
  • The pruning comparison retains dense Conv3D and contrasts keeping 6 of 14 residual branches with removing approximately 41.7% of channels at similar pre-recovery cost.Their RTX 5090 throughputs are 45.58 and 42.69 FPS, respectively.Source: §F.2 · pp. 21–22

F.3 INFERENCE BENCHMARKS

  • Decoding benchmarks use H100 and RTX 5090 with 81-frame Wan clips and 124-frame MiniMax-H3 clips; Wan2.2’s VBench-2.0 generation setting instead uses 121 frames.OOM cases are not assigned numerical throughput or speedup.Source: §F.3 · p. 22, Table G.1 · p. 22

G.1 FINE-GRAINED VBENCH-2.0 RESULTS

VBench-2.0 scores remain close to originals under decoder compression; in MiniMax-H3 reference-conditioned generation, joint encoder–decoder compression also accelerates both stages while outputs remain visually close.

G.1 FINE-GRAINED VBENCH-2.0 RESULTS

  • 99.51% of MiniMax-H3’s original overall VBench-2.0 score is retained by Lite; Wan2.1 Pro scores 53.1251 versus 52.9867 for the original decoder.Wan2.2 Pro and Lite score 57.5155 and 56.9366, respectively, compared with 56.8765 for the original decoder and 57.7332 for Turbo-VAED.Source: §G.1 · p. 22
  • Although aggregate generation scores stay close to originals, H3 Lite improves Motion Rationality and decreases Multi-View Consistency.The authors note that individual VBench-2.0 dimensions can change substantially under these compression settings.Source: §G.1 · p. 22

G.2 REFERENCE-CONDITIONED GENERATION

  • Lite reduces the MiniMax-H3 encoder from 12 to 6 residual main branches and the decoder from 36 to 12 Transformer blocks across five reference-input configurations.The configurations use one, three, six, or nine reference images, or nine images plus one reference video; required shortcuts, projections, and resampling operations are retained.Source: §G.2 · p. 23
  • Reference encoding is accelerated by 2.12×–2.68× and decoding by 2.68×–2.75× in the MiniMax-H3 reference-conditioned pipeline.Table G.2 reports latency separately for encoding and decoding, with speedup computed as original latency divided by Lite latency.Source: §G.2 · p. 23, Table G.2 · p. 24
  • The four displayed compressed outputs remain visually close to the original outputs in overall appearance and scene content.The figure presents three temporally ordered frames per example with the prompt summary below.Source: §G.2 · p. 23, Figure G.1 · p. 24

Related papers

Loading 2610.14726v1…