PaperlayerView exact revision on arXiv

Source-linked AI summary

VoRTeC: Taming Foundation Flow for One-step Real time Video Compression

Yichong Xia, Qinhong Wu, Bin Chen, Jinpeng Wang, Zeyuan Chen, Haoqian Wang

arXiv:2609.02291v2cs.CVcs.AI

TL;DR

Ultra-low-bitrate video compression struggles with oversmoothing, decoding latency, and temporal inconsistency. VoRTeC uses compact spatiotemporal latent codes, flow-trajectory estimation, multi-scale prior fusion, and frame-group communication to enable one-step decoding with a video-native foundation flow. It reports strong rate–perception performance, including 32 FPS decoding at 480p and substantial gains over diffusion-based baselines.

  • Problem

    Ultra-low-bitrate video compression must avoid the oversmoothing of neural codecs while overcoming the latency and temporal-consistency limitations of generative video compression.

  • Method

    VoRTeC compactly codes spatiotemporal latents, maps them onto Wan2.1 flow trajectories, fuses multi-scale priors, and communicates information across frame groups for one-step decoding.

  • Results

    32 FPS decoding at 480p is reported, alongside 58.12%–73.25% BD-rate savings in LPIPS against diffusion-based video compression baselines.

  • Takeaways & Limitations

    VoRTeC provides a foundation-flow-based route to fast video compression with an improved rate–perception trade-off and temporal continuity.

  • Takeaways & Limitations

    The compressed representation is assumed to follow a normal distribution during flow-path modeling.

Abstract

from arXiv · show

Ultra-low bitrate video compression still faces critical challenges: traditional neural video compression inevitably introduces blurring artifacts, while diffusion-based generative video compression suffers from excessive decoding latency and poor temporal consistency. To address these issues, we propose $\mathtt{VoRTeC}$, a Video Compression framework built upon a foundational flow model (Wan2.1). By compactly encoding latent video representations, predicting the positions of compressed representations along flow trajectories, and integrating multi-scale priors, $\mathtt{VoRTeC}$ enables the compressor to harness generative video flow priors effectively. Without accessing the parameters or gradients of flow matching networks, our framework achieves one-step decoding and reconstructions with high perceptual fidelity. Meanwhile, we maintain consistency across frame groups via tail-frame reuse and prior caching. Extensive experiments demonstrate that our method reduces bit consumption by 58\% compared to prior diffusion-based approaches, with decoding speed boosted by 3 to 197 times: $\mathtt{VoRTeC}$ achieves a decoding speed of 13 FPS at 720p and 32 FPS at 480p.

1 Introduction

Ultra-low-bitrate video compression must balance perceptual fidelity, temporal coherence, and decoding speed. VoRTeC addresses these constraints with a one-step foundation-flow framework that combines compact latent coding, prior fusion, and inter-group communication.

  • 1 Introduction: Ultra-low-bitrate compression remains unresolved because distortion-oriented neural codecs oversmooth textures and erase fine structures.Traditional and neural codecs can lose perceptual realism as bitrate becomes extremely low.
  • 1 Introduction: Video generative compression improves texture quality but must preserve temporal coherence, which image-domain priors cannot reliably model.Frame-wise enhancement can produce perceptual flickering as restored textures drift across frames.
  • 1 Introduction: Diffusion-based video codecs capture spatio-temporal priors but typically require at least five sequential denoising iterations, creating prohibitive decoding cost.Video-native flow models improve temporal stability, but multi-step flow-matching inference slows decoding.
  • 1 Introduction: VoRTeC introduces inter-group communication to propagate information between frame groups and improve temporal consistency without additional training or decoding overhead.The method uses preceding video-group information to support subsequent groups and reduce bitrate.
  • 1 Introduction: VoRTeC compresses spatiotemporal latents into compact codes, maps them onto flow trajectories, and uses multi-scale prior fusion for one-step decoding.The framework operates without access to the foundation flow-matching network’s parameters or gradients.
  • 1 Introduction: 58.12%–73.25% BD-rate savings in LPIPS are reported against diffusion-based video compression baselines, while VoRTeC+ improves perceptual performance by 20–30%.VoRTeC+ adds LoRA fine-tuning with 10M additional parameters while leaving inference speed unaffected.

2 Related Work

Related work progresses from distortion-oriented neural codecs toward generative priors and video-native flow models. The remaining trade-off is between perceptual realism, temporal modeling, and efficient decoding.

  • 2.1 Neural Video Compression: Neural video compression surpasses traditional hybrid codecs through end-to-end optimization of motion, residual or conditional coding, and entropy modeling.Successive systems enhance temporal context, feature modulation, and runtime efficiency.
  • 2.1 Neural Video Compression: MSE- or MS-SSIM-oriented neural codecs inevitably produce over-smoothed reconstructions lacking fine textures and perceptual realism at ultra-low bitrates.This limitation motivates moving beyond conventional rate–distortion objectives.
  • 2.2 Generative Compression: Generative compression extends rate–distortion optimization with a perceptual term measuring discrepancy between real and reconstructed data distributions.The coefficient γ controls the emphasis placed on perceptual realism.
  • 2.2 Generative Compression: Diffusion models provide stronger generative priors, but single-step image distillation methods remain limited to single-frame reconstruction without temporal modeling.This leaves video compression needing priors that represent spatio-temporal dependencies.
  • 2.3 Video Diffusion Models and Flow Matching: Wan2.1 advances video generation with a spatio-temporal VAE and flow matching, whose path is parameterized by xt = (1−t) x0 + t x1.The associated velocity network is DiT-based.
  • 2.3 Video Diffusion Models and Flow Matching: VoRTeC’s Flow Prior Multi-Fusion module aligns compressed latent structure with prior texture across patch granularities for compact encoding and high-fidelity reconstruction.Its stated design targets precise multi-scale alignment between the two representations.

3 Methodology

VoRTeC compresses spatiotemporal latents, estimates their positions along a foundation-flow trajectory, and fuses flow priors with compressed representations for reconstruction. Tail-frame reuse and prior caching connect frame groups, while multi-scale patchification and end-to-end rate–reconstruction optimization target temporal consistency, perceptual quality, and bitrate.

  • Spatiotemporal latent coding: A 3D causal encoder represents the first frame as an I-latent and subsequent frames as temporally downsampled P-latents within frame groups.The video is partitioned into key I-Groups and predictive P-Groups to organize temporal coding.
  • Spatiotemporal latent coding: The latent codec models latent-wise context using the I-latent and cached prior context to reduce redundancy during group compression.The I-latent provides group-wide context, while cached context is unavailable in the key group and is replaced by the compressed representation.
  • Flow-state estimation: Flow-State Estimation identifies the compressed representation’s approximate position along the flow path, and the transmitted rounded timestep enables FlowDIT denoising from that state.The timestep is rounded after estimating t⋆, and transmitting the resulting integer requires minimal, almost negligible bitrate.
  • Inter-group consistency: Tail-frame reuse connects groups by feeding the last I-Group frame into the next P-Group, while prior reconstructions are cached for subsequent prior fusion.The design maintains inter-group consistency without transmitting the key latent.
  • Flow-prior fusion: VoRTeC fuses flow-prior reconstructions with compressed latents through a ViT-based multi-scale fusion network without requiring foundation-network parameters or gradients.Coarse patches capture compressed representations’ structural information, while fine-grained prior patches attend to high-frequency details; FPMF also reduces bitrate through prior-guided modeling.
  • Flow-prior fusion: Coarse-grained patchification concentrates attention on semantically rich structures, whereas fine-grained patchification disperses attention across texture-rich regions.FPMF uses coarse patches for unreliable, low-frequency compressed features and fine patches for higher-frequency prior details.
  • Training objective: VoRTeC is optimized end-to-end with rate and reconstruction losses, including latent-domain reconstruction loss that constrains the compressor and FPMF module.The rate loss constrains the compressor’s bitrate, while the latent reconstruction term supervises representation reconstruction and fusion.

4 Experiments

Experiments evaluate VoRTeC against traditional, neural, and generative codecs across standard benchmarks, measuring perceptual quality, distortion, bitrate, temporal consistency, and throughput. Results show strong perceptual performance, high temporal continuity, fast decoding, and identifiable contributions from the proposed modules and losses.

  • Main Results: VoRTeC consistently outperforms generative and distortion-optimized baselines in perceptual quality on HEVC-B and UVG, while retaining competitive distortion performance.At 720p it also surpasses GLC-Video in PSNR; at 1080p its advantage narrows but remains leading.
  • Complexity Analysis: 12.60 fps decoding at 720p and 32.31 fps at 480p allow VoRTeC to outperform all learned baselines at lower resolutions.At 1080p, decoding reaches 3.93 fps and is 6.1× faster than GNVC-VD, 4.1× faster than YODA, and 3.1× faster than S2VC.
  • Qualitative Results: VoRTeC reconstructs more realistic frames than MSE-optimized codecs, with clearer facial contours and details; VoRTeC+ further improves textures while moderately reducing bitrate.The visual comparisons use HEVC and UVG sequences.
  • Temporal Consistency: Minimal Ewarp at the lowest bpp and consistently better FloLPIPS indicate stronger inter-frame continuity than the compared methods.Figure 6 evaluates FloLPIPS on UVG and Ewarp with corresponding bpp values on HEVC-ClassB.
  • Module Ablations: Replacing flow-state estimation with a fixed t=300 degrades perceptual quality, indicating difficulty learning the prior distribution pattern.This is reported in the module ablation experiments.
  • Module Ablations: Removing prior fusion, using identical mini-patch sizes, or omitting prior caching reduces perceptual quality or inter-frame stability.The reported mechanisms are prior redundancy, attention dispersion, and weakened information flow between frame groups.
  • Loss and Fine-tuning Ablations: LoRA fine-tuning adapts FlowDiT to compression noise and noticeably improves decoding perceptual performance.The paper also reports that adversarial and DINO losses improve visual realism, while adversarial loss more strongly improves depth-estimation accuracy.

5 Conclusion

VoRTeC enables one-step decoding with a pretrained Wan2.1 video flow-matching foundation model by embedding spatio-temporal representations into the flow path and refining them with multi-scale prior fusion. Its CGG scheme improves temporal continuity with negligible overhead, while the framework achieves a strong rate-perception tradeoff and real-time decoding.

  • Conclusion: VoRTeC enables one-step decoding using the pretrained Wan2.1 video flow-matching foundation model.It non-intrusively embeds spatio-temporal representations into the flow path.
  • Conclusion: 32 FPS decoding at 480p demonstrates substantially accelerated decoding compared with previous diffusion-based methods.The conclusion presents this as a real-time operating point.
  • Conclusion: The framework is compatible with flow-matching foundation models of various scales and architectures.This compatibility is presented as a new paradigm for generative video compression.

A.1 Mathematical Details

The appendix models compressor noise as zero-mean Gaussian and determines a flow-time state by minimizing Wasserstein distance. The optimum is characterized analytically, while experiments estimate the required variance from samples.

  • Noise Assumption: The compressor noise introduced by C(·) is assumed to follow a zero-mean Gaussian distribution.This assumption is based on a variational compression structure.
  • Flow State: For a given time step t, the flow state is computed from the normalizing-flow definition and follows a Gaussian distribution.The stated distribution is FL(t) ∼ N((1 −t)z, t2I).
  • Optimization: The mathematical problem is to solve for t⋆ by minimizing a distance between the relevant distributions.The distance is chosen as the Wasserstein Distance.
  • Solution: The minimum satisfies f(t⋆)=0, and experiments approximate σ with sample variance to estimate t⋆.The analytical minimum and practical estimation procedure are stated separately.

A.2.1 Model Traning Details

Training uses Wan2.1 1.3B as the foundation flow model alongside context and entropy modules from DCVC and ELIC. It proceeds in two stages with increasing group context, followed by optional LoRA fine-tuning for VoRTeC+.

  • Model and Stages: Wan2.1 1.3B provides the foundation flow model, while the context and entropy models use DCVC and ELIC modules.Training is divided into two stages.
  • Training Stages: The first training stage uses I-Group mode with GOP size 5 on Vimeo-90k, while the second expands the GOP size to 9 and uses 17-frame meta-groups.The second-stage configuration increases inter-group temporal context.
  • Optimization: AdamW is used throughout, with learning rates of 1 × 10−4 initially and 5 × 10−5 subsequently.These settings support finer parameter tuning in the later phase.
  • Fine-tuning: VoRTeC+ is fine-tuned with LoRA at rank 8, and experiments are conducted on an Nvidia A6000 GPU.The LoRA configuration is specific to the fine-tuning stage.

A.2.2 Test setting Detail

The experiments standardize video preprocessing, codec color handling, evaluation metrics, and baseline implementations across the test settings.

  • Preprocessing: Videos are converted from YUV420 to RGB using BT.709, with zero-padding for codecs requiring dimensions divisible by 64.Decoded frames are cropped back to the original dimensions; 720p testing uses center-cropped 1080p data.
  • Traditional Video Codecs: VTM-17.0 is evaluated in RGB while internally processing 10-bit YUV444 inputs under DCVC-series settings.The configuration specifies low-delay coding and includes frame rate, resolution, quantization parameter, and intra-period settings.
  • Baseline Implementations: DCVC-FM and DCVC-RT use officially released source code and pretrained weights.
  • Baseline Implementations: GLC-Video and SEVC are implemented using their publicly released code and pretrained weights.
  • Baseline Implementations: GNVC-VD, YODA, DiffVC, and FreeGVC use original paper-reported data under consistent experimental settings because their implementations are unavailable.
  • Metrics: S2VC bitrate data are taken directly from the original authors.LPIPS, DISTS, and FID are computed with the specified libraries and torchmetrics implementation, using feature size 2048 for FID.

A.3 Additional Experiment Results

Additional experiments evaluate VoRTeC across datasets, metrics, bitrate settings, and ablations. The results show strong low-resolution performance, competitive comparisons, improved bitrate allocation, scalable bitrate control, and better inter-group consistency with prior caching.

  • Quantitative Results: VoRTeC consistently outperforms generative methods across all metrics on low-resolution 480p data.On MCL-JCV, it surpasses GLC-Video on all metrics and performs comparably to GNVC-VD.
  • Quantitative Results: VoRTeC yields considerably higher PSNR than GLC-Video across the reported HEVC Class B, UVG, and MCL-JCV comparisons.Figure 10 reports the remaining distortion metrics on these datasets.
  • Quantitative Results: BD-Rate values are reported relative to VoRTeC across five datasets, with unavailable values marked by a dash when rate–distortion curves do not overlap.
  • Prior Ablation: Incorporating the flow prior reallocates bitrate toward specific objects rather than flat backgrounds by removing prior redundancy during end-to-end training.Figure 11 compares learned bitrate allocation maps with and without the flow prior.
  • Prior Cache Ablation: Prior caching resolves group-transition inconsistencies in texture, color, and shape, including changes in the eyes’ color and size.Without caching, inter-frame contextual learning becomes difficult and transitions become abrupt.
  • Meta Group Ablation: Reducing the number of P-Groups enables bitrate adjustment within a certain range through the scalable CGG design.The main setting uses a P-Group to I-Group ratio of 2, forming 25-frame meta-groups with one I-Group and two P-Groups.
Loading 2609.02291v2…