Source-linked AI summary
Neural Video Compression Based on Deformable Temporal Alignment and Difference-aware Fusion
Chuyue Shan, Songlin Sun, Wang Chenwei, Shen Zihan
TL;DR
Conditional video compression relies on accurate temporal context, but propagated features and compressed motion can become unreliable under complex motion, occlusion, and long prediction chains. The paper combines deformable temporal alignment with difference-aware spatial selective fusion to generate complementary context and select reliable information. Experiments report improved rate-distortion performance over DCVC-DC, including a 1.3% BD-rate reduction versus the baseline in an ablation.
Problem
Propagated temporal features and compressed motion are vulnerable to accumulated mismatch and local alignment errors, reducing temporal-context reliability in challenging regions.
Method
CTAM uses deformable alignment to generate complementary temporal context, while DSSF uses spatial selection weights to fuse it with propagated context.
Results
The proposed method improves overall rate-distortion performance over DCVC-DC, while an ablation reports a 1.3% BD-rate reduction versus the baseline method.
Takeaways & Limitations
More reliable temporal priors improve temporal-context representation and conditional-coding performance within the proposed framework.
Abstract
from arXiv · showhide
In conditional coding-based neural video compression, the quality of temporal context directly affects compression per- formance. Existing methods mostly construct context from prop- agated reference features, but they are vulnerable to motion esti- mation and local alignment errors in regions with complex mo- tion, occlusion, and high-frequency textures, resulting in inaccu- rate temporal information. To address this issue, this paper pro- poses a method combining deformable temporal alignment and difference-aware spatial selective fusion. A Context-aware Tem- poral Alignment Module is used to generate complementary tem- poral context, while a Difference-aware Spatial Selective Fusion module adaptively selects reliable temporal information and sup- presses misalignment. Experiments show that the proposed method achieves certain rate-distortion performance improve- ment over DCVC-DC.
I. INTRODUCTION
Conditional coding-based neural video compression benefits from temporal context, but propagated features and compressed motion can become unreliable under long-term propagation and complex motion. The proposed method adds CTAM and DSSF to generate complementary context and adaptively fuse more reliable information.
- Motivation: DCVC-style conditional coding uses temporal dependencies in feature-domain context to exploit inter-frame redundancy.The DCVC series constructs temporal context within a conditional coding framework.
- Motivation: Propagated reference features and compressed motion can produce inaccurate context around motion boundaries, occlusions, and complex textures.Long prediction chains may accumulate weakly related information, while quantized motion can cause local alignment errors.
- Proposed method: CTAM combines motion-compensated predictions with reconstructed references and uses deformable convolution to generate complementary aligned context.The module operates outside the original temporal context path and learns local sampling offsets in feature space.
- Proposed method: DSSF models differences between contexts and generates spatial selection weights to adaptively fuse more reliable temporal information.The framework therefore combines complementary temporal contexts rather than relying solely on propagated features.
- Background: Neural video compression is an end-to-end alternative to hand-crafted hybrid-codec components, with conditional coding remaining a major research direction.Existing methods include residual coding-based, conditional coding-based, and implicit neural representation-based approaches.
B. Temporal Context Learning
Temporal context is a key prior for coding and reconstructing the current frame, but propagated features can accumulate irrelevant information during long prediction chains. The paper addresses this reliability challenge with CTAM and DSSF.
- Context reliability: Temporal context directly influences coding efficiency and reconstruction quality in conditional coding-based neural video compression.It is generated from propagated reference features and motion information in systems such as DCVC-TCM and DCVC-DC.
- Context reliability: Long-term propagation can accumulate errors or mismatched information, motivating complementary context generation from adjacent reconstructed frames.Prior methods improve context modeling through refresh mechanisms or additional temporal context, but reliability and computational complexity remain challenges.
III. METHOD
The method extends the conditional coding-based DCVC-DC framework with CTAM for deformably aligned context and DSSF for adaptive fusion with propagated context.
- Framework: The framework uses motion coding, temporal context generation, and frame coding to process each input frame.Decoded motion and propagated reference features generate context that guides encoding and reconstruction.
- Framework: CTAM generates deformably aligned context from the reference and predicted frames.This context is designed to complement the propagated temporal context.
- Framework: DSSF adaptively fuses the aligned context with propagated context before subsequent context processing.The fusion output is used by the context encoder, entropy model, and context decoder.
B. Context-aware Temporal Alignment Module
CTAM uses decoded motion and adjacent reconstructed frames to learn deformable feature alignment, producing complementary context for later spatially selective fusion. Its offsets support alignment under complex motion, non-rigid deformation, and occlusion.
- Context construction: CTAM backward-warps the previous reconstructed frame with decoded optical representation to obtain a predicted current frame.The predicted and reconstructed frames provide inputs for learning feature-domain sampling offsets.
- Deformable alignment: The learned offsets drive deformable convolution to resample reference features and generate aligned deformable temporal context.At each alignment level, offsets are predicted from the current intermediate feature before updating it.
- Role in framework: The alignment module is part of a framework that combines CTAM with a separate difference-aware spatial selective fusion module.The cited figure captions identify the structures of the proposed CTAM and DSSF modules.
- Deformable alignment: Deformable convolution enables feature-domain alignment for complex motion, non-rigid deformation, and occlusion.The method learns sampling offsets for individual locations rather than using fixed sampling positions.
- Role in framework: CTAM provides robust aligned features for DSSF while reducing error accumulation without changing motion coding or bitstream structure.The aligned context is generated from decoded motion and propagated reference features during training.
C. Difference-aware Spatial Selective Fusion (DSSF)
DSSF fuses propagated and deformably aligned temporal contexts by estimating their differences and selecting reliable information spatially. This design targets non-uniform misalignment while preserving complementary temporal cues.
- Context complementarity: The propagated context preserves long-term temporal priors, while deformably aligned context captures local correlations for compensating misalignment.The latter is obtained from adjacent reconstructed and predicted frames and captures local motion, non-rigid deformation, and occlusion-related correspondences.
- Context complementarity: Because context reliability varies spatially, directly adding or concatenating the two branches may introduce redundancy or noise.Deformably aligned context can be more effective in large-motion or occluded regions, whereas propagated context can be more stable in flat or small-motion regions.
- Difference-aware selection: DSSF computes a difference map between the contexts to indicate potential motion-compensation errors or local misalignment.The difference map is defined as D_t = |C_t^0 − Č_t^0|.
- Difference-aware selection: A lightweight fusion network uses both contexts and their difference map to generate spatial selection weights for the propagated and deformably aligned branches.The weights are denoted A_b and A_c, respectively.
- Fused context: The weighted branches are fused into compensated temporal context for the subsequent context encoder, entropy model, and context decoder.This spatially selective fusion handles non-uniform misalignment more effectively than simple fusion and is intended to improve compression performance.
A. Experimental Settings
The method is trained on cropped Vimeo-90K videos and evaluated on UVG and HEVC datasets under a standardized low-delay configuration. Testing uses PSNR, BD-Rate, 96 frames, and an intra-period of 32.
- Datasets: Training uses Vimeo-90K videos cropped from 448*256 to 256*256, while testing uses UVG and HEVC Class B, C, D, and E datasets.The listed datasets define the training and evaluation data used in the experiments.
- Test conditions: The experiments use low-delay coding, BD-Rate for compression-rate changes, and PSNR for video quality evaluation.Positive BD-Rate values indicate bitrate increases, while negative values indicate bitrate savings.
- Test conditions: Each video is tested for 96 frames with an intra-period of 32 after conversion from YUV420 to RGB using FFmpeg.These settings support the reported RGB-space comparisons.
B. Performance Comparison
The proposed method improves compression efficiency over H.266/VVC under the reported RGB-space PSNR evaluation setting. The comparison uses 96 tested frames and an intra-period of 32.
- Rate-distortion results: 19.0% average bitrate saving is achieved compared with H.266/VVC in the RGB color space measured with PSNR.The evaluation uses 96 tested frames and an intra-period of 32.
C. Ablation Study
The experiments report overall rate-distortion comparisons and component ablations under RGB-space PSNR evaluation. The ablations use DCVC-DC as the baseline and quantify contributions from the proposed context components.
- Ablation setup: The ablation study evaluates average rate-distortion performance on the HEVC dataset with an intra-period of 32.Table 2 reports bitrate savings from each method relative to the DCVC-DC baseline.
- Comparisons: The reported visual and rate-distortion comparisons include representative test sequences and UVG, MCL-JCV, and HEVC Class C datasets.The rate-distortion curves use RGB colorspace, PSNR, and an intra-period of 32; the visual comparison includes zoomed-in regions.
- Component ablation: 1.3% BD-rate reduction is obtained when only CTAM is introduced compared with the baseline method.This result isolates the contribution of adding the additional temporal context through CTAM.
- Component ablation: 0.8% bitrate saving is achieved by M2 compared with baseline model M0 when directed temporal context is extracted directly from the predicted frame.M2 is used to evaluate context compensation separately from the baseline.
V. CONCLUSION
The proposed method enhances DCVC-DC’s temporal context representation through complementary deformable alignment and adaptive reliability-based fusion, improving overall rate-distortion performance.
- The method enhances DCVC-DC for conditional coding-based neural video compression.
- CTAM uses deformable convolution to generate complementary temporal context and reduce local misalignment.
- DSSF adaptively selects reliable contextual information for temporal context fusion.
- The proposed method improves temporal context representation and overall rate-distortion performance.