EVC: Towards Real-Time Neural Image Compression with Mask Decay

1 INTRODUCTION

• Neural image codecs’ large complexity can cause unsatisfactory real-world latency, while rate control often requires separate models for different trade-offs.Using a single model avoids loading a model selected for each target quality.Source: §1 · p. 1
• EVC runs at 30 FPS for 768 × 512 inputs and has rate-distortion performance on par with state-of-the-art models.An adjustable quantization step supports different rate-distortion trade-offs within one model.Source: §1 · pp. 1–2
• Mask decay improves the medium and small models by 50% and 30%, respectively, using a large model’s parameters.The method transforms the large model into the student structure by sparsifying inserted masks, then merging them into the original layers while preserving network functionality.Source: §1 · p. 2
• The scalable encoder progressively reduces residual representation by decaying the large encoder multiple times, supporting different encoding complexities with one decoder.The authors report better rate-distortion performance than training separate encoders and SlimCAE.Source: §1 · p. 2

2 RELATED WORKS

• Earlier neural image compression work introduced additive uniform noise for end-to-end training and then the hyperprior structure.Source: §2 · p. 2
• Recent neural codecs use improved probability estimation and more powerful backbones, yet still face high latency and separate models for each rate-distortion trade-off.The cited advances include autoregressive and Gaussian-mixture entropy models, attention, INNs, Transformers, and Swin backbones.Source: §2 · p. 3
• Model compression and knowledge distillation are mainly studied on semantic tasks, and the paper applies teacher-based improvement to small neural image compression models.Its proposed mask decay reuses teacher filters and adds a novel sparsity regularization loss.Source: §2 · p. 3

3.1 OUR EVC FOR IMAGE COMPRESSION

• Adjustable quantization steps let one EVC model support different rate-distortion trade-offs, while residual and depth-wise convolution blocks provide Small, Medium, and Large capacity configurations.The configurations use channel widths [64, 64, 128, 192], [128, 128, 192, 192], and [192, 192, 192, 192], respectively.Source: §3.1 · pp. 3–4

3.2 IMPROVE THE STUDENT BY MASK DECAY

• The proposed sparsity loss gives larger mask values stronger decay and smaller positive values a larger gradient toward zero, with a stationary point at x = 1.It is decoupled from the rate-distortion task loss as weight decay.Source: §3.2 · p. 5
• Mask decay inserts trainable masks into a pretrained teacher, sparsifies them, and merges them into convolution layers to form a smaller student while preserving network functionality.The resulting student is finetuned for a few epochs.Source: §3.2 · pp. 4–5

3.3 THE SCALABLE ENCODER

• On Tecnick and Kodak rate-distortion curves, EVC is on-par with SwinT, Entroformer, and STF and outperforms VTM.The figure caption describes the compared models as dramatically faster than those methods.Source: Figure 6 · p. 6
• Progressively compressing the encoder learns residual representations; applying the first k encoders provides a chosen encoding complexity while preserving earlier decodable bitstreams.Only the parameters of the encoder being compressed are optimized at each step.Source: Figure 5 · p. 5, §3.3 · p. 6

4 EXPERIMENTS

• The experiments use 200 training epochs, with 60 epochs for teacher-to-student mask decay and 140 epochs of student finetuning.Testing covers Kodak, HEVC test sequences, and Tecnick; the main metric is BD-Rate for PSNR versus BPP.Source: §4 · p. 6

4.1 COMPARISON WITH STATE-OF-THE-ART

• EVC is on par with state-of-the-art neural codecs and outperforms VTM 17.2, while running at 30 FPS for 768 × 512 inputs; its small model reaches 30 FPS at 1080P.The comparisons cover Tecnick and Kodak; other SOTA methods train separate models for different rate-distortion trade-offs.Source: §4.1 · p. 6

4.2 MASK DECAY

• Mask decay improves medium and small students by about 50% and 30%, respectively, demonstrating the method’s effectiveness and students’ benefit from reusing the teacher’s parameters.The encoder is more redundant than the decoder: with the large decoder, compressing the encoder from Large to Small loses 2.7% BD-Rate on Kodak, while only compressing the decoder loses 5.1%.Source: §4.2 · p. 7

4.3 THE SCALABLE ENCODER

• 15.5% BD-Rate improvement over SlimCAE; Scalable-EVC is on par with other state-of-the-art models.SlimCAE uses a hyperprior, conditional convolutions, and an autoregressive context model.Source: §4.3 · p. 8

4.4 ABLATION STUDIES

• The proposed sparsity loss outperforms L1 across tested decay rates, while L2 hardly works because its gradient vanishes near zero masks.A decay rate of 4e −5 gives the best BD-Rate; 8e −5 decays too quickly, while 1e−5 and 2e−5 leave too little finetuning time.Source: §4.4 · p. 8
• Avoidable Mask Decay achieves comparable or slightly better performance than raw mask decay while avoiding selected channels.At most 1% of channels are shared between A1B and F in the first block; A2B avoids F channels in both blocks.Source: §4.4 · p. 9
• AMD supports the conclusion that avoiding selected channels can retain comparable performance, suggesting optimization matters more than channel-importance ranking.The authors report no important channels and recommend focusing future work on the optimization process rather than each channel’s importance.Source: §4.4 · p. 9

5 CONCLUSIONS AND FUTURE WORKS

• EVC runs at 30 FPS for 768 × 512 inputs, is on par with SOTA neural codecs and superior to H.266/VVC, and its small model reaches 30 FPS at 1080P.The conclusion also reports 50% and 30% improvements for medium and small models, respectively, and says scalable EVC outperforms SlimCAE.Source: §5 · p. 9
• Determining each layer’s complexity in pre-defined student structures and finding more powerful pruning algorithms remain open problems.The authors advise focusing on optimization rather than each filter’s importance criterion.Source: §5 · p. 9

A THE FRAMEWORK OF EVC

• The EVC framework comprises an encoder, decoder, and entropy model with a hyperprior.Figure 9 is identified as the overall framework illustration.Source: Figure 9 · p. 13, A THE FRAMEWORK OF EVC · p. 13
• The encoder and decoder use predefined block output dimensions, residual blocks, and depth-convolution blocks, with stride-2 downsampling and sub-pixel-convolution upsampling.The depthwise convolution is 3 × 3; other layers in the depth-conv block use 1 × 1 convolutions.Source: A THE FRAMEWORK OF EVC · p. 13
• The dual spatial prior splits Y into Y1 and Y2: Y1 is coded from hyperprior side information, then reconstructed Y1 informs estimation of Y2.The authors say this prior supports parallel computation and fast inference compared with an autoregressive prior.Source: A THE FRAMEWORK OF EVC · pp. 13–14
• The paper compares Large, Medium, and Small model schemes, including parameter counts and MACs at 1920 × 1088 input resolution.The architecture statistics are identified in Table 3.Source: A THE FRAMEWORK OF EVC · p. 14, Table 3 · p. 14

B MASK DECAY

• The acceleration pipeline trains a cumbersome teacher, inserts and sparsifies mask layers, then merges masks into neighboring layers to form an efficient student.Figure 12 illustrates the process.Source: B MASK DECAY · p. 14, Figure 12 · p. 14
• After training, zero mask entries permit channel pruning, while positive entries are folded into the next convolution; the mask can then be removed without changing outputs.The claim applies to the described neighboring convolution layers and their linearity.Source: B MASK DECAY · pp. 14–15
• For channels outside residual connections, masks are placed before the residual addition to avoid degrading identity output and preserve identity-gradient back-propagation.The alternative placement after addition can degrade the identity output because most masks lie in [0, 1].Source: B MASK DECAY · p. 15

C THE SCALABLE ENCODER

• End-to-end scalable-encoder training requires large GPU memory and performs badly because it optimizes the whole network without using the pretrained large model.The method is described as inefficient.Source: C THE SCALABLE ENCODER · pp. 15, 17
• Separately trained small encoders with a shared frozen decoder and entropy model generate similar bitstreams, so adding encoders improves rate-distortion performance only marginally.The ensemble selects a bitstream using the raw image to compute rate-distortion scores.Source: C THE SCALABLE ENCODER · p. 17
• Progressive one-by-one training adds small encoders that learn residual representations, outperforming separately trained encoders by a significant margin.The approach mitigates homogenization by encouraging diversity among the small encoders.Source: C THE SCALABLE ENCODER · p. 17
• Mask decay prunes the large teacher encoder by sparsifying its masks until it transforms into the student structure, improving the scalable encoder’s rate-distortion performance.Together with residual representation learning, it treats the large encoder as a reference and significantly narrows the performance gap from the teacher.Source: C THE SCALABLE ENCODER · p. 17

D DETAILS FOR EXPERIMENTS

• Training uses Vimeo-90K septuplets, randomly cropped to 256 × 256 patches, with all models trained for 200 epochs.Mask decay takes 60 epochs to transform the teacher into the student, followed by 140 epochs of student fine-tuning; the default decay rate is 4e −5.Source: D DETAILS FOR EXPERIMENTS · p. 18
• Compression quality is measured by PSNR BD-Rate against VTM 17.2 on Kodak, HEVC test sequences, and Tecnick.Negative BD-Rate indicates bitrate savings, while positive values indicate bitrate increases.Source: D DETAILS FOR EXPERIMENTS · p. 18
• Latency testing measures both encoding and decoding, including arithmetic coding but excluding disk I/O, on systems with a 2080Ti or A100 GPU.Kodak contains 24 images at 768 × 512; Tecnick contains 100 images at 1200×1200, and HEVC tests use each sequence’s first frame.Source: D DETAILS FOR EXPERIMENTS · p. 18

E IMPLEMENT DETAILS FOR OTHER IMAGE CODECS

• Entroformer’s parallel model is evaluated on Tecnick and Kodak, but generating bitstreams requires large GPU memory.The passage identifies parallel and nonparallel model series.Source: E IMPLEMENT DETAILS FOR OTHER IMAGE… · p. 18
• The paper relies on published results for STF and ELIC because their authors did not release code, and compares Cheng2020 and Xie2021 despite their autoregressive components.The passage notes that Cheng2020’s autoregressive model takes more than one second per image on CPU; Xie2021 is also described as having high latency from that component.Source: E IMPLEMENT DETAILS FOR OTHER IMAGE… · p. 19
• For SwinT-ChARM, the authors use published latency because code was unavailable, and compare it with EVC using 768 × 512 inputs on a 2080Ti.SwinT does not reconstruct images during encoding, whereas EVC does.Source: E IMPLEMENT DETAILS FOR OTHER IMAGE… · p. 19, Table 6 · p. 19
• Ballé2018 uses a hyperprior structure adopted by many later models, including EVC, and achieves high inference speed without a context module.The passage characterizes its rate-distortion performance as poor compared with recent state-of-the-art models.Source: E IMPLEMENT DETAILS FOR OTHER IMAGE… · p. 19

F MORE EXPERIMENTAL RESULTS

• The ELIC comparison uses published numbers because code and models were unavailable, and its latency tests differ in determinism and arithmetic-coding treatment from the main-paper tests.The authors report that non-determinism reduces latency by an average of 16% but can cause catastrophic decoding failures; their GPU also differs from the TITAN Xp used in the referenced comparison.Source: F MORE EXPERIMENTAL RESULTS · pp. 19–20, Table 7 · p. 20
• EVC-LL achieves better rate-distortion performance and inference latency than Ballé2018, Minnen2018, Cheng2020, Minnen2020, and Xie2021, and is comparable to ELIC-sm.EVC uses a single model for all rate-distortion trade-offs, whereas ELIC trains separate models for each trade-off.Source: F MORE EXPERIMENTAL RESULTS · p. 20, Table 7 · p. 20
• EVC models outperform previous models in speed measured in megapixels per second.Table 8 defines higher speed as better and identifies Entroformer and STF as state-of-the-art methods.Source: F MORE EXPERIMENTAL RESULTS · p. 20, Table 8 · p. 20
• Mask decay significantly improves baseline models’ rate-distortion curves on Kodak and outperforms baselines by significant margins on CLIC2021 and Tecnick.The reported Kodak curves include models accelerating only the encoder, only the decoder, or both.Source: F MORE EXPERIMENTAL RESULTS · pp. 20–21, Figure 17 · p. 21, Table 9 · p. 21

• EVC-SS and EVC-LL reconstructions are almost the same, with no extra artifacts attributed to mask decay; EVC models encode and decode faster than VTM.Source: G VISUALIZATIONS · p. 23
• Figure 20 tuples give BPP, PSNR, encoding time, and decoding time; latency was measured on a 2080Ti GPU, and EVC-SS uses small encoder and decoder.M and L denote medium and large models, respectively.Source: Figure 20 · p. 23