MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression

1. Introduction

• Existing global context modules can consume substantial GPU memory and slow encoding and decoding, limiting high-resolution image coding.Capturing local, global, and channel-wise contexts together with acceptable linear complexity remains a challenge.Source: §1 · p. 2
• MEM++ combines channel-wise contexts from previously compressed slices with local and global context modeling for each slice.The model also integrates hyper-prior side information.Source: §1 · pp. 2–3
• A shifted-window checkerboard attention module captures local spatial context while retaining two-pass decoding and linear complexity.It is proposed to address performance degradation associated with checkerboard context modeling.Source: §1 · p. 3
• The global context modules decompose softmax to obtain linear complexity and use correlations from previously decoded slices without directly computing attention maps.The inter-slice module uses global correlations from all preceding slices.Source: §1 · pp. 2–3
• MLIC++ achieves state-of-the-art performance with linear complexity and a better trade-off between performance and GPU memory consumption.The paper also reports improved visual quality in a CLIC Professional Valid reconstruction.Source: §1 · pp. 2–3

2.1. Learned Image Compression

• The learned compression framework uses an analysis transform, quantization, a synthesis transform, and an entropy model to estimate rates.The latent is quantized for entropy coding, then used to reconstruct the image; quantization is addressed during training with uniform noise or a straight-through estimator.Source: §2.1 · p. 3
• Hyper-prior models estimate latent distributions from side information, with later variants using more flexible distributions.Examples include mean-scale Gaussian, asymmetric Gaussian, Gaussian mixture, and Gaussian-Laplacian-Logistic mixture models.Source: §2.1 · p. 4

2.2. Context-based Entropy Modeling

• Context-based entropy models capture local, global, or channel-wise dependencies, but a single model that covers all three with linear complexity remains challenging.Local methods can decode serially or in two checkerboard passes; global methods may have quadratic complexity or require extra side information, while channel-wise methods condition slices on previously decoded slices.Source: §2.2 · p. 4

3.1. Motivation

• Exploiting correlations in the latent representation can reduce the conditional entropy and save bits.The passage states that conditional entropy is bounded by entropy.Source: §3.1 · p. 4
• Latent channels show significant similarity, but spatial context modules use the same mask across channels and may miss some channel correlations.Figure 2 visualizes channels from a latent representation extracted by Cheng’20.Source: §3.1 · p. 4, Figure 2 · p. 5
• Spatial similarities motivate both local context for neighboring symbols and global context for distant symbols with similar features.The section points to similarity between bottom-left and bottom-right grass regions and notes that global-context complexity should be minimized for high-resolution coding.Source: §3.1 · p. 4, Figure 3 · p. 5
• Existing entropy models have limited interactions across channels or within a slice, and extending them to high-resolution coding at linear complexity is non-trivial.The proposed multi-reference model is motivated by capturing local, global, and channel-domain correlations together.Source: §3.1 · pp. 4–5

3.2.1. MLIC++

• MLIC++ uses analysis and synthesis transforms, plus hyper-analysis and hyper-synthesis transforms simplified from Cheng’20 with attention modules removed.Source: §3.2.1 · p. 5

3.2.2. MEM++

• MEM++ combines channel-wise, local spatial, intra-slice global, and inter-slice global context modules.The global modules extract context from within the current slice and from preceding slices.Source: §3.2.2 · p. 6
• For each slice, previously decoded slices provide channel-wise context, while a checkerboard split separates anchor and non-anchor parts for local context modeling.Source: §3.2.2 · p. 6

• The latent representation is evenly divided into slices, and slice ˆyi is conditioned on the preceding slices ˆy<i.Source: §3.3 · p. 7
• A three-layer 3 × 3 convolutional module extracts context from earlier slices, including symbols at the same and nearby positions.The module helps select relevant channels and extract information for probability estimation.Source: §3.3 · p. 7
• The model uses 32 channels per slice and 10 slices, with latent residual prediction estimating quantization error from decoded slices and hyper-priors.Source: §3.3 · p. 7
• Although slices are processed serially, their fixed channel count and small number keep encoding and decoding speeds fast enough.Source: §3.3 · p. 7

• The module uses input-dependent attention to provide content-adaptive local context, unlike fixed-weight CNN-based local context modules.Source: §3.4 · p. 7
• Overlapping K × K windows are formed with stride 1, and checkerboard masking prevents interactions between anchor and non-anchor parts.A convolution fuses local context before it is passed to a feed-forward network.Source: §3.4 · p. 7
• The module has linear complexity, Ω(K^4SL), where L = HW and S is the channel count per slice.Source: §3.4 · p. 7

• During decoding, the current symbol is unknown, making its global correlations difficult to determine without adding bits or computing similarity against all symbols.Source: §3.5 · p. 8
• Learned query, key, and value embeddings provide an adaptable attention-based alternative to fixed cosine similarity for estimating global correlations.The latent slices exhibit similar global correlations despite differences in magnitude.Source: §3.5 · p. 8
• The module uses decoded anchor symbols from the preceding slice as an approximation of global similarities, then refines neighboring similarities with a K × K convolution.Source: §3.5 · p. 8
• Decomposing softmax operations permits the attention calculation to be reordered for O(L) complexity, which scales linearly with resolution.Source: §3.5 · p. 9

3.6. Linear Complexity Inter-Slice Global Context Module

• For the current slice, symbols at the same positions in previous slices approximate its symbols, providing additional context to anchor and non-anchor parts.Attention measures similarity, and the softmax is split into two independent operations as in the intra-slice module.Source: §3.6 · p. 10
• The prior-slice representation is embedded with 1 × 1 and 3 × 3 depth-wise convolutions, the latter providing learnable position embedding.DepthRB enhances non-linearity in the module.Source: §3.6 · p. 10
• Computing the embedding during training and testing makes similarity implicit and keeps overall complexity O(L), linear with resolution.Source: §3.6 · p. 10

4.1.1. TRAINING DATASET

• 105 images above 512 × 512 from ImageNet, COCO 2017, DIV2K, and Flickr2K form the training dataset; JPEG images are further down-sampled to reduce compression artifacts.Source: §4.1.1 · p. 10

4.1.2. TRAINING STRATEGY

• Training uses 256 × 256 crops for the first 1.2M steps, then 512 × 512 crops for the remaining 0.8M steps to exploit global context.Models train for 2M steps with batch size 32 on a Tesla A100 GPU.Source: §4.1.2 · p. 10

4.2. Benchmarks and Metrics

• Evaluations use Kodak, Tecnick, and CLIC Professional Valid, with BD-Rate as the performance metric.Figure 10 presents PSNR–bit-rate and MS-SSIM–bit-rate curves.Source: §4.2 · pp. 10–11, Figure 10 · p. 11

4.3.1. QUANTITIVE RESULTS

• 13.39% BD-rate reduction over VTM-17.0 Intra on Kodak accompanies reductions of 17.59% on Tecnick and 13.08% on CLIC Professional Valid; MLIC++ is state of the art on all three.The table compares BD-rate for PSNR and MS-SSIM with VTM-17.0 Intra as anchor.Source: §4.3.1 · pp. 11–12, Table 3 · p. 12

4.3.2. QUALITIVE RESULTS

• On the Kodim07 reconstruction, MLIC++ is 1 dB higher in PSNR or 0.02 higher in MS-SSIM than VTM-17.0 Intra, while its MSE-optimized bpp is 84.4%–93.8% of competing models.The cited reconstruction comparison includes MLIC++, learned compression models, and VTM-17.0 Intra.Source: §4.3.2 · p. 12, Figure 11 · p. 13

4.4.1. ON GPU MEMORY COMSUPTION

• At 2048 × 2048, MLIC++ uses about 5.55 GB GPU memory versus MLIC+'s 22.37 GB; at 3584 × 3584 it uses 12.3 GB versus LIC-TCM's 45.95 GB.The study compares memory, encoding and decoding times, and forward MACs over multiple input resolutions.Source: §4.4.1 · p. 13, §4.4 · p. 13, Figure 12 · p. 14

4.4.2. ON ENCODING AND DECODING TIME

• At 2560 × 2560, MLIC++ encoding takes about half the time of MLIC+, and it is faster than MLIC+ for encoding and decoding across the listed resolutions.Against LIC-TCM, MLIC++ takes longer to decode low-resolution images but less time to encode high-resolution images.Source: §4.4.2 · p. 14, Figure 12 · p. 14

4.4.3. ON FORWARD MACS

• MLIC++ has lower MACs than MLIC+ and LIC-TCM, but higher MACs than ELIC, STF, and WACNN on high-resolution inputs, partly because it uses Cheng’20 transforms.The entropy model's MACs have modest effect on total MACs because the analysis transform downsamples the input four times.Source: §4.4.3 · p. 15

4.5.1. SETTINGS

• Ablations evaluate module contributions on Kodak using MSE-optimized models trained for 2M steps.Source: §4.5.1 · p. 15

4.5.2. ANALYSIS OF CHANNEL-WISE CONTEXT MODULE

• Adding channel-wise context reduces bit-rate by 4.89% versus the hyper-prior-only case, supporting redundancy across channels.The module references symbols at the same and nearby positions in preceding slices.Source: §4.5.2 · p. 15

4.5.3. ANALYSIS OF LOCAL SPATIAL CONTEXT MODULE

• The proposed checkerboard attention-based local module reduces bit-rate by 2.32% versus vanilla checkerboard context and retains two-pass decoding.It uses nonlinear dynamic attention, unlike the vanilla module's single linear convolution with fixed inference weights.Source: §4.5.3 · pp. 15–16

4.5.4. ANALYSIS OF INTRA-SLICE GLOBAL CONTEXT MODULE

• The intra-slice module captures distant anchor-to-non-anchor correlations that local modules cannot, and combining local and global contexts further improves performance.Its attention focuses on interactions within one slice, using the attention map of the previous slice to predict current-slice correlations.Source: §4.5.4 · p. 16

4.5.5. ANALYSIS OF INTER-SLICE GLOBAL CONTEXT MODULE

• The inter-slice module's visualized attention maps capture global dependencies, and adding it to the other context modules further enhances rate-distortion performance.The prose describes the rate-distortion comparison in Figure 15.Source: §4.5.5 · p. 16, Figure 15 · p. 16

4.5.6. ANALYSIS ON LEARNABLE POSITION EMBEDDING AND DEPTHRB

• Learnable position embedding and DepthRB contribute performance gains; sinusoidal position embedding yields negligible difference from omitting position embedding.Position embedding is not used in the local checkerboard module because its improvement is negligible and window boundaries already provide positional cues.Source: §4.5.6 · pp. 16–17, Table 6 · p. 17

4.5.7. COMPARISONS WITH MODEL WITH VANILLA GLOBAL SPATIAL CONTEXT MODULES

• Linear-complexity global attention does not degrade rate-distortion performance relative to vanilla quadratic-complexity attention.Figure 15 compares MLIC++ with its vanilla-attention variant.Source: Figure 15 · p. 16, §4.5.7 · p. 17

4.5.8. ANALYSIS OF TRAINING WITH LARGE PATCHES

• Using 512 × 512 rather than smaller patches during the final 0.8M training steps improves performance; the authors select this size considering overhead and performance.They attribute the benefit to better learning of long-range dependencies from larger latent representations, especially with sparse checkerboard attention.Source: Table 5 · p. 16, §4.5.8 · p. 17

5. Conclusion

• MEM++ captures local spatial context with checkerboard attention and global context with linear-complexity intra-slice and inter-slice modules.Source: §5 · p. 17
• MLIC++ achieves state-of-the-art performance and exhibits linear GPU memory consumption with resolution, suiting high-resolution image coding.Source: §5 · p. 17
• Future work will investigate asymmetrical analysis and synthesis transforms and a lighter linear-complexity multi-reference entropy model.Source: §5 · p. 17