Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation

• Human-vision codecs can discard semantic information needed by downstream machine tasks, motivating unified compression for both machine and human vision.The introduction notes that images are compressed and transmitted for use with cloud recognition models.Source: §1 · pp. 1–2
• Prior lightweight tuning methods preserve the base codec but underperform full fine-tuning and add computational and model complexity through task-specific designs.Earlier multi-task pipelines [17] also degrade human-vision rate-distortion performance and are restricted in adapting to new machine tasks.Source: §1 · p. 2
• Because base codecs use spatial and frequency components to improve human visual quality, Adapt-ICMH modulates intermediate features to reduce latent redundancy and adapt reconstructions to machine tasks.The stated goal is to balance machine-task performance and transmission cost across diverse tasks.Source: §1 · pp. 2–3
• SFMA combines spatial modulation to eliminate non-semantic redundancy with frequency modulation to amplify task-relevant and suppress redundant frequency components.For example, the introduction notes that high-frequency components can be more crucial for segmentation.Source: §1 · p. 3
• Adapt-ICMH plugs SFMA into the encoder and decoder of a shared base codec, reducing training and storage overhead while supporting almost all learned image compression models.The framework is described as architecture-agnostic and plug-and-play.Source: §1 · p. 3
• Experiments show Adapt-ICMH consistently outperforms other ICMH methods across machine-vision tasks with reduced model and computational complexity.The introduction reports this overall result without giving quantitative values.Source: §1 · p. 3

2 Related Work

• LIC improves nonlinear transforms and entropy models but primarily targets decoded-image human quality rather than downstream machine vision.Transform research progressed from CNNs to attention and transformer-based methods; entropy models evolved from factorized and hyperprior models toward spatial or channel-wise autoregression.Source: §2 · p. 3
• ICMH addresses tasks including classification, detection, segmentation, and pose estimation, but full task-specific codec fine-tuning requires storing and deploying a separate codec per task.This storage burden hampers practical applicability.Source: §2 · pp. 3–4
• Earlier multi-task pipelines [17] require task-specific decoders and scratch training, while prompt tuning lacks CNN compatibility and remains inferior to full fine-tuning.The spatial-channel mask approach does not easily extend to autoregressive entropy models and does not adapt nonlinear transforms.Source: §2 · p. 4
• The proposed framework generalizes across LIC models with different transforms and entropy models and achieves rate-accuracy performance comparable to full fine-tuning.It uses a pre-trained human-vision codec, unlike the cited prior approaches’ task-specific or limited adaptation strategies.Source: §2 · p. 4
• PEFT spans prompt tuning, adapters, and LoRA, but naive adapters ignore transmission costs across spatial components and image frequencies.The paper proposes a spatial-frequency modulation adapter for ICMH to address this limitation; adapters previously used for content-adaptive human-quality optimization serve a different methodology and application scenario.Source: §2 · p. 4

3.1 Empirical Findings by Full Fine-tuning

• Fine-tuned codecs prioritize semantic objects over complex backgrounds, while their frequency preferences vary by task: segmentation favors mid- and high-frequency components, whereas classification can use low and mid frequencies.The human-vision base codec instead encodes more background detail and a wider range of frequencies.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Framework Overview

• Adapt-ICMH plugs spatial-frequency modulation adapters into a pretrained codec’s encoder and decoder, freezes the base weights, and trains only the adapters using a rate-plus-task-specific perceptual-distortion objective.The distortion is measured with an off-the-shelf recognition model, and λ balances task performance against bitrate.Source: §3.2 · pp. 6–7, Figure 3 · p. 6
• After adaptation, the same base codec can serve human vision unchanged by removing the adapters, or support multiple machine-vision tasks by changing only lightweight adapter weights.This preserves the base codec’s human-vision rate-distortion performance.Source: §3.2 · p. 7

3.3 Spatial-Frequency Modulation Adapter

• Each SFMA adds parallel frequency and spatial modulation: FMA reduces frequency redundancy and amplifies task-important components, while SMA focuses features on semantic regions and reduces background redundancy.The adapted feature combines the original feature with both adapter outputs.Source: §3.3 · pp. 7–8

4.1 Training Details and Datasets.

• Training freezes pre-trained mbt2018-mean, cheng2020-anchor, or Lu2022-TIC codecs and tunes SFMA for classification, object detection, and instance segmentation on ImageNet-train or COCO2017-train.Classification uses 8 epochs and batch size 16; detection and segmentation use 40 epochs and batch size 8, with training images cropped and resized to 256 × 256.Source: §4.1 · p. 8

4.2 Evaluation

• Evaluation uses BPP for bitrate, top-1 accuracy on ImageNet-val for classification, and mAP at IoU 0.5 on COCO2017-val for detection and segmentation.Evaluation uses center-cropped 256 × 256 images for classification and images resized to 800 pixels on the short side for the COCO tasks.Source: §4.2 · pp. 8–9

4.3 Rate-Accuracy Comparison

• Adapt-ICMH outperforms across three machine-vision tasks, generalizes across three codecs, and matches or exceeds full fine-tuning with about 1–5% of base-codec parameters.Full fine-tuning requires a separate full-codec copy per task, increasing training and storage overhead.Source: §4.3 · p. 9, Figure 4 · p. 10, Table 1 · p. 11

4.4 Ablation study

• Ablations show that removing either SFMA branch degrades performance, FMA matters more than SMA, parallel branches outperform sequential ones, and duplicating SMA adds parameters while reducing performance.These comparisons support the proposed structure rather than model capacity alone as the source of improvement.Source: §4.4 · pp. 10–11, Figure 5 · p. 11, Table 2 · p. 12
• 69.54% bitrate savings for classification comes from placing SFMA after encoder and decoder stage1, versus 64.76% after stage3; adding SFMAs helps further.Both encoder- and decoder-side SFMAs are important: the encoder reduces task-feature redundancy, while the decoder adapts features for downstream tasks; increasing middle dimension improves performance but increases training complexity, motivating ˆC = 64.Source: §4.4 · pp. 11–12, Table 3 · p. 12, Table 4 · p. 12

4.5 Qualitative Results

• Adapt-ICMH produces superior instance-segmentation results at 0.147bpp, reducing background bits and enhancing high-frequency semantic-object edges despite low-frequency chromaticity and brightness distortions.The reported low-frequency distortions do not affect the segmentation result.Source: §4.5 · pp. 12–13, Figure 6 · p. 13

4.6 Towards scalable coding for machine and human vision.

• In scalable coding, adding SFMA to ICMH-Net significantly improves rate-accuracy performance while leaving full-bitstream human-vision reconstruction quality unchanged from the base codec.The setup uses the spatial-channel mask generator to create truncated and full bitstreams; SFMA is used for decoding the truncated stream for machine vision.Source: §4.6 · pp. 13–14, Figure 7 · p. 14

5 Conclusion

• Adapt-ICMH consistently outperforms other ICMH methods across machine-vision tasks with fewer trainable parameters and lower computational complexity.It fine-tunes a pretrained image codec optimized for human vision and is compatible with existing learned image compression models.Source: §5 · p. 14
• SFMA reduces latent redundancies in spatial and frequency domains and adapts decoded images for downstream machine-vision tasks.The conclusion describes the framework as plug-and-play and compatible with existing learned image compression models.Source: §5 · p. 14

A Preliminary of Learned Image Compression

• Mean squared error suits human-vision reconstruction, but the paper uses perceptual distortion loss to optimize adapters for machine-vision task performance.The passage identifies mean squared error as unsuitable when the reconstructed image is intended for machine vision.Source: A Preliminary of Learned Image… · p. 15

B More Discussion about ICMH

• Unlike the multi-task pipeline, which trains a codec from scratch and degrades human-vision PSNR by about 1dB, Adapt-ICMH fine-tunes pretrained codecs without compromising their R-D performance.The cited pipeline also incurs substantial training overhead and is restricted in adapting to newly arriving machine-vision tasks.Source: B More Discussion about ICMH · p. 16

Scalable coding framework

• Unlike ICMH-Net, which filters latent features with a binary mask and adapts only the deepest latent, SFMA uses spatial-frequency modulation and updates shallow latent.The passage describes ICMH-Net’s Gumbel-Softmax mask as suboptimal and difficult to optimize stably, with unsatisfactory machine-vision performance.Source: Scalable coding framework · p. 16
• SFMA can integrate with scalable-coding frameworks such as ICMH-Net to significantly improve rate-accuracy performance.The authors note that their original framework was not specifically designed for scalable coding.Source: Scalable coding framework · p. 16

Single bitstream framework.

• For single-task bitstreams, SFMA is lightweight and compatible with almost all learned image compression models, unlike approaches requiring encoder-side pre-analysis models, task-specific decoders, or added computation.The passage contrasts ROI coding, channel selection, and visual prompt tuning, noting the last is incompatible with CNN-based codecs.Source: Single bitstream framework · p. 16
• For multiple tasks in a single bitstream, prior methods learn omnipotent features or jointly optimize latents for human perception and classification, but finding an optimal task trade-off remains challenging.The passage notes that these approaches use one bitstream for multiple tasks.Source: Single bitstream framework · p. 16

Task-specific Perceptual Distortion Loss.

• The task-specific perceptual distortion loss enables SFMA training for machine vision tasks without accessing task-related labels.The loss is used to optimize the SFMAs.Source: Task-specific Perceptual Distortion Loss · p. 17
• For classification, object detection, and instance segmentation, the method uses pre-trained ResNet50, Faster RCNN, and Mask RCNN recognition models, respectively.These are used as off-the-shelf models.Source: Task-specific Perceptual Distortion Loss · p. 17
• ResNet50-FPN is the backbone network used by Faster RCNN and Mask RCNN.Figure 8 shows the ResNet50-FPN network architecture.Source: Task-specific Perceptual Distortion Loss · p. 17, Figure 8 · p. 17
• Classification perceptual loss averages MSE between original and reconstructed image features F1–F4 from ResNet50.The loss is one-quarter of the sum of the four feature-level MSE terms.Source: Task-specific Perceptual Distortion Loss · p. 17
• For object detection and instance segmentation, perceptual loss uses FPN features P2–P6 from the original and reconstructed images.The supplied equation defines this as the mean of five feature-level MSE terms.Source: Task-specific Perceptual Distortion Loss · p. 17, Hyperparamters of Training · p. 17

Hyperparamters of Training

• Table 5 details the training hyperparameters for the main-paper experiments across machine tasks.Experiments were conducted using an NVIDIA GeForce RTX 4090 and Intel Xeon Platinum 8260.Source: Hyperparamters of Training · p. 17, Table 5 · p. 18

D Details of Reproduction for Other Methods

• [47] and used different pre-trained recognition models, so their results could not be directly compared with Adapt-ICMH.The paper compares Adapt-ICMH with tuning-based methods.Source: D Details of Reproduction for Other… · p. 19
• TransTIC results were taken from the results published in its paper.Source: D Details of Reproduction for Other… · p. 19
• Methods and were reproduced using the same training, evaluation, and dataset settings, with hard Gumbel-Softmax temperature set to 1; Fig. 9 sketches their architectures.Source: D Details of Reproduction for Other… · p. 19, Figure 9 · p. 19

E Details of Architecture for Different Base Codecs

• Across base codecs with different nonlinear transforms and entropy models, SFMA achieves superior rate-accuracy performance without affecting base-codec visual quality.Source: E Details of Architecture for… · p. 19, Figure 10 · p. 20
• The paper provides implementation and pretrained-weight source paths for the base codecs in Table 6.Source: E Details of Architecture for… · p. 19, Table 6 · p. 21

F Details of Scalable Coding Pipeline

• The scalable framework injects SFMAs only in the base codec’s decoder and uses the mask generator to select machine-vision latents for transmission.Source: F Details of Scalable Coding Pipeline · p. 19, Figure 11 · p. 21
• The binary spatial-channel mask derived from entropy parameters selects quantized latents for the base layer, with the remaining latents assigned to the enhancement layer.Source: F Details of Scalable Coding Pipeline · p. 21
• The base layer is decoded with SFMAs to reconstruct an image for machine vision, while the full latent is decoded without SFMAs to reconstruct an image for human vision.Source: F Details of Scalable Coding Pipeline · p. 21
• SFMA significantly benefits ICMH-Net on classification and instance-segmentation tasks in scalable coding.Source: F Details of Scalable Coding Pipeline · p. 21

G Analysis on the Computational Complexity

• Channel selection reduces decoder complexity via a lightweight task-specific decoder, but loses pre-trained synthesis-transform knowledge and degrades rate-accuracy on complex detection and segmentation tasks.The method uses a separate lightweight decoder for each machine vision task.Source: G Analysis on the Computational… · p. 22
• Adapt-ICMH adds only small computational complexity, with latency increases described as ignorable; full fine-tuning avoids added computation but requires storing and deploying an entire codec copy.The comparison evaluates complexity on ImageNet-val [18] classification, excludes the off-the-shelf recognition model, and uses Lu2022-TIC [50] as the BD-acc anchor.Source: G Analysis on the Computational… · p. 21, Table 7 · p. 22

H Application on Larger Transformer-based Image Codecs

• Adapt-ICMH outperforms other ICMH frameworks on larger transformer-based codecs while using fewer than 1% of the base codec’s trainable parameters.The comparison reports classification results at one rate-accuracy point.Source: H Application on Larger… · p. 23, Table 8 · p. 23
• The larger-codec comparison reports ImageNet-val top-1 accuracy, and ICMH-Net cannot support the channel-wise autoregressive entropy models used by the compared codecs.The codecs are identified as STF, TCM, and FAT in the prose; the table caption names STF, TCM, and FLIC.Source: H Application on Larger… · p. 23, Table 8 · p. 23

I.1 Plugging SFMAs into Entropy Model

• Plugging SFMAs into the hyper encoder and hyper decoder entropy model does not significantly improve performance and adds model complexity.The authors therefore plug SFMAs only into the nonlinear transform.Source: §I.1 · pp. 23–24, Table 9 · p. 24
• The ablation attributes the key difference between human- and machine-vision-oriented compression to the nonlinear transform rather than the entropy model.This conclusion follows the finding that entropy-model adapters add complexity without significant performance improvement.Source: §I.1 · p. 24

I.2 Compared with Naive Adapter

• The naive adapter uses two linear layers with an intervening ReLU, limiting its ability to achieve SFMA’s spatial and frequency modulation.The comparison replaces SFMA with the naive adapter for feature adaptation.Source: §I.2 · p. 24
• Under similar trainable-parameter counts, the naive adapter is inferior to the proposed SFA and FMA.The paper presents this ablation as evidence of SFMA’s effectiveness.Source: §I.2 · p. 24, Table 10 · p. 24

J More Qualitative Results

• Qualitative detection and segmentation results show Adapt-ICMH reduces latent redundancy, achieving better task performance at lower bitrates.Source: J More Qualitative Results · p. 24, Figure 13 · p. 25, Figure 14 · p. 26
• The detection comparison includes decoded images, detection results, log power spectral density maps, and bit allocation maps.Source: Figure 13 · p. 25
• The instance-segmentation comparison includes decoded images, segmentation results, log power spectral density maps, and bit allocation maps.Source: Figure 14 · p. 26

K Pytorch Implementation of SFMA

• The frequency path applies a 3×3 depthwise convolution and computes a 2D real FFT with backward normalization.The code applies rfft2 to f_down(x), using dimensions (2, 3) and norm='backward'.Source: K Pytorch Implementation of SFMA · p. 27
• The implementation labels x as an intermediate feature and x_tilde as an adapted feature.Source: K Pytorch Implementation of SFMA · p. 27
• The spatial modulation path applies two down-projections, a depthwise 5×5 convolution, multiplication, ReLU, and an up-projection.The implementation expresses this as s_up(s_relu(s_dw(s_down1(x)) * s_down2(x))).Source: K Pytorch Implementation of SFMA · p. 27

L Limitation and Future Work

• Adapt-ICMH cannot directly achieve scalable coding for machine and human vision.The paper says SFMA can be incorporated into existing scalable-coding ICMH frameworks and boost their performance.Source: L Limitation and Future Work · p. 28
• Future work will extend the method to pose estimation [39] and person re-identification.Source: L Limitation and Future Work · p. 28
• The authors aim to extend SFMA beyond image compression to video coding for machines.They identify VCM as a current topic of interest and propose this extension as future work.Source: L Limitation and Future Work · p. 28