PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression

Haobo Xiong, Shaobo Liu, Kai Liu, Chongyang Ding

arXiv:2608.25568v1cs.CVcs.AI

TL;DR

Pretrained LIC adaptation needs to reduce deployment and retraining costs while addressing limited cross-layer coordination in existing PEFT methods. CrossMambaTuning combines a task-aware Mamba adapter with SICA for spatial modeling and cross-layer fusion, and reports SOTA performance across multiple machine vision tasks with 72% lower parameter overhead than a SOTA method.

  • Problem

    Existing PEFT methods for pretrained LIC models often lack explicit cross-layer coordination, while practical adaptation must reduce training, storage, and deployment overheads.

  • Method

    CrossMambaTuning integrates a task-aware Mamba adapter for local and long-range spatial modeling with a parameter-sharing SICA for cross-scale task-information fusion.

  • Results

    72% lower parameter overhead is reported for the Tiny variant versus the SOTA method [28], using 0.08M parameters while achieving comparable or superior performance.

  • Takeaways & Limitations

    The framework provides a unified, parameter-efficient adaptation approach that achieves SOTA results across classification, object detection, and instance segmentation.

  • Takeaways & Limitations

    The study focuses on high-level downstream tasks, while low-level vision applicability remains outside scope; runtime may also require simplification for extremely high-resolution or strict real-time settings.

Abstract

from arXiv · show

To reduce deployment cost and retraining overhead, adapting pretrained learned image compression (LIC) models to downstream machine vision tasks has attracted growing attention. However, existing methods typically insert fine-tuning modules independently into frozen backbones, lacking explicit mechanisms for cross-layer coordination. To address this limitation, we propose a novel framework named CrossMambaTuning, which integrates State Space Models with cross-layer interaction mechanisms for parameter-efficient fine-tuning. Specifically, we design an efficient Mamba adapter equipped with task-specific prompts and multi-scale branching to precisely capture both local features and global dependencies. Furthermore, we introduce a Scale-Invariant Cross-Layer Adapter (SICA) utilizing a parameter-sharing strategy to fuse task information across different scales and reduce redundancy. Extensive experiments demonstrate that CrossMambaTuning achieves state-of-the-art (SOTA) performance on multiple machine vision tasks, reducing parameter overhead by 72\% compared to SOTA methods. Code is available at https://github.com/rsr1123/CrossMambaTuning.

1 Introduction

CrossMambaTuning targets the deployment and retraining costs of adapting pretrained LIC models while addressing the lack of explicit cross-layer coordination in existing PEFT methods. It combines task-aware spatial modeling with scale-invariant cross-layer fusion, achieving strong performance with substantially fewer parameters.

  • Motivation: Pretrained LIC adaptation reduces the need to maintain separate task-specific codecs, addressing training, storage, and deployment overheads.Earlier unified approaches often trained multi-task networks from scratch, incurring substantial training and storage costs.
  • Limitations of Prior Work: Existing PEFT methods attach lightweight modules to pretrained LIC models but may impose rigid architectural constraints or operate independently across layers.Mask- and prompting-based approaches can constrain encoder structure, while LoRA- and adapter-based methods lack explicit cross-layer coordination.
  • Proposed Framework: CrossMambaTuning combines a task-specific Mamba adapter with multi-scale branching and selective scanning to model local features and long-range spatial dependencies.The framework is designed to capture task-relevant priors while adapting pretrained codecs efficiently.
  • Proposed Framework: SICA uses parameter sharing to fuse task information across different feature scales, addressing layer isolation while reducing compression redundancy.Its scale-agnostic cross-layer mechanism improves parameter utilization efficiency.
  • Results: 72% lower parameter overhead is achieved by the Tiny variant versus the SOTA method [28], using 0.08M parameters with comparable or superior performance.The result demonstrates the framework’s parameter efficiency relative to the cited state-of-the-art baseline.
  • Results: CrossMambaTuning integrates its Mamba adapter and cross-layer fusion mechanism and achieves SOTA performance across multiple machine vision tasks and network configurations.The reported scope includes diverse task settings and network configurations.

2 Related Work

Related work spans LIC architectures, unified machine-vision compression, and lightweight adaptation using State Space Models. CrossMambaTuning positions itself as an architecture-flexible PEFT framework for multiple machine vision tasks and codec families.

  • Learned Image Compression: LIC models are typically evaluated with PSNR and MS-SSIM and use transform and entropy modules to compress images.Prior transform designs include convolutions with GDN, residual learning, attention, and more recent Transformer-based architectures.
  • Machine Vision Compression: Human-perception-oriented LIC representations are not always aligned with downstream machine vision requirements.Unified and multi-task frameworks address diverse usage scenarios but can incur substantial training and storage overheads.
  • State Space Model Adaptation: State Space Models offer global modeling capacity competitive with Transformers while maintaining linear computational complexity.Their long-range modeling and computational properties have motivated their use as lightweight adaptation modules.
  • State Space Model Adaptation: Prior Mamba-based adaptation studies apply the model to settings including medical segmentation, point-cloud understanding, and vision-language fusion.These examples use Mamba for lightweight adaptation, cross-layer semantic fusion, or cross-model bridging in specialized scenarios.
  • Positioning of CrossMambaTuning: CrossMambaTuning is designed for pretrained LIC adaptation across classification, object detection, and instance segmentation.The framework is applicable to both CNN-based and Transformer-based architectures.

3 Method

CrossMambaTuning addresses long-range dependency and cross-layer coordination limitations in PEFT for LIC by combining a task-aware Mamba adapter with SICA. Its modules use multi-scale spatial processing, task prompts, cross-scale feature alignment, and shared parameters.

  • Motivation: Existing PEFT methods may miss long-range spatial dependencies and lack explicit cross-layer coordination, causing overlapping patterns across adapters.
  • Task-aware Mamba Adapter: The task-aware Mamba adapter is inserted after encoder and decoder stages except the last, using SSMs for efficient long-range dependency modeling.
  • Scale-Invariant Cross-Layer Adapter: SICA establishes adjacent-layer information pathways, aligns features across scales, and shares encoder or decoder parameters across stages for scale-independent semantic representations.
  • Task-aware Mamba Adapter: APC rectifies input features before the residual adapter, while the pipeline encodes an image into latent y and decodes it into a task-optimized reconstruction.
  • Task-aware Mamba Adapter: The Local Information Extractor uses 1×1, 3×3, and 5×5 depth-wise convolutions to fuse pixel-level, neighborhood, texture, and edge information.
  • Task-aware Mamba Adapter: TSPG combines shared task and layer-specific embeddings to initialize Mamba states and link distributed adapters through shared task information.

4 Experiments

Experiments evaluate CrossMambaTuning across classification, detection, and segmentation using Transformer- and CNN-based codecs. The method consistently delivers strong rate-accuracy performance while reducing trainable parameters, with ablations supporting its spatial, cross-layer, and prompt-based components.

  • Experimental setup: Experiments cover image classification, object detection, and instance segmentation on ImageNet and COCO2017 using TIC and ELIC base codecs.Evaluation reports bpp, Top-1 accuracy, mAP, BD-rate, and BD-acc/mAP under frozen-backbone adaptation.
  • Experimental results: CrossMambaTuning consistently outperforms competing methods on TIC rate-accuracy curves across the evaluated machine vision tasks.The comparison includes recent methods such as Adapt-ICMH and SVD-LoRA.
  • Experimental results: 51.7% of Adapt-ICMH’s trainable parameters are used by the Small model while surpassing that state-of-the-art method on classification.This result is reported for the TIC codec.
  • Experimental results: 27% of Adapt-ICMH’s parameters are used by the Tiny variant, which exceeds SVD-LoRA by 1.328 ∼1.533 mAP at equivalent bitrates on detection and segmentation.On ELIC, Tiny improves object-detection mAP by 0.484 at the same bitrate while reducing trainable parameters by 73%.
  • Ablation studies: Ablations attribute improvements to APC, multi-scale LIE, SICA, and task-specific TSPG, with SICA reducing BD-rate and cross-layer interaction reducing redundancy.The full combination achieves the best ablation performance.
  • Ablation studies: Spatial correlation decreases from 0.18 for the previous SOTA method to 0.05 for CrossMambaTuning, approaching 0.01 for full fine-tuning.The increased correlation without SICA further supports the adapter’s role in reducing redundancy.
  • Efficiency and reconstruction: The Tiny variant requires 0.08M trainable parameters compared with 7.51M for full fine-tuning, while remaining competitive in computational cost and latency.Removing adapters exactly restores the pretrained codec’s reconstruction RD performance.
  • Qualitative results: Qualitative comparisons show better preservation of fine-grained and task-relevant features, including detecting an apple and segmenting a glass missed by Adapt-ICMH.These examples are reported at lower bitrates.

5 Conclusion

The paper presents CrossMambaTuning for parameter-efficient adaptation of pretrained image compression codecs to machine vision tasks. Its Task-aware Mamba Adapter and Scale-Invariant Cross-Layer Adapter support strong performance across tasks and architectures with fewer trainable parameters.

  • Conclusion: CrossMambaTuning adapts pretrained image compression codecs to image classification, object detection, and instance segmentation.The framework is designed as a parameter-efficient fine-tuning approach.
  • Conclusion: The framework combines a Task-aware Mamba Adapter with a Scale-Invariant Cross-Layer Adapter for efficient codec adaptation.The conclusion identifies these as the proposed components.
  • Conclusion: Small and Tiny variants surpass current state-of-the-art methods while requiring substantially fewer trainable parameters.Additional experiments across diverse architectures support the reported robustness.

B Reconstruction Quality Under Different Adapter Settings

Adapter attachment prioritizes machine-vision objectives over explicit reconstruction fidelity, so reconstruction quality generally declines; deployment can retain both objectives as selectable codec modes.

  • PSNR and MS-SSIM are not explicitly preserved when adapters are trained with task-oriented feature loss instead of MSE reconstruction loss.The reported reconstruction metrics serve as references rather than a full rate–distortion comparison.
  • PSNR decreases from 28.92 dB to 23.90 dB, while MS-SSIM decreases from 0.971 to 0.923 with adapters attached.Comparable degradation is reported for other machine-vision compression frameworks.
  • The pretrained codec path supports human-vision compression, whereas the adapter-based path supports machine-vision compression within the same LIC system.Switching modes requires enabling or disabling a small set of task-specific parameters.

C Additional Complexity Comparison with PEFT Baselines

CrossMambaTuning is compared with full fine-tuning and representative PEFT baselines across trainable parameters, computation, latency, and downstream performance, with parameter efficiency identified as its main advantage.

  • Table 11 compares CrossMambaTuning with full fine-tuning and PEFT baselines on trainable parameters, computational cost, runtime latency, and downstream performance.Baselines follow official implementations when available and share the same hardware evaluation environment.
  • The framework’s main advantage is parameter efficiency rather than lower runtime cost.

D Task-Specific Perceptual Loss

Task-specific perceptual loss measures discrepancies between original and reconstructed images in pretrained downstream task-model feature spaces, using task-dependent feature layers.

  • Task-specific perceptual loss trains downstream adaptation using feature-based distortion rather than only reconstruction fidelity.Features are extracted from the original and reconstructed images with pretrained downstream models, and their MSE is computed.
  • The perceptual-loss feature layers are visually illustrated in Figure 7 using the ResNet50-FPN architecture.
  • Loss for Classification: For classification, the loss averages MSE across ResNet-50 feature layers F1, F2, F3, and F4.The classification feature extractor uses ResNet-50.
  • Loss for Detection and Segmentation: For object detection and instance segmentation, the loss uses FPN feature levels P2, P3, P4, P5, and P6 from Faster R-CNN and Mask R-CNN.Faster R-CNN supports detection, while Mask R-CNN supports instance segmentation.

E.2 Metric of Spatial Correlation

Spatial correlation quantifies linear dependence between normalized latent features at spatial offsets, averaged across channels; lower values indicate less spatial redundancy and are associated with improved rate–distortion performance.

  • The Pearson correlation coefficient measures linear correlation between variables and is invariant to scale.
  • Spatial correlation ρ(i, j) averages Pearson correlations between normalized latent features at positions (w,h) and (w+i,h+j) across channels.
  • Lower spatial correlation indicates reduced redundancy across latent spatial positions and is generally associated with improved rate–distortion performance.
  • The proposed SICA module and Mamba-Adapter significantly reduce ρ, indicating reduced model redundancy.
  • Wasserstein distance and KL divergence quantify discrepancies between feature distributions, providing complementary distribution-level measurements.Wasserstein distance measures transport cost, while KL divergence measures additional expected information cost from approximation.

H More Implementation Details

CrossMambaTuning combines task-aware Mamba adaptation with cross-layer interaction modules, using configuration choices that balance stability, capability, and parameter efficiency across model variants.

  • The Base model integrates APC, LIE, SICA, and TSPG into a Task-Aware Mamba-Adapter and Cross-Layer Interaction framework.These modules jointly provide the stated task-aware adaptation and cross-layer fusion configuration.
  • SS2D supplies selective scanning, while APC, LIE, and TSPG target distribution alignment, local dependency modeling, and task-aware conditioning.The scanning process is adopted without additional modifications.
  • Small and Tiny variants reduce intermediate dimensions to 32 and 16, respectively, and omit TSPG under stricter parameter budgets.In the Tiny variant, enabling TSPG raises trainable parameters from 0.08M to 0.09M for only marginal additional improvement.
  • Figure 8 depicts frozen codec layers with snowflakes and trainable layers with flames in the decoder-stage integration.The figure presents adapter insertion into various pretrained codecs.
  • Training uses Adam with a base learning rate of 4 × 10^-4, while batch sizes and schedules vary across classification, detection, and segmentation tasks.Classification models are trained for 8 epochs; detection and segmentation use longer schedules.

I Limitations

The study’s validation is bounded to high-level machine vision tasks and selected codec, bitrate, resolution, runtime, and deployment settings.

  • The framework is evaluated on high-level downstream machine vision tasks, leaving low-level vision applicability outside the study’s scope.
  • Runtime cost remains dependent on the underlying codec and may require simplification for extremely high-resolution or strict real-time deployment.
  • Extremely low bitrate regimes below 0.05 bpp are not studied, although compressed representations may become the dominant bottleneck there.
  • Broader evaluation across more diverse compression architectures and practical deployment environments remains future work.
Loading 2608.25568v1…