Source-linked AI summary
CRISP: Calibration-Aware Visual State Space Duality for Remote Sensing Semantic Segmentation
Kangning Wang, Haopeng Zhang, Zhiguo Jiang
TL;DR
Remote sensing segmentation needs to capture dense structures and complex boundaries efficiently at ultra-high resolution. CRISP calibrates VSSD with DCO and OMP, achieving state-of-the-art-level accuracy on Potsdam, Vaihingen, and LoveDA with approximately 30M parameters.
Problem
Remote sensing segmentation requires modeling dense small-scale structures and complex boundaries efficiently, while Transformer attention incurs quadratic memory and computation at ultra-high resolutions.
Method
CRISP couples a VSSD calibration branch using DCO with an OMP decoder head to restore high-frequency responses and preserve intra-class multimodal structure without sacrificing linear complexity.
Results
CRISP achieves accuracy on par with state-of-the-art methods on Potsdam, Vaihingen, and LoveDA using approximately 30M parameters.
Takeaways & Limitations
CRISP demonstrates a strong accuracy–efficiency trade-off for remote sensing semantic segmentation.
Takeaways & Limitations
CRISP is not latency-optimal because current state-space CUDA kernels remain less mature than convolution and attention kernels.
Abstract
from arXiv · showhide
State space models, especially Visual State Space Duality (VSSD), have emerged as efficient linear-time alternatives to Transformers for dense visual tasks. However, we observe that VSSD compresses spatial context into a global aggregation that suppresses high-frequency responses, causing excessive boundary smoothing in remote sensing semantic segmentation. To address this, we propose CRISP, a calibration framework with two components. Its core, the Duality Calibration Operator (DCO), restores local contrast and boundary responses through residual injection and frequency calibration within the VSSD backbone, without altering its linear complexity. To retain the recovered detail, an Orthogonal Multi-Prototype (OMP) head assigns multiple orthogonally constrained prototypes per class to model large intra-class variance. Extensive experiments on Potsdam, Vaihingen, and LoveDA show that, with approximately 30M parameters, CRISP achieves consistent gains in mean F1 (mF) and mIoU while remaining competitive with state-of-the-art methods. Code is available at https://github.com/crazylifeha/CRISP.
1 Introduction
CRISP targets the loss of high-frequency detail in efficient VSSD-based remote sensing segmentation while preserving linear complexity. It combines DCO calibration with an OMP decoder head to retain recovered intra-class multimodal structure and achieves state-of-the-art-level accuracy at approximately 30M parameters.
- Motivation: Remote sensing segmentation requires modeling dense small-scale structures and complex boundaries at ultra-high resolution for Earth-observation applications.These demands distinguish remote sensing imagery from natural images.
- Motivation: CNNs provide local representations, whereas ViTs model long-range dependencies but incur quadratic self-attention costs at ultra-high resolutions.This creates severe memory and computation bottlenecks.
- Motivation: SSMs such as Mamba and Mamba-2 with SSD offer promising alternatives, but causal scanning is mismatched to 2D images, motivating VSSD.SSD unifies sequence modeling and attention mechanisms, while VSSD adapts the approach to visual data.
- Method: CRISP combines an aggregation-aligned DCO backbone branch with an OMP decoder head to compensate VSSD while preserving linear complexity and intra-class multimodal structure.DCO reuses aggregation weights to recover attenuated high-frequency responses, while OMP learns orthogonally regularized sub-prototypes end-to-end.
- Results: At approximately 30M parameters, CRISP achieves accuracy on par with state-of-the-art methods on Potsdam, Vaihingen, and LoveDA, improving the accuracy–efficiency trade-off.The reported scope covers all three datasets and emphasizes efficiency alongside accuracy.
2 Related Work
Related work spans remote sensing segmentation, visual state space models, frequency- and boundary-preserving modeling, and prototype-based decoders. CRISP addresses the resulting gap by calibrating VSSD aggregation in place and using multi-prototype decoding to preserve detail and represent intra-class variation.
- Remote sensing semantic segmentation: Remote sensing segmentation requires preserving fine spatial detail because high-resolution imagery contains dense small objects and intricate boundaries.Prior methods include DANet, OCR, and UNetFormer, which model long-range context while retaining locality.
- Visual state space models: SSMs evolved from S4’s structured linear-time recurrence through Mamba’s selective recurrence and Mamba-2’s SSD formulation linking SSMs with linear attention.Vision Mamba, VMamba, PlainMamba, and EfficientVMamba adapt these models to images through different scanning strategies.
- Frequency- and boundary-preserving modeling: Prior frequency- and boundary-preserving methods reinject high-frequency signals through spatial post-processing, spectral branches, decoder fusion, or convolutional paths, but add computation and disrupt linear-scan continuity.PointRend sharpens masks by adaptively resampling uncertain boundary points after backbone processing.
- Frequency- and boundary-preserving modeling: CRISP’s DCO instead calibrates VSSD directional aggregation in place, reusing backbone aggregation weights to inject a state-level residual that compensates discretization-induced aliasing.This design restores high-frequency responses without appending a generic Fourier/DCT/wavelet module or decoder-side fusion stage.
- Prototype-based decoders: Conventional heads such as DeepLabv3+ [3] and UPerNet use one prototype per class, collapsing intra-class variation onto a single direction and limiting representation of diverse remote-sensing categories.The passage illustrates this issue with buildings that differ in roof materials.
3 Methodology
CRISP combines a DCO module within VSSD with an OMP segmentation head to counteract high-frequency attenuation while preserving linear-time processing. DCO restores weakened contrast and boundary responses, whereas OMP matches pixel descriptors against multiple class prototypes.
- 3.1 VSSD Background and Feature Attenuation: VSSD preserves O(L) complexity through multidirectional scanning and fusion, but its smooth global aggregation suppresses high-frequency information and over-smooths boundaries.The normalized aggregation behaves like a large-window low-pass operator, collapsing spatial tokens toward a low-rank summary.
- 3.2 Duality Calibration Operator: DCO performs state-level residual injection, token-level high-pass modulation, and channel-level DC/HC rebalancing between low-frequency semantics and high-frequency details.The residual is vj − v′j; high-pass modulation strengthens de-meaned responses, followed by mean/variation decomposition and channel reweighting.
- 3.2 Duality Calibration Operator: DCO is inserted after each VSSD fusion operation and reuses existing aggregation weights to recover local contrast and boundary details without quadratic interactions.It uses a near-identity residual gate, with λh initialized to 0.01, and preserves linear complexity.
- 3.3 Orthogonal Multi-Prototype Head: The OMP head uses a hyper-network to generate K prototypes per class and matches them with L2-normalized pixel descriptors from decoder feature maps.This design provides multiple class prototypes for matching per-location feature vectors.
4 Experiments
CRISP is evaluated on Potsdam, Vaihingen, and LoveDA under a unified training protocol, achieving consistently strong segmentation performance with 32.32M parameters. Experiments further show favorable efficiency, complementary DCO components, and synergy between DCO and OMP.
- Quantitative and qualitative results: Across LoveDA, Potsdam, and Vaihingen, CRISP achieves consistently strong mF and mIoU with a lightweight 32.32M parameter count.Improvements are especially pronounced for high-variance, irregular-contour categories such as Low-veg, Tree, and Forest.
- Efficiency: CRISP reduces parameters and computation versus VSSD while improving accuracy, yielding a favorable accuracy–efficiency trade-off.The reduction mainly comes from replacing UPerHead with MultiProtoDecoder, while DCO adds less than 10^-3M parameters.
- Quantitative and qualitative results: Qualitative comparisons show that CRISP improves boundary delineation and local shape fidelity over VSSD across Potsdam, LoveDA, and Vaihingen.
- DCO sub-components: DCO sub-components independently improve the 87.32 baseline by 0.33, 0.36, and 0.32 for residual injection, high-pass modulation, and DC/HC rebalancing, respectively.Their complementary roles restore contrast, re-amplify boundary responses, and recalibrate frequency components; near-zero gate initialization preserves an identity start.
- Component synergy: DCO alone raises the VSSD+UPerHead baseline from 87.32 to 88.10 mIoU, whereas OMP alone reaches 85.63 without restored high-frequency detail.The results show that OMP benefits from DCO-recovered structure, rather than gains arising solely from decoder capacity.
5 Conclusion
CRISP addresses high-frequency attenuation and boundary blurring in visual state-space backbones by combining DCO for response restoration with OMP for preserving recovered intra-class diversity.
- 5 Conclusion: CRISP mitigates high-frequency attenuation and boundary blurring in visual state-space backbones.
- 5 Conclusion: DCO treats duality error as a measurable, correctable signal and restores attenuated high-frequency responses through residual injection, local duality correction, and frequency-aware gating.
- 5 Conclusion: OMP preserves recovered intra-class diversity using orthogonally regularized sub-prototypes.
CRISP: Supplementary Material · 1 Efficiency and Qualitative Evidence
On Potsdam with 512 × 512 inputs, CRISP substantially reduces computational cost and parameters versus VSSD while improving FPS, although its current latency remains limited by immature state-space CUDA kernels. Supplementary error maps and component breakdowns provide qualitative and efficiency evidence for these findings.
- 1 Efficiency and Qualitative Evidence: The supplementary efficiency analysis evaluates CRISP on Potsdam using 512 × 512 inputs.Table S1 reports overall efficiency, while Table S2 breaks down component-wise parameters and FLOPs.
- 1 Efficiency and Qualitative Evidence: Qualitative visualizations complement the efficiency analysis, including error maps comparing VSSD and CRISP.In Fig. S1, red pixels indicate prediction errors.
- 1 Efficiency and Qualitative Evidence: CRISP cuts total FLOPs from 235.65G to 64.80G and parameters from 56.10M to 32.32M versus VSSD, while improving FPS under identical evaluation conditions.Tables S1 and S2 attribute the savings mainly to replacing the heavy UPerHead-style decoder with a lightweight multi-prototype decoder.
- 1 Efficiency and Qualitative Evidence: DCO contributes negligible parameters and modest FLOPs within CRISP’s component-wise efficiency breakdown.The lightweight multi-prototype decoder, rather than DCO, is identified as the main source of computational savings.
- 1 Efficiency and Qualitative Evidence: CRISP’s current latency disadvantage is attributed largely to immature CUDA kernels for state-space operators rather than an inherent model property.Convolution and attention kernels have benefited from years of engineering, including TensorRT, whereas state-space kernels are expected to improve as they mature.
- 1 Efficiency and Qualitative Evidence: CRISP’s practical payoff is framed around deployment settings where state-space operator latency limitations are less decisive.The supplied discussion indicates that the latency gap is an implementation artifact expected to narrow with more mature SSM kernels.
2 Effect of Prototype Number K
On Potsdam single-scale validation, CRISP performs best with three prototypes per class, while larger prototype counts plateau rather than improving performance.
- 2 Effect of Prototype Number K: Three sub-prototypes per class sufficiently cover dominant intra-class modes, while K≥4 plateaus without evidence of further benefit.The observed plateau suggests larger prototype counts do not improve validation performance and may risk overfitting.
3 Extended Boundary and Cross-Dataset Analysis
Extended analyses show that CRISP preserves boundaries through complementary DCO and prototype regularization, while its boundary-quality gains generalize across Potsdam, Vaihingen, and LoveDA. The full model is strongest in the Potsdam boundary-band diagnostic and consistently exceeds VSSD on cross-dataset mBF.
- Boundary-band diagnostic: The full CRISP model achieves the best boundary-band mIoU, boundary-band accuracy, and mBF on Potsdam, outperforming variants that remove DCO or prototype regularizers.Table S4 evaluates predictions within dilated ground-truth boundary bands.
- Boundary-band diagnostic: Removing either DCO or prototype regularizers consistently degrades all three boundary-band metrics, indicating that both components contribute to boundary preservation.The affected metrics are boundary-band mIoU, boundary-band accuracy, and mBF.
- Cross-dataset boundary quality: CRISP consistently outperforms the VSSD baseline in class-aware mean Boundary F-score across Potsdam, Vaihingen, and LoveDA.Table S5 reports the cross-dataset mBF comparison.
4 DCO Implementation Details
This section details DCO’s placement and forward computation within VSSD blocks. It combines residual recovery, kernelized injection, high-pass calibration, and DC/HC rebalancing while preserving linear-time processing.
- Insertion: DCO is inserted after the VSSD fusion operator in every VSSD block across all stages.This placement is illustrated in the detailed VSSD core diagram and implementation description.
- Kernelized residual injection: Residual injection reuses VSSD Q and K tensors with the non-negative ELU+1 decomposable kernel.The base fusion retains denominator clamping, whereas DCO uses an unnormalized read-out.
- Forward computation: The forward pass recovers duality residuals, injects them through reused VSSD aggregation statistics, then applies high-pass calibration and DC/HC rebalancing.Algorithm 1 follows this sequence from discretized-value comparison through calibrated output generation.
- Complexity: The accumulator is computed in one pass, adding O(L · dstate · D) work without asymptotic overhead beyond the existing linear attention pass.The implementation adds O(H+D) gate parameters per block.