PaperlayerView exact revision on arXiv

Source-linked AI summary

SDF-Net: Structure-Aware Disentangled Feature Learning for Opticall-SAR Ship Re-identification

Furui Chen, Han Wang, Yuhan Sun, Jianing You, Yixuan Lv, Zhuang Zhou, Hong Tan, Shengyang Li

arXiv:2603.12588v2cs.CV

TL;DR

Optical–SAR ship ReID is challenged by severe radiometric differences between passive optical and coherent SAR sensing, while existing alignment methods overlook stable ship geometry. SDF-Net uses intermediate geometric constraints and disentangled residual fusion, achieving 60.9% mAP and 69.9% Rank-1 accuracy on HOSS-ReID under All-to-All.

  • Problem

    Optical–SAR ReID lacks an explicit way to use modality-invariant ship geometry while handling modality-dependent radiometric distortion.

  • Method

    SDF-Net uses normalized gradient energy from intermediate layers for structure consistency, then fuses shared identity and modality-specific features through parameter-free additive refinement.

  • Results

    60.9% mAP and 69.9% Rank-1 accuracy are achieved on HOSS-ReID under the All-to-All protocol, outperforming existing methods.

  • Takeaways & Limitations

    The results support explicitly incorporating geometric structure as a physical prior into cross-modal representation learning for optical–SAR ship ReID.

  • Takeaways & Limitations

    Performance degrades below approximately 3.0 m GSD and below approximately 15° incidence, while validation remains limited by HOSS-ReID being the only public benchmark.

Abstract

from arXiv · show

Cross-modal ship re-identification (ReID) between optical and synthetic aperture radar (SAR) imagery is fundamentally challenged by the severe radiometric discrepancy between passive optical imaging and coherent active radar sensing. While existing approaches primarily rely on statistical distribution alignment or semantic matching, they often overlook a critical physical prior: ships are rigid objects whose geometric structures remain stable across sensing modalities, whereas texture appearance is highly modality-dependent. In this work, we propose SDF-Net, a Structure-Aware Disentangled Feature Learning Network that systematically incorporates geometric consistency into optical--SAR ship ReID. Built upon a ViT backbone, SDF-Net introduces a structure consistency constraint that extracts scale-invariant gradient energy statistics from intermediate layers to robustly anchor representations against radiometric variations. At the terminal stage, SDF-Net disentangles the learned representations into modality-invariant identity features and modality-specific characteristics. These decoupled cues are then integrated through a parameter-free additive residual fusion, effectively enhancing discriminative power. Extensive experiments on the HOSS-ReID dataset demonstrate that SDF-Net consistently outperforms existing state-of-the-art methods. The code and trained models are publicly available at https://github.com/cfrfree/SDF-Net.

I. INTRODUCTION

Optical–SAR ship ReID must bridge fundamentally different sensing physics while preserving stable ship geometry and accommodating modality-dependent appearance. SDF-Net addresses this gap with structure consistency, disentangled representations, and parameter-free fusion, achieving state-of-the-art HOSS-ReID performance.

  • Motivation: Optical imagery offers visual detail, whereas SAR provides all-weather and day-night observation for continuous ship monitoring and long-term tracking.The two sensing modalities play complementary roles in maritime surveillance.
  • Physical Prior: Ships provide a physical cross-modal prior because hull contour, aspect ratio, and superstructure layout remain stable across modalities, particularly under near-nadir satellite observation.Texture is sensor-dependent, whereas macroscopic geometry forms a distortion-tolerant anchor for representation alignment.
  • Challenge: Optical–SAR ReID faces nonlinear radiometric distortion because passive optical reflectance and coherent active microwave backscatter measure fundamentally different quantities.Modality-specific artifacts include SAR speckle, corner-reflector responses, and geometric distortions, alongside optical cloud, sunglint, and illumination effects.
  • Proposed Method: SDF-Net uses a ViT backbone and a Structure Consistency Constraint to enforce cross-modal geometric alignment at intermediate feature stages.The constraint uses normalized gradient energy statistics to obtain scale-invariant structural descriptors and reduce sensitivity to radiometric distortion.
  • Proposed Method: SDF-Net disentangles terminal representations into modality-invariant shared features and modality-specific features, then combines them through parameter-free additive residual fusion.The modality-specific component refines the shared identity representation without introducing additional computation.
  • Results: Extensive experiments on HOSS-ReID demonstrate state-of-the-art performance for the proposed physics-guided disentanglement approach.The reported experiments are presented as validation of the method’s efficacy for maritime target association.

II. RELATED WORK

Related work spans cross-modal ReID, disentangled and structure-aware representation learning, and optical–SAR ship analysis. Existing VI-ReID principles do not transfer directly because optical–SAR matching involves different sensing physics, rigid targets, and modality-specific information.

  • Cross-Modal Re-Identification: Cross-modal ReID research commonly learns modality-robust shared embeddings, semantic correspondences, affinity reasoning, or contrastive alignment across heterogeneous sensors.These approaches aim to reduce modality gaps by aligning global or local representations.
  • Cross-Modal Re-Identification: VI-ReID methods do not transfer directly to optical–SAR ship ReID because the domains differ in sensing physics, object deformation, and the role of modality-specific information.The mismatch is categorical rather than merely quantitative.
  • Cross-Modal Re-Identification: Optical–SAR sensing combines active coherent microwave backscatter with passive solar reflectance, producing severe nonlinear radiometric distortion and SAR-specific speckle and reflector responses.These responses violate statistical assumptions commonly used for VI-ReID distribution alignment.
  • Disentangled and Structure-Aware Representation Learning: Disentangled representation learning separates identity-relevant information from modality-dependent variation, while SDF-Net preserves modality-specific sensor footprints as residual information.This differs from approaches that discard modality-specific features as style noise.
  • Disentangled and Structure-Aware Representation Learning: Intermediate feature layers preserve spatial organization and structural cues, motivating normalized gradient energy statistics as scale-invariant descriptors for optical–SAR ship ReID.The approach aligns intermediate representations rather than relying only on output-level features.
  • Optical–SAR Ship Analysis: Geometric structural properties are established as useful modality-invariant descriptors in remote sensing matching, including methods based on local geometric structure.This supports combining explicit geometric cues with appearance information rather than discarding all appearance characteristics.

C. Optical–SAR Image Analysis and Ship ReID

Optical–SAR ship ReID addresses identity association across heterogeneous sensing modalities with sharply different physical measurements and radiometric behavior. SDF-Net builds on a ViT-based TransOSS architecture and adds structure-aware and disentangled representation learning.

  • HOSS-ReID provides a dedicated benchmark for optical–SAR association under diverse scattering conditions and complex maritime environments.
  • Existing methods use implicit attention, statistical or generative alignment, and geometry-guided structural priors to bridge the optical–SAR modality gap.
  • SDF-Net encodes structural invariance as a core learning objective and integrates modality-specific cues through residual refinement.
  • The model learns a unified latent embedding that reduces intra-class variation while preserving identity-critical geometric signatures across modalities.
  • SDF-Net adds intermediate Structure-Aware Consistency Learning and terminal Disentangled Feature Learning to the TransOSS ViT-B/16 framework.

C. Cross-Modal Feature Tokenization

SDF-Net uses modality-specific input tokenization and intermediate latent features to separate stable ship geometry from sensor-dependent appearance. Gradient-energy aggregation provides a structural representation that is less sensitive to local radiometric disturbances.

  • Cross-Modal Feature Tokenization: Independent optical and SAR projection heads map modality-specific P × P patches into a shared C-dimensional latent space.
  • Intermediate Stage: Structure-Aware Consistency: Intermediate feature maps retain spatial topology while avoiding the raw noise and excessive abstraction associated with lower and higher network layers.
  • Intermediate Latent Geometry Excavation: Gradient operators capture modality-agnostic structural transitions from spatial intensity variations rather than absolute amplitudes.
  • Intermediate Latent Geometry Excavation: Aggregating absolute horizontal and vertical gradient magnitudes across the spatial grid yields per-channel descriptors summarizing global structural intensity.
  • Intermediate Latent Geometry Excavation: The integrated descriptor fstruct = ex + ey acts as a distortion-tolerant structural anchor that captures macroscopic ship contours despite localized SAR scattering peaks.

2) Scale-Invariant Instance Normalization:

SDF-Net applies Instance Normalization to gradient-energy descriptors so that structural information is less dependent on modality-specific radiometric distributions.

  • Instance Normalization standardizes gradient-energy descriptors across channels to improve radiometric robustness.
  • The normalization removes modality-specific amplitude and illumination styles while preserving geometric content for cross-modal association.

3) Prototype-level Consistency Loss:

SDF-Net aligns normalized structural descriptors at the identity-prototype level and disentangles terminal representations into shared and modality-specific subspaces. Additive fusion then uses the modality-specific branch as complementary residual refinement.

  • Prototype-level Consistency Loss: Structural descriptors are aligned through optical and SAR prototypes for each identity, avoiding instance-level overfitting to sample noise.
  • Prototype-level Consistency Loss: The structure consistency loss prioritizes modality-invariant geometric signatures over transient textural features.
  • Prototype-level Consistency Loss: Selecting Transformer block Bs = 6 balances spatial-detail preservation and semantic abstraction in the 12-layer backbone.
  • Disentangled Feature Learning and Residual Fusion: Terminal projection heads factorize representations into shared identity and modality-specific subspaces, while orthogonality regularization isolates sensor-independent cues.
  • Disentangled Feature Learning and Residual Fusion: Additive fusion retains modality-specific nuances as residual refinement for distinguishing highly similar ships.

F. Joint Optimization Objective

SDF-Net jointly optimizes identity supervision, feature disentanglement, and structural consistency to produce a robust cross-modal identity representation. Its forward pipeline extracts intermediate structural cues, disentangles terminal features, and fuses them additively.

  • Joint Optimization Objective: The unified objective combines label-smoothed cross-entropy and weighted triplet identity losses with orthogonality and structural-consistency terms.λorth and λstruct balance disentanglement and structural consistency during optimization.
  • Forward Processing Pipeline: The forward pipeline processes each optical or SAR image through modality-specific tokenization and Transformer blocks before intermediate structure extraction.The algorithm specifies separate optical and SAR tokenizers followed by feature extraction through layer Bs.
  • Forward Processing Pipeline: Structure-aware consistency learning computes normalized gradient features from intermediate representations and optimizes Lstruct across modalities.The pipeline obtains spatial gradients Gx and Gy and applies instance normalization to the structural feature.
  • Forward Processing Pipeline: Terminal features are decoupled into shared and specific components, regularized by orthogonality, and combined through parameter-free additive residual fusion.The final representation is defined as ffuse = fsh + fsp.
  • Evaluation Setting: HOSS-ReID evaluation uses instance-level ship crops from optical and SAR imagery, with coordinated multi-angle acquisitions and human-annotated cross-modal identities.The benchmark includes crops from Jilin-1 optical and TY-MINISAR SAR constellations, while labels rely on spatial proximity, temporal continuity, and ship characteristics.

B. Evaluation Metrics

The evaluation measures cross-modal retrieval with ranking accuracy and gallery-wide precision–recall, using HOSS-ReID comparisons across model families and retrieval protocols. SDF-Net achieves superior reported performance, including 60.9% mAP and 69.9% Rank-1 under the All protocol.

  • Evaluation Metrics: Rank-1, Rank-5, and Rank-10 CMC scores measure whether a correct identity appears within the top-k retrieved results.CMC evaluates identity matching capability at multiple retrieval depths.
  • Evaluation Metrics: mAP summarizes retrieval efficacy by averaging Average Precision across queries and accounting for precision and recall throughout the gallery.For each query, AP integrates the precision–recall curve before averaging over the query set.
  • Comparison with State-of-the-Art Methods: The benchmark comparison categorizes evaluated methods into general vision backbones, single-modality ReID models, and cross-modal retrieval frameworks.The comparison is conducted on the HOSS-ReID benchmark.
  • Comparison with State-of-the-Art Methods: TransReID achieves 20.9% mAP under SAR-to-Optical, illustrating the difficulty of appearance-based matching under radar-specific distortions.The reported limitation concerns coherent speckle noise and geometric artifacts in SAR imagery.
  • Comparison with State-of-the-Art Methods: 60.9% mAP and 69.9% Rank-1 are achieved by SDF-Net under the All protocol, improving over TransOSS by 3.5% and 4.0%, respectively.Under SAR-to-Optical, mAP rises from 38.7% to 46.6%, a 7.9% absolute increase.

E. Ablation Studies

Ablations show that structural consistency and disentangled feature learning are complementary, while structural extraction works best at an intermediate Transformer depth. SCL stabilizes cross-modal representations, and DFL improves discriminability when supplied with that geometric anchor.

  • Effectiveness of Proposed Modules: SCL alone raises SAR-to-Optical mAP from 44.5% to 46.6%, indicating improved retrieval stability from intermediate geometric consistency.The improvement is attributed to anchoring representations against radiometric fluctuations.
  • Effectiveness of Proposed Modules: SCL alone improves SAR-to-Optical mAP by 2.1% but reduces Rank-1 by 1.5%, exposing a robustness–discriminability trade-off.Geometry improves matches distributed across the ranked list but can be less discriminative among visually similar ships.
  • Effectiveness of Proposed Modules: DFL alone reduces SAR-to-Optical mAP from 44.5% to 41.4% and Rank-1 from 38.8% to 31.3% without structural guidance.The orthogonality constraint alone cannot prevent modality-specific subspaces from drifting into incompatible regions.
  • Effectiveness of Proposed Modules: Combining SCL and DFL yields 46.6% SAR-to-Optical mAP, 38.8% Rank-1, 60.9% All-to-All mAP, and 69.9% All-to-All Rank-1.SCL stabilizes the shared space, allowing DFL to add modality-specific residual information without the drift observed in DFL-only training.
  • Sensitivity Analysis of Structural Extraction Layer: Bs = 6 achieves 60.9% mAP and 69.9% Rank-1 under All, outperforming shallow and deep structural extraction settings.Shallow layers retain radiometric noise, whereas deeper layers lose fine-grained geometric layout through spatial collapse.

3) Evaluation of Feature Fusion Strategies:

SDF-Net’s additive fusion combines shared identity and modality-specific features without expanding feature dimensionality, achieving the strongest reported fusion performance. Its robustness remains stable across hyper-parameter settings and moderate resolution changes, with degradation at extreme downsampling.

  • Feature fusion: 59.2% mAP: the shared identity feature alone outperforms the modality-specific feature alone at 58.7% under the All protocol.The shared branch isolates modality-invariant cues, while discarding modality-specific information loses fine-grained details.
  • Feature fusion: 60.9% mAP and 69.9% Rank-1: additive fusion fsh+fsp achieves the peak All-to-All performance.It also dominates the more challenging SAR-to-Optical task while preserving the original feature dimensionality.
  • Computational complexity: SDF-Net introduces zero additional parameters relative to the baseline through parameter-free spatial-gradient integration, instance normalization, and element-wise additive fusion.The comparison is reported in Table VI.
  • Sensitivity and robustness: Performance is tolerant across broad orthogonality and structural-consistency weight combinations, with λstruct optimal around 1.0 and λorth reaching peak capacity at 10.0.Departures from these settings cause gentle rather than catastrophic degradation.
  • Resolution robustness: At 0.25× resolution, SCL-only reaches 53.8% versus 53.2% for the baseline, while SDF-Net performs best from native resolution through 0.50×.At extreme downsampling, the ship spans only a few pixels and modality-specific features become unreliable.

2) Incidence Angle and Projective Distortion:

SDF-Net is most robust under near-nadir and moderate projective-distortion conditions, where structural consistency preserves hull-level geometry. Under severe blur or near-grazing incidence, SCL-only can outperform the full model because fine-grained modality-specific cues become unreliable.

  • Projective distortion: The dataset’s near-nadir geometry uses θ < 25°, where layover and foreshortening are largely confined to superstructures and do not distort the hull footprint used by SCL.Global spatial averaging of gradient magnitudes is therefore tolerant to mild local displacement.
  • Projective distortion: At θ = 5°, SCL-only reaches 13.3% mAP versus 11.4% for the baseline, while performance degrades as incidence angles become shallower.The evaluation applies uniform warps without geometric augmentation training.
  • Projective distortion: At moderate incidence angles, SDF-Net achieves the best results, whereas SCL-only leads in the near-grazing case.The consistent SCL-only advantage under subsampling, low-pass filtering, and shearing identifies structural consistency as the robustness mechanism.
  • Motion blur: Under 9-pixel blur, SCL-only reaches 56.8% versus 55.8% for the baseline, while SDF-Net leads under mild blur.At stronger blur, azimuthal defocusing suppresses the fine-grained features used by DFL, while macroscopic hull edges survive moderate filtering.
  • Class activation mapping: Grad-CAM shows consistent attention to rigid hull contours and spatial layout across optical and SAR images while suppressing sea clutter and speckle noise.The comparison layout places optical input, optical attention, SAR attention, and SAR input from left to right within each group.

2) Visual Analysis of Layer-wise Feature Evolution:

Layer-wise visualizations indicate that intermediate Transformer features provide the clearest geometric anchor for cross-modal matching. Retrieval examples show improved robustness, but failures remain for geometrically similar ships and extremely low-resolution targets.

  • Layer-wise evolution: Layers 2 and 4 retain high spatial resolution but are dominated by modality-specific artifacts, including illumination variation, backscatter points, and speckle noise.These shallow representations are unsuitable for direct geometric alignment.
  • Layer-wise evolution: Layer 6 produces a concentrated ship layout that filters radiometric distortions while preserving physical proportions and topology.This intermediate depth provides the reported optimal structural anchor for cross-modal matching.
  • Layer-wise evolution: Layers 8–12 progressively lose spatial fidelity as global context aggregation causes spatial collapse and overly abstract representations.These deeper features still encode discriminative semantic cues for final retrieval.
  • Retrieval visualization: Across All, Optical-to-SAR, and SAR-to-Optical protocols, SDF-Net retrieves a higher density of correct matches than the baseline.The qualitative comparison attributes this robustness to anchoring representations on modality-invariant geometric skeletons.
  • Failure analysis: SDF-Net confuses ships with similar hull contours, aspect ratios, and superstructure layouts because structural anchoring suppresses potentially disambiguating appearance cues.This exposes a robustness–discriminability trade-off in structure-centric matching.
  • Failure analysis: Very low-resolution targets fail when ships span too few pixels to generate reliable intermediate-level gradient energy statistics.This agrees with the 0.25× resolution boundary identified in the stability analysis.

V. DISCUSSION

SDF-Net combines structure-aware gradient statistics with disentangled, parameter-free feature fusion to improve optical–SAR ship ReID. Its benefits are supported by benchmark results and analyses, but performance remains bounded by image quality, viewing geometry, benchmark scope, and detector quality.

  • Method and design principle: SCL extracts intermediate gradient energy to isolate rigid hull structure, while DFL separates shared and modality-specific representations for additive refinement.The two modules are described as complementary components anchoring ReID on modality-invariant geometric structures.
  • Failure analysis: Failures predominantly arise from geometrically similar ships and low-resolution targets with weak structural cues.The failure analysis covers both SAR-to-Optical and Optical-to-SAR retrieval directions.
  • Empirical findings: 60.9% mAP and 69.9% Rank-1 accuracy are achieved on HOSS-ReID under the All-to-All protocol, outperforming existing methods.These results are reported as benchmark evidence for SDF-Net’s effectiveness.
  • Broader implications: Intermediate gradient energy statistics provide a robust, parameter-free structural descriptor that may generalize to sensor pairs such as LiDAR–camera fusion.The analysis identifies spatial derivatives, global averaging, and instance normalization as components of this descriptor.
  • Limitations: Performance degrades below approximately 3.0 m GSD and θ ≈15° incidence, and generalization beyond HOSS-ReID remains unvalidated.The paper also notes that DFL requires SCL and that detector errors propagate to ReID.
  • Conclusion and future work: The proposed geometric prior and disentangled fusion achieve state-of-the-art optical–SAR ship ReID while adding zero parameters.Future work includes physics-informed projective augmentation, adaptive fusion weighting, and extension beyond near-nadir observation.
Loading 2603.12588v2…