Source-linked AI summary
ODSS-Net: Orthogonally Decoupled Shared-Specific Representations for Multi-Modal Object Re-Identification
Zhifeng Ma, Yuhong Feng, Chongyi Huang, Mingyang Zhou, Minwen Deng
TL;DR
Multi-modal object ReID seeks to use complementary sensors despite challenges in cross-modal alignment and fusion. ODSS-Net decouples shared and specific representations and adaptively refines and fuses modality-specific features. Experiments on three benchmarks demonstrate its effectiveness.
Problem
Cross-modal alignment can entangle shared structure with modality-specific cues, while fixed fusion does not account for sample-wise differences in modality reliability.
Source: §1 · p. 1Method
ODSS-Net constructs shared structural features, removes their aligned component from identity-relevant local features by orthogonal projection, then uses shared context to adaptively refine and fuse modality-specific features.
Source: §1 · pp. 1–2, §2.1 · p. 2, §2.2 · p. 3Results
Experiments on three benchmarks demonstrate ODSS-Net's effectiveness for multi-modal object ReID.
Source: §1 · p. 2Takeaways & Limitations
The paper supports orthogonally decoupled shared-specific learning with adaptive fusion as an effective approach to multi-modal object ReID.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Multi-modal object re-identification (ReID) leverages complementary information from heterogeneous sensors. Cross-modal alignment and fusion remain challenging. Existing methods mainly reduce modality gaps via feature alignment and fusion. However, alignment may entangle modality-shared structural information with modality-specific cues, while conventional fusion may not fully exploit complementary information. To address these issues, we propose ODSS-Net, a framework for orthogonally decoupled shared-specific representation learning and adaptive fusion of complementary modality-specific features. First, a Shared Structural Feature Construction (SSFC) module suppresses modality-dependent appearance variations and extracts modality-shared structural information. Then, an Orthogonal Specific Feature Learning (OSFL) module aggregates identity-relevant local information and removes the shared-aligned component via geometric orthogonal projection. Finally, a Sample Adaptive Mixture of Experts (SAMoE) module uses the shared structure as routing context to adaptively refine and fuse modality-specific features. Experiments on three benchmarks demonstrate the effectiveness of ODSS-Net for multi-modal object ReID.
1. INTRODUCTION
Multi-modal object ReID uses complementary sensor information, but must separate shared and modality-specific representations while adapting fusion to sample-wise cue reliability. ODSS-Net addresses these challenges with orthogonal decoupling and sample-adaptive fusion.
1. INTRODUCTION
- Multi-modal ReID uses complementary RGB, near-infrared, and thermal information, yet modality appearance discrepancies and variable cue reliability make shared-specific disentanglement and adaptive fusion challenging.Conventional RGB appearance can become unreliable under illumination variation and low-light conditions.Source: §1 · p. 1
- Feature alignment may entangle shared and specific information, fixed fusion overlooks sample-wise reliability, and prior separation is not explicitly enforced during forward transformation.These limitations motivate explicit shared-specific decoupling and sample-adaptive fusion.Source: §1 · p. 1
- ODSS-Net constructs shared structural features, orthogonally removes their aligned component from identity-relevant local features, and uses shared context to adaptively refine and fuse modality-specific features.Experiments on three benchmarks validate its effectiveness.Source: §1 · pp. 1–2
2. METHODOLOGY
The methodology builds a shared structural representation, derives modality-specific features by removing their shared-aligned component, and adaptively refines and fuses them using shared context.
2.1. Shared Structural Feature Construction (SSFC)
- SSFC suppresses modality-dependent appearance variation, applies token-wise instance normalization and positional embeddings, captures regional dependencies with self-attention, then pools and projects tokens into a shared feature.The shared feature is used for subsequent modality-specific learning and sample-adaptive fusion.Source: §2.1 · pp. 2–3
2.2. Orthogonal Specific Feature Learning (OSFL)
- OSFL uses shared structure to aggregate patch tokens into identity-relevant local information, then subtracts the shared-aligned residual by geometric orthogonal projection to obtain a modality-specific feature orthogonal to the shared feature.The resulting feature is used for subsequent expert refinement.Source: §2.2 · p. 3
2.3. Sample Adaptive Mixture of Experts (SAMoE)
- SAMoE forms a routing context from stacked shared features; modality-specific routers select top-k experts, whose residual-MLP outputs refine each feature before shared-guided cross-attention and projection produce the retrieval embedding.Only selected experts receive nonzero routing weights.Source: §2.3 · p. 3
S + MHA
The section presents the retrieval embedding construction and objective functions: shared and specific features are combined, while losses encourage cross-modal shared-feature consistency and preserve identity-discriminative cues.
S + MHA
- Feature-wise concatenation combines shared and refined specific features, and a final projection produces the retrieval embedding.Source: S + MHA · p. 3
2.4. Objective Functions
- The consistency loss minimizes pairwise differences between shared features across RGB–NIR, RGB–TIR, and NIR–TIR modality pairs.Source: §2.4 · p. 3
- A modality-shared identity classifier is applied to preserve identity-discriminative cues in the orthogonalized specific features.The loss uses one-hot identity labels and the shared classifier.Source: §2.4 · p. 3
- Cross-entropy and triplet losses supervise the entire network alongside the stated objective components.Source: §2.4 · p. 3
3. EXPERIMENT
ODSS-Net is evaluated on three multi-modal ReID benchmarks using mAP and CMC Rank-K metrics. The reported comparisons show strong benchmark performance, while ablations and similarity distributions examine its modules and orthogonal projection.
Datasets and Evaluation Metrics
- Evaluation covers RGBNT201, RGBNT100, and MSVR310, with mAP and CMC at Rank-1, Rank-5, and Rank-10 reported.RGBNT201 contains 4,787 aligned RGB, NIR, and TIR triplets from 201 identities; RGBNT100 and MSVR310 are vehicle ReID datasets.Source: §3 · p. 3
Comparison with State-of-the-Art Methods
- 82.1% mAP and 87.1% Rank-1 on RGBNT201 rank first across reported metrics, while ODSS-Net records the best mAP on RGBNT100 and MSVR310.On RGBNT100 and MSVR310, mAP is 88.1% and 54.0%, respectively; Rank-1 is 96.2% and 63.3%.Source: Table 1 · p. 4, §3 · p. 4
Effect of GOP
- With GOP, RGBNT201 mAP rises from 77.0% to 82.1% and Rank-1 from 81.1% to 87.1%.The passage reports cleaner modality-specific cues for subsequent adaptive fusion.Source: Table 3 · p. 4, §3 · p. 4
Cosine Similarity Distributions
- After GOP, cosine similarities between shared and specific features concentrate around zero across all three benchmarks.This concentration indicates reduced shared-specific correlation.Source: Figure 3 · p. 4, §3 · p. 4
Ablation Study
- The full model improves RGBNT201 over the baseline by 9.2 percentage points in mAP and 9.0 points in Rank-1.SSFC, OSFL, and SAMoE are added successively in the reported ablation.Source: §3 · p. 4, Table 2 · p. 4
4. CONCLUSION
The conclusion summarizes ODSS-Net as a shared-specific representation framework for multi-modal object ReID, combining structural representation, orthogonal projection, and sample-adaptive refinement and fusion.
4. CONCLUSION
- ODSS-Net combines shared structural representations, orthogonal projection for modality-specific cues, and shared-guided sample-adaptive refinement and fusion.Source: §4 · p. 4
- The paper reports effectiveness and highly competitive performance on three multi-modal ReID benchmarks.Source: §4 · p. 4
- SSFC constructs shared structure, OSFL separates specific cues, and SAMoE refines and fuses them under shared guidance.Source: §4 · p. 4