Source-linked AI summary
Dynamic Alignment and Calibration for Multimodal Learning
Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi
TL;DR
Static alignment overlooks sample-wise differences, and confidence-aware fusion may not account for feature magnitude and confidence gaps. ACML adapts alignment and attention regularization to those differences. It reports the best performance across datasets under clean and noisy conditions.
Problem
Existing methods may apply uniform alignment across samples and confidence-aware fusion without adequately accounting for feature magnitude and confidence differences.
Source: §1 · p. 2Method
ACML combines confidence-dependent triplet alignment, including decorrelation for positive pairs with larger confidence gaps, with attention regularization adjusted for feature magnitude and confidence differences.
Source: §1 · p. 3, §3.2 · p. 6Results
ACML achieves the best performance across all datasets under clean and noisy conditions, with clean-setting relative average accuracy gains over the strongest baseline of 1.82% on NYU Depth V2, 1.64% on SUN RGB-D, and 0.41% on FOOD101.
Source: §4.1 · p. 9Takeaways & Limitations
DCTA is designed to preserve complementary information and mitigate noise propagation while promoting semantic consistency for high-confidence positive pairs.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.
1 Introduction
Real-world modality quality and reliability fluctuate, but existing methods do not adequately adapt alignment and fusion to sample-level variation. ACML combines confidence-aware cross-modal alignment with attention calibration based on feature magnitude and confidence differences.
1 Introduction
- Dynamic modeling is needed because modality data quality and reliability fluctuate with device heterogeneity, environmental noise, and occlusions.Source: §1 · p. 2
- Static alignment applies uniform constraints across samples, which can over-align ambiguous or noisy pairs and diminish complementary information.Strong alignment benefits high-confidence samples, but forcing all positive pairs closer can degrade the feature space.Source: §1 · p. 2
- Confidence-aware fusion can impose unreasonable contribution reallocation when modality feature magnitudes differ greatly or confidence differences are small.These constraints may interfere with effective fusion or restrict potential cross-modal synergy.Source: §1 · p. 2
- DCTA enforces semantic consistency for high-confidence positive pairs while promoting diversity between high- and low-confidence pairs to preserve complementarity and mitigate noise propagation.For low-confidence samples, it relaxes alignment and introduces orthogonality encouragement.Source: §1 · p. 3
- DAC adjusts attention regularization according to feature magnitude and confidence differences, avoiding unreasonable over-regularization of attention weights.Source: §1 · p. 3
2 Related Work
Prior work spans explicit and implicit alignment, contrastive objectives, and fusion at different stages. Dynamic fusion adapts modality contributions, while this work identifies norm and confidence differences as reasons to vary pairwise constraints.
2.1 Multimodal Alignment
- Contrastive alignment pulls positive pairs together and pushes negative pairs apart, while ACML extends this objective with triplet alignment to balance consistency and complementarity.Prior approaches include explicit similarity-based alignment and implicit joint representations learned through translation or prediction objectives.Source: §2.1 · p. 3
2.2 Multimodal Fusion
- Fusion methods combine modalities at the data, feature, or prediction level, while dynamic approaches adapt contributions across samples or time.Homogeneous fusion constraints may overlook norm and confidence differences; ACML instead adjusts pairwise constraint strength using both.Source: §2.2 · p. 4
3 Method
ACML estimates sample-level modality confidence, then uses it to guide cross-modal alignment and attention calibration. Its final objective combines these constraints with fused and unimodal classification losses.
3.1 Feature Extraction and Confidence Estimation
- During training, stochastic inference is used to obtain robust, stable confidence estimates; a probabilistic auxiliary path applies dropout to the same feature vector to mitigate over-confidence from a single deterministic forward pass.The confidence score is the empirical mean of ground-truth probabilities across K stochastic predictions; dropout probability is 0.1 and K = 10.Source: §3.1 · pp. 4–5
3.2 Dynamic Cross-modal Triplet Alignment
- LDCTA weights alignment by paired-sample confidence, encourages decorrelation for positive pairs with larger confidence gaps, and adds repulsion for negative pairs.The combined alignment, decorrelation, and repulsion terms form a batch-wise modality-pair loss.Source: §3.2 · pp. 5–6
3.3 Difference-aware Attention Calibration
- LDAC adjusts attention calibration using feature-magnitude and confidence differences, relaxing constraints for large magnitude gaps and emphasizing comparable magnitudes with distinguishable confidence.The stated weighting jointly models magnitude and confidence differences to adapt calibration across modality pairs.Source: §3.3 · p. 7
3.4 Fusion and Optimization Objective
- ACML reweights modality features, mean-fuses them for classification, and optimizes fused and unimodal classification losses alongside LDCTA and LDAC.The overall objective uses λ1 and λ2 to balance the two constraint losses.Source: §3.4 · p. 8
4 Experiments and Analysis
The experiments evaluate ACML’s performance, robustness, component contributions, representation effects, hyperparameter stability, missing-modality applicability, and scalability. Across these settings, the reported results favor ACML over baselines and support its use across varied multimodal scenarios.
4.1 RQ1 (Effectiveness & Robustness)
- ACML achieves the best performance across all three datasets in clean and noisy settings, with larger gains under severe corruption on NYU Depth V2 and SUN RGB-D.Clean-setting relative average-accuracy gains are 1.82%, 1.64%, and 0.41% over the strongest baseline on NYU Depth V2, SUN RGB-D, and FOOD101; at ε = 10, gains on the first two datasets are 6.00% and 5.36%.Source: §4.1 · p. 9, Table 1 · p. 10
4.2 RQ2 (Ablation Study)
- On NYU Depth V2, either LDCTA or LDAC improves the baseline, while full ACML performs best on average and worst-case accuracy across noise conditions.Replacing LDCTA with LSC or LCSC, or LDAC with unweighted LMSE, degrades performance.Source: §4.2 · pp. 10–11, Table 2 · p. 11
4.3 RQ3 (Visualization Analysis)
- Figure 2 t-SNE visualizations show that after LDCTA, high-confidence samples form more compact clusters with improved cross-modal consistency, while low-confidence samples, shown in lighter colors, remain relatively scattered.Figure 2 uses distinct symbols for image (△) and text (□) modalities; color marks class, and darker shades indicate higher confidence.Source: §4.3 · p. 11, Figure 2 · p. 11
4.5 RQ5 (Applicability to Missing Modalities)
- On FOOD101, adding ACML improves DCP and SyP across missing-modality ratios and patterns on validation and test sets, with gains persisting at 90% missing.The backbones are frozen while prompts and ACML are updated.Source: Table 3 · p. 12, §4.5 · p. 13
4.4 RQ4 (Hyper-parameter Analysis)
- Across NYU Depth V2 and FOOD101, excessively small or large λ1 or λ2 values degrade performance, while values such as 0.5 and 1 yield greater stability.The pattern is more evident on the larger-scale FOOD101 dataset.Source: §4.4 · p. 12, Figure 3 · p. 12
4.6 RQ6 (Generalization and Scalability)
- On BRCA and ROSMAP multi-omics tasks with three modalities, ACML achieves consistent performance gains over six state-of-the-art multimodal methods.The experiments report classification metrics across both datasets and describe the results as evidence of generalization to diverse multimodal scenarios.Source: Table 4 · p. 13, §4.6 · p. 13
Conclusion
ACML addresses static alignment and fusion strategies that lack difference-aware modeling. Its DCTA and DAC modules adapt alignment and attention regularization, and experiments report superior performance and robustness across multimodal benchmarks.
Conclusion
- ACML combines dynamic cross-modal alignment with difference-aware attention calibration to support robust, adaptive multimodal representation learning.The conclusion describes DCTA as adapting alignment to confidence and DAC as adjusting attention regularization to modality feature-magnitude and confidence differences.Source: Conclusion · p. 14
- The authors report superior performance and robustness across multiple multimodal benchmark datasets.Source: Conclusion · p. 14
- Future work will explore integrating ACML with large-scale pre-trained foundation models and extending it to medical multimodal analysis, understanding, and generation.Source: Conclusion · p. 14