PaperlayerView exact revision on arXiv

Source-linked AI summary

Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing

Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang

arXiv:2608.24010v1cs.CV

TL;DR

Multi-label cross-modal hashing is limited by deterministic point proxies that cannot accommodate conflicting semantic gradients without collapse or oscillation. KDPH uses anisotropic Kent distributions, separating stable mean directions from adaptable directional variance and training the geometry with a Cayley-transform-based loss. Across three benchmarks, it outperforms state-of-the-art methods, while its added flexibility represents a trade-off relative to point proxies in single-label regimes.

  • Problem

    Deterministic point proxies cannot accommodate conflicting gradients from label co-occurrence, causing optimization collapse in multi-label cross-modal hashing.

  • Method

    KDPH represents class proxies as learnable anisotropic Kent distributions and uses a tailored Cayley-transform-based loss to decouple centroid and directional variance.

  • Results

    KDPH achieves state-of-the-art performance across three benchmark datasets and outperforms all compared baselines across hash code lengths.

  • Takeaways & Limitations

    Directional distributions let proxies maintain stable semantic centroids while accommodating diverse label correlations and contextual variance.

  • Takeaways & Limitations

    KDPH introduces redundancy as a trade-off, whereas point-proxy advantages remain reliable in single-label regimes but vanish in multi-label scenarios.

Abstract

from arXiv · show

Supervised proxy-based deep cross-modal hashing has become the dominant paradigm for large-scale retrieval. However, prevalent methods model class proxies as deterministic points in the embedding space. This rigid assumption causes severe gradient conflicts in multi-label scenarios, where gradient conflicts arising from label co-occurrence lead to severe gradient contention and optimization collapse. To resolve this, we propose Kent-based Distributional Proxy Hashing (KDPH), a novel framework that shifts proxy representation from static points to flexible anisotropic Kent distributions on the hypersphere. Unlike point proxies that must shift their positions to accommodate conflicting gradients, KDPH absorbs these conflicts by dynamically adjusting its directional variance. This allows the proxy to maintain a stable semantic mean direction while stretching to cover diverse label correlations. Furthermore, to ensure stable training of these geometric parameters, we derive a tailored loss function incorporating the Cayley transform to enforce strict orthogonality. To the best of our knowledge, KDPH is the first framework to successfully introduce the Kent distributions into cross-modal hashing. Experiments on three benchmark datasets demonstrate that KDPH mitigates proxy collapse and chaotic oscillation, significantly outperforms state-of-the-art methods. Code is available at https://github.com/Senmo996/KDPH-official-code.

1 Introduction

Supervised deep cross-modal hashing struggles with multi-label semantic overlap because deterministic point proxies cannot accommodate conflicting gradients. KDPH replaces these points with anisotropic Kent distributions that absorb conflicts through directional variance while preserving semantic centroids.

  • 1 Introduction: Multi-label instances create overlapping semantic correlations that existing supervised hashing frameworks struggle to represent.This motivates more robust modeling strategies for complex, non-exclusive semantics.
  • 1 Introduction: Deterministic point proxies lack spatial capacity for conflicting gradients, driving optimization collapse in multi-label scenarios.Conflicting labels simultaneously pull samples toward divergent proxies.
  • 1 Introduction: KDPH models each class proxy as a learnable anisotropic Kent distribution instead of a static point.The distribution provides semantic shape and volume for contextual variance from label co-occurrence.
  • 1 Introduction: KDPH absorbs gradient conflicts by stretching directional variance rather than shifting the proxy centroid.A Cayley-transform-based loss enforces strict orthogonality for the distribution’s principal axes.
  • 1 Introduction: KDPH achieves state-of-the-art performance on three benchmarks, with superior robustness in complex multi-label scenarios.The experiments empirically validate the gradient-decoupling strategy.

2 Related Work

Cross-modal hashing has progressed from deterministic and isotropic representations toward probabilistic embeddings, but isotropic or lightweight non-isotropic models remain too rigid for entangled multi-label semantics. KDPH addresses this gap with Kent distributions and a Cayley-transform-based training strategy.

  • 2 Related Work: Recent cross-modal hashing methods span CNN-RNN, adversarial, graph, transformer, pairwise, gradient-based, and geometric designs.These approaches reflect continued algorithmic development in supervised deep cross-modal hashing.
  • 2 Related Work: Probabilistic embedding learning replaces deterministic point estimates, including spherical von Mises-Fisher models for hyperspherical feature spaces.The progression follows the geometry of the target embedding space.
  • 2 Related Work: vMF models assume uniform variance across directions, limiting their ability to represent anisotropic semantic manifolds in multi-label data.Lightweight variance scaling and geometric regularization provide non-isotropy but often lack sufficient flexibility for densely entangled semantics.
  • 2 Related Work: KDPH introduces Kent-based distributional proxies to provide richer geometric flexibility for multi-label cross-modal hashing.Its framework combines a Kent-based proxy learner with a CLIP-based feature extraction module.

3 Method

KDPH models class proxies as anisotropic Kent distributions on the hypersphere rather than deterministic points, using directional variance and manifold-constrained optimization to handle multi-label semantics. Its training combines distribution-aware scoring with irrelevance regularization, cross-modal alignment, quantization, and Cayley-based orthogonality.

  • Kent-based Distributed Proxy Mechanism: KDPH replaces deterministic class points with learnable Kent distributions whose centroid and directional variances represent semantic structure.The Kent proxy uses a mean direction, concentration, orthogonal axes, and anisotropic shape parameters.
  • Kent-based Distributed Proxy Mechanism: Distinct shape parameters βc,i let each proxy stretch or compress along selected semantic directions without drastically shifting its centroid.This accommodates variance introduced by multi-label co-occurrences.
  • Distribution-Aware Proxy Triplet Loss: Training enforces positive-proxy energy above negative-proxy energy by a margin, aggregates valid multi-label triplets, and adds irrelevance, cross-modal alignment, and quantization objectives.Irrelevance regularization penalizes high similarity between pairs sharing no labels, while symmetric InfoNCE aligns matched image-text pairs.
  • Orthogonal Frame Construction via Cayley Transform: The Cayley transform generates strictly orthogonal proxy frames, with the first rotation-matrix column as the centroid and subsequent columns as shape axes.This avoids expensive orthogonalization while preserving the required geometric constraints.
  • Anisotropic Energy Scoring and Parameterization: The anisotropic energy score reinterprets Kent log density for discriminative ranking, avoiding the intractable high-dimensional Bessel-function normalization constant.A logarithmic regularizer constrains parameter magnitude while retaining Kent geometry.

4 Experiment

Experiments on three multi-label cross-modal datasets compare KDPH with state-of-the-art baselines and analyze performance, proxy stability, collapse, and embedding structure. KDPH consistently improves retrieval and robustness, especially under semantic complexity and compressed codes.

  • Experimental Settings: KDPH is evaluated on MIRFLICKR-25K, NUS-WIDE, and MS COCO against nine state-of-the-art deep cross-modal hashing methods.The experiments use standard multi-label benchmarks and compare KDPH with methods including DCPH, DSPH, and nivMF.
  • Main Results: KDPH outperforms all state-of-the-art baselines across hash code lengths, exceeding nivMF by over 7.92% on MS COCO for I2T @ 16 bits.The comparison attributes the advantage to anisotropic Kent distributions and their ability to model elliptical semantic variance.
  • Main Results: Approximately 6.07% improvement on MS COCO over the second-best method demonstrates KDPH’s bit-efficiency at 16 bits.KDPH maintains discriminative power in highly compressed Hamming spaces and balances visual and textual modalities.
  • Proxy Oscillation and Stability: In high-co-occurrence classes, the baseline exhibits severe proxy oscillation and jagged precision curves, whereas KDPH mitigates chaotic oscillations.For low-co-occurrence classes, both methods converge stably, indicating that rigid point proxies suffice for simpler manifolds.
  • Proxy Collapse: A 41.1% improvement in five-label scenarios shows KDPH’s stronger prevention of proxy collapse under high semantic complexity.The analysis tracks subordinate-label performance and associates the improvement with preserving semantic variance for tail concepts.
  • Ablation Study: On MS COCO, mAP reaches 0.7693 with isotropic vMF, 0.7615 with rigid point proxies, and higher performance with the anisotropic Kent model.The ablation attributes the Kent model’s advantage to capturing anisotropic semantic conflicts and recalibrating semantic interactions.
  • Embedding Visualization: t-SNE shows baseline proxies for co-occurring classes collapsing onto a central anchor, while KDPH produces coherent radial clusters and clearer semantic separation.The visualization uses the complex Person category on MS COCO and contrasts DSPH with KDPH.

5 Conclusion

The conclusion presents KDPH as a solution to gradient contention in multi-label cross-modal hashing. By replacing point proxies with anisotropic Kent distributions, it reports improved performance across three benchmarks.

  • 5 Conclusion: KDPH addresses gradient contention by replacing point proxies with anisotropic Kent distributions that adjust directional variance.This preserves stable semantic centroids while accommodating diverse label correlations.
  • 5 Conclusion: Experiments on three benchmarks show that KDPH outperforms state-of-the-art methods.

A Extended Analysis for Main Results: Semantic Topological Evolution

The extended topological analysis evaluates semantic organization by grouping MS COCO categories into super-groups and measuring intra- and inter-group similarity. KDPH yields higher separation through stronger cohesion and lower cross-group similarity.

  • Semantic Topological Evolution: The analysis aggregates 80 MS COCO categories into 12 semantic super-groups to evaluate semantic disentanglement.It measures Intra-Group Similarity and Inter-Group Similarity to characterize organization of the learned proxy space.
  • Semantic Topological Evolution: KDPH achieves higher Separation Scores across nearly all categories than the baseline.The Separation Score is defined as the difference between Intra- and Inter-group similarities.
  • Semantic Topological Evolution: KDPH improves semantic structure through reduced Inter-Group similarity and increased Intra-Group cohesion.Vehicle and outdoor groups show particularly strong separation improvements.

B Extended Analysis for Proxy Oscillation and Stability

The appendix broadens the training-dynamics analysis across twelve MS COCO classes stratified by semantic co-occurrence. KDPH remains stable in complex categories while preserving the baseline’s adequate behavior in simpler ones.

  • Extended Stability Analysis: The appendix evaluates twelve additional MS COCO classes divided into high- and low-co-occurrence groups.This tests KDPH across varying degrees of semantic complexity.
  • High Co-occurrence Categories: For high-co-occurrence categories, baseline Proxy Flip Rate fluctuates between 5% and 20% late in training, alongside jagged and suboptimal precision curves.The categories include fork, microwave, potted plant, mouse, keyboard, and laptop.
  • Low Co-occurrence Categories: For low-co-occurrence categories, KDPH mirrors the baseline’s stability while maintaining low flip rates and high precision.The categories include sandwich, boat, cow, banana, airplane, and elephant.

B.3 Quantitative Correlation between Proxy Chaos and Performance

The Chaos Index summarizes proxy-trajectory instability, and high chaos is associated with degraded retrieval performance in complex multi-label settings. KDPH reduces chaos and preserves comparable stability when semantic interference is low.

  • Chaos Index: The Chaos Index aggregates path efficiency, directional consistency, movement entropy, and autocorrelation to quantify proxy-learning trajectory stability.It is normalized between 0 and 1.
  • High-co-occurrence classes: High-co-occurrence classes (Avg ACL ≈4.1) show an inverse relationship between Chaos Index and Precision@50.The baseline has elevated chaos, while KDPH produces a more directed optimization path and higher Precision@50.
  • High-co-occurrence classes: KDPH’s lower Chaos Index is accompanied by higher path efficiency and substantially improved retrieval performance in complex classes.The comparison links stabilization of proxy trajectories with superior Precision@50 scores.
  • Low-co-occurrence classes: In low-co-occurrence classes (Avg ACL ≈1.1), both methods maintain low Chaos Indices and high, comparable Autocorrelation scores.Overlapping performance curves indicate that KDPH retains stable learning when gradient contention is absent.

C Extended Analysis for Quantifying Proxy Collapse via GTSD

GTSD measures the confidence gap between the best- and worst-recognized ground-truth labels, providing a fine-grained view of proxy collapse. As label density increases, KDPH maintains tighter score distributions and reduces divergence relative to the baseline.

  • GTSD definition: GTSD quantifies the disparity between a sample’s highest- and lowest-confidence ground-truth labels.High GTSD indicates dominance by head labels, whereas low GTSD indicates greater intra-sample equity.
  • Evaluation setting: The analysis uses MS COCO and examines how GTSD changes as the number of labels increases from 3 to 8.This provides a complexity-stratified evaluation of proxy collapse.
  • Complexity scaling: 14.8%: KDPH reduces GTSD for samples with 8 labels relative to the baseline.The reported benefit increases with semantic complexity and is attributed to using Kent shape parameters to cover multiple-label semantics.

D.1 Impact of Loss Coefficients

The loss coefficients control distinct aspects of KDPH’s optimization, and performance is sensitive to their values. The reported settings balance irrelevance regularization against preservation of positive semantic and cross-modal relationships.

  • Irrelevance Regularization (ϵ): mAP improves as ϵ increases from 1 to 40, but excessive regularization at ϵ = 100 causes sharp performance degradation.The paper attributes the decline to suppression of positive semantic correlations by an overly dominant regularization gradient.
  • Irrelevance Regularization (ϵ): ϵ = 40 provides the reported balance for Multi-Modal Irrelevance Regularization.The objective penalizes high similarity between semantically disjoint pairs.
  • Cross-Modal Alignment (ζ): ζ = 0.3 yields peak performance for the cross-modal alignment loss.An overly large ζ may prioritize instance-level matching over class-level semantic structure.

D.2 Impact of Anisotropic Geometry (Naxes)

KDPH’s anisotropic geometry requires an appropriate number of principal axes: too few limit semantic flexibility, while too many destabilize accuracy. The training analyses further show that its benefits are concentrated in complex, high-co-occurrence scenarios without harming simple cases.

  • Principal-axis selection: Naxes = 8 achieves the best reported result for modeling the Kent shape parameter on 64-bit codes.This corresponds to K/8 for 64-bit codes, and performance does not increase monotonically with Naxes.
  • Principal-axis selection: Naxes = 4 provides insufficient degrees of freedom to capture complex, multidirectional semantic variances.The limitation arises from using too few shaping parameters.
  • Principal-axis selection: Naxes values beyond 16 substantially reduce accuracy through dimensionality and overfitting effects.The paper identifies Naxes = K/8 as a parsimonious configuration for anisotropic semantic manifolds.
  • Training dynamics: On high-co-occurrence classes, KDPH achieves rapid stabilization and superior convergence compared with the baseline’s chaotic oscillation.The accompanying performance analysis reports lower chaos and higher retrieval precision for KDPH.
  • Training dynamics: On low-co-occurrence classes, both methods maintain high stability, showing that KDPH does not negatively affect simple semantic scenarios.Both methods exhibit low Chaos Indices and high Autocorrelation in these settings.
  • Complexity-dependent benefit: In dense multi-label settings, KDPH reduces GTSD by up to 14.8%, indicating stronger gains as semantic complexity increases.The baseline’s divergence widens with label count, whereas KDPH maintains lower divergence.
Loading 2608.24010v1…