PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models

Hayeon Kim, Ji Ha Jang, Junghun James Kim, Se Young Chun

arXiv:2603.22042v3cs.CVcs.AI

TL;DR

VLMs and existing hyperbolic approaches do not fully capture hierarchical part-to-whole structure when parts differ in semantic representativeness. UNCHA models that representativeness with hyperbolic uncertainty, integrates it into contrastive and entailment learning, and achieves state-of-the-art performance across several benchmarks. The method produces more accurate part-whole ordering for compositional vision-language understanding.

  • Problem

    VLM embeddings have limited ability to capture hierarchical part-to-whole relationships, while existing hyperbolic approaches do not model varying part representativeness in complex multi-object scenes.

  • Method

    UNCHA represents part-to-whole semantic representativeness with hyperbolic uncertainty and incorporates it into uncertainty-guided contrastive and entailment losses with entropy-based regularization.

  • Results

    UNCHA achieves state-of-the-art performance on zero-shot classification, retrieval, and multi-label benchmarks, including complex multi-object scenes.

  • Takeaways & Limitations

    UNCHA learns more accurate part-whole orderings and supports compositional understanding by calibrating uncertainty to semantic representativeness.

  • Takeaways & Limitations

    The entailment loss becomes zero once q is fully contained in the cone, preventing further fine-grained alignment; the formulation also parameterizes only the space component in the tangent space.

Abstract

from arXiv · show

While Vision-Language Models (VLMs) have achieved remarkable performance, their Euclidean embeddings remain limited in capturing hierarchical relationships such as part-to-whole or parent-child structures, and often face challenges in multi-object compositional scenarios. Hyperbolic VLMs mitigate this issue by better preserving hierarchical structures and modeling part-whole relations (i.e., whole scene and its part images) through entailment. However, existing approaches do not model that each part has a different level of semantic representativeness to the whole. We propose UNcertainty-guided Compositional Hyperbolic Alignment (UNCHA) for enhancing hyperbolic VLMs. UNCHA models part-to-whole semantic representativeness with hyperbolic uncertainty, by assigning lower uncertainty to more representative parts and higher uncertainty to less representative ones for the whole scene. This representativeness is then incorporated into the contrastive objective with uncertainty-guided weights. Finally, the uncertainty is further calibrated with an entailment loss regularized by entropy-based term. With the proposed losses, UNCHA learns hyperbolic embeddings with more accurate part-whole ordering, capturing the underlying compositional structure in an image and improving its understanding of complex multi-object scenes. UNCHA achieves state-of-the-art performance on zero-shot classification, retrieval, and multi-label classification benchmarks. Our code and models are available at: https://github.com/jeeit17/UNCHA.git.

1. Introduction

Existing VLMs struggle to preserve hierarchical and compositional relationships in complex multi-object scenes. UNCHA addresses this by modeling part-to-whole representativeness as uncertainty and incorporating it into hyperbolic alignment, achieving superior performance across diverse benchmarks.

  • Motivation: Euclidean VLM embeddings can distort hierarchical structures and struggle with compositional relations in complex multi-object scenes.These limitations affect models such as CLIP, ALIGN, and ALBEF.
  • Motivation: Hyperbolic geometry offers an efficient foundation for embedding hierarchical and fine-grained relational structures.Its negative curvature and exponential volume growth support hierarchy-aware vision-language learning.
  • Research gap: Existing hyperbolic approaches do not model that different parts have different levels of semantic representativeness to the whole scene.Treating all parts equally can degrade multi-object alignment and waste embedding-space capacity.
  • Method: UNCHA assigns lower uncertainty to more representative parts and higher uncertainty to less representative parts, then uses uncertainty-guided contrastive and entailment losses.The entailment loss includes entropy-based regularization to calibrate uncertainty and promote richer use of hyperbolic space.
  • Results: UNCHA outperforms prior hyperbolic VLMs across zero-shot image classification, retrieval, compositional, and multi-object benchmarks.The results validate its modeling of part-to-whole representativeness and compositional understanding.

2. Related Works

Vision-language models benefit from hyperbolic geometry because Euclidean embeddings poorly represent hierarchies and complex multi-object relations. The related work motivates explicitly modeling varying part-to-whole representativeness to avoid structural collapse and limited distinction among parts.

  • Vision-language models: VLMs align image and text representations effectively but Euclidean space makes hierarchical relationships difficult to represent.CLIP also exhibits biases in complex multi-object scenes, including emphasis on first-mentioned or larger objects.
  • Part-to-whole representativeness: Without distinguishing varying part-to-whole representativeness, hyperbolic embeddings can lose structural separation and collapse toward a narrow region.This limits compositional understanding in multi-object settings.
  • Hyperbolic representation learning: Hyperbolic space provides exponential volume growth and tree-like geometry for near distortion-free hierarchical embeddings.Prior work applies it to text, images, graphs, and vision-language representation learning.
  • Uncertainty modeling: Hyperbolic radius has been studied as a proxy for concept abstractness or uncertainty across pixel classification, image retrieval, and multimodal language understanding.However, prior approaches do not differentiate the varying strengths of relationships among parts.

3. Method

UNCHA extends hyperbolic vision-language alignment by representing part-to-whole semantic representativeness as uncertainty and using it in contrastive and entailment learning. Its entailment formulation preserves fine-grained alignment while calibrating uncertainty across parts.

  • 3.1. Preliminaries: Hyperbolic VLMs represent hierarchical relations through entailment, with abstract concepts nearer the origin and specific concepts farther away.MERU models inter-modal image-text entailment, while HyCoCLIP adds intra-modal entailment between parts and whole scenes.
  • 3.2.1. Uncertainty model of semantic representativeness: UNCHA uses hyperbolic radius, implemented through a smooth monotonic transformation of Euclidean norm, as a differentiable uncertainty measure.Points near the origin have higher uncertainty, while radius is inversely related to uncertainty.
  • 3.2.2. Uncertainty-guided contrastive loss: UNCHA adds local contrastive alignment and assigns separate temperatures to global, local, and global-local objectives for different information levels.The local loss aligns part images with corresponding text, while separate temperatures model global, local, and global-local relationships.
  • 3.2.2. Uncertainty-guided contrastive loss: Uncertainty-guided contrastive learning adaptively scales temperatures according to each part's estimated uncertainty, reducing the contribution of less representative parts.Higher uncertainty produces a larger temperature and a smaller contrastive-loss contribution.
  • 3.2.3. Entailment loss for uncertainty calibration: The calibrated entailment loss adds an angular term to the original cone-based objective, preserving a small gradient after entailment is satisfied.This Leaky-ReLU-like relaxation encourages fine-grained alignment while maintaining smooth optimization continuity.

4. Experiments

Experiments evaluate UNCHA on zero-shot classification, retrieval, hierarchical metrics, part-level alignment, multi-label classification, and multi-object representation. Across these evaluations, UNCHA consistently outperforms prior approaches and produces more structured part–whole hyperbolic embeddings.

  • Zero-shot classification: UNCHA consistently outperforms prior approaches across 16 zero-shot image-classification benchmarks and two vision-encoder sizes.Top-1 accuracy is reported for all datasets, with ViT-S and ViT-B configurations evaluated.
  • Retrieval and hierarchical classification: UNCHA achieves superior performance across zero-shot retrieval and hierarchy-aware ImageNet metrics.The hierarchy-aware metrics assess preservation of class-label structure within the hyperbolic embedding space.
  • Part-level alignment: UNCHA achieves substantial gains on part-level alignment with hard negatives, testing fine-grained matching and ordering between corresponding and non-matching captions.The benchmark uses densely annotated image parts and region-specific hard negatives.
  • Ablation study: Removing uncertainty-aware contrastive scaling, uncertainty calibration, or entropy regularization consistently reduces performance.The ablation study evaluates the contribution of each component on classification and retrieval benchmarks.
  • Multi-object evaluation: UNCHA outperforms all baselines across multi-label classification and multi-object representation benchmarks.ComCo contains realistic object compositions, while SimCo contains synthetic scenes with diverse geometric shapes; evaluation uses classification mAP.
  • Embedding analysis: UNCHA produces more dispersed, distinctive, and semantically structured hyperbolic embeddings, separating part representations from whole-scene representations.Compared with HyCoCLIP, part embeddings lie closer to the origin than whole-scene embeddings, improving part–whole separation.

5. Conclusion

UNCHA integrates hyperbolic uncertainty as a measure of part-to-whole semantic representativeness into contrastive and entailment learning. The paper reports state-of-the-art performance across zero-shot, retrieval, multi-label, and complex multi-object benchmarks.

  • Conclusion: UNCHA integrates part-to-whole representativeness, quantified as hyperbolic uncertainty, into contrastive and entailment learning.Uncertainty is calibrated using part-to-whole entailment relationships and entropy-based regularization.
  • Conclusion: Extensive experiments demonstrate state-of-the-art performance across zero-shot classification, retrieval, multi-label benchmarks, and complex multi-object scenes.The conclusion highlights uncertainty-guided alignment as important for compositional understanding in vision-language learning.

S.1.1. Model architecture

The model uses a CLIP-style text Transformer and a Vision Transformer for images, with two tested vision-encoder capacity configurations.

  • Text encoder: The text encoder is a 12-layer, 512-dimensional Transformer with a maximum input length of 77 tokens and a 49,408-token vocabulary.Its design follows CLIP.
  • Image encoder: The image encoder is a Vision Transformer using patch size 16, evaluated in ViT-S and ViT-B configurations.These architectural choices are consistent with prior hyperbolic vision-language models.
  • Training setup: Training applies the same image augmentations as OpenCLIP, including random cropping.The passage introduces the augmentation setup as part of the model configuration.

S.1.2. Model initialization

Model initialization uses learnable curvature, representation scaling, and temperature parameters, together with fixed entailment and regularization coefficients.

  • Curvature and scaling: Lorentz-space curvature is initialized at κ = 1.0, learned within [0.1, 10.0], and converges to κ = 0.1.The range is used for numerical stability, and the final value matches prior hyperbolic methods.
  • Temperature parameters: Learnable temperatures are initialized as τgl = 0.06 and τl = τg = 0.07, with all temperature values clipped at 0.01 minimum.These parameters control global-local, local, and global logit scales.

S.1.3. Optimizer and hardware

Training uses 500K steps on four A100 GPUs with AdamW optimization, cosine learning-rate decay, and linear warm-up.

  • 500K training steps use four A100 GPUs and a batch size of 768.
  • AdamW uses β1 = 0.9, β2 = 0.98, and weight decay of 0.2, excluding selected learnable parameters from decay.Excluded parameters include temperature parameters, curvature, and scaling factors cimg and ctxt.
  • A cosine learning-rate scheduler reaches 5 × 10^-4 after a 4k-step linear warm-up.

S.2.1. Training details on other models

The supplementary experiments evaluate UNCHA across part-level alignment, multi-object representation, segmentation, bounding-box classification, embedding structure, and loss-component behavior. Across these settings, the reported results show strong or superior performance, improved separation, and more effective compositional representations.

  • Part-level alignment: UNCHA achieves the best or second-best performance across nearly every part-level alignment setting, especially challenging All Pick5-SCM and All Pick5-Neg cases.
  • Multi-object representation: UNCHA consistently achieves superior multi-object representation across object counts and datasets, distinguishing individual objects in complex scenes.
  • Zero-shot semantic segmentation: UNCHA produces strong zero-shot semantic segmentation performance with more coherent region assignments and reliable separation of multiple object instances.
  • Bounding-box classification: 32.89% is the largest reported Top-1 accuracy margin for UNCHA across COCO, LVIS, and OpenImages bounding-box classification.The results also report consistently superior performance across all three datasets and Top-1 and Top-5 metrics.
  • Hyperbolic embedding analysis: UNCHA embeddings are farther from the origin, more widely dispersed, and less overlapping between part and whole representations than HyCoCLIP embeddings.The analysis interprets this distribution as more effective use of available hyperbolic volume.
  • Hyperbolic embedding analysis: Part-level objects distribute across different hyperbolic regions according to uncertainty, while the global bedroom concept lies farther from the origin.
  • Loss-component analysis: The uncertainty-guided contrastive loss improves relational alignment, while uncertainty calibration prevents embedding contraction and preserves a well-distributed space.Removing calibration concentrates embeddings near approximately 0.57R, whereas removing contrastive guidance weakens relational alignment.
  • Distribution analysis: MMD, W1, and W2 rise sharply early in training and gradually stabilize, quantitatively confirming increasing divergence between part and whole image distributions.
Loading 2603.22042v3…