PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat, Seifeldin Elkerdany, Weixing Wang, Gerard de Melo

arXiv:2608.29313v1cs.CVcs.AI

TL;DR

Global Euclidean VLMs and hyperbolic or query-conditioned methods separately address alignment and hierarchy, leaving their combination incomplete. Hyper3-CLIP joins query-conditioned visual pooling with hyperbolic hierarchy-aware training, improving retrieval and multi-label classification while remaining competitive on hierarchy metrics. Its main scope boundary is that the parent-child specificity ordering is an explicit semantic assumption.

  • Problem

    Existing VLMs, hyperbolic models, and query-conditioned methods do not jointly provide fine-grained visual alignment and hierarchical geometric structure.

  • Method

    Hyper3-CLIP builds text-derived caption, sentence, localized, and phrase queries whose pooled visual nodes are trained in hyperbolic space with contrastive and entailment objectives.

  • Results

    Hyper3-CLIP improves COCO and Flickr R@5/R@10 retrieval and VOC/COCO multi-label classification while remaining competitive on hierarchy metrics.

  • Takeaways & Limitations

    Query-conditioned visual representations provide a supported way to inject fine-grained, multi-granular structure into hyperbolic VLM training.

  • Takeaways & Limitations

    The parent-child text ordering assumes that a parent query is more specific because it contains additional contextual information.

Abstract

from arXiv · show

CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.

1 Introduction

Hyper3-CLIP addresses the limited hierarchical structure of global Euclidean VLM representations by combining hyperbolic learning with query-conditioned visual alignment. It integrates caption, sentence, localized, and phrase queries into hierarchy-aware training while retaining grounded supervision.

  • Most VLMs use a single Euclidean image-caption embedding, limiting representation of visual part-whole and language parent-child structure.
  • Hyperbolic VLMs model hierarchy through entailment, while grounded methods use region-level or query-driven representations for fine-grained alignment.
  • Hyper3-CLIP combines query-conditioned visual pooling with hyperbolic hierarchical representation learning.Textual hierarchies use captions, sentence fragments, localized descriptions, and extracted phrases; each query attends to image patches and produces a hyperbolic visual representation.
  • Hyperbolic hierarchical representations combined with query-conditioned visual alignment improve image-text retrieval and hierarchy-sensitive image-caption alignment.
  • The contribution integrates existing query decomposition, text-conditioned pooling, and grounded entailment components through parent-linked hyperbolic query nodes.Query-conditioned pooling is used only during training, leaving inference as a standard dual encoder without additional pooling computation.

2 Method

Hyper3-CLIP extends grounded hyperbolic image-text training with a text-derived query hierarchy and query-conditioned visual nodes. These nodes are trained with weighted entailment relations alongside inherited global, local, and global-local alignment objectives.

  • Each training sample contains a full image-caption pair and grounded image-box/text-box pairs.The text boxes are typically noun or phrase annotations corresponding to localized image crops.
  • The inherited objective provides contrastive and entailment losses over full-image and localized nodes, while Hyper3-CLIP adds hierarchical query construction, query-conditioned pooling, and query-level entailment.
  • 2.1 Hierarchical Query Construction: The query hierarchy uses the caption as root, with sentence fragments, localized text boxes, and extracted phrases as additional query types.Each query stores a parent link and reliability weight; the caption root receives zero query weight because the global objective already supervises it.
  • 2.2 Query-Conditioned Visual Pooling: Each text query attends over full-image patch tokens to produce a separate query-specific visual node.The resulting visual representation is projected and lifted into hyperbolic space alongside the corresponding text representation; pooling is training-only.
  • 2.3 Training Objective: The training objective retains inherited global, local, and global-local contrastive alignments and adds weighted hyperbolic query entailment terms.For non-root queries, the added relations connect query visual nodes to query text, the full image to query visual nodes, and parent text queries to child queries.
  • 2.3 Training Objective: The full image and query-conditioned visual node are ordered by semantic specificity, with the full image treated as more specific because it retains complete-scene context.The text-side parent-child ordering likewise treats the parent as more specific because it contains additional contextual information.

3 Experiments

The experiments evaluate Hyper3-CLIP on retrieval, hierarchy-sensitive classification, and multi-label recognition using zero-shot protocols and lightweight text-derived query hierarchies. Results show improved retrieval and multi-label performance, while hierarchy metrics remain competitive and prompt choice materially affects classification.

  • Experimental setup: Hyper3-CLIP constructs query hierarchies from captions, sentence queries, text-box descriptions, and extracted phrases using rule-based string processing.Candidates are added in a fixed order, normalized, deduplicated case-insensitively, and filtered by a three-character minimum.
  • Zero-shot retrieval and hierarchy: Table 2 evaluates zero-shot image-text retrieval on COCO and Flickr30K alongside ImageNet hierarchy metrics in the ViT-B/16 setting.Retrieval reports paired R@5/R@10 columns for text and image retrieval; TIE and LCA are hierarchy metrics where lower is better.
  • Zero-shot retrieval and hierarchy: Hyper3-CLIP improves over UNCHA across all eight COCO/Flickr retrieval settings while remaining competitive on ImageNet hierarchy metrics.It does not improve TIE or LCA.
  • Zero-shot classification: Hyper3-CLIP reaches 47.84 mean-per-class accuracy on the 16-dataset zero-shot classification benchmark versus 47.41 for UNCHA under the stated prompt protocol.Gains concentrate on several datasets, while weaknesses remain for Country211, DTD, and EuroSAT.
  • Zero-shot multi-label classification: Hyper3-CLIP improves over UNCHA by 2.69 mAP on VOC and 3.35 mAP on COCO under a shared evaluator.The comparison avoids mixing UNCHA values reported under a different protocol.

4 Ablation Study

The ablations test which query-conditioned entailment relations and training controls drive hierarchy-sensitive alignment while preserving retrieval. Additional analyses examine prompt sensitivity and the localized-part budget used during training.

  • Query-Conditioned Objective Components: The full objective improves caption-hierarchy AP by 3.64 points over the no-query baseline while leaving retrieval essentially unchanged.The text parent-to-query relation recovers 73% of the full AP gain, or 2.65/3.64 points.
  • Query-Conditioned Objective Components: The jointly trained objective improves AP by 0.99 over the text-only variant, although the query image-text and visual whole-to-query terms do not improve AP individually.Their contribution appears only when the relations are trained together.
  • Query-Conditioned Objective Components: Reversing the visual hierarchy reduces AP from 79.85 to 76.06, slightly below the no-query baseline of 76.21, while average retrieval remains near baseline.This control tests whether the hierarchy gain depends on the proposed I⪯Iq direction.
  • Prompt Sensitivity and Zero-Shot Protocol: Hyper3-CLIP collapses under the official prompt template on three prompt-sensitive datasets but recovers with the shared Photo prompt.The audit evaluates Food-101, CUB, and Flowers102 using both prompt regimes while keeping the other 13 dataset columns unchanged.
  • Parts per Image: Five localized parts per image yields the best average R@1, retaining 99.38% of part instances and leaving 99.20% of examples untruncated.Using all parts marginally improves Avg R@10 but does not improve Avg R@1.

5 Conclusion

Hyper3-CLIP introduces query-conditioned visual nodes into grounded hyperbolic image-text training, combining textual query hierarchies with hierarchy-aware entailment relations. Evaluations support the design across retrieval, classification, and hierarchy-sensitive alignment, while analyses identify prompt wording and localized-part budgets as practical considerations.

  • 5 Conclusion: Hyper3-CLIP builds query hierarchies from captions, fragments, localized descriptions, and phrases, then embeds query-conditioned visual nodes alongside query text in hyperbolic space.Each query pools full-image patch tokens through query-conditioned cross-attention.
  • 5 Conclusion: The model supervises visual and textual nodes with visual-text, image-query, and parent-child entailment relations while retaining global, local, and global-local contrastive alignment.Query-conditioned pooling is used only during training, preserving standard dual-encoder inference cost.
  • 5 Conclusion: Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, achieves strong zero-shot classification and multi-label results, and remains competitive on hierarchy metrics.Controlled ablations report higher caption-hierarchy entailment AP without reducing average retrieval.
  • 5 Conclusion: Prompt wording can sharply affect zero-shot classification, while a moderate localized-part cap retains nearly all grounded supervision and avoids long-tail training overhead.These analyses provide practical diagnostics for the evaluated setting.
  • 5 Conclusion: The results suggest that textual query hierarchies can inject fine-grained, multi-granular structure into hyperbolic vision-language training.Future work includes inference-time query-conditioned nodes, richer hierarchies, and broader compositional evaluations.

A Base Hyperbolic Objective Details

The appendix summarizes the geometric and objective components inherited by Hyper3-CLIP. The main method adds query-conditioned visual nodes to these existing components.

  • A Base Hyperbolic Objective Details: The appendix provides geometric background inherited by Hyper3-CLIP.
  • A Base Hyperbolic Objective Details: The appendix also provides objective background inherited by Hyper3-CLIP.
  • A Base Hyperbolic Objective Details: Hyper3-CLIP adds query-conditioned visual nodes to the inherited components.

A.1 Hyperbolic Space and the Lorentz Model

Hyper3-CLIP represents image and text embeddings on the Lorentz manifold, using hyperbolic distance for contrastive retrieval and cone penalties for entailment relations.

  • A.1 Hyperbolic Space and the Lorentz Model: Hyperbolic space’s exponential volume growth makes it suitable for tree-like or part-whole structure.
  • A.1 Hyperbolic Space and the Lorentz Model: The Lorentz model represents embeddings x on a manifold with curvature magnitude c > 0 and constraint ⟨x, x⟩L = −1/c, x0 > 0.
  • A.1 Hyperbolic Space and the Lorentz Model: Image and text encoder features are projected onto the Lorentz manifold, and contrastive retrieval uses −dL(x, y) as similarity.
  • A.1 Hyperbolic Space and the Lorentz Model: Hyperbolic entailment uses cone penalties E(a ⪯b) to penalize a specific node when it falls outside the cone rooted at a more general node.

A.2 Inherited HyCoCLIP and UNCHA Losses

The inherited objective combines directional contrastive alignment with grounded image-box and text-box supervision. UNCHA further calibrates part-whole terms using uncertainty, while Hyper3-CLIP adds query-conditioned entailment terms.

  • A.2 Inherited HyCoCLIP and UNCHA Losses: The directional contrastive loss matches paired embeddings of specified types and treats other batch elements as negatives.
  • A.2 Inherited HyCoCLIP and UNCHA Losses: HyCoCLIP extends global image-caption alignment with grounded image-box and text-box supervision.
  • A.2 Inherited HyCoCLIP and UNCHA Losses: The inherited entailment objective covers cross-modal global and local relations plus within-modality part-whole relations.The listed relations are I ⪯T, Ibox ⪯T box, I ⪯Ibox, and T ⪯T box.
  • A.2 Inherited HyCoCLIP and UNCHA Losses: UNCHA reduces the influence of ambiguous or weakly representative parts using uncertainty derived from hyperbolic radius.
  • A.2 Inherited HyCoCLIP and UNCHA Losses: Hyper3-CLIP retains the base objective while adding query-conditioned entailment terms.
Loading 2608.29313v1…