Hyperbolic Contrastive Learning for Visual Representations beyond Objects

• The object-centric scene hierarchy places visually similar objects at its roots and scenes containing them as descendants, reflecting that object classes can occur in different scenes.Source: §1 · p. 2
• Co-occurring scene-object pairs are positives and objects absent from a scene are negatives for the auxiliary hyperbolic objective, addressing the combinatorial growth of possible scenes.The paper motivates hyperbolic space by noting that infinite trees can be embedded without distortion in 2D hyperbolic space.Source: §1 · p. 2
• Joint training preserves object clustering while giving scene representations a hierarchical structure, and representation properties support zero-shot vision tasks.The reported zero-shot tasks include label uncertainty quantification and out-of-context object detection.Source: §1 · p. 2
• The proposed hyperbolic contrastive loss organizes scene representations around an object-centric hierarchy using positive and negative pairs sampled from it.Source: §1 · p. 2
• HCL representations transfer better than vanilla contrastive representations across object detection, semantic segmentation, and linear classification.Source: §1 · p. 2

2.2. Representation Learning beyond Objects

• Because object compositions yield exponentially many possible scenes, the method uses hyperbolic space to represent hierarchical structure.The paper contrasts this with Euclidean space, which may require arbitrarily many dimensions for faithful embedding, and notes that trees can be embedded in 2D hyperbolic space without distortion.Source: §2.1 · p. 2

2.1. Object-Centric Scene Hierarchy

• The hierarchy links images, object boxes, and multi-object scene regions across the dataset, with edges defined by containment.Scene regions are partial image areas containing multiple objects; natural scene images are designated as leaf nodes.Source: §2.1 · pp. 2–3

2.2.1 Hyperbolic Space

• The Poincaré ball supplies the hyperbolic model, where points near its boundary reproduce tree paths through a common ancestor.The method maps Euclidean network outputs into the ball with the exponential map and clips the input norm to avoid numerical issues.Source: §2.2.1 · p. 3

2.2.2 Hyperbolic Contrastive Learning

• HCL learns object representations with a Euclidean contrastive loss and scene representations with a hyperbolic loss over hierarchy-based positive and negative pairs.In practice, it pairs a scene region with an object inside it and uses other objects outside that region as negatives; the hyperbolic branch replaces the cross-entropy inner product with negative hyperbolic distance.Source: §2.2.2 · pp. 3–4, Figure 2 · p. 4
• The combined objective weights the hyperbolic and Euclidean losses using λ, which controls their trade-off.The method motivates placing objects, as nodes near the hierarchy root, close to the center when minimizing positive-pair distances.Source: §2.2.2 · p. 4

3.1. Implementation Details

• Pretraining uses COCO and OpenImages with three contrastive-learning baselines, and evaluates linear classification plus COCO object detection and semantic segmentation.Object boxes come from ground truth or selective search; the models are evaluated on VOC, ImageNet-100, ImageNet-1k, and COCO.Source: §3.1 · pp. 4–5

3.2. Main Results

• HCL improves ImageNet linear accuracy by 0.94 percentage points and VOC linear classification by 1.61 mAP when pretrained on OpenImages.Similar improvements occur with COCO pretraining and with either ground-truth or selective-search boxes; gains are generally larger on OpenImages than COCO.Source: §3.2 · pp. 5–6, Table 2 · p. 5
• HCL consistently improves COCO object detection and semantic segmentation across MoCo-v2, Dense-CL, and ORL, and improves linear classification on VOC and ImageNet datasets.Source: Table 1 · p. 5, §3.2 · p. 5, Table 2 · p. 5

3.3.1 Label Uncertainty Quantification

• On ImageNet-ReaL, HCL representation norms rank images by label uncertainty comparably to supervised ResNet-50 entropy, while combining the rankings reaches 0.717 NDCG.The norm and entropy provide complementary signals; smaller norms tend to indicate single-object images, while larger norms tend to indicate scenes.Source: §3.3.1 · p. 6, Figure 3 · p. 6, Table 3 · p. 6
• On COCO validation, HCL norms achieve much better NDCG than supervised ResNet-50 entropy and can be applied to new data without further training.The authors suggest the signal could help guide label reassessment or provide an extra signal for model training.Source: §3.3.1 · p. 6

3.3.2 Out-of-Context Detection

• For out-of-context object ranking, HCL reaches 0.61 mAP, compared with 0.52 for MoCo similarity and 0.44 for random ranking.The evaluation uses object and masked-scene representations from SUN09 and ranks objects by their hyperbolic distance to the scene.Source: §3.3.2 · pp. 6–7, Figure 4 · p. 7

Ablation setup

• The ablations train on an OpenImages subset and report ImageNet-100 linear-evaluation top-1 accuracy.Source: §4 · p. 7

Similarity measure and the center of the scene-object hierarchy

• Replacing negative hyperbolic distance with Euclidean similarity impairs downstream accuracy, and a scene-centric hierarchy yields substantially lower accuracy.The Euclidean-similarity variant performs worse than the baseline without a scene-object loss.Source: §4 · p. 7

Trade-off between the Euclidean and hyperbolic losses

• A small λ = 0.01 gives marginal improvement, while larger values can destabilize or stall RSGD training as strong early hyperbolic gradients push representations toward the boundary.Replacing RSGD with SGD permits larger λ after increasing ε from 1e−5 to 1e−1, but SGD performs worse than RSGD.Source: §4 · p. 7

5. Related Work

• Hyperbolic space has been applied to hierarchy-related settings including language, graphs, and vision, with generality, uncertainty, and compositionality motivating its use.Prior vision applications include image classification, segmentation, zero-/few-shot learning, action recognition, and video prediction.Source: §5 · pp. 7–8
• Self-supervised approaches to multi-object images include dense, pixel-level, and object-level methods, while this paper targets representations of the scene-object hierarchy.The cited approaches include Dense-CL, DetCon, PixContrast, Pixel-to-Pixel Contrast, Self-EMD, and ORL.Source: §5 · p. 8

6. Conclusion

• HCL learns visual representations for objects and scenes in a shared space using a hyperbolic contrastive objective based on an object-centric scene hierarchy.The paper evaluates it on image classification, object detection, and semantic segmentation benchmarks.Source: §6 · p. 8
• The learned representations are demonstrated in zero-shot settings including out-of-context object detection and quantifying label uncertainty in datasets.Source: §6 · p. 8
• The authors hope the work encourages holistic visual representation spaces and further attention to non-Euclidean representation learning.Source: §6 · p. 9

A. Experiment setups

• Selective Search proposals are filtered by a 96-pixel minimum scale, maximum IoU of 0.5, and aspect ratio from 1/3 to 3; up to 100 proposals are generated per image.One proposal is randomly selected as the object image.Source: A. Experiment setups · p. 9
• Pretraining uses full OpenImages with about 1.9 million images and a balanced subset requiring at least two classes and 900 instances per class.The subset averages 12 objects per image.Source: A. Experiment setups · p. 9
• Object boxes of width × height ≤56 × 56 are dropped; Euclidean-branch boxes ≤256 × 256 may be expanded and jittered, while hyperbolic-branch boxes are filtered and resized to ≤224 × 224.Scene crops merge the selected object box with another 1 to 5 sampled boxes.Source: A. Experiment setups · p. 9
• Training follows MoCo-v2 protocols, uses a base learning rate of 0.3, and runs for 200 epochs on COCO and the OpenImages subset or 75 on full OpenImages.HCL takes 0.757 sec/iter versus MoCo's 0.616 sec/iter, with one additional scene-representation forward pass as the only stated overhead.Source: A. Experiment setups · p. 9

B.1. Robustness under Corruption.

• 1.9 mCE improvement over the baseline without HCL loss was reported for HCL trained on OpenImages and lineval on ImageNet.Source: §B.1 · p. 9

B.2. Fine-grained class classification

• The model provides little performance improvement on fine-grained classification datasets.The authors suggest similar scene contexts across classes may explain the limited gain.Source: §B.2 · p. 9, Table 7 · p. 9

B.3. More ImageNet Examples

• Figure 6 shows additional ImageNet training-set images sorted by representation norms.Source: Figure 6 · p. 10
• Figure 7 shows additional ImageNet training-set images sorted by representation norms.Source: Figure 7 · p. 11

• The ablations train on OpenImages and report top-1 accuracy on ImageNet-100 or ImageNet-1k.Source: C. Additional ablation studies · p. 12

Radius of the Poincaré ball

• The hyperbolic objective improves performance across all tested Poincaré-ball radii, though a too-small radius yields a smaller improvement.The authors attribute the smaller gain to stronger regularization.Source: C. Additional ablation studies · p. 12, Table 8 · p. 12

Configuration of the encoder head

• A separate hyperbolic head makes training more stable at larger learning rates but does not improve performance.The Euclidean and hyperbolic branches otherwise share backbone and head weights.Source: C. Additional ablation studies · p. 12, Table 9 · p. 12

Hyperbolic linear evaluation

• In hyperbolic linear evaluation, SGD can easily cause overflow, whereas Adam is more stable with appropriate learning rates.The evaluation tests whether representations are more linearly separable in hyperbolic space.Source: C. Additional ablation studies · p. 12, Table 10 · p. 12