HSG: Hyperbolic Scene Graph

1 Introduction

• Scene graphs model objects and their semantic and spatial relationships, while MSG-based methods jointly represent place and object nodes but typically learn in Euclidean space.These methods use visual encoders and contrastive learning to align semantically related entities.Source: §1 · p. 1
• Real-world places entail objects, but Euclidean embeddings may require higher dimensions or organize them suboptimally, weakening structural consistency and scene graph quality despite strong place recognition.Places and objects also exhibit hierarchical semantic structure.Source: §1 · pp. 1–2
• Hyperbolic geometry offers inherent hierarchy and exponentially increasing capacity with distance from the origin, making it suitable for modeling entailment [60].HSG’s place-object entailment cone constrains valid object embeddings within a place-defined cone.Source: §1 · p. 2, Figure 1 · p. 2
• HSG embeds places and objects in a Lorentz hyperbolic manifold, adds an entailment loss for their hierarchical constraints, and maps embeddings to tangent space for MSG graph construction.The origin represents the most abstract concept, with more specific entities farther away.Source: §1 · pp. 2–3
• Qualitative analysis shows HSG embeddings have clear hierarchical organization aligned with semantic entailment and are more structured than Euclidean baseline embeddings.Evaluations also report improved structural consistency and graph quality.Source: §1 · p. 3
• 33.17 PP IoU and 33.51 Graph IoU: HSG surpasses the best AoMSG variant (25.37) by 8.14 on Graph IoU while maintaining competitive Recall@1 against SepMSG and AoMSG.The largest gains are in graph-level metrics.Source: §1 · p. 3

Scene graph

• Scene graphs model objects and semantic or spatial relations; 3D and multiview approaches add spatial grounding, with MSG building graphs from unposed images.MSG uses multiview visual correspondences and can integrate object-object relation edges to enrich semantic structure.Source: §2 · pp. 3–4

Hyperbolic representations in computer vision

• Hyperbolic vision methods span image embeddings, segmentation, contrastive learning, and vision-language models, representing semantic hierarchies beyond Euclidean embeddings.MERU demonstrates zero-shot transfer across downstream tasks; later works [41] explore alignment, retrieval, and structured scene understanding.Source: §2 · p. 4

Entailment embeddings

• Order embeddings [51] encode hierarchical relations as partial orders in Euclidean space, while hyperbolic entailment cones model them as geodesically convex cones.Methods for partially ordered NLP and knowledge-graph data use asymmetric constraints to represent hierarchical relations.Source: §2 · pp. 4–5

• Hyperbolic space can represent hierarchical or tree-like data because subset volumes grow exponentially with radius, unlike flat Euclidean space.Source: §3 · p. 5
• The Lorentz model represents hyperbolic space as an upper half of a two-sheeted hyperboloid with constant negative curvature −c, using a Lorentz inner product that subtracts the time-coordinate product from the spatial dot product.Source: §3 · p. 5
• Figure 3 shows that HSG replaces MSG’s L2-normalized hyperspherical embeddings and cosine similarity with Lorentz hyperboloid embeddings and negative Lorentzian distance, adding an entailment loss for hierarchical structure.Source: Figure 3 · p. 6
• Vectors are projected onto a tangent space and then mapped onto the Lorentz manifold with the exponential map.Source: §3 · p. 6
• Because hyperbolic space is a Hadamard manifold, the exponential map is a global diffeomorphism and the logarithmic map is uniquely defined.Source: §3 · p. 6

4 The Proposed Method

• Inspired by MERU, HSG maps embeddings to the Lorentz hyperboloid and jointly optimizes Lorentzian contrastive and entailment losses for place–object hierarchy.Source: §4 · p. 7

4.1 Hyperbolic Deep Learning

• Decoder embeddings are lifted from the origin’s tangent space via the exponential map, with a parameterization that stays on the hyperboloid and clips tangent norms to avoid overflow.Source: §4.1 · pp. 7–8
• Object and predicate supervision use InfoNCE with negative Lorentz geodesic distance as similarity, encouraging aligned object–predicate pairs to lie close while curvature allocates capacity to higher-level abstractions.Source: §4.1 · p. 8
• Using hyperbolic entailment cones, place embeddings define regions for valid object children; general places induce wider cones, while specific concepts produce narrower ones.Source: §4.1 · p. 8

4.2 Scene Graph

• Across calibrated views, objects are associated by geometric consistency and feature similarity, features and predicates are fused into a global graph, then message passing supports object classification and predicate prediction.Source: §4.2 · pp. 9–10

5 Experiments

• Experiments use non-overlapping ARKitScenes splits of 4,492 training scenes and 200 test scenes, with DINOv2 visual features and the AoMSG decoder.Training follows the MSG protocol; evaluation measures PP IoU, PO IoU, and object-box GIoU against ground truth.Source: §5 · p. 10

5.1 Results

• curv_init around 80 yields the strongest performance: values ≤20 lose hierarchical capacity, while values ≥300 cause metric collapse through numerical instability.The authors attribute collapse at high curvature to boundary-concentrated embeddings, vanishing gradients, and lost discriminative structure; they use curv_init = 80 as default. Fixed c = 1 collapses entailment losses and harms graph metrics despite high Recall@1; removing entailment loss causes moderate but consistent declines, while Euclidean InfoNCE substantially reduces PP IoU and Graph IoU.Source: Figure 4 · p. 10, §5.1 · p. 12
• 33.51 Graph IoU is HSG’s highest result, alongside 33.17 PP IoU; its Graph IoU exceeds the best AoMSG variant’s 25.37 by +8.14.HSG also attains Recall@1 of 98.39 versus 98.61 for AoMSG-B-4, while SepMSG-Linear’s higher PO IoU of 55.67 comes with markedly worse PP IoU and Graph IoU.Source: Table 1 · p. 10, §5.1 · p. 11
• HSG performs best with a 1024-dimensional projector; 512 under-parameterises it, while 2048 reduces PP IoU to 29.5.AoMSG-4 and SepMSG-Linear vary only marginally across dimensions, so 1024 is adopted for all experiments.Source: Table 2 · p. 11, §5.1 · p. 11
• DINOv2-Base achieves the best overall backbone performance, while ConvNeXt-Base and DINOv2-Large tie at 32.94 PP IoU and 33.22 Graph IoU.DINOv2-Small’s high Recall@1 but poor graph metrics shows retrieval alone is insufficient to assess backbone suitability; Transformer models generally outperform CNNs on graph-level metrics.Source: §5.1 · p. 11, Table 3 · p. 12
• Changing the aperture threshold from 1.0 to 0.01 yields nearly identical results across all evaluation metrics, indicating robustness to this setting.The results describe hyperbolic entailment as providing stable hierarchical supervision without sensitive aperture-threshold tuning.Source: §5.1 · p. 13

[ROOT] embedding

• Following MERU, HSG defines [ROOT] as the Lorentz-hyperboloid origin, while Euclidean models use the global average of training embeddings as an approximate root.Distance to [ROOT] measures semantic specificity; Euclidean models use ℓ2 distance for comparison.Source: §6 · p. 13

Embedding distances from [ROOT]

• HSG places place embeddings at smaller root distances than object embeddings, while AoMSG and SepMSG show weak or inconsistent ordering.In hyperbolic space, root distance reflects generality, so more abstract concepts are expected to lie closer to the origin.Source: §6 · p. 14

7 Limitations and Future Work

• Adaptive or multi-stage curvature optimization could better accommodate nonuniform geometry and improve embedding expressiveness and training stability.The paper identifies curvature optimization as an avenue for further improvement.Source: §7 · p. 14
• Stronger encoders, including DINOv3 [48], more accurate open-vocabulary detectors such as GroundingDINO, and large-scale or temporal scene graph frameworks may improve robustness, generalization, and scalability.The paper notes that overall performance is closely tied to the quality of the underlying encoders.Source: §7 · p. 14
• Joint optimization with downstream tasks or the incorporation of multimodal cues could unlock additional benefits for HSG.Source: §7 · p. 14
• The paper characterizes hyperbolic scene graph representations as a promising, flexible basis for scalable hierarchical visual understanding and structured reasoning in complex scenes.Source: §7 · p. 14

8 Conclusion

• HSG improves hierarchical structure quality while achieving competitive performance against Euclidean baselines.The conclusion does not provide quantitative comparisons in the supplied passage.Source: §8 · p. 14
• HSG combines hyperbolic contrastive learning with an entailment-aware objective to capture hierarchical place–object relationships.The framework learns scene-graph representations in hyperbolic space and remains compatible with existing MSG pipelines.Source: §8 · p. 14
• HSG produces structured, semantically consistent embeddings while remaining compatible with existing MSG pipelines.Source: §8 · p. 14
• HSG extends hyperbolic representation learning to scene understanding and recognition tasks.Source: §8 · p. 14
• The framework offers a foundation for incorporating additional modalities or downstream tasks in structured visual reasoning.The conclusion presents this flexibility as broader potential for hyperbolic embeddings.Source: §8 · p. 14
• The conclusion calls for further exploration of geometry-aware visual representations.Source: §8 · p. 14

A Hyperparameter setting

• Table 5 identifies the hyperparameters used in HSG’s main experiments.The supplied passage gives the table’s scope but no hyperparameter values.Source: Table 5 · p. 19

B.1 IoU Between Two Adjacency Matrices

• Graph IoU is the intersection-over-union of aligned ground-truth and predicted adjacency matrices, equivalently TP/(TP + FP + FN) for binary edge prediction.This counts correctly predicted edges against all true-positive, false-positive, and false-negative edges.Source: §B.1 · pp. 19–20
• The edge-based IoU penalizes both extra predicted edges and ground-truth edges that are missing from the prediction.These correspond to false positives and false negatives, respectively.Source: §B.1 · p. 20

B.2 Place-Place and Place-Object IoU.

• Place-place and place-object adjacency subgraphs receive separate IoU scores, while the final graph IoU evaluates alignment of the entire multiview scene graph.Each score is the intersection size divided by the union size for its corresponding edge type.Source: §B.2 · p. 20

Object Truth-to-Result Matching.

• For place-object IoU, ground-truth and predicted objects are matched one-to-one by maximizing accumulated generalized IoU across frames where both objects appear.The resulting permutation reorders predicted object nodes before scoring the place-object adjacency matrices.Source: Object Truth-to-Result Matching · p. 21

C Entailment Loss Derivations

• The entailment-loss derivation maps points between the Poincaré ball and Lorentz hyperboloid models and preserves cone aperture across these isometric representations.The cone aperture depends on the point and the curvature and scale parameters.Source: C Entailment Loss Derivations · p. 21
• In the Lorentz model, entailment holds when the exterior angle between the text and image embeddings, viewed from the origin, is no greater than the cone aperture centered at the text embedding.The exterior angle is derived from the hyperbolic triangle formed by the origin and the two embeddings.Source: C Entailment Loss Derivations · pp. 21–22

D.1 More Ablations

• τ = 0.1 yields HSG’s highest PP IoU, PO IoU, and Graph IoU; performance slightly degrades at τ = 0.01.The experiments vary only temperature while keeping the base HSG hyperparameters unchanged.Source: §D.1 · p. 22, Table 6 · p. 23
• Decreasing τ from 0.5 to 0.2 significantly improves performance, and HSG is relatively robust across τ values in [0.05, 0.12].The results identify τ = 0.1 as the best overall temperature.Source: §D.1 · p. 22, Table 6 · p. 23
• Smaller temperatures can produce stronger contrastive-learning gradients that may benefit from a smaller learning rate for stable optimization.Around τ = 0.1, HSG achieves its best performance under the base learning rate.Source: §D.1 · p. 23
• Embedding dimensions from 128 to 1024 consistently improve PP IoU, PO IoU, and Graph IoU, with dimension 1024 yielding the highest scores across all metrics.Curvature is fixed at c = 80, with other hyperparameters unchanged.Source: Table 7 · p. 23, §D.1 · p. 23
• Increasing the embedding dimension to 2048 degrades performance, suggesting that excessive dimensionality may introduce redundancy and make optimization more difficult.The results indicate a moderate embedding dimension balances representation capacity and training stability.Source: Table 7 · p. 23, §D.1 · p. 23

E.1 Projecting onto the Poincaré ball

• As training progresses, place embeddings move toward the Poincaré-disk origin while object embeddings remain farther away, forming a hierarchical organization in which places tend to entail objects.The observed radial ordering is consistent with earlier distance histograms showing smaller radial distances for place embeddings than for object embeddings.Source: Figure 6 · p. 24, §E.1 · p. 24

E.2 Distribution of embedding distances from root

• With DINOv2-base, place embeddings (µ=0.0584) have slightly smaller root distances than object embeddings (µ=0.0610), yielding a mild but consistent ordering despite partial overlap.Source: Figure 7 · p. 25, §E.2 · p. 25
• ConvNeXt-base shows more pronounced separation, but its inverted ordering indicates limited alignment with the intended hierarchy.Source: Figure 7 · p. 25, §E.2 · p. 25
• ViT-base produces heavily overlapping place and object distributions (µ=0.0477 vs. µ=0.0388) with comparable spread, reflecting weak hierarchical organization.Source: Figure 7 · p. 25, §E.2 · p. 25
• In hyperbolic space, root proximity reflects semantic generality, so the more general place concept is expected to lie closer to the root than objects.Source: §E.2 · p. 25

E.3 Qualitative real-world experiment

• Figure 8 shows a pretrained Grounding DINO detector with the trained baseline producing coherent, meaningful outputs on videos beyond the training dataset, alongside an interactive place–object graph.The visualization marks place nodes in blue and object nodes in orange.Source: Figure 8 · p. 26, §E.3 · p. 26

E.4 Other visualisations

• Figure 9 groups each set of three side-by-side images into connected MSG nodes recognized as the same place.Source: Figure 9 · p. 27
• Figure 10 groups views of the same physical object into one object node and uses matching colors for same-object instances within a scene.Some images were captured sideways, and their original orientation is retained in the visualization.Source: Figure 10 · p. 28