PaperlayerView exact revision on arXiv

Source-linked AI summary

HSG: Hyperbolic Scene Graph

Liyang Wang, Zeyu Zhang, Hao Tang

arXiv:2604.17454v1cs.CV

TL;DR

Euclidean scene graph embeddings do not explicitly capture hierarchical entailment between places and objects, limiting structural consistency. HSG learns place and object representations in hyperbolic space with entailment-aware constraints, improving graph structure while retaining competitive retrieval. It achieves a Graph IoU of 33.51, exceeding the best AoMSG variant’s 25.37 by 8.14.

  • Problem

    Euclidean scene graph embeddings inadequately capture hierarchical place–object entailment, which can lead to suboptimal structural organization and scene graph quality.

  • Method

    HSG learns scene graph representations in a Lorentz hyperbolic manifold and uses an entailment loss to enforce hierarchical constraints between places and objects.

  • Results

    33.51 Graph IoU is achieved by HSG, outperforming the best AoMSG variant’s 25.37 by 8.14 while maintaining competitive retrieval performance.

  • Takeaways & Limitations

    HSG improves hierarchical structure quality while remaining compatible with existing MSG pipelines and retaining competitive performance against Euclidean baselines.

  • Takeaways & Limitations

    Performance remains closely tied to the quality of the underlying encoders, motivating stronger encoders and detectors for improved robustness, generalization, and scalability.

Abstract

from arXiv · show

Scene graph representations enable structured visual understanding by modeling objects and their relationships, and have been widely used for multiview and 3D scene reasoning. Existing methods such as MSG learn scene graph embeddings in Euclidean space using contrastive learning and attention based association. However, Euclidean geometry does not explicitly capture hierarchical entailment relationships between places and objects, limiting the structural consistency of learned representations. To address this, we propose Hyperbolic Scene Graph (HSG), which learns scene graph embeddings in hyperbolic space where hierarchical relationships are naturally encoded through geometric distance. Our results show that HSG improves hierarchical structure quality while maintaining strong retrieval performance. The largest gains are observed in graph level metrics: HSG achieves a PP IoU of 33.17 and the highest Graph IoU of 33.51, outperforming the best AoMSG variant (25.37) by 8.14, highlighting the effectiveness of hyperbolic representation learning for scene graph modeling. Code: https://github.com/AIGeeksGroup/HSG.

1 Introduction

Existing scene graph methods use Euclidean embeddings that inadequately capture hierarchical place–object entailment, motivating HSG’s hyperbolic scene graph representation. HSG introduces hyperbolic geometry and entailment constraints, improving structural metrics while maintaining competitive retrieval.

  • Motivation: Euclidean embeddings may fail to efficiently represent hierarchical and entailment relationships between places and objects.This can produce suboptimal structural organization and scene graph quality despite strong place recognition.
  • Approach: HSG learns scene graph representations in hyperbolic space to model hierarchical place–object relationships.Places and objects are embedded in a Lorentz hyperbolic manifold, with more abstract concepts nearer the origin.
  • Approach: HSG introduces an entailment loss that explicitly enforces hierarchical constraints between place and object embeddings.The loss encourages structurally meaningful organization in the learned hyperbolic space.
  • Results: HSG improves structural metrics such as PP IoU and Graph IoU while maintaining competitive Recall@1 against Euclidean baselines.Qualitative analysis also reports clear hierarchical organization aligned with semantic entailment.
  • Results: 33.51 Graph IoU is achieved by HSG, surpassing the best AoMSG variant’s 25.37 by 8.14.HSG records a PP IoU of 33.17 and the highest Graph IoU among the reported comparisons.

2 Related Work

Related work spans conventional and 3D scene graphs, hyperbolic visual representations, and entailment embeddings. These lines of research motivate structured representations that capture spatial grounding and latent semantic hierarchies.

  • Scene Graphs: Conventional scene graphs model visual entities as nodes and semantic or spatial relationships as edges for compositional reasoning.Transformer-based and open-vocabulary methods extend relational modeling and category generalization.
  • Scene Graphs: 3D scene graphs add spatial grounding by representing objects, rooms, or viewpoints as nodes in spatial-environment graphs.They can be constructed from 3D meshes, point clouds, or RGB-D observations.
  • Hyperbolic Representations: Hyperbolic representations model hierarchical and structured data through the exponential volume growth of hyperbolic space.Prior computer-vision work reports improved hierarchical organization relative to Euclidean embeddings.
  • Entailment Embeddings: Entailment embeddings represent asymmetric semantic relations such as is-a directly in embedding space.Order embeddings use partial orders, while hyperbolic entailment cones model entailment as geodesically convex cones.

3 Preliminaries

Hyperbolic geometry models hierarchical scene-graph relationships using a Lorentz hyperboloid, with distances and maps defined through Lorentzian geometry.

  • Hyperbolic space has constant negative curvature and exponentially growing volume, making it suitable for hierarchical or tree-like data.
  • The Lorentz model represents the upper half of a two-sheeted hyperboloid in (n + 1)-dimensional Minkowski spacetime.
  • The Lorentz hyperboloid contains vectors x satisfying ⟨x, x⟩L = −1/c for curvature parameter c > 0.
  • The Lorentz inner product combines spatial similarity with a negative time-component product: ⟨x, y⟩L = ⟨xspace, yspace⟩ − xtimeytime.
  • HSG computes geodesic distance as c^-1/2 cosh^-1(−c⟨x, y⟩L), then uses tangent-space projection and exponential or logarithmic maps.
  • HSG replaces MSG’s hyperspherical embeddings and cosine similarity with Lorentz hyperboloid embeddings and negative Lorentzian distance.

4 The Proposed Method

HSG extends a multiview scene-graph pipeline with Lorentzian embeddings, contrastive learning, and entailment constraints to organize places and objects hierarchically.

  • HSG maps embeddings to the Lorentz hyperboloid and jointly optimizes a Lorentzian contrastive objective with an entailment loss.
  • Decoder outputs are lifted onto the hyperboloid so every embedding satisfies ⟨x, x⟩L = −1/c during training.
  • HSG uses the exponential map at the hyperboloid origin, where the tangent vector’s Lorentz norm reduces to the Euclidean norm of its spatial component.
  • The model clips large tangent norms before applying the exponential map to avoid overflow from rapidly growing sinh and cosh values.
  • Hyperbolic contrastive learning defines similarity through negative Lorentz distance for object- and predicate-level supervision.
  • Entailment cones assign broader places wider cones and specific concepts narrower cones, while an aperture-threshold loss enforces place–object hierarchy.
  • The total loss combines predicate, object, and weighted entailment losses: Ltotal = Lpr+Lobj+λLent.
  • The multiview pipeline associates objects across calibrated views using 3D geometric consistency and feature similarity, then fuses their features into global nodes.

5 Experiments

Experiments show that HSG improves graph-level scene-graph quality while retaining competitive retrieval, with performance depending on curvature, projector dimensions, and encoder choice. Qualitative and ablation analyses further examine its hierarchical behavior and design components.

  • Curvature initialization: curv_init = 80 is the performance optimum, balancing hierarchical capacity and training stability.Very small or large curvature values cause metric collapse due to limited hierarchical capacity or instability.
  • Main results: 33.51 Graph IoU is HSG’s highest reported graph-level score, surpassing the best AoMSG variant’s 25.37 by +8.14.HSG also achieves a PP IoU of 33.17.
  • Main results: 98.39 Recall@1 keeps HSG competitive with AoMSG-B-4 at 98.61 while HSG maintains stronger graph quality.SepMSG-Linear achieves higher PO IoU at 55.67 but performs markedly worse on PP IoU and Graph IoU.
  • Hierarchical structure: Place embeddings become closer to [ROOT] than object embeddings in HSG, whereas Euclidean baselines show no evident hierarchical structure.The comparison uses Lorentzian distances for HSG and L2 distances for AoMSG and SepMSG.
  • Ablations: Aperture thresholds from 1.0 to 0.01 produce nearly identical results across all evaluation metrics.This indicates robustness to the aperture threshold without sensitive hyperparameter tuning.

6 Qualitative Analysis

HSG’s qualitative analysis evaluates semantic hierarchy through distances to a root representing the most generic concept. Its embeddings place more general places nearer the root than more specific objects.

  • Root-distance analysis: [ROOT] is the Lorentz-hyperboloid origin, and distance to it reflects semantic abstraction.For Euclidean models without a canonical root, the global average of training embeddings approximates [ROOT].
  • Root-distance analysis: HSG separates place and object embeddings by root distance, with places closer to [ROOT] and objects farther away.This ordering matches the intended entailment structure in which places are more general than objects.

7 Limitations and Future Work

The paper identifies several avenues for improving HSG’s expressive capacity, robustness, generalization, and scalability. These include adaptive curvature, stronger encoders and detectors, larger temporal frameworks, downstream-task optimization, and multimodal cues.

  • Curvature optimization: Adaptive or multi-stage curvature optimization could better accommodate nonuniform geometric structures.The authors associate this direction with potential gains in embedding expressiveness and training stability.
  • Encoders and scalability: Performance remains closely tied to the quality of the underlying encoders.The paper suggests stronger foundation models, more accurate open-vocabulary detectors, and large-scale temporal scene-graph frameworks.
  • Future directions: Joint optimization with downstream tasks and multimodal cues are proposed as further improvement directions.The authors suggest these additions could provide additional benefits.

8 Conclusion

HSG learns hyperbolic scene-graph representations that explicitly capture hierarchical place–object relationships while remaining compatible with existing MSG pipelines. The reported results show improved hierarchical structure quality and competitive Euclidean-baseline performance, positioning hyperbolic embeddings as a flexible basis for structured visual reasoning.

  • Conclusion: HSG combines hyperbolic contrastive learning with an entailment-aware objective to produce structured, semantically consistent embeddings.The framework remains compatible with existing MSG pipelines.
  • Conclusion: HSG improves hierarchical structure quality while achieving competitive performance against Euclidean baselines.The conclusion frames this as an extension of hyperbolic representation learning to scene understanding and recognition.
  • Conclusion: Hyperbolic scene-graph representations provide a flexible foundation for additional modalities and downstream tasks.The paper presents this as evidence of broader potential for geometry-aware visual reasoning.

B Details of evaluation metrics

The evaluation aligns predicted and ground-truth graphs before measuring adjacency overlap, separately evaluates place-place and place-object edges, and reports graph-level IoU. Object matching uses accumulated generalized IoU across frames to obtain a one-to-one permutation.

  • B.1 IoU Between Two Adjacency Matrices: Graph IoU is computed as adjacency-matrix intersection divided by union after aligning graph vertices.The equivalent edge formulation is TP/(TP+FP+FN), penalizing both extra predicted edges and missing ground-truth edges.
  • B.2 Subgraph IoU: PP IoU and PO IoU separately evaluate place-place and place-object adjacency subgraphs.The final Graph IoU evaluates alignment of the entire multiview scene graph.
  • B.3 Object Matching: Object existence is represented with binary indicators for each frame, and accumulated generalized IoU scores determine object correspondences.The indicators specify whether each ground-truth or predicted object exists in frame t.
  • B.3 Object Matching: A one-to-one permutation aligns predicted objects with ground-truth objects before place-object adjacency evaluation.The permutation maximizes accumulated matching scores across object detections and reorders predicted nodes before computing PO IoU.
  • C Entailment Loss Derivations: The entailment constraint is satisfied when the exterior angle between two Lorentz-model points does not exceed the cone aperture centered at the reference point.The cone aperture is derived in the Poincaré ball and transferred to the Lorentz model through an isometric mapping.

D.1 More Ablations

The ablations show that HSG performance depends on contrastive temperature and embedding dimension, while training progressively separates place and object embeddings into a hierarchical organization.

  • Effect of temperature τ: τ = 0.1 achieves the best PP IoU, PO IoU, and Graph IoU, whereas τ = 0.01 slightly degrades performance.Reducing τ from 0.5 to 0.2 improves performance significantly, but overly small temperatures are less effective.
  • Effect of embedding dimensions: Embedding dimension 1024 performs best, with larger dimensions from 128 to 1024 consistently improving PP IoU, PO IoU, and Graph IoU.These experiments fix curvature at c = 80 and keep other hyperparameters identical to the base configuration.
  • Embedding evolution: As training progresses, place embeddings move toward the origin while object embeddings remain farther away, forming a hierarchical organization.Place embeddings occupy more central hyperbolic regions and tend to entail object embeddings.

E.2 Distribution of embedding distances from root

Root-distance distributions assess whether place embeddings are more central and abstract than object embeddings, with encoder choice substantially affecting this hierarchy.

  • Root-distance interpretation: Place is expected to lie closer to [ROOT] than objects because it represents a more general concept.HSG defines [ROOT] as the origin, so distance from it reflects semantic specificity.
  • Encoder comparison: DINOv2-base yields place and object means of 0.0584 and 0.0610, respectively, indicating mild but consistent hierarchical ordering.The distributions partially overlap but remain distinguishable.
  • Encoder comparison: ConvNeXt-base reverses the expected ordering, while ViT-base produces heavily overlapping distributions with weak hierarchical organization.These comparisons show that encoder choice influences alignment with the intended hierarchy.
  • Qualitative visualization: Figure 8 illustrates coherent place-object connections on videos beyond the training dataset using a pretrained Grounding DINO detector.The visualization includes place nodes in blue and object nodes in orange.
Loading 2604.17454v1…