PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept-Guided Exploration: Building Persistent, Actionable Scene Graphs

Noé Zapata, Gerardo Pérez, Alejandro Torrejón, Pedro Núñez, Pablo Bustos

arXiv:2608.23650v1cs.RO

TL;DR

Robots need incremental, task-driven scene graphs that can be built from an empty representation while supporting human-interpretable spatial understanding. This paper uses asynchronous room and door concept agents with hierarchical constraints to build actionable graphs without a pre-existing global metric map, demonstrating architectural viability and scalability potential.

  • Problem

    Robots need incremental, opportunistic scene-graph construction that supports task-driven understanding beyond limited-semantic occupancy grids.

  • Method

    Asynchronous room and door agents cooperatively build a shared scene graph, with validated rooms constraining door instantiation through graph-based communication.

  • Results

    The experiments confirm the viability of concept-first actionable scene-graph construction without requiring a pre-existing global metric map.

  • Takeaways & Limitations

    The architecture offers a scalable, human-interpretable representation from connected local reference frames, while sacrificing dense geometric fidelity relative to metric-first systems.

  • Takeaways & Limitations

    The current implementation is limited to rectangular rooms in structured environments and may struggle with clutter, non-rectangular spaces, and dynamic obstacles.

Abstract

from arXiv · show

The perception of 3D space by mobile robots is rapidly moving from flat metric grid representations to hybrid metric-semantic graphs built from human-interpretable concepts. While most approaches first build metric maps and then add semantic layers, we explore an alternative, concept-first architecture in which spatial understanding emerges from asynchronous concept agents that directly instantiate and manage semantic entities. Our robot employs two spatial concepts (room and door), implemented as autonomous processes within a cognitive distributed architecture. These concept agents cooperatively build a shared scene graph representation of indoor layouts through active exploration and incremental validation. The key architectural principle is hierarchical constraint propagation: Room instantiation provides geometric and semantic priors to guide and support door detection within wall boundaries. The resulting structure is maintained by a complementary functional principle based on prediction-matching loops. This approach is designed to yield an actionable, human-interpretable spatial representation without relying on any pre-existing global metric map, supporting scalable operation and persistent, task-relevant understanding in structured indoor environments.

1 Introduction

The paper introduces a concept-first architecture in which asynchronous agents representing spatial concepts cooperatively build and refine a shared 3D scene graph. The system supports incremental, opportunistic exploration from an initially empty representation without relying on a prior free/occupied grid.

  • Incremental construction: The architecture builds scene graphs incrementally from an empty representation while proposing actions that reduce uncertainty during real-time operation.These actions may coincide or compete with requests from higher-priority tasks, making the representational system opportunistic.
  • Concept-first architecture: Autonomous room and door agents asynchronously construct a 3D scene graph by instantiating and relating human-interpretable spatial concepts.The agents cooperatively build the representation through geometric and semantic relationships between concept instances.
  • Concept-driven representation: The graph is constructed directly from room and door concepts, avoiding a previous free/occupied grid representation.A 3D LiDAR supplies a stream of scene points for concept-driven construction.
  • Agent organization: Each concept agent manages its concept’s life-cycle by creating, refining, and predicting instances within a global shared representation.Agents access the shared representation for contextual information that constrains creation and updates, while contributing opportunistically to the whole graph.

2 Related works

Related work spans metric–semantic mapping, hierarchical 3D scene graphs, geometric and neural layout inference, and active online perception. The paper positions its concept-first architecture as an incremental, room-centered alternative that uses explicit spatial structure and hierarchical semantic constraints rather than metric-first or object-centric pipelines.

  • Metric–semantic mapping: Classical geometric SLAM supports localisation and mapping but lacks high-level semantics, motivating semantic labels, object-centric landmarks, and metric–semantic representations.These limitations affect human–robot interaction, long-term robustness, and task-oriented reasoning.
  • 3D scene graphs: 3D Scene Graphs unify floors, rooms, objects, and cameras in a hierarchical structure grounded in reconstructed 3D geometry.Subsequent robotic extensions operationalise this representation for actionable perception.
  • Room-layout inference: Neural room-layout models infer geometrically coherent layouts from single panoramic images at low inference cost, but depend on annotated datasets and often assume regular environments.Their single-shot, black-box inference cannot incrementally refine partial or ambiguous evidence encountered during robot exploration.
  • Active perception: Active-SLAM and situational-graph research motivate incrementally refined, multi-layer representations in which perception actively gathers information to validate or refute semantic hypotheses.Information-theoretic methods additionally provide principled sensing-action selection under uncertainty.
  • Concept-first architecture: Unlike metric-first and object-centric systems, the proposed architecture constructs an actionable, persistent graph online without prior global maps, treating rooms and connectivity as first-class structure and propagating semantic constraints top-down.Its contribution is architectural: asynchronous communicating concept agents organise simple perception methods into a system for building and maintaining the scene graph.

3 Method … 3.4 Overview of the spatial representation construction process

The method builds an actionable indoor scene graph directly from human-interpretable concepts through asynchronous CORTEX agents, active exploration, and incremental validation. Shared Working Memory coordinates hierarchical room-and-door construction while preserving geometric structure, semantic relationships, and responsiveness to task demands.

  • 3.1 Architectural Motivation: Concept-first representation instantiates human-interpretable spatial entities directly, rather than adding semantic layers after metric grid mapping.This is intended to support transparent reasoning, human-robot communication, and high-level task planning.
  • 3.1 Architectural Motivation: Hierarchical constraint propagation narrows later detection search spaces using accumulated spatial context, but its efficiency and robustness remain intended outcomes rather than quantitative results.The architecture is designed to apply concept-driven constraints to perception and planning.
  • 3.2 The CORTEX architecture: CORTEX separates cognitive memories and asynchronous agents from a subcognitive level connected bidirectionally to the robot body.Its distributed Working Memory is a low-latency, high-throughput graph that agents can edit to represent the robot and environment.
  • 3.2 The CORTEX architecture: Agents communicate and coordinate by editing shared Working Memory, with atomic graph modifications preventing race conditions and preserving data integrity.Robot-body, mission-monitoring, and concept-aware agents react to graph changes according to local or supervisory goals.
  • 3.3 Agent Interaction and Coordination via the Working Memory: Working Memory enables implicit collaboration: agents create, modify, or delete nodes and edges, while mission_monitoring arbitrates competing intentions according to the robot’s high-level task.C_room signals exploration through has_intention, and C_door activates only after a validated current_room exists.
  • 3.4 Overview of the spatial representation construction process: Croom and Cdoor cooperatively construct an actionable scene graph from no prior spatial knowledge through active exploration, concept instantiation, and incremental validation.Supporting agents include robot_body, energy_optimiser, long_term_spatial_memory, and mission_monitoring.
  • 3.4 Overview of the spatial representation construction process: Under a Manhattan-world assumption, rooms are rectangular with orthogonal walls, while doors remain within wall boundaries, with at most one door per wall and exactly two connected rooms.Room-centred frames and child wall frames organize the scene’s spatial relationships.
  • 3.4.1 Assumptions and limitations: The resulting distributed graph supports navigation, manipulation planning, and human communication while combining continuous localisation with semantic concept instantiation.Operational limits include cluttered or non-rectangular environments and dynamic obstacles; mission_monitoring may interrupt initialisation, leaving partial representations for later refinement.

3.5 Conceptual agents and intentional actions

Conceptual agents autonomously instantiate, maintain, and refine concept instances through defined properties, priors, affordances, initialization, and behavior-tree lifecycles. Room and door agents use active actions and prediction-based validation to build actionable spatial entities in working memory.

  • Conceptual agents: A conceptual agent is an autonomous process responsible for instantiating, maintaining, and refining a specific concept class.Each concept class is defined by measurable properties F, creation priors Φ, affordances Aff, initialization I, and lifecycle L.
  • Conceptual agents: Concept affordances expose actionable intentions: rooms support visit, while doors support visit and cross.Visit creates a has_intention edge notified to mission_monitoring; crossing creates an aff_cross_x node attached to the associated door.
  • Conceptual agents: The behavior-tree lifecycle matches incoming measurements to existing instances, triggers initialization when matching error exceeds threshold, and validates instances through predicted features.Established predictions also serve as landmarks for robot localisation through energy minimisation, while initialization accumulates evidence through intentional actions.
  • Room agent: Room instances are parameterized by centre, angle, and dimensions, and are actively searched when the robot starts or crosses a door.The room agent inserts a temporary room node when detected corners satisfy Φroom and no room already exists, then navigates toward the estimated room centre.
  • Door agent: Door detection begins only when a current room is available and filters point-cloud evidence to wall-adjacent points using δ = 0.15m.It analyzes a polar scan at K = 1.7m, detects abrupt distance changes, and tests anchor-point pairs against width and wall constraints.

3.6 Long-Term Spatial Memory

The CORTEX long-term spatial memory maintains a persistent, topological graph of locally consistent, concept-centric metric frames linking rooms through doors. Dynamic loading and unloading of relevant rooms bounds computational requirements as environments grow while preserving actionable, explainable spatial memory.

  • The LTSM agent stores previously visited rooms and their door connections in a persistent igraph-based topological graph of locally consistent metric frames.This design prioritizes local precision and long-term scalability over global metric consistency.
  • Doors are stored as dual objects with coordinates in both adjacent rooms’ frames, preserving the minimal information needed for an actionable connected-room graph.When crossing into an unfamiliar room, the system retains the departing room until the destination is initialized and made current.
  • For known-room insertion, door dual coordinates orient the destination room so corner matching can determine whether the room connection is accepted.The procedure transforms room corners into the door frame and aligns both door frames at the same spatial point.
  • LTSM dynamically loads rooms relevant to the robot’s location and planned actions while unloading distant or irrelevant spaces, keeping computation bounded as environments expand.The architecture avoids maintaining a complete monolithic global map in active memory and supports real-time operation in large facilities.
  • This bounded-memory design supports long-term operational scalability while retaining the transparency and explainability of the concept-driven approach.The paper presents qualitative rather than formal asymptotic analysis of the computational complexity.

3.7 Navigating through the scene graph

Once a nominal room is established, an energy-optimiser agent continuously estimates robot pose with a sliding-window GTSAM factor graph. Validated corners and instantiated doors become time-associated landmarks, linked through statistically weighted measurement factors.

  • Pose estimation: The energy-optimiser updates the robot’s position at 10 Hz using a GTSAM factor graph initialized with nominal room corners and the robot’s working-memory pose.Fresh odometry adds robot poses up to 30 nodes under a FIFO policy, while sliding-window iSAM2 computes the optimal pose.
  • Measurement validation: Each new pose triggers a Mahalanobis-distance pre-association check between projected nominal corners and current measurements against a statistical threshold.The check uses validated corner values maintained in working memory by the corresponding concept agent.
  • Measurement validation: Validated matches create graph factors representing statistically weighted differences between measured and predicted features, incorporating measurement and propagated prediction uncertainty.Propagated uncertainty combines nominal-corner position uncertainty with the robot’s current-pose uncertainty.
  • Landmark integration: Validated corners enter the factor graph as measurements associated with corresponding landmarks and connected to the temporally closest pose node.The connection uses the pose timestamp most closely matching the corner observation timestamp.
  • Landmark integration: When a door is instantiated in working memory, its nominal position is inserted as a graph landmark, and door measurements are handled like corner observations.Door measurements follow the same graph-integration procedure as corner observations.

4 Results

Results from Webots experiments show that the concept agents incrementally construct and extend actionable scene graphs across rooms and doors, including loop closure in a ten-room scenario. The evaluation reports strong geometric and topological fidelity, with low and stable resource consumption, while remaining limited to simplified structured environments.

  • Experimental scope: The experiments used simplified, structured proof-of-concept environments with varied room dimensions and door counts, while more complex layouts and dynamic obstacles remain future work.Detection parameters and thresholds were tuned empirically for the specific robotic setup and test environments.
  • Scene-graph evolution: The first experiment demonstrates room acquisition, door detection and affordance execution, followed by new-room acquisition and cross-room door matching.Croom initializes missing rooms, updates the robot’s pose through room-relative coordinate frames, and Cdoor promotes provisional doors to acquired status while adding crossing affordances.
  • Scene-graph evolution: A ten-room experiment shows incremental construction of local and global representations and loop closure connecting several rooms.All agents and sub-cognitive components ran on different cores of an Intel i9 13th generation onboard computer with an NVidia RTX system.
  • Geometric and topological accuracy: The generated maps exhibit strong positional and dimensional fidelity, with low mean room-placement error, minimal variance, and negligible room and door-width errors.The results were aggregated across multiple generation runs and compared against simulator layouts.
  • Geometric and topological accuracy: The system identified the correct room-to-room and room-to-door connectivity in all test cases despite minor geometric deviations.The evaluation emphasizes topological-map consistency and correctness as the most relevant outcome.
  • Computational performance: Mean RAM usage remained between 0.22% and 0.33% for each process, with low standard deviation indicating stable and predictable memory consumption.The measurements came from an experimental scene-graph construction run across several rooms.

5 Noise Sensitivity and Detection Mistakes

The current concept-first representation is vulnerable to sensor and odometry noise, as well as detection mistakes that persist because validated concept instances are not revised. Proposed extensions add continuous fitness evaluation, hierarchical correction, alternative hypotheses, and graceful degradation.

  • Noise Sensitivity: Sensor and odometry noise degrade the final spatial representation, with LiDAR noise impairing corner detection and odometry drift compounding during exploration.Systematic noise evaluation remains future work; preliminary simulations identify these challenges.
  • Noise Sensitivity: Future experiments will vary LiDAR noise (σ = 0.01m to 0.1m), odometry drift rates (0.5◦to 5◦of distance travelled), and angular uncertainty (0.5◦to 5◦).The intended result is a noise tolerance envelope for acceptable spatial accuracy.
  • Detection Mistakes: Validated instances remain fixed, so early incorrect models and initially correct models later contradicted by new observations cannot be revised.Examples include misreading a partially observed L-shaped space as rectangular and discovering walls or openings incompatible with an existing room model.
  • Detection Mistakes: The proposed lifecycle adds continuous fitness monitoring that can invalidate inconsistent models and child concepts, using recent observations for responsiveness and historical consistency for stability.A persistent low fitness score would trigger re-evaluation.
  • Detection Mistakes: A graduated response would adjust parameters for minor discrepancies, revise structure for major geometric changes, or replace untenable models entirely.The broader proposal also maintains alternative hypotheses and partial representations when no satisfactory model exists.

6 Discussion

The discussion confirms the viability of the concept-first architecture for building actionable scene graphs while emphasizing qualitative comparison over direct quantitative evaluation. Its main strengths are independence from a pre-existing global metric map and explicit, interpretable reasoning about environmental structure, with sparse geometry as a trade-off.

  • The experimental results confirm the viability of the concept-first architecture for building actionable scene graphs.The discussion prioritizes contextualizing contributions and limitations through qualitative comparison because of fundamental architectural differences with metric-first systems.
  • The system builds its world model from connected local reference frames without relying on a pre-existing global metric map.This contrasts with frameworks such as Hydra, which ground their 3D scene graph in a dense mesh.
  • Explicitly modelling rooms and walls as first-class graph entities enables hierarchical constraint propagation from room geometry to door search.This top-down mechanism differs from the primarily bottom-up, object-centric approach of many open-vocabulary systems.
  • The approach provides clear, explicit mechanisms for reasoning about environmental structure.Its sparse, concept-based representation trades away dense geometric fidelity.

7 Conclusions and future works

The work demonstrates a concept-first architecture in which asynchronous room and door agents incrementally build semantic scene graphs without prior metric maps. Future work targets multi-agent operation, broader concept and room support, more robust learning-based detection, and transparent spatial representations for robot operation and collaboration.

  • Conclusions: Asynchronous room and door agents build semantic scene graphs incrementally within the CORTEX architecture without requiring prior metric maps.The architecture uses hierarchical constraint propagation, with room instantiation providing priors for door detection.
  • Future works: Future research could extend the architecture to multi-robot scenarios using shared WM and LTSM to improve graph consistency through inter-robot constraints.This direction could mitigate aggregated sensor noise but introduces data-association challenges between agents’ detections.
  • Future works: Open topics include non-rectangular rooms, more robust data-driven detection functions, differentiable online geometry adjustment, and additional concept classes.Examples of additional concepts include furniture and household objects.
  • Conclusions: The long-term goal is transparent, human-interpretable spatial representation supporting autonomous robot operation and effective human-robot collaboration in real-world environments.The work is framed as an initial step toward cognitive architectures reasoning about space with human-meaningful concepts rather than metric primitives.

Funding · Institutional Review Board Statement · Informed Consent Statement

The work received partial support from European and Spanish public research funding programs.

  • Funding: The study was partially funded by FEDER Project 0124_EUROAGE_MAS_4_E and Spanish Ministry of Science and Innovation grant PID2022-137344OB-C31.The grants were associated with the 2021–2027 POCTEP Program and funded through MCIN/AEI/10.13039/501100011033/FEDER, UE.
Loading 2608.23650v1…