PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering

Albert Gassol Puigjaner, Kostas Alexis

arXiv:2609.26360v1cs.RO

TL;DR

EQA requires agents to explore unseen environments because the visual evidence needed to answer questions may not yet be available, and prior approaches typically leave floorplan structure unused for targeting unseen rooms. HFLEX-EQA combines online scene graphs, hierarchical VLM planning, semantic frontiers, and floorplan-guided room discovery. It outperforms existing EQA baselines on OpenEQA and ExploreEQA and is demonstrated on a quadruped robot.

  • Problem

    EQA exploration commonly relies on locally observed information, while existing approaches do not use available floorplans to reason about question-relevant unseen rooms.

    Source: §I · p. 1

  • Method

    HFLEX-EQA combines online hierarchical scene graphs, VLM planning over observations and exploration history, and floorplan-guided exploration toward relevant unobserved room types.

    Source: §I · pp. 1–2

  • Results

    HFLEX-EQA with floorplan priors outperforms existing EQA baselines on OpenEQA and ExploreEQA and is deployed on a quadruped robot in previously unseen environments.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results support combining hierarchical VLM planning with structural floorplan priors for EQA.

    Source: B. Simulation Results · p. 7, §VI · p. 8

Abstract

from arXiv · show

Embodied Question Answering (EQA) requires an agent to explore a previously unseen environment, gather relevant information, and answer questions about the scene. Recent approaches leverage Vision-Language Models (VLMs) together with semantic maps or scene graphs to guide exploration. However, exploration is typically driven only by local observations, while structural priors about the environment remain largely unused. We propose HFLEX-EQA, a hierarchical EQA framework that combines online scene graph construction, VLM- based planning, semantic frontier exploration, and floorplan priors. The system incrementally builds a hierarchical scene graph and an open-vocabulary occupancy map from RGB-D observations, enabling a VLM to jointly reason over the scene graph, task-relevant visual observations, exploration history, and an estimated topological floorplan. Furthermore, we introduce a room-discovery strategy that leverages the floorplan and open-vocabulary frontier semantics to guide exploration toward semantically relevant yet currently unobserved room types. We evaluate HFLEX-EQA on the OpenEQA and ExploreEQA benchmarks and demonstrate deployment on a quadruped robot in real indoor environments. Our results demonstrate the benefit of combining VLM-based hierarchical planning with structural floorplan priors for the EQA task.

I. INTRODUCTION

EQA requires active information gathering because the evidence needed to answer may lie in unseen parts of an environment, while existing exploration often relies on local observations. HFLEX-EQA combines hierarchical VLM planning with scene-graph and floorplan reasoning, and is evaluated on two benchmarks and deployed on a quadruped robot.

I. INTRODUCTION

  • Existing EQA exploration typically uses local observations, although questions may require discovering a room, improving an object view, or exploring a known area.Structural priors such as floorplans could help identify question-relevant unseen rooms, but existing EQA approaches do not exploit them for this purpose.Source: §I · p. 1
  • HFLEX-EQA combines online scene-graph construction with hierarchical VLM planning over observations, exploration history, and floorplan priors.Dedicated low-level planners handle room exploration, object inspection, or floorplan-guided room discovery.Source: §I · pp. 1–2
  • HFLEX-EQA introduces floorplan-guided room discovery using topological floorplans and open-vocabulary frontier semantics to target relevant unobserved room types.Source: §I · p. 2
  • HFLEX-EQA with floorplan priors outperforms existing EQA baselines on OpenEQA and ExploreEQA, and is deployed in previously unseen environments on a quadruped robot.Source: §I · p. 2

II. RELATED WORK

Prior work brings semantic maps, scene graphs, and VLM reasoning to exploration and EQA, but exploration is commonly guided by locally observed information. The problem formulation considers active answering in unseen indoor environments using RGB-D observations and an estimated floorplan graph.

II. RELATED WORK

  • Prior EQA systems use structured semantic memory and VLM reasoning, but typically guide exploration with a single strategy based on locally observed information.HFLEX-EQA contrasts this with hierarchical planning modes and a topological floorplan for reasoning about relevant unseen room types.Source: §II · p. 2

III. PROBLEM STATEMENT

  • The task takes a question, optional answer choices, online RGB-D observations, and an estimated floorplan graph as input.The floorplan graph represents semantically labeled room instances as nodes and room connectivity as edges.Source: §III · p. 2
  • The agent must actively gather the visual information needed to answer, returning a choice when available or a free-form answer otherwise.The estimated floorplan may be incomplete or inaccurate, including missing or misclassified rooms and incorrect connectivity.Source: §III · p. 2

IV. METHOD

HFLEX-EQA incrementally constructs a semantic scene representation from online observations and combines it with question-guided VLM planning. Its hierarchical graph includes room, frontier, navigation, object, and metric-semantic mesh layers.

IV. METHOD

  • The high-level VLM planner uses the question, optional answer choices, scene graph, relevant visual memory, action history, and floorplan to decide whether to answer or explore.Exploration is assigned one of three modes: go to objects, explore a room, or find a room; a low-level planner selects targets and paths.Source: Figure 2 · p. 3

A. Hierarchical Scene Graph

  • The online hierarchical scene graph combines room, frontier, navigation, object, and metric-semantic mesh layers.Hydra supports reconstruction and graph optimization, while the method augments the graph with semantic room labels, key views, frontier nodes, and a navigation layer.Source: A. Hierarchical Scene Graph · p. 3
  • Room nodes store semantic labels and key views, while frontier nodes represent explored-to-unexplored boundaries and carry open-vocabulary semantic features.Navigation nodes encode traversability, and frontier nodes connect to nearby navigation nodes and objects.Source: A. Hierarchical Scene Graph · p. 3
  • Object nodes store their centroid, bounding box, semantic label, and associated mesh, which provides a dense representation for constructing higher-level graph layers.Source: A. Hierarchical Scene Graph · p. 3
  • An open-vocabulary occupancy map fuses RGB-D-projected semantic features, while a generalized Voronoi diagram and wavefront frontier detection provide navigation and frontier layers.The map maintains occupancy state and running averages of semantic features for its cells.Source: A. Hierarchical Scene Graph · p. 3

B. Floorplan Prior

HFLEX-EQA uses an estimated floorplan graph as a weak global structural prior alongside the online scene graph. Its room labels and connectivity guide reasoning about which room types may lead toward a question-relevant place, without requiring exact floorplan localization.

B. Floorplan Prior

  • The floorplan graph represents room instances as semantically labeled nodes connected by edges, providing global topology rather than metric geometry or agent localization.Source: B. Floorplan Prior · p. 3
  • The floorplan is a weak prior that may omit rooms, mislabel them, or contain incorrect connectivity, so the method does not assume a complete or fully accurate map.Source: B. Floorplan Prior · p. 3
  • Online room labels and floorplan connectivity help identify room types likely to lead toward a question-relevant place without exact floorplan localization.Simulation floorplans are constructed from HM3D semantic annotations; real-world floorplans are made manually from known room layouts.Source: B. Floorplan Prior · p. 4

C. High-Level VLM Planner

The high-level VLM planner reasons over scene and floorplan graphs, task-relevant visual memory, the question, and exploration history. It selects an exploration mode and target or returns an answer with confidence.

C. High-Level VLM Planner

  • The planner selects explore room, go to objects, or find room with a corresponding room, object, or room-class target, or returns an answer and confidence.Source: C. High-Level VLM Planner · p. 4
  • Relevant visual memory selects task-relevant images from the current view, room key views, and available object-inspection views instead of using the complete observation history.Source: C. High-Level VLM Planner · p. 4
  • Before task planning, a VLM predicts each room’s semantic label from its key views and the semantic labels of objects in that room.Source: C. High-Level VLM Planner · p. 4
  • At each iteration, the planner combines serialized scene and floorplan graphs with visual memory, the question, decision history, and optional answer choices.Source: C. High-Level VLM Planner · p. 4
  • An answer is returned and the episode ends when its confidence meets κ_min; otherwise, the low-level planner executes the selected mode and target.Source: C. High-Level VLM Planner · p. 4

D. Low-Level Planning

Specialized low-level planners execute the selected exploration mode by exploring known rooms, obtaining more informative object views, or searching for an undiscovered room type. Their goals use room, frontier, and object nodes, with floorplan structure and semantic cues guiding room discovery.

D. Low-Level Planning

  • For a known room, the planner predicts question-relevant object labels and navigates to the reachable frontier with the strongest semantic match, oriented toward unexplored space.Source: D. Low-Level Planning · p. 4
  • For a relevant observed object, candidate viewpoints are scored for visibility and image coverage; the chosen view is added to visual memory, while angle and illumination are not modeled.Source: D. Low-Level Planning · pp. 4–5
  • For an undiscovered target room type, the planner matches the current room label to possible floorplan locations and uses neighboring room classes and transition distances to guide the search.Source: D. Low-Level Planning · p. 5

A. Ground truth occupancy map and rooms

HFLEX-EQA uses the floorplan and frontier semantics to direct exploration toward a target room type. Its planner selects a frontier using target-room transition cues, floorplan progress, and unexplored-region size.

A. Ground truth occupancy map and rooms

  • Floorplan weight w(c) favors room classes closer to the target in the floorplan graph; it is zero when the current room cannot be matched or no path exists.When the agent is not localized, all instances of its current room type are treated as possible locations.Source: Figure 3 · p. 5
  • The transition score uses maximum cosine similarity between frontier features and target-room prompt embeddings, favoring frontiers resembling entrances into that room.The prompts include descriptions such as a hallway to, or opening into, the target room.Source: A. Ground truth occupancy map and rooms · pp. 5–6
  • The highest-scoring frontier becomes the next exploration target, favoring directions consistent with the floorplan and larger unexplored regions.The floorplan-guided term and normalized frontier size contribute alongside semantic frontier scores.Source: Figure 3 · p. 5, A. Ground truth occupancy map and rooms · p. 6
  • The high-level VLM planner chooses the missing information and exploration mode, while low-level planning uses the scene graph, semantic frontiers, and floorplan to gather it.Source: A. Ground truth occupancy map and rooms · p. 6

A. Implementation Details

The implementation fixes room-view, planner, and exploration parameters across the reported experiments.

A. Implementation Details

  • Implementation retains at most three question-relevant room views and sets find-room weights to (α, β, γ) = (0.35, 0.55, 0.1).Explore-room uses N = 6; go-to-objects uses ϵ = 1 and (λvis, λimg) = (0.7, 0.3). GPT-5.5 is used for real-world experiments, and κmin = 0.8 throughout.Source: A. Implementation Details · p. 6

B. Simulation Results

HFLEX-EQA achieves the highest success rate across evaluated VLMs on both benchmarks, while its ablations and ground-truth-semantics analysis support the reported benefits of its components and structural prior.

  • Using Habitat on HM3D, HFLEX-EQA evaluates 108 OpenEQA and 114 ExploreEQA single-floor episodes with semantic annotations; generated topological floorplans encode semantic room labels and connectivity, not metric geometry or agent localization.Source: B. Simulation Results · p. 6
  • Across VLMs, HFLEX-EQA has the highest success rate on both datasets, reaching up to 75.0% on OpenEQA and 63.2% on ExploreEQA with stronger models.Source: Table I · p. 6, Table IV · p. 7
  • HFLEX-EQA’s higher success accompanies more planning than GraphEQA, whose earlier termination comes with lower success; ExploreEQA generally uses more iterations and longer trajectories.Source: B. Simulation Results · p. 7
  • Ablations show success falls without the floorplan prior, viewpoint selection, or semantic frontiers, while adding floorplans to GraphEQA’s prompt has mixed dataset effects.Source: B. Simulation Results · pp. 7–8
  • With ground-truth semantics, both HFLEX-EQA and GraphEQA improve, but HFLEX-EQA remains ahead.Source: B. Simulation Results · p. 8

C. Real-world Results

Real-world tests deploy HFLEX-EQA on a quadruped in unseen indoor settings, answering diverse question types; replayed no-choice answers remain correct and planner modes match 47/50 multiple-choice runs. The conclusion presents hierarchical planning, structured representations, and floorplan priors as beneficial for EQA.

C. Real-world Results

  • Perception, mapping, and hierarchical planning run onboard in real time; only VLMs are queried through their APIs.Source: C. Real-world Results · p. 8
  • 47/50 planning modes matched the multiple-choice runs, and replayed free-form answers remained correct without answer choices.Source: C. Real-world Results · p. 8
  • Planner mode consistency remained stable under visual perturbations, while answer success decreased under strong blur.Source: C. Real-world Results · p. 8

VI. CONCLUSIONS

  • The conclusion presents combining hierarchical planning, structured scene representations, and floorplan priors as beneficial for EQA.Source: Figure 5 · p. 8, §VI · p. 8
Loading 2609.26360v1…