Bird's-Eye-View Scene Graph for Vision-Language Navigation

1. Introduction

• Panoramic observations lack scene layouts and 3D structure, while occlusion makes it difficult to identify instructed objects and landmarks.Multiple candidate nodes can correspond to the same subview, and panoramic views make spatial relations such as which chair is furthest from a door difficult to ground.Source: §1 · p. 1
• At each step, BSG predicts BEV grid-level and graph-level decision scores, combined with a panoramic subview-selection score.Local BEV representations collected at navigable nodes are connected topologically into a global scene graph.Source: §1 · p. 2
• Multi-view image features are transformed into BEV representations, where 3D detection predicts oriented boxes encoding object geometry and semantics.Node embeddings use neighboring BEV grids and are updated by querying overlap regions across steps.Source: §1 · p. 2
• On REVERIE val unseen, BSG outperforms the state-of-the-art method by 5.14% and 3.21% in SR and RGS, respectively, and improves R2R test SR and SPL by 4% and 3%, respectively.The approach was evaluated on REVERIE, R2R, and R4R.Source: §1 · p. 2

2. Related Work

• VLN research has developed transformer-based embeddings, additional supervision, planning strategies, and larger benchmarks and platforms.Planning strategies include self-correction, global action spaces, map building, knowledge prompts, and IL/RL ensembles.Source: §2 · p. 2
• Panoramic subviews can cause action ambiguity and miss geometric cues, while navigation maps range from dense metric maps to sparse topological and combined topo-metric maps.Metric maps include geometry-based SLAM and learnable semantic maps; topological maps model relations among sparse nodes.Source: §2 · p. 2
• Existing map-based methods neglect 3D perception, whereas BSG uses 3D detection to encode scene layouts and geometric cues for path planning.Scene representations are described as needing both object semantics and layout composition.Source: §2 · p. 2
• BSG uses camera-based BEV perception supervised by oriented 3D bounding boxes from Matterport3D to capture object-level geometry for navigation.This contrasts with the cited previous point-cloud-based detection approaches.Source: §2 · p. 2

3. Approach

• BSG projects multi-view images into BEV, encodes the features with 3D detection, and fuses graph-level and grid-level scores for action prediction.The approach also uses panoramic subview selection scores.Source: Figure 2 · p. 3

3.1. BSG Construction

• Each navigable location receives a node representation from neighboring BEV grids; cross-attention across overlapping regions integrates observations from successive steps.The BEV feature is also used to predict bounding boxes carrying object-level geometry information.Source: §3.1 · p. 3

3.2. BEV-based Navigation Action Prediction

• The agent predicts global graph-level and local grid-level scores, pools grid scores into candidate scores using distance-dependent weights, then fuses local and global scores.For unconnected nodes, a backtracking score sums decision scores of visited candidate nodes.Source: §3.2 · p. 4

3.3. BEV Representation Encoding

• A 3D detection head trained on Matterport3D2 uses bipartite matching with focal classification and L1 box-regression losses on BEV features.The authors state BSG is not limited to a particular BEV model.Source: §3.3 · pp. 4–5

3.4. Implementation Details

• BEV detection and navigation policy networks are trained separately; the implementation uses BEVFormer features, pretraining and finetuning, and online BSG updates at inference.The default BEV queries are 11 × 11 with four reference points per query, and the fused weight is 0.5.Source: §3.4 · p. 5

4. Experiment

• The experiments first report benchmark performance, then test core model designs and analyze BEV perception’s impact on navigation.The diagnostic studies use val unseen splits of REVERIE and R2R.Source: §4 · p. 5, §4.2 · p. 7

4.1. Performance on VLN

• BSG outperforms prior methods across REVERIE, R2R, and R4R; on REVERIE val unseen, it exceeds DUET by 5.14% SR, 1.86% SPL, and 3.21% RGS.On R2R test unseen, SR improves from 69 to 73, SPL from 59 to 62, and NE from 3.65 to 3.19; on R4R, SR rises from 45 to 47.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.2. Diagnostic Experiment

• Adding the BEV branch improves the basic agent across metrics, while detection loss adds 2.87% REVERIE SR, 3.15% REVERIE RGS, and 2.13% R2R SPL.Ablations also find that both graph- and grid-level decision spaces help navigation; Bivariate Gaussian weights outperform uniform weights.Source: §4.2 · pp. 7–8, Table 7 · p. 7, Table 4 · p. 7

4.3. Analysis on BEV Encoding

• BEVFormer leads the tested BEV models with 0.299 mAP and 0.488 mAR; oriented bounding boxes improve perception and navigation across tasks.Scale and perception range affect detection accuracy, and the appropriate range should account for indoor dimensions and structure; reference points sample multi-view features into BEV features.Source: §4.3 · p. 8, Table 9 · p. 8, Table 10 · p. 8

• Panoramic VLN agents lack 3D layouts and geometric cues, which the conclusion says limits their planning ability.Source: §5 · p. 8
• BSG is proposed for 3D-perception-based VLN, giving agents access to scene and object layouts.Source: §5 · p. 8
• Fusing BSG-based and BEV grid-level action scores yields promising results and highlights BEV perception's potential in VLN.Source: §5 · p. 8

6. Model Details

• The combined model converts actions to nodes for the discrete VLN environment and uses a panoramic branch alongside BSG.BSG complements methods centered on panoramic decision spaces.Source: §6 · p. 9

6.1. Different Decision Space

• Panoramic decision spaces can map multiple candidate nodes to one view, while BEV grid decisions represent each node through neighboring grid features.The described panoramic space discretizes views into 36 angles; earlier low-level control uses six actions.Source: §6.1 · p. 9

6.2. Complementary to Existing Methods

• The model combines global scores over navigable nodes with local scores first predicted at BEV-grid level, complementing panoramic methods.Global scoring includes previously visited and observed nodes; local scores cover navigable nodes from the current node.Source: §6.2 · p. 9

6.3. Detailed Network Architecture on REVERIE

• For REVERIE, object features are combined with BEV features before a cross-modal transformer predicts grid-level decisions and object scores.Object features use ViT-B/16 and orientation features for heading and elevation.Source: §6.3 · p. 9

6.4. Pretraining Objectives

• R2R/R4R use MLM, MRC, and SAP-PM in pretraining, while REVERIE adds OG; MRC is panoramic-only and BEV adds progress monitoring.MLM masks 15% of instruction tokens, and OG averages matching scores from the panoramic and BEV branches.Source: §6.4 · p. 10

6.5. Finetuning Objectives

• Finetuning alternates Teacher-Forcing and Student-Forcing behavior cloning for action prediction, with REVERIE also using OG loss.The OG loss is balanced with the other losses using a predefined weight of 0.20.Source: §6.5 · p. 10

7.1. Evaluation Metrics

• Evaluation uses SR, TL, OSR, SPL, and NE for VLN, plus grounding metrics for REVERIE and path-fidelity metrics for R4R.The section defines SR using a final-position threshold of less than 3 m from the goal.Source: §7.1 · p. 10

7.2. Training Details

• The combined model is pretrained for 100k iterations and finetuned for 25k, with best checkpoints selected using validation-unseen metrics.REVERIE selection uses SPL; R2R and R4R use the sum of SR and SPL.Source: §7.2 · p. 10
• BEVFormer uses a static detection model, while LSS and BEVDepth use ResNet-50 with 256 × 704 images and train for 24 epochs.The BEVFormer detector uses ViT-B/16 and trains for 500 epochs; LSS and BEVDepth use AdamW.Source: §7.2 · p. 11

• Detailed R2R instructions provide additional information that enables a more accurate navigation strategy in the same-path comparison.Source: §8 · p. 11
• Matterport3D2 detection reports AP of 0.535 for bed and 0.394 for sofa, while nearly flat objects such as picture and plant are harder to detect.The passage notes that Matterport3D2 detection performance could be improved further.Source: §8 · p. 11

9.1. Detailed Annotation Process

• Raw camera images preserve projection parameters absent from stitched skybox panoramas; they were used at 1280×1024 resolution for 3D detection.Each panorama contains 6 × 3 raw images; prior skybox processing stitches them and the simulator splits them into 12×3 images.Source: §9.1 · p. 11
• Oriented bounding boxes were annotated under the LiDAR coordinate system using PCA on object-segment x and y coordinates, fitting object outlines more tightly than axis-aligned boxes.The original dataset supplied axis-aligned boxes, which the authors considered insufficiently accurate for 3D detection.Source: §9.1 · p. 11

9.2. Detailed Dataset Statistics

• Matterport3D2 collects six multi-view images only at viewpoints containing objects and uses the existing train-seen, validation-unseen, and test-unseen splits.The dataset statistics are presented in Table 11.Source: §9.2 · p. 11

10. Discussion

• The authors report that BSG improves model performance and enhances decision-making with grid-level scores, while Matterport3D2 provides oriented boxes for future indoor 3D-detection research.The code is released under MIT; the study uses publicly available datasets and implements models on MMDetection3D.Source: §10 · p. 11
• The agents were developed and evaluated in virtual environments and trained primarily with stationary objects, so moving-object collisions remain a real-world deployment concern.The authors call for further safety research, including speed sensors and additional environments for studying potential damage risks.Source: §10 · pp. 11–12
• Figure 7 presents visual results for the same ground-truth path on REVERIE and R2R; Table 12 reports Matterport3D2 test-unseen results.The supplied captions identify these scopes but do not provide result values.Source: Figure 7 · p. 12, Table 12 · p. 12