Source-linked AI summary
Cross-view Semantic Segmentation for Sensing Surroundings
Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, Bolei Zhou
TL;DR
The paper addresses the lack of spatial information in ordinary image-level semantic masks and the scarcity of real-world top-down annotations. It introduces cross-view semantic segmentation and VPN, which learns from simulated multimodal, multi-angle observations and transfers to real scenes through domain adaptation. Experiments in synthetic environments and on a robot show effective spatial understanding from 2D image input.
Problem
Image-level semantic masks do not encode surrounding spatial layout, while real-world top-down semantic annotations are difficult to collect.
Method
VPN aggregates features from multiple first-view angles and modalities, trains in simulation, and uses pixel-level and output-space domain adaptation for real-world transfer.
Results
VPN improves with additional views, achieves 84.7% pixel acc. and 33.2% mIoU on CARLA using six RGB views, and outperforms the 3D geometric method.
Takeaways & Limitations
The resulting lightweight top-down semantic map supports surrounding awareness for mobile robots and can be an alternative to costly 3D methods when object height is not essential.
Abstract
from arXiv · showhide
Sensing surroundings plays a crucial role in human spatial perception, as it extracts the spatial configuration of objects as well as the free space from the observations. To facilitate the robot perception with such a surrounding sensing capability, we introduce a novel visual task called Cross-view Semantic Segmentation as well as a framework named View Parsing Network (VPN) to address it. In the cross-view semantic segmentation task, the agent is trained to parse the first-view observations into a top-down-view semantic map indicating the spatial location of all the objects at pixel-level. The main issue of this task is that we lack the real-world annotations of top-down-view data. To mitigate this, we train the VPN in 3D graphics environment and utilize the domain adaptation technique to transfer it to handle real-world data. We evaluate our VPN on both synthetic and real-world agents. The experimental results show that our model can effectively make use of the information from different views and multi-modalities to understanding spatial information. Our further experiment on a LoCoBot robot shows that our model enables the surrounding sensing capability from 2D image input. Code and demo videos can be found at \url{https://view-parsing-network.github.io}.
I. INTRODUCTION
The paper introduces cross-view semantic segmentation to infer a spatial top-down semantic map from first-view observations, addressing the limits of image-level masks and scarce real-world annotations. VPN aggregates multi-angle, multimodal observations, learns in simulation, and transfers to real-world scenes through domain adaptation.
- Motivation: Standard semantic segmentation recognizes image content but does not provide the spatial information needed to understand surroundings and support navigation.
- Task: Cross-view semantic segmentation predicts a top-down semantic map from first-view observations, representing object layout and surrounding spatial structure.The map is a 2.5D representation covering discrete objects and stuff classes such as floor and wall.
- Challenge: Real-world top-down semantic annotations are difficult to collect, motivating the use of simulation environments with controllable viewpoints and modalities.Synthetic data is transferred to real scenes through domain adaptation to reduce the domain gap.
- Approach and evaluation: VPN aggregates features from multiple first-view angles and modalities into a top-down semantic map, and is evaluated in House3D, CARLA, and real-robot demonstrations.
II. RELATED WORK
Related work includes image-based semantic segmentation, semantic mapping, layout estimation, free-space estimation, road-layout estimation, and view synthesis. The proposed framework estimates a top-down semantic map directly from images rather than using an intermediate 3D reconstruction step.
- Semantic understanding: Prior semantic segmentation methods assign labels pixel-wise within a single image, while semantic mapping abstracts environments for communication with robots.
- Layout estimation: Previous layout-estimation methods commonly use layout annotations or geometric constraints, whereas this framework estimates the top-down map directly from images.
- View and domain transfer: View synthesis and simulation-based recognition provide related directions for transferring visual models across viewpoints and domains.
C. Learning in Simulation Environments
VPN is trained in simulation to transform multiple first-view feature maps into a top-down representation. Its View Transformer Module addresses the need for each top-down pixel to use information from all input views rather than a local receptive field.
- Learning in Simulation Environments: Simulation environments provide controllable camera placement and multimodal observations for training cross-view semantic segmentation models.
- Framework: VPN encodes observations from multiple angles and modalities, transforms their feature maps, and aggregates them into one top-down-view feature map before decoding.The described pipeline uses N angles and M modality-specific encoders; Fig. 2 gives N = 6 and M = 2 as an example.
- View Transformer Module: A standard encoder-decoder performs poorly because each top-down pixel must consider all first-view feature maps rather than only a local input region.
- View Transformer Module: The View Transformer Module learns dependencies across first-view and top-down spatial locations and combines View Relation and View Fusion modules.Each input has its own View Relation Module, after which the resulting top-down feature maps are fused.
C. Sim-to-real Adaptation
The framework adapts VPN from simulated environments to real-world scenes through pixel-level semantic-mask transfer and adversarial output-space alignment.
- Pixel-level adaptation: Pixel-level adaptation converts real-world RGB images into synthetic-style semantic masks to reduce the domain shift.Semantic masks provide sufficient information without a texture gap and are easy to transfer.
- Pixel-level adaptation: The real RGB image is parsed into a semantic mask and remapped across real-world and simulation categories before VPN processing.The mapping constructs concept correspondences between the real world and the simulation environment.
- Output space adaptation: Output-space adaptation initializes the generator with a simulation-trained VPN and adversarially aligns target-domain predictions with source-domain outputs.The generator uses segmentation loss on source images, while a discriminator distinguishes source- from target-domain feature maps after softmax.
- Output space adaptation: The generator objective combines cross-entropy segmentation loss with an adversarial loss weighted by λ.The discriminator is optimized with binary source-and-target classification cross-entropy.
D. Network configuration
VPN uses lightweight view-specific processing and a View Transformer Module designed to preserve spatial detail while aggregating views and modalities.
- View encoder and decoder: ResNet-18 serves as the encoder, with the last residual block and average-pooling layer removed to retain higher-resolution feature maps.The decoder uses a pyramid-pooling module.
- View Transformer Module: The View Transformer Module is evaluated through an ablation study focused on its contribution to the network.The supplied passage identifies the study but does not report its numerical results.
- View Transformer Module: Each view relation module uses a two-layer MLP, while the view fusion module adds features to preserve shape consistency and efficiency.The view relation module has equal input and output dimensions of H_IW_I.
- Sim-to-real: The sim-to-real configuration uses a 4-view VPN generator, a five-convolution-layer discriminator, and HRNet pretrained on CityScapes for real-image semantic masks.Each discriminator convolution is followed by leaky ReLU with parameter 0.2 except the last layer.
IV. EXPERIMENTS
The experiments progress from dataset overview to synthetic evaluation and then real-world evaluation of VPN trained in simulation.
- IV. EXPERIMENTS: The evaluation covers dataset descriptions, synthetic House3D and CARLA performance, and real-world performance after simulation training.The experiments are organized into Sections IV-A, IV-B, and IV-C.
A. Benchmarks
The benchmarks include two synthetic cross-view datasets and one real-world dataset, spanning indoor, driving, and multi-directional observations.
- Datasets: The benchmark suite contains synthetic House3D and CARLA cross-view datasets plus the real-world nuScenes dataset.These datasets provide the evaluation settings for cross-view semantic segmentation.
- House3D cross-view dataset: House3D pairs eight views captured 45 degrees apart with top-down semantic masks and RGB, depth, and semantic modalities.Its training and validation sets contain 143k pairs from 342 scenes and 20k pairs from 68 scenes, respectively.
- NuScenes dataset: nuScenes supplies six-direction RGB observations in different modalities, with 919 samples for unsupervised training and 515 masked samples for evaluation.The evaluation masks are binary top-down-view masks.
- CARLA cross-view dataset: CARLA provides 28,000 annotated pairs from 14 driving episodes, each containing six view sets from fixed directions.The pairs include different input modalities and top-down annotations.
- Evaluation: Performance is reported with pixel accuracy and mean IoU, measuring correctly classified pixels and predicted-versus-ground-truth intersection over union.The comparison baselines are a depth-and-RGB 3D geometric method and the X-Fork cross-view synthesis architecture.
1) Results of VPNs:
VPN performance improves with more views and complementary modalities, while its view transformer and direct top-down prediction outperform simpler baselines and geometric reconstruction.
- 91.4% pixel acc. and 41.2% mIoU form the House3D upper bound from direct top-view RGB inputs.
- 84.7% pixel acc. and 33.2% mIoU are achieved on CARLA with a 6-view RGB-input VPN.
- More input views rapidly improve single-modality segmentation, while Depth-Semantic VPN achieves the best multi-modal performance.
- VPN outperforms a feature-summing encoder-decoder baseline, whose multi-view performance can fall below its single-view result because of poor fusion.
- VPN outperforms the 3D geometric method, which fails to reconstruct directly unobserved objects such as desks behind chairs.
C. Results of sim-to-real adaptation
The model is transferred from simulated CARLA data to real-world nuScenes data through unsupervised domain adaptation, producing qualitative surrounding maps and real-world evaluation results.
- A 6-view semantic VPN trained on CARLA predictions is transferred to nuScenes using unsupervised domain adaptation.
- After adaptation, VPN roughly segments road shapes such as crossroads and sketches relative locations of cars and buildings.
- The exploration procedure updates the agent with predicted top-down-view free-space and state maps while assuming the agent is centered in each map.
A. Result and comparison
The exploration system evaluates navigation against random, ground-truth, and imitation-learning baselines using repeated coverage measurements, and VPN-based maps support efficient environment exploration.
- The comparison includes random walk, top-down-view navigation with a ground-truth map, and imitation learning without a top-down-view.
- The imitation-learning navigation agent uses four first-view depth images, a state map, and 729 training plus 121 validation trajectories.
- Figure 5 presents sample exploration trajectories, with start and end points marked red and green respectively.
- With VPN's predicted top-down-view map, the agent can efficiently explore the environment.
VI. REAL ROBOT EXPERIMENT
The LoCoBot experiment evaluates whether VPN can support semantic navigation from RGB observations. The results demonstrate surrounding awareness through a lightweight top-down-view semantic map, especially when object height is not essential.
- Semantic navigation: The LoCoBot robot performs semantic navigation by identifying and reaching targets specified by semantic tokens such as “go to the bench”.At the initial location, the robot takes 8 RGB images from different angles.
- Experimental outcome: VPN can be applied to mobile robots to facilitate surrounding awareness through a lightweight and efficient top-down-view semantic map.
- Practical scope: When object height is not essential, VPN could be an alternative to traditional 3D-based methods because those methods are costly in data memory and computation.