PaperlayerView exact revision on arXiv

Source-linked AI summary

Symphonize 3D Semantic Scene Completion with Contextual Instance Queries

Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, Xinggang Wang

arXiv:2306.15670v2cs.CVcs.RO

TL;DR

Voxel-wise SSC approaches tend to neglect instance semantics and scene context. Symphonies uses image-derived instance queries to model scenes, reporting state-of-the-art results on two benchmarks while noting that absent instance-level annotations may affect performance.

  • Problem

    Voxel-wise modeling and pixel-voxel projection tend to neglect higher-level instance semantics in 3D scene completion.

    Source: §1 · p. 1

  • Method

    Symphonies uses image-derived contextual instance queries and Serial Instance-Propagated Attentions to integrate instance semantics into scene modeling.

    Source: §1 · p. 1

  • Results

    15.04 mIoU on SemanticKITTI and 18.58 mIoU on SSCBench-KITTI-360; Symphonies reports state-of-the-art performance on both benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Symphonies integrates instance-centric semantics and scene context from images and volumes to address limitations posed by geometric ambiguity in prior voxel-wise modeling.

    Source: §5 · p. 8

    The absence of instance-level annotations may affect the performance of instance-based methods.

    Source: §5 · p. 8

Abstract

from arXiv · show

`3D Semantic Scene Completion (SSC) has emerged as a nascent and pivotal undertaking in autonomous driving, aiming to predict voxel occupancy within volumetric scenes. However, prevailing methodologies primarily focus on voxel-wise feature aggregation, while neglecting instance semantics and scene context. In this paper, we present a novel paradigm termed Symphonies (Scene-from-Insts), that delves into the integration of instance queries to orchestrate 2D-to-3D reconstruction and 3D scene modeling. Leveraging our proposed Serial Instance-Propagated Attentions, Symphonies dynamically encodes instance-centric semantics, facilitating intricate interactions between image-based and volumetric domains. Simultaneously, Symphonies enables holistic scene comprehension by capturing context through the efficient fusion of instance queries, alleviating geometric ambiguity such as occlusion and perspective errors through contextual scene reasoning. Experimental results demonstrate that Symphonies achieves state-of-the-art performance on challenging benchmarks SemanticKITTI and SSCBench-KITTI-360, yielding remarkable mIoU scores of 15.04 and 18.58, respectively. These results showcase the paradigm's promising advancements. The code is available at https://github.com/hustvl/Symphonies.

1. Introduction

Voxel-wise 3D scene completion often overlooks instance semantics, motivating Symphonies to use image-derived instance queries for scene modeling and 2D-to-3D reconstruction. The paper reports state-of-the-art results on SemanticKITTI and SSCBench-KITTI-360.

1. Introduction

  • Voxel-wise modeling and pixel-voxel projection tend to neglect higher-level instance semantics, leaving 3D scene modeling vulnerable to occlusion and perspective errors.The introduction asks how instances can steer scene modeling and 2D-to-3D reconstruction.Source: §1 · p. 1
  • Symphonies uses image-derived contextual instance queries to integrate instance semantics and scene context into scene modeling.Serial Instance-Propagated Attentions interact with image and voxel features, while query fusion enriches contextual scene reasoning.Source: §1 · pp. 1–2
  • The Depth-Rectified Voxel Proposal Layer refines initial geometry by elevating 2D image features to the scene’s implicit surface.The introduction presents this layer alongside contextual reasoning as part of Symphonies.Source: §1 · p. 2
  • 15.04 mIoU on SemanticKITTI and 18.58 mIoU on SSCBench-KITTI-360; Symphonies reports state-of-the-art performance on both benchmarks.The introduction describes these results as significantly outperforming previous vision-based methods.Source: §1 · p. 2

2. Related Works

Prior SSC work spans volume-based and view-volume approaches, while camera-based perception also includes BEV representations and feature aggregation. Symphonies distinguishes itself by using instance queries to model semantics and context without additional BEV or occupancy prediction networks.

2. Related Works

  • SSC methods include TSDF-driven volume networks and view-volume networks that convert RGB or depth features into 3D volumes.Camera-based SSC work also includes TPV representations, two-stage voxel diffusion, implicit stereo depth, and Transformer-based decoding.Source: §2 · p. 2
  • Symphonies integrates instance queries to enrich scene context and mitigate geometric ambiguity without additional BEV or occupancy prediction networks.The paper also notes occupancy as a general scene descriptor for a wide scope of driving tasks.Source: §2 · p. 3
  • Camera-based 3D perception includes 2D detectors extended to 3D boxes, Transformer approaches using BEV or position embeddings, and IPM-based BEV segmentation.Other described approaches use angle- and radius-specific embeddings or cross-attention over BEV queries.Source: §2 · p. 3

3. Scene from Instances

Symphonies models 3D scenes from image-derived instance queries, combining instance semantics with scene context. Its method initializes geometry-aware voxel proposals, then uses serial attention to exchange information between image, instance, and scene representations.

3.1. Overview

  • RGB images are encoded into multi-scale features, while learnable instance queries and volumetric scene features form the decoder’s inputs.The instance-aware encoder uses a Deformable Transformer and MaskDINO-pretrained weights; the depth estimator supplies depth for the proposal layer and instance reference points.Source: §3.1 · pp. 3–4

3.2. Depth-Rectified Voxel Proposal Layer

  • Depth-rectified voxel proposals initialize features on the depth-derived implicit surface, providing coarse geometry awareness for later instance-level aggregation.Camera-to-world mapping places image points in the scene volume, and Deformable Attention aggregates multi-scale image features at proposal locations.Source: §3.2 · p. 4

3.3. Symphonies Decoder Layer

  • Serial Instance-Propagated Attentions exchange information across image, instance, and scene features, while instance self-attention captures query relations and context.Scene-instance attention aggregates scene features from instance queries, and scene self-attention propagates voxel features across the scene; the field of view excludes invisible voxels.Source: §3.3 · pp. 4–5

3.4. Losses

  • The training objective combines Scene-Class Affinity Loss with class-frequency-weighted cross-entropy, and applies auxiliary decoder-layer losses at half scale.The Scene-Class Affinity Loss is applied to semantic and geometric predictions.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate Symphonies on SemanticKITTI and SSCBench-KITTI-360, with comparisons, component ablations, and visual analyses. Results cover benchmark performance as well as the contributions of the model’s components.

4.1. Dataset and Metric

  • Evaluation uses RGB-only input on two densely annotated urban-driving datasets, reporting IoU and mIoU for occupied voxels and voxel-wise semantics.SemanticKITTI has 20 classes and SSCBench-KITTI-360 has 19; training uses 30 epochs on four NVIDIA 3090 GPUs.Source: §4.1 · p. 5, §4 · p. 5, §4.2 · p. 5

4.3. Main Results

  • 2.72 mIoU on SemanticKITTI and 4.77 mIoU on SSCBench-KITTI-360 over camera-based counterparts, with particular strength on instance classes.VoxFormer reports marginally higher IoU on SemanticKITTI, while Symphonies outperforms LiDAR-based methods on mIoU.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 6

4.4. Ablation Studies

  • 2.15 mIoU from encoder pretraining, 0.75 mIoU from the Depth-Rectified Voxel Proposal Layer, and 5.38 IoU from the Symphonies Decoder support the component design.The decoder gain is attributed to dynamic instance modeling and context capture; the proposal-layer gain to more accurate geometry.Source: Table 3 · p. 7, §4.4 · p. 7
  • Instance-image and instance-scene cross-attention improve decoder performance, while instance self-attention adds 0.52 IoU.Instance-image cross-attention contributes less, which the authors attribute to image features already being captured in the Voxel Proposal Layer.Source: Table 4 · p. 7, §4.4 · p. 7
  • The stereo Voxel Proposal Layer improves occupancy prediction over FLoSP and the monocular AdaBins-based layer.The authors link the improvement to more precise depth estimation and reduced geometric ambiguity.Source: Table 5 · p. 7, §4.4 · p. 7
  • With a pre-trained encoder, adding instance queries improves performance by 1.36 mIoU; pretraining alone contributes 0.48 mIoU.The authors attribute the query benefit to the encoder’s enhanced instance awareness.Source: Table 6 · p. 7, §4.4 · p. 7

4.5. Visualizations

  • Visualizations show more detailed car and trunk predictions and clearer building and vegetation layouts than MonoScene, while attention maps show queries attending to corresponding image and scene regions.The attention analysis also reports activation of semantically related scene regions during scene-instance cross-attention.Source: §4.5 · pp. 7–8, Figure 4 · p. 8

5. Conclusion

Symphonies integrates instance-centric semantics and scene context into 3D Semantic Scene Completion, addressing geometric ambiguity associated with prior voxel-wise modeling. The authors report superiority over existing methods while identifying absent instance-level annotations as a limitation.

5. Conclusion

  • Symphonies integrates instance-centric semantics and scene context from images and volumes to address geometric ambiguity in prior voxel-wise modeling.The authors report that extensive experiments demonstrate superiority over existing methods.Source: §5 · p. 8
  • The authors describe Symphonies as an initial step in using instance-centric queries, noting that absent instance-level annotations may affect instance-based methods.Source: §5 · p. 8
  • Extending the paradigm to multi-view and temporal scenarios is identified as future work.Source: §5 · pp. 8–9

A. Additional Experimental Results

Additional SemanticKITTI validation results are reported in Table 7, where Symphonies consistently achieves state-of-the-art performance in line with the paper's primary findings.

A. Additional Experimental Results

Loading 2306.15670v2…