IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection

1. Introduction

• Point clouds provide 3D geometry but are sparse and lack detailed texture, while synchronized multi-view images add dense semantic and texture information.The paper notes that point clouds may be especially sparse beyond 100 meters in outdoor nuScenes scenarios.Source: §1 · p. 1
• Scene-level BEV fusion can overlook foreground instances, which occupy fewer grid cells than background and have limited local context.Prior approaches may rely on decoder refinement, while some object-level methods do not model collaboration between scene and instance features.Source: §1 · pp. 1–2
• IS-Fusion uses HSF for multi-granularity scene context and IGF to select foreground candidates, model instance relationships, aggregate multimodal context, and return instance information to the scene.An Instance-to-Scene transformer yields a BEV representation intended for instance-aware 3D detection.Source: §1 · p. 2
• 72.8% mAP on the nuScenes validation set, outperforming BEVFusion by 4.3% mAP.The paper also reports surpassing CMT and SparseFusion by 2.5% and 1.8% mAP, respectively.Source: §1 · p. 2

2. Related Work

• Point-cloud detectors are grouped as point-based, voxel-based, and point-voxel fusion networks; voxel methods quantify points into regular grids for standard convolutions.Point-based methods can incur substantial computation in large scenes, while some point-voxel approaches use multiple proposal-refinement stages.Source: §2 · p. 2
• Multimodal 3D detection fusion is categorized as early, middle, or late according to when modalities are combined.Early fusion can be more sensitive to calibration errors; late fusion may limit interaction during proposal generation.Source: §2 · pp. 2–3
• Middle fusion aligns point-cloud features with multi-view image features on a unified BEV plane to simplify fusion; Yang et al. also fuse BEV and image perspective-view spaces to preserve modality-specific information.Source: §2 · p. 3
• IS-Fusion combines local instance-level and global scene-level fusion, unlike methods focused on either global scene fusion or instances alone.It exchanges information between instances and the scene for instance-centric tasks such as 3D object detection.Source: §2 · p. 3

3.1. Overall Framework

• Point-cloud BEV and image perspective-view features are fused by an encoder comprising HSF and IGF, then decoded into 3D detections.The point-cloud and image inputs use modality-specific encoders; the decoder uses a transformer architecture with attention layers and a feed-forward detection head.Source: §3.1 · pp. 3–4

3.2. Hierarchical Scene Fusion

• Point-to-Grid assigns image features to projected LiDAR points, compares points within each pillar to address calibration noise, and max-pools point information.The point features are obtained by bilinearly interpolating image features at projected image-plane coordinates.Source: §3.2 · p. 4
• Grid-to-Region groups grid features into M×M regions and uses shifted-region attention to let grids interact across regions and capture long-range dependencies.Grouping regions avoids the computational expense of global self-attention over all grid cells.Source: §3.2 · pp. 4–5
• HSF propagates information from individual points through BEV grids to regions, integrating local and global multimodal scene context.Point-to-Grid models inter-point or pixel correlations within each grid, while Grid-to-Region mines inter-grid and inter-region context.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

3.3. Instance-Guided Fusion

• IGF selects top-K instance candidates using predicted centerness, reasons over instance relationships, and aggregates local multimodal context from nearby feature locations.Its instance aggregation uses self-attention between candidates and considers a small set of neighbor locations to save computation.Source: §3.3 · p. 5
• Instance-to-Scene cross-attention lets each BEV grid feature acquire information from enriched instance features, producing the BEV representation used for detection.The enriched instances incorporate multimodal context before being merged into the scene feature.Source: §3.3 · pp. 5–6

4.1. Experimental Setup

• The nuScenes evaluation covers 10 object categories and uses mAP and NDS, with NDS combining translation, scale, orientation, velocity, and attribute.The dataset has 700 training, 150 validation, and 150 test sequences, each with approximately 40 frames and six calibrated images per point cloud; the implementation trains end-to-end for 10 epochs.Source: §4.1 · p. 6

4.2. Performance Benchmarking

• On nuScenes test, IS-FUSION improves over its TransFusion-L-based LiDAR baseline by 7.5% in mAP and 5.0% in NDS; on validation, it surpasses CMT and SparseFusion by 2.5% and 1.8% in mAP, respectively.The validation comparison reports comparable inference speed.Source: Table 2 · p. 7, §4.2 · p. 7

4.3.1 Component-wise Ablation

• On nuScenes validation, the full model with HSF and IGF achieves 72.8% mAP and 74.0% NDS; HSF improves mAP/NDS by 2.2%/1.6%, while IGF beats Baseline-LC by 1.5%/1.2%.Source: Table 3 · p. 7, §4.3.1 · p. 8
• Swin-T outperforms ResNet-50 and CSPNet, and increasing image resolution to 384 × 1056 brings a slight performance improvement.The comparisons vary image encoders and input resolutions.Source: Table 4 · p. 8, §4.3.1 · p. 8

4.3.2 Analysis of HSF

• Point-to-Grid adds 0.5% mAP/0.3% NDS over Baseline-LC, Grid-to-Region adds 1.8% mAP/1.2% NDS, and full HSF adds 2.2% mAP/1.6% NDS.The passage attributes the larger Grid-to-Region gain to its exploration of inter-grid and inter-region features.Source: Table 4 · p. 8, §4.3.2 · p. 8

4.3.3 Analysis of IGF

• K=200 and D=16 yield 70.9% mAP and 72.8% NDS; larger settings add no improvement, while IGF produces higher-response, more complete BEV patterns.K is the number of instance candidates and D the number of sampled locations on the multimodal feature.Source: Table 4 · p. 8, §4.3.3 · p. 8

• IS-FUSION achieves superior performance on the competitive nuScenes benchmark.Source: §5 · p. 8
• HSF captures hierarchical scene context with Point-to-Grid and Grid-to-Region transformer attentions, while IGF mines instances, explores inter-instance relationships, and incorporates local multimodal context.Source: §5 · p. 8
• Instance-to-Scene transformer attention encourages collaboration between instance and scene representations, a perspective potentially beneficial to instance-centric tasks.Source: §5 · p. 8