PaperlayerView exact revision on arXiv

Source-linked AI summary

MatchFusion: Explicit-Implicit Instance Matching for Spatio-Temporal Multimodal Autonomous Driving

Xiaoyu Li, Jiajia Fu, Long Shi, Tianyu Du, Ruihang Li, Xian Wu, Lijun Zhao, Yingtao Zhang, Lining Sun, Ruifeng Li

arXiv:2609.25860v1cs.CVcs.RO

TL;DR

Sparse instance interaction needs reliable correspondences despite geometric and semantic differences, while attention and structural association each have limitations. MatchFusion combines structural affinities with selective semantic refinement to guide spatial and temporal information exchange. On nuScenes it improves perception across front ends while reducing computation and memory, and temporal integration also improves SparseDrive without additional supervision.

  • Problem

    Sparse instance interactions need reliable correspondences despite geometric discrepancies and heterogeneous semantics, while attention-based alignment can be costly and structural association lacks contextual evidence for ambiguous matches.

    Source: §I · pp. 1–2

  • Method

    MatchFusion initializes affinities from geometry and category consistency, selectively refines plausible matches with instance embeddings, and uses a soft matchmap to guide residual aggregation for spatial and temporal interaction.

    Source: §I · p. 2

  • Results

    55.3% fewer FLOPs and 39.3% less GPU memory than SparseFusion accompany higher perception accuracy across nuScenes front ends; temporal integration also improves SparseDrive without additional supervision.

    Source: §I · p. 2

  • Takeaways & Limitations

    MatchFusion provides a unified, efficient instance-interaction mechanism for spatial and temporal multimodal perception, including integration into an end-to-end driving framework.

    Source: §V · p. 8

Abstract

from arXiv · show

Sparse instance representations provide a compact interface for spatial LiDAR-camera and temporal past-current interaction in multimodal perception and E2EAD. Effective interaction requires reliable instance correspondences despite geometric discrepancies and heterogeneous semantic representations. Attention-based methods exploit contextual semantics but often require specialized representation alignment, increasing computational overhead. In contrast, association based on structured object states is efficient and interpretable but lacks contextual evidence to resolve ambiguous matches. To combine these complementary strengths, we propose MatchFusion, a learnable instance matching and fusion module for spatio-temporal multimodal autonomous driving. MatchFusion initializes pairwise affinities using geometric similarity and category consistency, then selectively refines structurally plausible associations using instance embeddings. The resulting soft matchmap guides a common residual aggregation operator for adaptive information exchange. This unified matching-fusion formulation supports spatial LiDAR-camera and temporal past-current interaction, using multi-view image-plane geometry and motion-compensated BEV geometry as the respective structural priors. Experiments on nuScenes demonstrate consistent perception gains across diverse front-end configurations. Compared with a prior instance-centric fusion method, the MatchFusion-equipped system achieves higher perception accuracy while reducing FLOPs by 55.3% and GPU memory usage by 39.3%, with the matching-fusion module accounting for only 3.7% of total perception latency. Integrating temporal MatchFusion into SparseDrive further improves perception within an E2E framework without additional supervision. These results establish explicit-implicit matching as an effective and efficient mechanism for spatio-temporal instance interaction.

I. INTRODUCTION

MatchFusion addresses the need for reliable, efficient instance correspondences across modalities and time by combining structural matching with semantic refinement. On nuScenes, it improves perception across front ends while reducing computational and memory costs.

I. INTRODUCTION

  • Sparse instance interactions need reliable correspondences despite geometric discrepancies and heterogeneous semantic representations across modalities and time.Attention methods exploit contextual semantics but can require specialized alignment; structural association is efficient and interpretable but lacks contextual evidence for ambiguous matches.Source: §I · pp. 1–2
  • MatchFusion initializes affinities from geometric similarity and category consistency, selectively refines plausible matches with instance embeddings, and uses a soft matchmap for residual aggregation.Spatial interaction uses multi-view image-plane geometry; temporal interaction uses motion-compensated BEV geometry.Source: §I · p. 2
  • Compared with SparseFusion, MatchFusion achieves higher perception accuracy while reducing FLOPs by 55.3% and GPU memory usage by 39.3%.The matching-fusion module accounts for 3.7% of total perception latency, and temporal integration further improves SparseDrive without additional supervision.Source: §I · p. 2
  • MatchFusion's unified formulation supports spatial LiDAR-camera and temporal past-current interaction through setting-specific geometric priors and common residual aggregation.The module operates on instance representations and is decoupled from front-end feature extraction.Source: §I · p. 2

II. RELATED WORK

Prior work uses either attention-based interaction, which exploits contextual semantics, or explicit structural association, which is efficient and interpretable. MatchFusion combines structural affinity initialization with semantic refinement and soft matchmap-guided aggregation.

II. RELATED WORK

  • Sparse instance representations also appear in spatio-temporal perception and E2EAD, including methods that propagate queries or connect perception with motion prediction and planning.Examples include StreamPETR, Sparse4D [8], StreamCMT, and BridgeAD.Source: §II · p. 2
  • SparseFusion, MV2DFusion, and temporal methods, use attention or query interaction to integrate instance features and contextual semantics.Specialized representation processing can increase architectural complexity, computation, and memory use.Source: §II · p. 2
  • Box-centric methods [11], associate instances using geometric, categorical, or motion cues, enabling efficient and interpretable matching in shared geometric space.Hard assignments limit exchange to selected correspondences, while structural costs alone cannot use contextual embeddings to resolve ambiguous matches.Source: §II · p. 2
  • MatchFusion combines structural association with semantic interaction through affinity initialization, selective refinement, and soft matchmap-guided residual aggregation.The formulation adapts geometric priors to spatial LiDAR-camera and temporal past-current interaction.Source: §II · p. 2

A. Overview and Instance Formulation

MatchFusion operates on sparse instance representations from upstream query-based perception models, separating interaction from front-end feature extraction. It matches structured object states using query semantics and aggregates information into enhanced representations for downstream tasks.

A. Overview and Instance Formulation

  • The module operates on sparse representations from upstream query-based models, decoupling matching and fusion from front-end feature extraction.Adapting it to spatial or temporal interaction requires specifying the geometric prior and aggregation direction.Source: A. Overview and Instance Formulation · p. 3
  • Each input instance has a structured state containing its 3D center, dimensions, yaw angle, and predicted category, alongside a query embedding.The query embedding captures contextual semantics associated with the instance.Source: A. Overview and Instance Formulation · p. 3
  • Given two instance sets, MatchFusion initializes correspondences from structural cues, selectively refines them using query semantics, and uses the soft matchmap to guide residual feature aggregation.Enhanced instance representations are produced for downstream perception and planning.Source: A. Overview and Instance Formulation · p. 3

B. Explicit–Implicit Matching-Fusion

MatchFusion pairs structural priors from object states with semantic evidence from query embeddings in a coarse-to-fine matching process. Its soft matchmap then guides residual information exchange between instance sets.

B. Explicit–Implicit Matching-Fusion

  • Structural initialization combines geometric similarity with a hard category-consistency constraint to form pairwise affinities.Category-inconsistent pairs are excluded from later semantic refinement and matchmap-guided aggregation.Source: B. Explicit–Implicit Matching-Fusion · p. 3
  • Semantic refinement uses query embeddings to adjust structurally plausible, category-consistent associations that remain ambiguous.The refinement is conditioned on the initial masked affinity, and a finite threshold controls which pairs receive semantic corrections.Source: B. Explicit–Implicit Matching-Fusion · p. 3
  • The resulting soft matchmap preserves graded many-to-many affinities and guides residual aggregation of complementary instance features.The common operator maps source embeddings into the receiving feature space; spatial interaction applies it in both directions, while temporal interaction applies it once.Source: B. Explicit–Implicit Matching-Fusion · p. 4

C. Spatio-Temporal Instantiation

MatchFusion uses a common matching and residual-aggregation principle for spatial and temporal interaction, with geometry priors tailored to each setting. The resulting updates enrich instance representations with complementary evidence.

C. Spatio-Temporal Instantiation

  • The shared aggregation operator supports both settings, while its direction determines how complementary evidence updates instance representations.Spatial updates operate in both directions, whereas temporal interaction enriches current candidates with past context; spatially fused instances retain hypotheses from both modalities.Source: C. Spatio-Temporal Instantiation · pp. 4–5
  • Spatial LiDAR–camera matching uses multi-view image-plane geometry to reduce sensitivity to camera depth discrepancies, then refines affinities with query semantics.Visibility-aware averaging combines view-specific affinities, and the matchmap guides bidirectional residual updates using the original input queries.Source: C. Spatio-Temporal Instantiation · p. 4
  • Temporal interaction motion-compensates historical instances and matches them to augmented current candidates using relative BEV positions and sizes.Semantic refinement addresses ambiguities associated with residual localization and motion-estimation errors; residual aggregation incorporates historical evidence into current queries.Source: C. Spatio-Temporal Instantiation · pp. 4–5

A. Datasets and Implementation Details

The evaluation uses nuScenes perception and planning metrics, alongside computational-efficiency measures. Perception experiments freeze pretrained front-ends, while the E2E setting jointly trains MatchFusion-T within SparseDrive under the original objectives.

A. Datasets and Implementation Details

  • nuScenes evaluation covers detection, tracking, planning, and computational efficiency with task-specific metrics.Detection uses NDS and mAP; tracking uses AMOTA and MOTA; planning uses trajectory L2 error and collision rate; efficiency uses FLOPs, GPU memory, and inference latency.Source: A. Datasets and Implementation Details · p. 5
  • Perception experiments freeze pretrained front-ends and report results on the nuScenes validation set without test-time augmentation or ensembling.The experiments compare multiple front-end configurations; MatchFusion is trained for 16 epochs, and MatchFusion-T uses existing downstream objectives without additional losses.Source: A. Datasets and Implementation Details · p. 5
  • For E2EAD, MatchFusion-T is integrated into SparseDrive and the entire system is trained from scratch under the original perception, motion, and planning objectives.This setting uses instance representations without spatial LiDAR–camera fusion or additional supervision, with matched baseline and equipped-system protocols.Source: A. Datasets and Implementation Details · p. 5

B. Comparative Evaluation

MatchFusion improves perception across front-end configurations and compares favorably with SparseFusion on accuracy and system efficiency. Temporal MatchFusion also improves perception in SparseDrive without additional supervision, while planning performance remains comparable.

Perception Gains Across Front-Ends

Accuracy–Efficiency Trade-off

  • 55.3% fewer FLOPs and 39.3% less memory than SparseFusion, with higher NDS; MatchFusion-ST uses 254.2 GFLOPs and 3.7 GB memory.The matching-fusion module accounts for 3.7% of total perception latency.Source: B. Comparative Evaluation · p. 5, Table III · p. 6

Integration into End-to-End Driving

  • In SparseDrive, MatchFusion-T raises NDS and mAP by 0.8 and 0.5 points, and AMOTA and MOTA by 2.7 and 1.7 points.Planning remains comparable: CR decreases from 0.123% to 0.119%, while L2 error increases slightly from 0.63 to 0.64 m.Source: B. Comparative Evaluation · pp. 5–6, Table I · p. 6

C. Ablation Studies

The ablations examine the contributions of spatial and temporal interaction, matching and aggregation choices, category consistency, training efficiency, and robustness to perturbed priors.

Ablation Studies

  • Ablations use Sparse4Dv3 with ResNet-50 and TransFusion-L with VoxelNet as the camera and LiDAR front-ends, respectively, unless otherwise specified.The studies examine spatial and temporal interaction, matching-fusion design, and robustness to spatial and temporal misalignment.Source: C. Ablation Studies · p. 6

Spatio-Temporal Fusion

  • Adding temporal interaction raises NDS from 72.5 to 73.0, AMOTA from 49.0 to 62.3, and MOTA from 43.6 to 60.8.MatchFusion-S raises NDS from 70.1 for the LiDAR-only baseline to 72.5; temporal aggregation complements spatial fusion across broad 3D perception tasks.Source: C. Ablation Studies · p. 6, Table IV · p. 6
  • MatchFusion-T improves detection and tracking in SparseDrive without spatial LiDAR–camera fusion, supporting its use beyond standalone perception.The integration uses SparseDrive's original training objectives without additional supervision.Source: C. Ablation Studies · p. 6

Matching–Fusion Construction

  • MatchFusion-S improves over cross-attention by 0.7 NDS and 1.7 mAP points.The comparison evaluates matchmap construction and aggregation strategies; MatchFusion uses structural initialization and semantic refinement to guide residual aggregation.Source: C. Ablation Studies · p. 6, Table V · p. 6
  • Enforcing category consistency raises NDS from 72.1 to 73.0 and mAP from 69.0 to 70.4.The authors report that excluding cross-category associations provides more reliable correspondences for refinement and aggregation.Source: C. Ablation Studies · p. 7, Table VI · p. 7

Efficiency of Instance-Level Interaction

  • Structural initialization with selective semantic refinement requires 1.7× training time and 1.3× memory, versus 1.9× and 2.2× for semantic-only matching.With frozen front-ends, MatchFusion-ST achieves 73.0 NDS and 70.4 mAP; the unfrozen configuration requires 2.1× memory and 5.5× training time.Source: C. Ablation Studies · p. 7, Table VII · p. 7, Table VIII · p. 7

Spatio-Temporal Fusion Robustness

  • Spatial translations of 5–20 cm leave NDS unchanged at reported precision, while 1°, 2°, and 4° rotations reduce it from 73.0 to 71.9, 71.8, and 71.7.At 4°, results remain above the LiDAR-only baseline by 1.6 NDS and 2.9 mAP points.Source: C. Ablation Studies · p. 7, Table IX · p. 7
  • Across tested temporal perturbations, MatchFusion-ST maintains at least 72.3 NDS, 69.2 mAP, 61.2 AMOTA, and 60.1 MOTA.Detection remains above both unimodal baselines, and tracking exceeds the camera-only baseline.Source: C. Ablation Studies · p. 7, Table IX · p. 7

D. Qualitative Results

The qualitative results illustrate complementary front-end errors addressed by fusion and the roles of structural and semantic cues in instance matching.

D. Qualitative Results

  • MatchFusion corrects individual front-end errors and recovers objects missed by a single modality.The visualization presents qualitative BEV detection results for Sparse4Dv3, TransFusion-L, and the MatchFusion-equipped system on nuScenes validation.Source: D. Qualitative Results · p. 7
  • In the matching visualization, box thickness encodes detection confidence and line thickness encodes matching score.The figure visualizes explicit–implicit cross-modal matching on nuScenes validation.Source: Figure 4 · p. 8
  • Structural priors identify plausible associations, while query semantics distinguish corresponding instances from geometrically ambiguous candidates.The refinement gate concentrates semantic corrections on structurally plausible pairs.Source: D. Qualitative Results · p. 8

V. CONCLUSIONS

MatchFusion combines geometric and categorical priors with selective semantic refinement to support spatial and temporal instance interaction. The conclusions report consistent perception gains, robustness to perturbations, and compatibility with end-to-end optimization, while identifying uncertainty modeling and multi-source ensembling as future work.

V. CONCLUSIONS

  • MatchFusion combines geometric and categorical priors with selective semantic refinement to form soft correspondences for residual feature aggregation across spatial and temporal interaction.The formulation remains decoupled from front-end feature extraction.Source: §V · p. 8
  • Experiments on nuScenes show consistent perception gains across diverse front-ends, computational and memory efficiency, and robustness to spatial and temporal perturbations.Integrating MatchFusion-T into SparseDrive further improves perception under the original training objectives.Source: §V · p. 8
  • Future work will model uncertainty in geometric priors and extend MatchFusion to multi-source ensembling for offline perception.Source: §V · p. 8

Related papers

Loading 2609.25860v1…