PaperlayerView exact revision on arXiv

Source-linked AI summary

MV2GF: Multi-view Pedestrian Detection with a Visual Geometric Foundation Model

Taiga Yamane, Satoshi Suzuki, Ryo Masumura, Shota Orihashi, Tomohiro Tanaka, Mana Ihori, Naoki Makishima

arXiv:2608.20639v1cs.CV

TL;DR

Existing MVPD methods struggle with visual geometry and projection distortions when camera configurations differ from training. MV2GF combines foundation-model geometry with task-specific features and pointmap-based projection, and experiments show stronger generalization, with a measured inference-speed cost.

  • Problem

    Existing MVPD methods struggle to generalize to unseen camera configurations because they inadequately capture cross-view geometry and depend on configuration-specific distortion patterns.

  • Method

    MV2GF fuses task-specific ResNet features with geometric features from frozen DA3 and projects image features into 3D locations using DA3 pointmaps.

  • Results

    MV2GF improves generalization over prior methods, exceeding the previous state of the art in MODA by 4.6 points on GMVD, 4.7 on MVPerception, and 2.2 on Wildtrack.

  • Takeaways & Limitations

    Foundation-model geometric features and pointmaps improve MVPD generalization to camera configurations absent from training data.

  • Takeaways & Limitations

    MV2GF is 1.2–1.9 FPS slower than previous methods, although its speed is sufficient for datasets comprising multi-view videos at 2.0 FPS.

Abstract

from arXiv · show

Multi-View Pedestrian Detection (MVPD) aims to detect pedestrians in the form of a bird's eye view map from multi-view images. Recent MVPD methods adopt a unified framework that projects 2D image features into a 3D world space and aggregates them into a single feature. Although they are effective, they struggle to generalize to unseen camera configurations during training due to two main issues. First, they are difficult to capture accurate visual geometry across views in unseen camera configurations. Second, they make detection models highly dependent on distortion patterns during training arising from their image feature projection. To address these, we leverage a visual geometric foundation model and propose MV2GF. This foundation model has exhibited strong generalization in capturing visual geometry across views and predicting accurate 3D attributes in diverse camera configurations. MV2GF fuses task-specific features with general-purpose geometric features extracted by the foundation model to effectively capture the visual geometry even in unseen camera configurations. Furthermore, MV2GF projects each pixel in the image features to an appropriate 3D location using 3D pointmaps predicted by the foundation model, preventing the detection model from depending on distortion patterns during training. Our experiments demonstrate the effectiveness of leveraging a visual geometric foundation model for MVPD and that MV2GF generalizes better than existing methods.

1 Introduction

MVPD uses multi-view images to produce bird’s eye view pedestrian maps, but existing projection-based methods generalize poorly to unseen camera configurations. MV2GF addresses this with a visual geometric foundation model, feature fusion, and pointmap-based 3D projection, improving generalization.

  • MVPD aggregates complementary information from multiple camera views to detect pedestrians as a bird’s eye view map and mitigate occlusion.
  • Existing methods extract 2D image features, project them into 3D world space using calibrated cameras, aggregate them into BEV features, and predict a BEV map.
  • Unseen camera configurations expose failures in learning 2D–3D correspondence and compensating for configuration-dependent distortion patterns.
  • MV2GF leverages a pretrained visual geometric foundation model whose multi-view features capture general-purpose geometry across diverse camera configurations.
  • TGF fuses trainable ResNet features with frozen foundation-model features, while FPA uses 3D pointmaps to place image features at appropriate world locations.
  • 4.6 points on GMVD testing, 4.7 points on MVPerception, and 2.2 points on Wildtrack: MV2GF’s MODA gains over the previous state of the art.

2 Related Work

MVPD research has evolved from aggregating single-view detections to unified feature-projection frameworks, while visual geometric foundation models provide a complementary source of geometry for diverse multi-view settings.

  • Multi-view Pedestrian Detection: MVPD comprises traditional detection-and-aggregation methods and unified frameworks that project multi-view image features into 3D space before predicting BEV maps.
  • Multi-view Pedestrian Detection: Traditional methods aggregate per-view pedestrian detections, but occlusion in individual views limits their final detection performance.
  • Multi-view Pedestrian Detection: Many unified methods use single-view encoders or specialized feature extraction, including foreground-focused pooling designed to emphasize pedestrian geometry.
  • Multi-view Pedestrian Detection: Perspective-based projection and derivative transformations introduce shadow-like distortions that later networks learn to compensate for.
  • Visual Geometric Foundation Models: Visual geometric foundation models learn cross-view geometry from large-scale multi-view data and generalize across depth estimation, reconstruction, and camera-pose tasks.

3 Proposed Method

MV2GF uses Depth Anything 3 to provide general-purpose geometric features and 3D pointmaps, combining them with task-specific features for multi-view pedestrian detection. It aggregates features at pointmap-derived 3D locations to produce a BEV representation that is less dependent on projection distortions.

  • Task-specific and Geometric Information Fusion: TGF fuses trainable ResNet features emphasizing pedestrians with frozen DA3 features encoding general-purpose visual geometry across views.The fusion produces high-resolution image features intended to capture geometry across diverse, including unseen, camera configurations.
  • Feature Pointmap Aggregation: FPA uses DA3 pointmaps to project each TGF feature pixel to an appropriate 3D world location before aggregating features into a BEV representation.Features are assigned to voxels according to their predicted 3D locations and aggregated with max pooling, then the vertical dimension is compressed.
  • Feature Pointmap Aggregation: FPA creates voxel grids from multi-view pointmaps and features, max-pools vectors within each voxel, and converts the result into a BEV feature.The voxel grid uses reduced horizontal resolution for computational efficiency and compresses the vertical dimension before CNN processing.
  • Training and Inference: The detection head predicts a BEV pedestrian occupancy map and an offset map, while training uses focal loss for occupancy and L1 loss for offsets.Per-view head and foot occupancy predictions are also used as an auxiliary focal-loss objective.

4 Experiments

Experiments evaluate MV2GF on multiple datasets, unseen and matched camera configurations, and component variants. MV2GF improves generalization and maintains strong detection performance, with gains tied to its fusion, pointmap aggregation, and geometric foundation model features.

  • Datasets: Experiments use Wildtrack, MVPerception, and GMVD, with GMVD-D testing unseen camera configurations and GMVD-S testing matched configurations.Wildtrack contains 400 frames from 7 cameras, MVPerception contains 800 frames from 6 cameras, and GMVD spans 7 scenes and 13 camera configurations.
  • Implementation Details and Evaluation Metrics: MODA is the primary evaluation metric because it accounts for both false positives and false negatives.A detection is a true positive when it lies within 0.5 meters of ground truth; MODP, precision, and recall are also reported.
  • Comparison with Previous Methods: MV2GF outperforms previous methods across unseen-camera tests, exceeding the highest prior MODA by 4.6 points on GMVD-D, 4.7 on MVPerception, and 2.2 on Wildtrack.It leads all metrics on GMVD-D and MVPerception and achieves the highest MODA, MODP, and precision on Wildtrack.
  • Comparison with Previous Methods: Under matched camera configurations, MV2GF achieves superior MODA and recall to previous methods across all evaluated datasets and is comparable to MSMVD.On MODA, it exceeds MSMVD by 0.2, 0.1, and 0.1 points for GMVD-S, MVPerception, and Wildtrack, respectively.
  • Ablation Studies: Combining TGF and FPA improves the baseline by 11.4 points on MODA, while combining ResNet and DA3 features improves MODA over either feature source alone.Using both feature types exceeds ResNet-only and DA3-only configurations by 3.5 and 7.3 points, respectively.
  • Ablation Studies: DA3 features, four transformer blocks, and the default θz = 50 cm improve component performance, while MV2GF remains stronger than prior methods across foundation-model choices and input-view counts.MV2GF degrades less than MSMVD when views are reduced, and DA3 outperforms MapAnything and Pi3X in the tested configuration.

5 Limitations

MV2GF’s visual geometric foundation model improves generalization but slightly reduces inference speed. The measured slowdown remains above the 2.0 FPS rate of most MVPD datasets.

  • 1.2–1.9 FPS slower: MV2GF incurs a small inference-speed gap versus previous methods.Measurements used an Nvidia 80GB A100 GPU and six views from the GMVD-D testing split.
  • Inference-speed comparisons are reported in Table 5 for six-view GMVD-D testing data on an Nvidia 80GB A100 GPU.
  • Most MVPD datasets use multi-view videos at 2.0 FPS, making MV2GF’s measured speed sufficient for processing them.The authors identify model compression, such as knowledge distillation, as future work for acceleration.

6 Discussion and Conclusion

MV2GF uses foundation-model features and 3D pointmaps to improve MVPD generalization across camera configurations. It generalizes better than prior methods in mismatched configurations, but remains weaker when configurations match testing conditions.

  • MV2GF combines foundation-model multi-view features and 3D pointmaps to capture geometry and place image features appropriately in 3D space.The design avoids shadow-like distortions associated with previous projection methods.
  • MV2GF generalized better than previous methods when training and testing used different camera configurations.
  • MV2GF underperformed previous methods trained with the same camera configurations as testing data.The authors identify configuration-specific optimization as an important direction for future work.

Supplementary Material

The supplementary material reports ablations for MV2GF design choices, including offset prediction, auxiliary loss, feature extraction, pooling, and loss weighting.

  • Table 6 reports the effects of offset prediction and auxiliary loss in MV2GF.
  • The supplementary material evaluates multiple implementation and training choices rather than only the core model configuration.
  • The supplementary ablations examine ResNet size, pooling choice, trainable-ResNet features, and loss weighting.All models use the GMVD-D training and testing split, with default settings marked in gray.

A Implementation Details

Implementation details specify the FPA convolutions, occupancy-map prediction heads, optimizer settings, Gaussian-kernel size, hardware, memory use, and GMVD-S evaluation protocol.

  • FPA uses three dilated convolutional layers, while additional convolutional processing predicts per-view pedestrian-head and feet occupancy maps.
  • Adam uses β1 = 0.9 and β2 = 0.99 without weight decay, with a 10-pixel Gaussian-kernel diameter for down-sampled ground-truth maps.
  • All experiments run on an Nvidia 80GB A100 GPU.
  • DA3 has 1.4B parameters and trainable modules have 23.0M parameters; memory use is 27.8 GB during training and 21.5 GB during inference on six views.
  • Table 7 evaluates MSMVD’s multi-scale BEV design on GMVD-S, where training and testing camera configurations are identical.

B Additional Ablation Studies

Additional ablations show that offset prediction, per-view auxiliary supervision, larger task-specific backbones, max pooling, and trainable ResNet features each improve MV2GF’s detection performance.

  • Offset prediction improved all detection metrics, including MODP by 4.1 points over omitting the offset map.
  • Per-view auxiliary supervision improved MODA by 1.0 points and made TGF image features more discriminative.
  • ResNet101 outperformed ResNet18 by 0.7 points on MODA, indicating larger task-specific features improve generalization to unseen camera configurations.
  • Max pooling outperformed mean pooling by 0.3, 1.3, and 0.4 points on MODA, MODP, and recall, respectively.Max pooling selects the most relevant and informative features from multiple views within each voxel.
  • Trainable ResNet features substantially outperformed DINOv2 and DINOv3 features, which were only slightly better than using DA3 features alone.The comparison attributes the advantage to task-specific information in ResNet features.

C Effect of MSMVD’s multi-scale BEV design

Adding MSMVD’s multi-scale BEV design to MV2GF greatly improved MODP and precision under matched training and testing camera configurations, yielding better performance than MSMVD on all metrics.

  • Adding MSMVD’s multi-scale BEV design to MV2GF improved MODP and precision and surpassed MSMVD on all metrics.The design was especially important for MODP and precision in the matched-camera setting.
  • The MultiviewX-to-Wildtrack setting evaluates generalization when training data are smaller, less diverse, and contain fewer scenes.

E Visual Comparison of Detection Results

Visual comparisons present MV2GF and MSMVD detection results across GMVD-D, MVPerception, and Wildtrack, while DA3 pointmaps are shown across the same datasets.

  • Figures 3–5 compare predicted BEV maps from MV2GF and MSMVD across GMVD-D, MVPerception, and Wildtrack.Green points denote detected pedestrians, blue circles false positives, and red circles false negatives.
  • Figures 6–8 visualize DA3-predicted 3D pointmaps on GMVD, MVPerception, and Wildtrack.The text reports high-quality pointmaps across all three datasets, enabling projection of image-feature pixels to appropriate 3D locations.

G Visual Comparison of BEV features

MV2GF’s BEV features are compared with ground truth and MVFP, while the accompanying visualizations cover detection results and DA3 pointmaps across multiple datasets.

  • Figure 9 compares the ground-truth BEV map with MV2GF and MVFP BEV features on GMVD-D.The ground-truth green points represent pedestrians.
  • MV2GF exhibited fewer distortions around pedestrians than MVFP and more precisely represented pedestrian locations.The passage attributes this to using DA3-predicted 3D pointmaps for projecting image features into 3D world space.
  • Figures 3–5 encode detected pedestrians, false positives, and false negatives with green points, blue circles, and red circles, respectively.
  • Figures 6–8 visualize DA3 3D pointmaps on GMVD, MVPerception, and Wildtrack.
Loading 2608.20639v1…