Source-linked AI summary
Multiview Detection with Feature Perspective Transformation
Yunzhong Hou, Liang Zheng, Stephen Gould
TL;DR
Multiview pedestrian detection must resolve occlusion-related ambiguity while aggregating cues and spatial information across cameras. MVDet uses anchor-free feature-map projection to the ground plane and fully convolutional spatial aggregation, achieving 88.2% MODA on Wildtrack, 14.1% above the previous state of the art, with competitive MultiviewX results.
Problem
Occlusions and crowdedness make pedestrian existence, counts, and locations ambiguous, requiring multiview and spatial aggregation.
Method
MVDet projects convolutional feature maps onto the ground plane for anchor-free multiview aggregation and applies large kernels in a fully convolutional spatial-aggregation approach.
Results
88.2% MODA on Wildtrack is a 14.1% improvement over the previous state of the art, and MVDet achieves competitive results across occlusion levels on MultiviewX.
Takeaways & Limitations
The results support feature-map projection and fully convolutional spatial aggregation as an effective design for multiview pedestrian detection.
Abstract
from arXiv · showhide
Incorporating multiple camera views for detection alleviates the impact of occlusions in crowded scenes. In a multiview system, we need to answer two important questions when dealing with ambiguities that arise from occlusions. First, how should we aggregate cues from the multiple views? Second, how should we aggregate unreliable 2D and 3D spatial information that has been tainted by occlusions? To address these questions, we propose a novel multiview detection system, MVDet. For multiview aggregation, existing methods combine anchor box features from the image plane, which potentially limits performance due to inaccurate anchor box shapes and sizes. In contrast, we take an anchor-free approach to aggregate multiview information by projecting feature maps onto the ground plane (bird's eye view). To resolve any remaining spatial ambiguity, we apply large kernel convolutions on the ground plane feature map and infer locations from detection peaks. Our entire model is end-to-end learnable and achieves 88.2% MODA on the standard Wildtrack dataset, outperforming the state-of-the-art by 14.1%. We also provide detailed analysis of MVDet on a newly introduced synthetic dataset, MultiviewX, which allows us to control the level of occlusion. Code and MultiviewX dataset are available at https://github.com/hou-yz/MVDet.
1 Introduction
MVDet addresses occlusion-driven ambiguity in multiview pedestrian detection by aggregating projected feature maps on the ground plane and reasoning over spatial neighbors with fully convolutional operations. It achieves 88.2% MODA on Wildtrack, improving over the previous state of the art by 14.1%.
- 1 Introduction: The system takes synchronized frames from N cameras, aggregates cues where their views overlap on the ground plane, and outputs a pedestrian occupancy map.The overview frames multiview cue aggregation and spatial-neighbor aggregation as the system’s two core questions.
- 1 Introduction: Occlusions and crowdedness create ambiguity about pedestrian existence, counts, and locations, making multiview and spatial aggregation central challenges.Multiview systems use synchronized, calibrated cameras with overlapping fields of view and evaluate occupancy on the ground plane.
- 1 Introduction: MVDet replaces potentially inaccurate anchor-box aggregation with anchor-free projection and concatenation of convolutional feature maps from multiple views.Camera calibration maps image coordinates to world locations, enabling projection onto the ground plane.
- 1 Introduction: For spatial aggregation, MVDet uses a fully convolutional solution with large-kernel operations instead of CRF or mean-field inference.The design aims to aggregate neighboring spatial information for comprehensive pedestrian occupancy decisions.
- 1 Introduction: 88.2% MODA on Wildtrack, a 14.1% increase over the previous state of the art, demonstrates MVDet’s effectiveness on multiview pedestrian detection.The system is also evaluated on the synthetic MultiviewX dataset, where it achieves competitive results under multiple occlusion levels.
2 Related Work
Prior work combines information across monocular, multimodal, and multiview sources for detection and 3D understanding. Multiview pedestrian methods aggregate anchor features or detection results and use CRF or mean-field inference for spatial neighbors, while geometric-transformation methods provide related projection mechanisms.
- 2 Related Work: Anchor-free detection methods emerged partly because predefined anchors can limit performance, including in monocular pedestrian detection.Examples detect pedestrian head-foot points or center and scale rather than predefined bounding-box anchors.
- 2 Related Work: Multiple information sources, including RGB views, depth, point clouds, and stereo images, have been studied for 3D object understanding.Prior work includes maximum pooling across 2D views and feature aggregation for 3D anchors.
- 2 Related Work: Multiview pedestrian detection commonly fuses multiple RGB cameras through 2D anchor features or single-view detection results.Anchor-based methods calculate multiview boxes from assumed human dimensions, whereas other approaches fuse single-view detections.
- 2 Related Work: CRF and mean-field inference are established approaches for aggregating spatial-neighbor information in multiview pedestrian detection.These methods cast occupancy reasoning as structured inference over the scene.
- 2 Related Work: Geometric transformations such as affine and perspective transformations model visual phenomena using explicitly calculated or learned parameters.Related work applies spatial transformers to 2D inputs and projects 2D keypoints from 3D skeletons.
3 Methodology
MVDet aggregates calibrated multiview evidence by projecting anchor-free feature maps onto the ground plane, then uses spatial aggregation to produce occupancy decisions. Its design replaces inaccurate anchor-box representations and non-convolutional spatial inference with feature-map transformation and large-kernel convolutions.
- 3 Methodology: MVDet projects calibrated feature maps from multiple views onto the ground plane and applies large-kernel convolutions for joint occupancy decisions.The architecture concatenates projected view features with ground-plane coordinates before spatial aggregation.
- 3 Methodology: The shared CNN extracts per-view feature maps before resizing them for projection and multiview aggregation.The feature extractor shares weights across the N input images and uses dilated convolutions to maintain spatial resolution.
- 3 Methodology: Anchor-free feature vectors avoid inaccurate anchor-box shapes by sampling corresponding feature-map points for each ground-plane location.Anchor boxes derived from assumed human dimensions can describe background when targets differ in size or pose, such as a sitting person.
- 3 Methodology: Perspective transformation maps quantized ground-plane locations to calibrated image features, padding locations outside each camera’s view with zeros.The method uses a parameterized sampling grid based on camera calibration and appends a two-channel X-Y coordinate map.
- 3 Methodology: Feature-map projection preserves more useful information than projecting RGB images or single-view detection results, and achieves the highest MODA among the compared choices.Feature maps are less affected by broken pixel neighborhoods because semantic information is concentrated within individual features, while single-view outputs discard cues under occlusion.
- 3 Methodology: MVDet trains ground-plane occupancy regression with a Gaussian-soft target and Euclidean loss.The supplied methodology passages describe the ground-plane loss but do not include the complete combined-loss expression.
4 Experiment
MVDet is evaluated on Wildtrack and MultiviewX using standard multiview detection metrics and comparisons with prior methods and targeted variants. Results support anchor-free feature projection and large-kernel spatial aggregation, while performance varies with dataset realism, camera coverage, and occlusion.
- 4.3 Method Comparisons: 88.2% MODA on Wildtrack and 83.9% on MultiviewX surpass prior comparisons by 14.1% and 8.7%, respectively.MVDet also achieves the highest MODP and recall on both datasets, while Deep-Occlusion retains higher precision.
- 4.4 Evaluation of MVDet: Anchor-free feature projection outperforms anchor-based aggregation by 9.1% MODA on Wildtrack and 7.2% on MultiviewX.Without spatial aggregation, MVDet reaches 76.9% MODA on Wildtrack and 77.2% on MultiviewX.
- 4.4 Evaluation of MVDet: Projecting RGB images yields 26.8% and 19.5% MODA, while projecting single-view results yields 11.3% and 18.7% on Wildtrack and MultiviewX.Replacing clustering with large-kernel convolution raises projected-result performance to 68.2% and 73.2% MODA, respectively.
- 4.4 Evaluation of MVDet: Large-kernel spatial aggregation increases MODA by 11.3% on Wildtrack and 6.7% on MultiviewX.These gains exceed the reported CRF and mean-field improvements, although the paper does not assert superiority over CRF-based methods.
- 4.1 Experiment Setup: Performance gains from spatial aggregation are smaller on MultiviewX, where each ground-plane location is covered by 4.41 cameras versus 3.74 on Wildtrack.The datasets use different crowding and data-generation settings, with MultiviewX providing controllable synthetic configurations.
- 4.4 Evaluation of MVDet: Removing the single-view detection loss reduces performance by 1.2% on one dataset and 2.0% on the other, while remaining competitive.The authors attribute limited additional benefit to existing ground-plane supervision of foot points.
5 Conclusion
MVDet addresses multiview pedestrian-detection ambiguities caused by occlusion through anchor-free multiview aggregation and large-kernel spatial aggregation, achieving strong results on Wildtrack and MultiviewX.
- 88.2% MODA on Wildtrack, outperforming the previous state of the art by 14.1%.MVDet also achieves very competitive results on the synthetic MultiviewX dataset.
- MVDet aggregates projected feature maps without anchors and applies large kernels to resolve spatial ambiguity in a fully convolutional system.The method targets multiview and spatial aggregation under occlusion.
- The authors present MVDet as a strong baseline intended to encourage further research in multiview pedestrian detection and related fields.