Source-linked AI summary
EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
Lan Xu, Weipeng Xu, Vladislav Golyanik, Marc Habermann, Lu Fang, Christian Theobalt
TL;DR
Markerless high-speed capture is sparsely explored, and high-frame-rate imaging brings bandwidth and lighting constraints. EventCap combines asynchronous event tracking with low-rate intensity-image pose cues and batch optimization; the authors report detailed 1000-fps capture and robustness across scenarios.
Problem
Markerless capture of fast human motion is sparse, while high-frame-rate image methods face large bandwidth demands and well-lit-scene requirements.
Source: §1 · p. 1Method
EventCap jointly uses asynchronous event trajectories and low-rate intensity-image pose estimates in batch-based 3D motion optimization, followed by event-based pose refinement.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors describe EventCap as a step toward markerless high-speed capture, with potential applications in AR, VR, gaming, entertainment, and sports performance evaluation.
Source: §5 · p. 8
Abstract
from arXiv · showhide
The high frame rate is a critical requirement for capturing fast human motions. In this setting, existing markerless image-based methods are constrained by the lighting requirement, the high data bandwidth and the consequent high computation overhead. In this paper, we propose EventCap --- the first approach for 3D capturing of high-speed human motions using a single event camera. Our method combines model-based optimization and CNN-based human pose detection to capture high-frequency motion details and to reduce the drifting in the tracking. As a result, we can capture fast motions at millisecond resolution with significantly higher data efficiency than using high frame rate videos. Experiments on our new event-based fast human motion dataset demonstrate the effectiveness and accuracy of our method, as well as its robustness to challenging lighting conditions.
1. Introduction
Markerless capture of high-speed motion remains sparse, while high-frame-rate imaging brings substantial bandwidth and computation demands. EventCap combines event streams with low-rate intensity images to track motion at high temporal resolution and introduces an evaluation dataset.
1. Introduction
- Markerless high-speed motion capture is sparse, and high-frame-rate methods face substantial data-bandwidth and computation demands.Capturing VGA RGB video at 1000 fps for one minute yields 51.5 GB of data.Source: §1 · p. 1
- Event cameras asynchronously measure per-pixel intensity changes and offer high temporal resolution, 140 dB dynamic range, low power consumption, and low data bandwidth.Source: §1 · p. 1
- Sparse events between high-temporal-resolution frames have low signal-to-noise ratio, while temporal-change-only measurements complicate initialization and can allow tracking drift.Accumulating events into images makes measurements dense but loses their temporal information.Source: §1 · p. 2
- EventCap jointly optimizes asynchronous event trajectories and low-rate intensity-image pose detections, using batch-based 3D motion estimation and event-based pose refinement.CNN-based 2D and 3D pose estimates help address tracking drift and monocular depth ambiguity.Source: §1 · p. 2
- The paper introduces a dataset for event-camera fast-human-motion evaluation and provides motion-capture results at 1000 fps.Source: §1 · p. 2
2. Related Work
Prior capture approaches trade off capture speed, system complexity, lighting constraints, and data volume. EventCap’s related work contrasts these approaches with event-based tracking, including an earlier multi-event-camera method that accumulates events into frames.
2. Related Work
- Marker-based multi-view systems capture fast motion but are costly and intrusive, while markerless multi-camera systems retain synchronization, calibration, and data-processing burdens.Source: §2 · p. 2
- Multi-camera motion capture can reduce cost or support out-of-studio use, but synchronization, calibration, and high-frame-rate data processing remain burdensome.Depth-camera approaches include a synthetic 240 fps system, while active IR limits outdoor use and raises power consumption.Source: §2 · p. 2
- Monocular RGB methods estimate 3D poses or joint angles, but high-frame-rate tracking errors can exceed inter-frame motion and data redundancy remains an issue.Batch optimization and image-guided registration can alleviate tracking error; high-frame-rate RGB also requires well-lit scenes.Source: §2 · p. 2
- Event-camera tracking has covered 2D objects and faces, while an earlier 3D human-capture method used multiple event cameras and triangulated detected joints.That method accumulated events into frames, undermining their asynchronous, high-temporal-resolution information and limiting high-frame-rate capture.Source: §2 · p. 3
3. EventCap Method
EventCap tracks a rigged actor template using event and low-rate intensity streams to capture motion at 1000 fps. It generates asynchronous event trajectories, jointly optimizes poses across each batch, then refines them using event-based silhouette alignment.
3. EventCap Method
- EventCap tracks a scanned, SMPL-rigged actor template from event and low-rate intensity streams, targeting 1000-fps motion capture.The skeleton parameters include joint angles, global rotation, and root translation.Source: §3 · p. 3
3.1. Asynchronous Event Trajectory Generation
- For each adjacent-image batch, EDI-sharpened images support forward/backward feature tracking; stitched trajectories are fit continuously, then sliced at millisecond intervals.Forward and backward tracks are aligned at the batch midpoint, with matches beyond a four-pixel threshold discarded.Source: §3.1 · p. 4, Figure 3 · p. 4
3.2. Hybrid Pose Batch Optimization
- Batch optimization jointly fits skeletal poses using event correspondences and CNN 2D/3D detections, with plausible joint-angle bounds and stabilization for untracked body parts.The CNN detections help address tracking drift and monocular depth ambiguity.Source: §3.2 · pp. 4–5
3.3. Event-Based Pose Refinement
- Finally, ICP refinement matches projected mesh boundaries to nearby events and keeps poses near batch estimates; four iterations improve pose estimates.The event search uses a local 8 × 8 spatial patch and a batch-duration temporal window.Source: §3.3 · pp. 5–6
4. Experimental Results
EventCap is evaluated on a new fast-motion dataset, with results reported for its components, template meshes, and comparisons to monocular RGB methods. The experiments report accurate high-frequency tracking with substantially lower data throughput than high-frame-rate image methods.
4. Experimental Results
- On the proposed dataset, EventCap captures high-frequency motion details at 1000 fps and produces plausible poses even in an outdoor night sequence.The dataset contains 12 sequences from six actors, captured with a DAVIS240C event camera; a sequence also has multi-view markerless ground truth.Source: §4 · pp. 6–7
4.1. Ablation Study
- The full pipeline consistently has the lowest average per-joint error and standard deviation, while its batch optimization reduces drift and pose refinement corrects remaining image misalignment.Ground-truth comparisons use Procrustes-aligned poses and every tenth tracking frame.Source: Figure 7 · p. 7, §4.1 · p. 7
- The 3D-scanned template yields lower average per-joint error than SMPL (73.72mm vs 77.88mm), although their pose estimates are comparable.The scanned mesh also improves latent-image overlay because SMPL cannot model the clothes.Source: Figure 9 · p. 7, §4.1 · p. 7, Figure 8 · p. 7
4.2. Comparison to Baselines
- EventCap achieves the highest tracking accuracy in the high-frame-rate setting, uses 3.4% of the bandwidth required by high-frame-rate image methods, and uses only 10% more bandwidth than low-frame-rate upsampling.Against high-frame-rate reference-image methods, it also achieves better tracking accuracy while using 3.4% of their bandwidth.Source: Table 1 · p. 8, §4.2 · p. 8, Table 2 · p. 8
5. Discussion and Conclusion
EventCap presents the first markerless 3D human motion capture approach using a single event camera, alongside a new high-speed motion dataset. Experiments report effective, robust capture across varied scenarios, while future work targets occlusions, topological changes, and runtime.
5. Discussion and Conclusion
- The authors present the first markerless 3D human motion capture approach using a single event camera and a new high-speed human motion dataset.Source: §5 · p. 8
- Batch optimization uses hybrid image and event streams, and a new event-based pose refinement approach further refines captured motion.Source: §5 · p. 8
- Experiments demonstrate EventCap's effectiveness and robustness for capturing fast human motions in varied scenarios.Source: §5 · p. 8
- Future work will address large occlusions and topological changes and improve runtime performance.Source: §5 · p. 8