PaperlayerView exact revision on arXiv

Source-linked AI summary

EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset

Ryosei Hara, Wataru Ikeda, Masashi Hatano, Mariko Isogawa

arXiv:2609.17189v1cs.CV

TL;DR

Conventional cameras struggle with low light and motion blur, while egocentric event cameras introduce dense background events and prior methods lack left/right hand identity. EventEgoHands++ combines instance-level hand detection with visibility-adaptive attention and expanded synthetic and real datasets, outperforming existing baselines on both datasets.

  • Problem

    Egocentric event-based hand reconstruction is hindered by wearer-generated background events and prior binary masks that do not distinguish left and right hands.

  • Method

    EventEgoHands++ jointly estimates instance-level hand boxes and masks and dynamically selects attention operations according to detected hand visibility.

  • Results

    EventEgoHands++ outperforms the best existing method, reducing MPJPE by 21.82 mm (33.7%) on N-HOT3D and 7.90 mm (18.8%) on EEH-R.

  • Takeaways & Limitations

    The framework and datasets support event-based egocentric hand reconstruction across synthetic, real-world, well-lit, and low-light conditions.

Abstract

from arXiv · show

3D hand mesh reconstruction is a challenging yet essential task for downstream applications, including human-robot interaction and AR/VR. Although conventional cameras have been widely adopted for this task, methods that rely on them struggle in low-light environments and under severe motion blur. To address these limitations, event-based cameras have recently attracted attention for their high dynamic range and high temporal resolution. However, applying event cameras to egocentric hand reconstruction remains challenging because camera wearer's motion produces dense background events that obscure hand-specific signals. Although the first egocentric event-based approach mitigates this issue using hand segmentation, its binary hand mask does not distinguish between left and right hands. As a result, the model lacks instance-level hand information and predicts both hands even when only one or neither hand is present. This limitation leads to incorrect inter-hand relationships and degraded reconstruction accuracy. In this paper, we propose EventEgoHands++, a framework for event-based 3D hand mesh reconstruction from an egocentric viewpoint. The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands. Moreover, we introduce Adaptive Attention, which dynamically gates the attention based on these detection results to accurately learn the spatial relationship and mutual interactions between the hands. To train and evaluate our framework, we extend the synthetic N-HOT3D dataset and newly construct EEH-R, the largest real-world event-based egocentric hand dataset to date, comprising approximately 1M annotated frames captured in environments including low-light conditions. Extensive experiments on both synthetic and real datasets demonstrate that our method consistently outperforms the baselines.

I. INTRODUCTION

EventEgoHands++ addresses egocentric event-based hand reconstruction, where camera motion creates background events and prior binary masks fail to distinguish hand identities. It combines instance-level detection, visibility-adaptive attention, expanded datasets, and strong synthetic and real-world results.

  • I. INTRODUCTION: Egocentric reconstruction is important for interaction applications but is challenged by motion blur, illumination changes, and dense background events from wearer motion.Event cameras offer high temporal resolution and dynamic range for fast motion and low-light conditions, but egocentric background activity obscures hand signals.
  • I. INTRODUCTION: Prior EventEgoHands uses a binary hand mask that lacks left/right identity, reconstructs both hands when one or neither is visible, and degrades inter-hand relative positions.Its unconditional interaction modeling further leaves visibility-dependent hand relationships unresolved.
  • I. INTRODUCTION: The Hand Detector jointly estimates instance-level bounding boxes and masks, while Adaptive Attention selects inter-hand attention operations according to detected hand visibility.Together, these components identify each present hand and model spatial relationships only when corresponding hands are visible.
  • I. INTRODUCTION: EEH-R provides approximately 1M annotated real-world frames, while the extended N-HOT3D provides approximately 480K annotated frames with refined masks and bounding boxes.EEH-R includes well-lit and low-light conditions and supports real egocentric event-based evaluation.
  • I. INTRODUCTION: EventEgoHands++ reduces MPJPE by 21.82 mm (33.7%) and MPVPE by 20.57 mm (34.0%) on N-HOT3D, and by 7.90 mm (18.8%) and 7.13 mm (18.1%) on EEH-R.These reductions are measured against the best existing method on the synthetic and real datasets, respectively.

II. RELATED WORK

Related work spans egocentric hand understanding, 3D pose and mesh reconstruction, and event-camera sensing. Event cameras address motion blur and low light, motivating their use for egocentric hand reconstruction.

  • II. RELATED WORK: Egocentric hand analysis supports interaction understanding, action recognition, anticipation, forecasting, and human–robot collaboration or augmented-reality interfaces.The literature has expanded from localization and segmentation toward hand–object interaction and temporal hand-motion understanding.
  • II. RELATED WORK: 3D egocentric hand-pose datasets increasingly address bimanual interaction and articulated objects, progressing from FPHA to H2O, HoloAssist, ARCTIC, Assembly101, Ego-Exo4D, and HOT3D.This progression reflects increasingly complex hand and object interactions.
  • II. RELATED WORK: 3D hand reconstruction methods use parametric models such as MANO for anatomical priors, non-parametric vertex regression with graph networks, and Transformer architectures for global interactions.Parametric approaches remain favored for robustness to occlusion and valid hand topology.
  • II. RELATED WORK: Event cameras provide high dynamic range and temporal resolution that help address low-light conditions and motion blur affecting conventional RGB or depth cameras.Their asynchronous sensing can preserve motion and geometric cues during fast movements.

C. EVENT-BASED 3D HAND MESH RECONSTRUCTION

Event-based hand reconstruction has largely focused on fixed third-person views, while egocentric sensing introduces background events from wearer motion. EventEgoHands++ addresses this with instance-aware extraction and reconstruction from event data.

  • C. EVENT-BASED 3D HAND MESH RECONSTRUCTION: Event-based hand reconstruction methods have mainly targeted fixed third-person views, leaving egocentric event-based reconstruction relatively underexplored because wearer motion obscures hand events.EventEgoHands mitigated this issue by estimating coarse hand regions and filtering events within them.
  • C. EVENT-BASED 3D HAND MESH RECONSTRUCTION: EventEgoHands++ uses event data to reconstruct 3D joints and mesh vertices for both hands through Hand Extraction and Hand Reconstruction stages.The first stage detects hand regions; the second estimates 3D hand poses using learned visual features and adaptive attention.
  • C. EVENT-BASED 3D HAND MESH RECONSTRUCTION: The Hand Detector predicts instance-level bounding boxes and masks, and the masks extract separate event frames for the left and right hands.Jointly learning boxes and masks is used instead of mask-only prediction because it is more robust.
  • C. EVENT-BASED 3D HAND MESH RECONSTRUCTION: The detector uses LNES event representations and YOLO26 to jointly estimate hand bounding boxes and segmentation masks.LNES preserves temporal information through temporal weighting.

B. HAND RECONSTRUCTION STAGE

The hand reconstruction stage extracts spatial features from detected hand regions and applies attention operations conditioned on which hands are visible. Refined features are decoded into 3D hand pose and mesh outputs.

  • Detected hand event frames are encoded with a shared EfficientNetV2-S backbone into spatial feature maps for the left and right hands.The backbone processes the filtered hand regions produced by the hand extraction stage.
  • Adaptive Attention: When both hands are detected, self-attention refines each hand independently before bidirectional cross-attention models inter-hand dependencies.The sequence captures intra-hand spatial relationships first, then exchanges information between hands.
  • Adaptive Attention: When only one hand is detected, cross-attention is skipped and self-attention operates only on the detected hand.The unavailable hand does not participate in subsequent attention processing.
  • Attention-pooled feature maps are decoded into MANO parameters and converted into 3D hand joints and mesh vertices.The decoder produces per-hand outputs through the MANO model.
  • Adaptive Attention: When neither hand is detected, the sample’s subsequent reconstruction processing is skipped.This condition prevents reconstruction from proceeding without a valid hand region.
  • Adaptive Attention: Adaptive Attention switches attention operations themselves rather than masking attention scores, avoiding corruption from an undetected hand.Masked attention still computes with the undetected hand, whereas this design omits that hand’s processing entirely.

3) MANO Decoder

The MANO decoder maps refined hand features into pose, shape, translation, and rotation parameters, then generates 3D joints and mesh vertices.

  • 3) MANO Decoder: Each refined spatial feature map is attention-pooled and linearly mapped to MANO pose, shape, translation, and rotation parameters.The pose vector θ has dimension 45 and the shape vector β has dimension 10.
  • 3) MANO Decoder: The MANO model decodes these parameters into 20×3 3D joint coordinates and 778×3 mesh-vertex coordinates.Joints represent articulated hand locations, while vertices represent the hand surface.

C. TRAINING OBJECTIVE

Training combines joint, inter-hand interaction, mesh-vertex, and MANO parameter losses into a weighted hand reconstruction objective.

  • C. TRAINING OBJECTIVE: The 3D joints loss uses L1 distance, while the interaction loss uses L2 distance between predicted and ground-truth relative joint offsets.The interaction term measures relative positioning between the left and right hands.
  • C. TRAINING OBJECTIVE: The vertices loss uses L1 distance between predicted and ground-truth 3D mesh-vertex positions.It evaluates reconstruction of the hand mesh structure.
  • C. TRAINING OBJECTIVE: The MANO loss measures discrepancies between predicted and ground-truth pose θ and shape β parameters.This term assesses the accuracy of the parametric hand representation.
  • C. TRAINING OBJECTIVE: The training objective linearly combines four losses: joints, inter-hand relative offsets, mesh vertices, and MANO pose and shape parameters.The four terms are balanced by λγ, λδ, λϵ, and λζ, respectively.

IV. DATASET COLLECTION

The study constructs complementary synthetic and real egocentric event datasets with hand annotations, controlled splits, and challenging lighting conditions. N-HOT3D provides scalable simulation, while EEH-R supplies over one million real annotated frames.

  • A. SYNTHETIC DATASET: N-HOT3D: N-HOT3D is generated by applying v2e to HOT3D Aria-glasses data from nine subjects, producing synthetic egocentric event data.The simulator uses HOT3D MANO and camera parameters and outputs 346×260 event frames.
  • A. SYNTHETIC DATASET: N-HOT3D: The extended N-HOT3D dataset contains 480,120 frames with refined segmentation masks and newly added bounding-box annotations.These updates support detection training in addition to segmentation.
  • B. REAL DATASET: EEH-R: EEH-R contains 1,019,716 ground-truth annotations from eight subjects across 85 sequences and more than two hours of real event-camera recordings.Recordings include hand-object interactions in both well-lit and dark conditions.
  • B. REAL DATASET: EEH-R: EEH-R captures precise hand and camera poses using MoCap gloves, fabric gloves, and a 16-camera OptiTrack system.The setup preserves natural hand appearance while recording fine-grained finger articulation and global positioning.
  • B. REAL DATASET: EEH-R: EEH-R covers well-lit scenes averaging 457 lux and dark scenes averaging 3.5 lux across kitchen and workspace categories.The dataset includes eight subjects and two scene categories under two lighting conditions.
  • B. REAL DATASET: EEH-R: EEH-R is split by subject identity into 636,433 training, 164,727 validation, and 218,556 evaluation frames.The subject-disjoint split assigns P05, P07, P08, and P09 to training; P03 and P04 to validation; and P06 and P10 to evaluation.

V. EXPERIMENTS

The experiments train and evaluate the detector and reconstruction stages using event data, pretrained backbones, and specified optimization settings, while comparing against established event-based hand reconstruction baselines.

  • Experimental Setup: Inputs are center-cropped and resized to 224 × 224 pixels, and experiments use the official dataset splits on a single NVIDIA RTX 6000 Ada GPU.
  • Hand Detector: The Hand Detector uses a YOLO26-based segmentation model adapted for two-channel event input and fine-tuned for the task.Predicted masks are dilated with a 7 × 7 kernel for one iteration.
  • Hand Reconstruction: The reconstruction stage uses an ImageNet-pretrained EfficientNetV2-S backbone with C = 1280 feature channels and 7 × 7 spatial resolution.It is optimized with AdamW for 30 epochs using a batch size of 32 and specified hand-loss weights.
  • Baselines: EventHands, Ev2Hands, and EventEgoHands provide comparison baselines for event-based 3D hand mesh reconstruction.EventHands predicts one hand at a time, Ev2Hands predicts both hands from point-cloud event features, and EventEgoHands is an earlier egocentric method using mask-only estimation.

C. EVALUATION METRICS

The evaluation measures detector quality, per-hand pose and mesh accuracy, and relative configuration between hands, with quantitative and qualitative comparisons on synthetic N-HOT3D and real EEH-R data.

  • Evaluation Metrics: The evaluation combines PCK, AUC, R-AUC, RR-AUC, MPJPE, and MPVPE to assess per-hand accuracy, inter-hand configuration, and mesh reconstruction error.R-AUC is wrist-relative per hand, whereas RR-AUC measures both hands relative to the right wrist.
  • N-HOT3D Evaluation: On N-HOT3D, EventEgoHands++ outperforms all existing methods across metrics, reducing MPJPE by 33.7% and MPVPE by 34.0% versus EventEgoHands.Table 3 reports quantitative results over the synthetic dataset.
  • Qualitative Evaluation: Qualitative comparisons show more accurate hand shapes, finger details, rotations, and inter-hand relationships than the compared methods on both datasets.The method also identifies when only one hand is visible and reconstructs the visible hand without treating the absent hand as present.

B. EVALUATION ON EEH-R DATASET

On the real EEH-R dataset, EventEgoHands++ performs best across metrics and preserves reconstruction quality in low-light event-camera scenarios while improving hand segmentation over key detector baselines.

  • Quantitative Evaluation: EventEgoHands++ achieves the best EEH-R performance across metrics, reducing MPJPE by 7.90 mm and MPVPE by 7.13 mm versus the best existing method.The reductions are 18.8% for MPJPE and 18.1% for MPVPE.
  • Dataset Characteristics: EventHands and Ev2Hands outperform EventEgoHands on several EEH-R metrics because desk-based sequences keep hands large, close, and nearly constant in the frame.The authors connect this setting to easier hand detection, as reflected by detector mAP.
  • Qualitative Evaluation: The method remains effective on real event-camera data, including challenging low-light environments, and produces meshes closest to ground truth with more diverse poses.Existing methods often miss inter-hand relationships and fine intra-hand shape details.
  • Efficiency: When only one hand is detected, Adaptive Attention skips selected operations, making inference slightly faster.
  • Hand Detection: The Hand Detector improves N-HOT3D mAP@50 from 0.272 to 0.750 and mAP@50–95 from 0.042 to 0.407 relative to U-Net.The larger gains occur where hands occupy only a small portion of the frame.
  • Hand Detection: On EEH-R, the Hand Detector slightly trails U-Net on mAP@50 but surpasses it on stricter mAP@50–95, with a substantial advantage under dark illumination.

D. ABLATION STUDY

Ablations show that the Hand Detector supplies the main accuracy gains, Adaptive Attention strengthens inter-hand positioning, and targeted supervision and threshold choices improve robustness across conditions.

  • Proposed Components: Without the Hand Detector, both hands are always treated as present and attention processes the full frame, including background noise.This prevents reliable localization and separation of hand instances.
  • Proposed Components: Adaptive Attention alone slightly improves R-AUC but slightly degrades MPJPE, showing that attention requires properly localized hand regions.
  • Proposed Components: Adding the Hand Detector improves all metrics, while combining it with Adaptive Attention yields a 12.6% relative RR-AUC gain from 0.469 to 0.528.The detector primarily improves overall accuracy, whereas Adaptive Attention specifically strengthens relative positioning between hands.
  • General Applicability: The Hand Detector consistently improves R-AUC and RR-AUC when integrated into three existing methods, indicating benefits beyond the proposed reconstruction framework.
  • Attention Components: Self-attention mainly improves wrist-relative pose accuracy, whereas cross-attention improves RR-AUC by exchanging information between hands; combining both performs best.
  • Detection Threshold Analysis: Detection F1 remains stable across a threshold range before dropping, motivating confidence thresholds of 0.5 on synthetic data and 0.8 on real data.The higher real-data threshold limits false positives that would introduce nonexistent hands into reconstruction.
  • Dark-Scene Supervision: Under dark conditions, mAP@50–95 saturates after approximately 25% of the dark training set is added to well-lit supervision.This mixed-supervision strategy is more effective than training only on dark scenes.

E. FAILURE ANALYSIS

The proposed method fails mainly under severe hand–object occlusion, sparse events during near-static motion, and frame-by-frame temporal jitter. These limitations motivate explicit hand–object modeling and temporal processing for more robust reconstruction.

  • Severe occlusion can cause missed hand detection and inaccurate reconstruction of hidden parts such as fingertips.The paper identifies hand–object interaction modeling and separate hand/object representations as promising directions.
  • Sparse events during nearly static hand and head motion can prevent detection, especially in low-light environments.The paper suggests accumulating past events or adding temporal processing to mitigate this failure.
  • Frame-by-frame estimation produces temporal jitter in reconstructed meshes, yielding less-smooth wrist trajectories than the ground truth.The paper proposes extending the framework to 4D hand mesh reconstruction using the temporal nature of event streams.
  • The method improves reconstruction performance over existing approaches on both synthetic and real datasets, while the failure analysis identifies remaining robustness gaps.The reported limitations concern occlusion, sparse-event conditions, and temporal consistency rather than the overall benchmark comparison.
Loading 2609.17189v1…