PaperlayerView exact revision on arXiv

Source-linked AI summary

EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation

JiaCheng Ge, SiYu Zhang, ShengJie Li, XinTong Yang

arXiv:2610.06970v1cs.LGcs.HC

TL;DR

Egocentric bimanual pose estimation must contend with degraded visual clues, motivating the question of whether vision and wrist EMG can interact directly during pose formation. EVFormer uses sequential bidirectional cross-attention and feature-wise gating to regress 44 joint angles; on one participant's recording, it reports lower test MAE than its internal controls. The result is preliminary, with broader evaluation needed to establish generalizability.

  • Problem

    Self-occlusion, hand contact, and object manipulation reduce first-person visual clues, motivating a test of whether interacting vision and wrist EMG can improve continuous joint-angle prediction.

    Source: §1 · p. 1

  • Method

    EVFormer exchanges visual spatial and bilateral-wrist EMG temporal information through sequential bidirectional cross-attention, then applies feature-wise gating to regress 44 current hand angles.

    Source: §2.4 · p. 3, §4.5 · p. 7

  • Results

    11.482° test MAE was below vision-only (13.228°), EMG-only (13.610°), late fusion (13.387°), and training-mean pose (13.320°) in the single-recording comparison.

    Source: §8 · p. 22

  • Takeaways & Limitations

    The internal comparison provides preliminary evidence of an overall advantage, while not establishing improvement in every gesture class or random seed.

    Source: §1 · p. 2, §8 · p. 22

    Evaluation uses one participant and one recording without independent test users, limiting claims about generalization across users and sessions.

    Source: §7.3 · p. 19

Abstract

from arXiv · show

Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often degraded by self-occlusion, hand-hand contact, and object manipulation. We propose EVFormer, a multimodal framework that combines the current RGB frame with the preceding 200 ms of bilateral wrist surface electromyography (sEMG) to estimate 44 finger and wrist joint angles. EVFormer separately encodes visual spatial features and sEMG temporal features, enables cross-modal information exchange through sequential bidirectional cross-attention, and integrates the two modalities using feature-wise gated fusion. We evaluate EVFormer in a single-participant feasibility study using one synchronized public EgoEMG recording with chronologically separated training, validation, and test splits. On 296 test samples, EVFormer achieves a mean absolute error of 11.482 degrees, compared with 13.228-13.610 degrees for vision-only, sEMG-only, late-fusion, and training-mean baselines. This corresponds to relative error reductions of 13.20% compared with the vision-only model and 14.23% compared with late fusion. EVFormer also achieves the lowest error in four of the five evaluated gesture classes. These results provide preliminary evidence that feature-level interaction between egocentric vision and sEMG can improve bimanual hand pose estimation. Further evaluation across participants, recording sessions, sensor placements, and real-world interaction conditions is required to establish the generalizability of the approach.

1. Introduction

EVFormer asks whether directly interacting first-person vision and bilateral wrist EMG can improve continuous bimanual joint-angle prediction. It uses sequential cross-modal attention and gated fusion, and reports lower overall test error than the current controls on a single public recording.

1. Introduction

  • EVFormer exchanges image spatial tokens and recent EMG temporal tokens through sequential bidirectional cross-attention, then uses feature-wise gating to regress 44 joint angles.The current RGB frame forms 16 tokens, while the preceding 200 ms of wrist EMG forms 25 time tokens.Source: §1 · p. 1
  • Self-occlusion, hand contact, and object manipulation reduce visual clues, while wrist EMG provides a distinct observation related to forearm muscle contraction.Because the modalities differ in sampling, timescale, and noise, adding EMG does not automatically improve accuracy.Source: §1 · p. 1
  • 11.482° overall test MAE is lower than vision-only (13.228°), EMG-only (13.610°), late fusion (13.387°), and training-mean pose (13.320°).The reported relative reduction is 13.20% versus vision-only.Source: §1 · p. 2
  • The study uses a public synchronized recording from one participant, so it does not replace evaluation with new users or complete multi-participant partitions.The paper identifies synchronization, filtering, and partitioning as methods and experiments topics.Source: §1 · p. 2
  • The paper's contributions include a 44-dimensional bimanual angle model, fixed single-segment comparisons summarized by sample-weighted MAE, and gesture-category analysis that preserves local failures.It also identifies conditions needed for a complete multi-user evaluation.Source: §1 · p. 2

2. Related Work

Prior work spans EMG-based continuous pose estimation, egocentric hand reconstruction, and multimodal fusion, but these settings and outputs are not directly interchangeable. EVFormer places fusion at the pose-representation stage and tests it against internal controls, with conclusions bounded by the single-segment evaluation.

2.1 Surface EMG and Continuous Hand Pose Estimation

  • EMG studies include continuous finger-angle estimation and wearable hand-pose tracking, while electrode placement, subjects, and tasks can alter signal distributions.EgoEMG provides synchronized first-person images, bilateral EMG, and continuous bimanual angles; same-person time blocks cannot substitute for new-user evaluation.Source: §2.1 · pp. 2–3

2.2 Egocentric Vision, Bimanual Interaction, and Angle Representation

  • Egocentric hand datasets differ in cameras, annotations, and evaluation goals, so angle MAE is not directly comparable to 3D keypoint or MANO outputs.The paper's 44-dimensional target is the joint angle of both hands.Source: §2.2 · p. 3

2.3 Where to Fuse Vision and EMG

  • EVFormer exchanges visual-position and EMG-time tokens through bidirectional attention before gating, but its gating is not a calibrated sensor-quality score.The current internal advantage does not establish a mechanism contribution without parameter-matched ablation; the late-fusion and EVFormer configurations differ in trainable parameters.Source: §2.3 · p. 3

2.4 Motivation for EVFormer

  • EVFormer combines current-image spatial cues with historical-EMG movement cues to regress 44 current hand angles, achieving 11.482° test MAE in the current single-segment experiment.That MAE is 13.20% lower than the visual-only result, but still needs review on an official multi-user division.Source: §2.4 · pp. 3–4

2.5 Relationship to EgoEMG and Research Questions

  • The internal sample-weighted MAE supports an overall advantage on the current segment, but does not replace an EgoEMG residual-fusion comparison or official multi-user evaluation.It also does not establish improvement in every category or random seed.Source: §2.5 · p. 4

3. Dataset and Pose Annotations

The study predicts both hands' joint angles from a public EgoEMG mirror recording using synchronized image and EMG inputs. Chronological splits and filtering define the pilot evaluation, whose single-recording data and possible offline EMG processing constrain its scope.

3.1 Data Source and Bimanual Angle Targets

  • The target contains 44 scalars: 20 finger angles and two wrist angles per hand, with left-hand values at positions 0–21 and right-hand values at 22–43.The main metric is absolute angular error averaged over valid test samples and output dimensions, in degrees.Source: §3.1 · p. 4

3.2 Synchronization, Preprocessing, and Label Filtering

  • The inputs use 400 EMG points before each center, averaged in groups of four, plus a fixed lower-center 64×64 RGB crop from the center frame.After label and input checks, 1,968 synchronized candidates were reduced to 1,964 samples; EMG normalization statistics came from training blocks only.Source: §3.2 · p. 4

3.3 Dataset Size and Experimental Splits

  • Chronological boundaries yield 1,375 training, 293 validation, and 296 test samples, with non-overlapping EMG windows across blocks.Test gesture categories were absent from training, but category 21 appeared in validation, so the test categories were not all unseen during model selection.Source: §3.3 · p. 5

3.4 Data Quality and Scope

  • The public clip is a single-participant mirror recording, and potentially offline-filtered EMG means a pre-center input window alone does not establish real-time causal processing.Online evaluation would need explicit causal filtering and accounting for timestamps, video decoding, and inference delays.Source: §3.4 · pp. 5–6

4. Unified Tasks and Model Baselines

The section defines aligned vision-only, EMG-only, and fused pose-estimation tasks, then describes the branches and sequential cross-modal fusion used to predict 44 joint angles. It also distinguishes the single-fragment pilot from the fuller protocol needed for formal evaluation.

4.1 Three Unified Tasks

  • The three conditions use synchronized samples and the same 44-angle targets, valid-label mask, and test centers, keeping their evaluation frames and output dimensions consistent.The EMG-only condition uses both wrists, the vision-only condition uses the current RGB image, and the fusion condition uses both inputs.Source: §4.1 · p. 6

4.2 Evaluation Protocol and Training Settings

  • MAE averages absolute errors across valid test centers and 44 angles, while grouped results must be weighted by sample count to match the overall test set.The paper identifies 11.482° as the sample-weighted EVFormer main value.Source: §4.2 · p. 6

4.3 EMG-to-Pose Branch

  • The EMG branch converts the 100×16 binned sequence into 25 64-dimensional tokens, then uses temporal position embeddings and a four-head Transformer encoder.Training-block statistics normalize channels, and normalized absolute values are clipped to 8.Source: §4.3 · p. 6

4.4 Vision-to-Pose Branch

  • The vision branch turns a 64×64 RGB image into 16 64-dimensional tokens and applies a four-head Transformer encoder for intra-image self-attention.The lightweight branch is intended for single-segment feasibility experiments, not as a competitive alternative to larger pretrained backbones.Source: §4.4 · pp. 6–7

4.5 Vision-EMG Fusion: EVFormer

  • EVFormer applies sequential bidirectional cross-attention with residual scaling, then gates the modality summaries and regresses 44 joint angles.Vision queries EMG first; EMG then queries the updated visual representation. The attention uses 16 visual and 25 EMG tokens, and the gate is not a calibrated sensor-reliability score.Source: §4.5 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

4.6 Connection to Formal Benchmarks

  • The single-fragment pilot does not replace formal evaluation, which requires the official EgoEMG data and gesture, user, and combined test divisions.The paper also reports that matching seed-level artifacts are unavailable, so some detailed results cannot be independently recovered.Source: §4.6 · pp. 8–9, Table 1 · p. 8

5. Experiments

On the single-segment test, EVFormer had the lowest overall MAE among the reported controls. Regional EVFormer results and perturbation effects remain unverified, and the chronological splits have several possible sources of distribution shift.

5.1 EMG-to-Pose

  • EMG-only test MAE was 13.610° (three-seed SD 0.485°), versus validation MAE of 10.691°; its wrist error was 17.492° across four wrist dimensions.The passage notes that the increase in test error and seed fluctuation indicate instability for this single-segment setup.Source: §5.1 · p. 9

5.2 Vision-to-Pose

  • The visual-only branch had 13.228° overall test MAE, but EVFormer’s regional results were not verified, precluding hand-, finger-, or joint-level improvement claims.EVFormer’s regional values could not be reconciled with its primary overall value and require re-aggregation from matching predictions.Source: §5.2 · p. 10

5.3 Fusion of EMG and Vision

  • Late fusion had 13.387° MAE versus EVFormer’s 11.482° (14.23% relative improvement), but parameter differences prevent attributing the full gap to cross-attention alone.Late fusion retains both encoders and gating but omits token-layer exchange.Source: §5.3 · p. 10
  • 11.482° was EVFormer’s overall test MAE, below vision-only (13.228°), EMG-only (13.610°), late fusion (13.387°), and training mean (13.320°).Figure 6 compares overall test MAE on the same 296 test centers and 44-dimensional targets.Source: §5.1 · p. 9, Figure 6 · p. 11

5.4 Sensitivity to Input Changes

  • Input-perturbation effects cannot be quantified from the reviewed results; clean-reference consistency and matched checkpoints and test predictions remain unresolved.The proposed review covers image zeroing, blur, channel zeroing, EMG noise, and combined conditions; software perturbations do not substitute for real-world or cross-user evaluation.Source: §5.4 · pp. 11–12

5.5 Time Periods and Data Distribution

  • Validation MAE was 10.415–10.691° for vision-only, EMG-only, and late fusion, and 8.720° for EVFormer (three-seed SD 0.080°); test errors were higher for every learning model.The validation-to-test gap cannot be attributed entirely to unseen gestures because time, fatigue, camera angle, muscle activation, and labeling efficiency may also vary.Source: §5.5 · p. 12
  • The final 74 retained test centers were more sparsely distributed in time as labeling efficiency declined, so equal-sized sample groups do not represent uniform observations across periods.The first three groups are around 200–225 seconds, while the last extends to 281 seconds.Source: §5.5 · p. 12

6. Fine-Grained Experimental Results and Diagnostics

The diagnostics report EVFormer’s sample-weighted test error alongside gesture-level patterns, data-availability constraints, and unresolved checks that limit interpretation of subgroup and robustness results.

6.1.2 Descriptive Test Errors by Gesture Class

  • EVFormer had the lowest MAE in four gesture classes, but class 21 was an exception at 12.989°, above vision-only at 12.834°.Category 27 contains only 16 samples; Figure 7 places class errors beside sample counts.Source: Figure 7 · p. 13, §6.1.2 · p. 13

6.1.3 Why Aggregation Matters

  • 11.482° sample-weighted EVFormer MAE was lower than vision-only (13.228°), EMG-only (13.610°), and late fusion (13.387°).The calculation weights the five gesture-class errors by their sample counts; category 22 contributes the most.Source: §6.1.3 · p. 13
  • The five-class macro average is 10.6248°, but the paper’s main indicator is the sample-weighted 11.482° overall MAE.The five categories contain 51, 142, 59, 28, and 16 samples; category 22 has the largest weight.Source: §6.1.3 · p. 13, §6.1.2 · p. 13

6.2.1 Changes in Validity over Recording Time

  • Valid bilateral labels decline from nearly all raw rows in the first 50 seconds to about 3.3% at 250–300 seconds and zero after 300 seconds.These rates describe label availability, not prediction success; Figure 8 visualizes the reported time intervals.Source: §6.2.1 · pp. 13–14, Figure 8 · p. 14

6.1.1 Per-Seed Results; 6.2.2 Independence and Statistical Units

  • Adjacent test centers may remain correlated within the same recording session, and the corrected EVFormer result lacks a verified seed-level dispersion or confidence interval.Time-block separation avoids overlapping input windows across splits but does not remove same-session correlation.Source: §6.1.1 · p. 12, §6.2.2 · p. 14

6.3.1 Definition and Unit of Pairwise Comparisons; 6.3.2 Equal-Count Temporal Groups in the Test Set

  • Frame-level pairwise wins and EVFormer’s four temporal-group results remain unverified, so the overall ranking does not establish improvement in every frame or time group.The four groups contain 74 samples each, but their physical time spans differ; the fourth extends to 281.0 seconds.Source: §6.3.1 · p. 15, §6.3.2 · p. 15

6.4.1 Left/Right Hand, Finger, and Wrist Errors; 6.4.2 Descriptive Statistics of Training, Validation, and Test Targets

  • EVFormer’s regional and joint-level errors were removed because the corrected aggregate could not be reconstructed, while target-distribution statistics do not explain validation-to-test error changes.The 40 finger and 4 wrist dimensions require recalculation from matched predictions; training-mean pose was close to the visual model’s test MAE.Source: §6.4.1 · p. 16, §6.4.2 · p. 16

6.5.1 Instantiated and Trainable Parameters; 6.5.2 Theoretical Interaction Size and Measured Latency; 6.6.2 Checkpoint Consistency and Per-Seed Audit; 6.6.3 Checkpoints, Result Files, and Interpretive Limits

  • Late fusion and EVFormer differ by 33,280 gradient-active parameters, and perturbation results, latency, and several EVFormer subgroup values still require matched-checkpoint review.The parameter difference prevents attributing the full gain to cross-attention alone; the reported input transformations are not verified robustness results.Source: §6.5.1 · p. 17, §6.5.2 · p. 17, §6.6.2 · p. 18

7. Discussion and Future Evaluation

EVFormer had the lowest overall error in the reported internal comparison, but the study’s single-participant scope and unresolved controls limit what can be concluded. The discussion outlines broader evaluation, robustness, and deployment checks needed to assess its generality and practical value.

7.1 Main Findings

  • EVFormer’s test MAE was 11.482°, below vision-only (13.228°), EMG-only (13.610°), late fusion (13.387°), and training-mean pose (13.320°).It was lowest in four of five gesture classes, with a local disadvantage in class 21.Source: §7.1 · p. 19

7.2 Scope of Evaluation Relative to Baselines and Dataset; 7.3 Limits of Interpretation

  • The comparison uses one participant and one recording; it does not establish superiority across all gesture classes or published methods.The official multi-user EgoEMG division and a matching residual-fusion baseline were not used.Source: §7.2 · p. 19, §7.3 · p. 19

7.4 Sequence of Follow-Up Experiments

  • Follow-up work should test image crops and hand visibility, then re-evaluate input perturbations using checkpoints consistent with the 11.482° result.True electrode displacement and multi-user signal quality require separate data collection.Source: §7.4 · p. 19

7.4 Sequence of Follow-Up Experiments; 7.5.2 Robustness and Ablation Studies

  • Capacity-matched ablations should vary cross-attention direction and EMG history length to distinguish interaction effects from parameter count or input choices.The current EVFormer comparison has more effective training parameters than late fusion.Source: §7.4 · pp. 19–20, §7.5.2 · p. 20

7.5.1 Data Splits and Comparisons

  • A fuller benchmark should use official gesture, user, and combined splits, compare matched baselines, and report results separately for each division.Per-user distributions and prespecified gesture-family results can complement joint-angle MAE.Source: §7.5.1 · p. 20

7.5.2 Robustness and Ablation Studies; 7.5.3 Reporting Requirements and Failure Criteria

  • Robustness tests need independent visibility or preset-perturbation criteria, while uncertainty should be based on participants or acquisition groups rather than adjacent frames.The reporting plan also calls for explicit hardware measurements of training time, memory, parameters, and latency.Source: §7.5.2 · p. 20, §7.5.3 · p. 20

7.6.1 Data Verification and Baseline Reproduction; 7.6.2 Prespecified Outcomes and Resource Reporting

  • Before formal comparison, researchers should verify data and loaders, reproduce published baselines, and prespecify outcomes, perturbation subsets, and resource reporting.The proposed stress tests include predetermined EMG timing offsets and matched perturbations across methods.Source: §7.6.1 · pp. 20–21, §7.6.2 · p. 21

7.6.3 Three Possible Conclusions from a Complete Study; 7.7.1 What the Current Model Predicts; 7.7.2 Evaluating Practical Value for Control

  • The model estimates current posture from past EMG and a current image; offline angle MAE alone does not establish future-intention prediction or online control value.A complete evaluation should interpret results against prespecified outcomes, including conditions where the current design is not supported.Source: §7.6.3 · p. 21, §7.7.1 · pp. 21–22, §7.7.2 · p. 22

8. Conclusion

EVFormer combines egocentric vision and bilateral-wrist EMG to regress bimanual joint angles, and achieved a lower overall error than the reported controls in a single-recording comparison. The conclusion treats this as preliminary evidence and calls for broader evaluation.

8. Conclusion

  • EVFormer uses sequential bidirectional cross-attention and feature-wise gating to regress 44 bimanual joint angles.It exchanges information between visual spatial tokens and bilateral-wrist EMG temporal tokens.Source: §8 · p. 22
  • 11.482° sample-weighted test MAE was below vision-only (13.228°), EMG-only (13.610°), late fusion (13.387°), and training-mean pose (13.320°).The comparison used one participant and one recording; EVFormer was best in four of five gesture classes, with a local disadvantage in class 21.Source: §8 · p. 22
  • The results support an overall accuracy gain in this setting, while generalization, real-input robustness, and online performance require further study.Unverified seed, regional, temporal-group, and perturbation results are not used to support the conclusion.Source: §8 · p. 22

Appendix A: Data Fields and Angle Semantics

The appendix describes how synchronized EMG and video fields relate to the bimanual angle targets, including their index semantics and validity requirements.

A.1 Input and Target Fields

  • The input fields include left and right filtered EMG, a center-frame index, joint-angle arrays, wrist angles, and label-validity flags.Gesture class is used only for post-training diagnosis, not as a model input or for checkpoint selection.Source: §A.1 · p. 24
  • Each hand’s 22 output dimensions comprise 20 finger angles and two wrist angles, with the right-hand indices shifted by 22 in the 44-value vector.The index ranges assign four angles to each finger and two dimensions to the wrist.Source: §A.1 · p. 24, Table A1 · p. 24

A.2 Unit Conversion and Validity Checks

  • Finger targets are converted from radians to degrees while wrist values remain in degrees; samples require valid labels for both hands and both wrists.This strict bilateral validity rule deletes an observation if either hand’s label is invalid.Source: §A.2 · p. 24

B.1 Exact Sample-Selection Procedure

The sample-selection procedure uses incremental candidate centers, validity checks, and temporal boundary spacing to create deterministic train, validation, and test blocks. The retained test segment ends before the source recording ends.

  • At 2 kHz, candidate centers began at row 400 and advanced by 200 rows, with each 400-row EMG history ending just before its center.Source: §B.1 · p. 24
  • Validity and frame-index checks followed by 70% and 85% sequence cut points produced the final 1,375/293/296 split after four boundary samples were removed.The boundary removals prevent the 400-row EMG history from overlapping adjacent blocks.Source: §B.1 · p. 24
  • The test retention ends at 281.0 s although the source extends to approximately 345.7 s, so it does not cover the entire acquisition.Source: §B.1 · pp. 24–25

B.2 Gesture IDs in Each Data Block

Gesture-category overlap between training and testing is limited, and the section also specifies the pre-experimental network configuration.

B.2 Gesture IDs in Each Data Block

  • Table B1 indicates little overlap between labeled training and test gestures; the unlabeled category -1 appears only in the training block.Source: §B.2 · p. 25, Table B1 · p. 25
  • Testing contains 51, 142, 59, 28, and 16 samples across five categories; strict gesture generalization still requires official multi-participant classification [1].Source: §B.2 · p. 25

C.1 Lightweight Model Configuration

  • The 171,116-parameter pre-experimental network uses 64-dimensional tokens, four attention heads, 16 visual tokens, and 25 EMG tokens.Its visual and EMG backbones use convolutional layers, while cross-attention residuals are scaled by 0.1 and a sigmoid gate combines modality features.Source: §C.1 · p. 25

C.2 Training and Evaluation Implementation

Training-only statistics and validation-based checkpoint selection govern evaluation, while the reported overall score still requires archived artifacts and reconstruction checks.

  • EMG channel statistics are estimated only from training samples; images are scaled to 0–1, and targets are divided by 90 for training before predictions are rescaled for angular MAE.Source: §C.2 · p. 25
  • Validation MAE alone drives early stopping, and each mode and seed's optimal checkpoint is reloaded before test metrics are calculated.Source: §C.2 · p. 25
  • The reported 11.482° sample-weighted EVFormer score still needs archived seeds, checkpoints, prediction arrays, and logs for reconstruction verification.Hardware, software version, and operating configuration also need to be documented.Source: §C.2 · p. 25
Loading 2610.06970v1…