PaperlayerView exact revision on arXiv

Source-linked AI summary

EMG-GPT: Predictive Pretraining on Residual-Quantized EMG Tokens for Hand Pose Estimation

Ettore Magni, Rolandos Alexandros Potamias, Stefanos Zafeiriou, Konstantinos Barmpas

arXiv:2610.05235v1cs.LGcs.AI

TL;DR

The paper asks whether unlabeled sEMG can yield reusable predictive representations for continuous hand-pose estimation. EMG-GPT predicts future codes from frozen NeuroRVQ representations, combining within-frame integration with causal temporal modeling; its states transfer to Regression and Tracking with competitive reported results.

  • Problem

    Whether unlabeled EMG supports reusable predictive representations for continuous pose estimation remains unestablished by supervised decoding results alone.

    Source: §1 · p. 1

  • Method

    EMG-GPT keeps NeuroRVQ representations fixed, integrates structured features within each frame, models frames causally, and predicts future codes coarse-to-fine.

    Source: §1 · p. 2, §2.2 · p. 2, §3.2 · p. 4

  • Results

    Across all three conditions, Regression has lower angular and landmark errors than SensingDynamics and NeuroPose, while Tracking remains marginally behind vEMG2Pose.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The experiments support transfer from EMG-only pretraining to continuous hand-pose decoding in Regression and Tracking.

    Source: §4 · p. 5, §5 · p. 6

    External comparisons are descriptive, not evidence of significance or benchmark superiority, because frontend, augmentation, temporal alignment, and scoring grid differ.

    Source: §4 · p. 5

Abstract

from arXiv · show

Surface electromyography (sEMG) is a low-power, cost-effective biosignal for hand-pose estimation and gesture classification. In this work, we examine whether self-supervised pretraining on sEMG can yield transferable representations for continuous hand-pose estimation. We introduce EMG-GPT, a causal transformer-based model that operates on discrete sEMG representations from a frozen residual vector quantization (RVQ) tokenizer and learns temporal dynamics through depth-autoregressive future-code prediction. The model combines within-frame integration with causal temporal modeling while preserving the geometry of the pretrained codebook. EMG-GPT shows competitive results in both Regression and Tracking tasks, supporting EMG-only pretraining as a viable approach for learning transferable sEMG representations.

1 Introduction

The introduction motivates continuous hand-pose estimation from wearable sEMG and asks whether unlabeled EMG can support reusable predictive representations. EMG-GPT addresses this with structured-token future prediction and tests whether the learned states transfer to pose decoding.

1 Introduction

  • Continuous pose estimation must recover evolving articulated states, but supervised decoding results do not establish whether unlabeled EMG supports reusable predictive representations.The emg2pose benchmark provides 370 hours of paired sEMG and motion capture from 193 participants, with generalization tests across users, sensor placements, and activity stages.Source: §1 · p. 1
  • EMG-GPT keeps the pretrained NeuroRVQ tokenizer and codebooks fixed, predicts future-frame codes in RVQ depth order, and avoids making temporal sequence length depend on retained depth K.This separates temporal pretraining from tokenizer adaptation.Source: §1 · p. 2
  • Because token predictability need not imply kinematic utility, the study tests learned states on emg2pose Regression and Tracking under held-out generalization conditions.The introduction also situates wearable, non-invasive sEMG as a modality for estimating hand movements.Source: §1 · pp. 1–2

2 Background

The background describes NeuroRVQ's structured EMG codes and the modeling challenge they pose: prediction must preserve both temporal order and the within-frame residual hierarchy.

2.1 Frozen NeuroRVQ EMG representations

  • NeuroRVQ maps multichannel EMG into branch-specific RVQ code stacks at each electrode; each code index selects a codebook row rather than carrying geometric or ordinal meaning itself.The structured frame representation retains electrode, branch, and residual-depth axes.Source: §2.1 · p. 2

2.2 Structured causal prediction

  • A temporal predictor must model causal dependencies across frames while accounting for the ordered, branch-specific coarse-to-fine code hierarchy within each frame.These are distinct structures: time-ordered context and residual-code order.Source: §2.2 · p. 2

3 Method

EMG-GPT represents each EMG frame by integrating structured NeuroRVQ features, then models frame evolution causally and predicts future codes in residual-depth order. Its pre-head states transfer to separate Regression and Tracking decoders under frozen or full-backbone adaptation.

3 Method

  • A bidirectional intra-frame encoder integrates simultaneous electrode–frequency features into one state per frame, followed by a causal transformer over frame states.Only token-frame modeling is causal; increasing retained RVQ depth enriches each frame without increasing the temporal sequence length.Source: §3 · p. 3, Figure 1 · p. 3, §3.1 · p. 4

3.1 Structured RVQ Frame Encoding

  • The encoder sums frozen codebook vectors across retained RVQ depth, then applies normalization, projection, and channel and frequency-branch embeddings to form its input features.A learned frame-summary token and bidirectional intra-frame transformer produce the frame state.Source: §3.1 · pp. 3–4

3.2 Hierarchical Predictive Pretraining

  • Causal temporal states predict future codes coarse-to-fine: finer depths condition on realized coarser codes, while same-depth channel–branch targets factorize conditionally and score against matching frozen codebooks.The pretraining objective bypasses neither the fixed tokenizer nor its codebooks; no gradient reaches NeuroRVQ or its codebooks.Source: §3.2 · p. 4

3.3 Transfer to Hand Pose

  • Regression interpolates features without an initial pose; Tracking repeats them and receives the boundary pose, while frozen transfer fixes the backbone and full adaptation also trains the EMG-GPT encoder.Both tasks use the pre-head states and bypass the categorical prediction head.Source: §3.3 · pp. 4–5, §3.2 · p. 4

4 Evaluation

Evaluation compares EMG-GPT with published baselines on Regression and Tracking under held-out generalization conditions. Regression errors are lower than those of SensingDynamics and NeuroPose, while vEMG2Pose is marginally stronger; Tracking is also marginally behind vEMG2Pose.

4 Evaluation

  • Evaluation uses the emg2pose partitions and reports Regression separately from Tracking, with User, Stage, and combined User–Stage holdouts.Tracking receives a ground-truth boundary pose; the reported evaluation uses K = 4, a 25-Hz representation, and full-backbone adaptation.Source: §4 · p. 5
  • Across all three conditions, Regression has lower angular and landmark errors than SensingDynamics and NeuroPose, while vEMG2Pose remains marginally stronger.The table reports mean ± sample SD across users within each generalization condition.Source: §4 · p. 5, Table 1 · p. 5
  • External comparisons are descriptive rather than claims of significance or benchmark superiority because frontend, augmentation, temporal alignment, and scoring grid differ.Source: §4 · p. 5

5 Conclusions

EMG-GPT combines frozen NeuroRVQ geometry with within-frame integration, causal temporal modeling, and depth-autoregressive future-code prediction. Experiments support transfer from EMG-only pretraining, while the authors identify adaptations and broader pretraining as future directions.

5 Conclusions

  • EMG-GPT combines frozen NeuroRVQ geometry, bidirectional within-frame integration, causal temporal modeling, and depth-autoregressive future-code prediction.Source: §5 · p. 6
  • The experiments support transfer from EMG-only pretraining.Source: §5 · p. 6
  • Future work will investigate joint tokenizer–predictor adaptation, motion-preserving objectives, causal streaming inference, and scaling across users, devices, and datasets.The authors describe scaling as a possible route toward a transferable sEMG foundation model for pose estimation, gesture recognition, and human–machine interaction.Source: §5 · p. 6

A Frozen NeuroRVQ EMG Tokenization Interface

The section specifies the frozen NeuroRVQ-EMG token interface and EMG-GPT's predictive interface. It preserves branch–depth codebook structure while producing one causal temporal state per frame and predicting codes autoregressively across depth.

A Frozen NeuroRVQ EMG Tokenization Interface

A.1 Codebook assignment and residual recurrence

  • For each frame, electrode, and branch, the tokenizer starts with latent z, assigns code indices from depth-specific codebooks, and defines reconstruction from the first K depths.Each codebook has M tokens; the residual initialization is r(0) = z and the assignment uses a numerical stabilizer ε > 0.Source: §A.1 · p. 7

A.2 Structured EMG output

  • The structured output retains electrode, frequency-branch, and RVQ-depth axes; the tokenizer has four branches and up to 16 depths, with 8192 tokens of dimension 128 per codebook.Codebooks are shared across electrodes but not across branches or depths, and pretraining uses 16 global electrodes.Source: §A.2 · p. 7

A.3 Identifiers and frozen geometry

  • Code indices are addresses rather than geometric coordinates; the matching frozen branch–depth codebook vectors carry the representation geometry.Relabeling indices and permuting the corresponding codebook rows leaves the selected vectors and reconstruction unchanged.Source: §A.3 · p. 7

B EMG-GPT Predictive Interface

  • The predictive interface takes windows of structured code indices and frozen codebooks, then constructs frame-level states for temporal modeling.The stored input has shape [N, L, C, B, K], and the codebooks have shape [B, K, M, de].Source: B EMG-GPT Predictive Interface · p. 7, §B.1 · p. 7

B.1 Frame and temporal backbone

  • A bidirectional frame encoder integrates electrode–branch features into one state per frame before causal temporal blocks produce states dependent only on frames 1:t.The [FRAME] token is prepended per time step; training pairs windows with targets Δ token frames ahead, and discontinuities split sequences.Source: §B.1 · p. 8

B.2 Depth-autoregressive codebook head

  • At each RVQ depth, queries incorporate the partial reconstruction from coarser levels; the head scores predictions against the matching frozen codebooks.Same-depth queries exchange context without receiving realized same-depth targets; training uses ground-truth coarser codes, while generation uses previously generated coarser codes.Source: §B.2 · p. 8

B.3 Effect of retained depth and transfer tensor

  • Increasing retained depth K adds reconstruction vectors and ordered prediction targets, but leaves intra-frame positions, context length, and transferred temporal states unchanged.Transfer uses H after the backbone LayerNorm, not code logits or predicted future indices.Source: §B.3 · p. 8

C Implementation and Pretraining Details

Pretraining uses complete contiguous windows and predicts token frames ahead with frozen codebooks. The selected configuration is 25 Hz with K = 4.

C Implementation and Pretraining Details

  • At 25 Hz with a 40-ms hop and Δ = 5, the prediction target is 200 ms ahead.Patch width is recorded separately and does not define the forecast interval.Source: C Implementation and Pretraining Details · p. 9
  • Missing storage blocks split recordings into separate sequences, so training windows do not span discontinuities; codebooks remain frozen while the input path and categorical head use the same tables.Frame-encoder, temporal, and query-projection parameters are learned during pretraining.Source: C Implementation and Pretraining Details · p. 9
  • The selected EMG-GPT configuration uses a 25-Hz representation and K = 4.Table 2 identifies this as the selected model configuration.Source: Table 2 · p. 9

D Regression and Tracking Protocols

Both tasks use the final normalized states of a pretrained K = 4 EMG-GPT with NeuroRVQ and its codebooks fixed. They use separate pose-decoding and loss protocols for Regression and Tracking.

D Regression and Tracking Protocols

  • Both tasks use 150 token frames at 25 Hz with 25 left-context frames, and score a five-second pose tail at 50 Hz.Invalid inverse-kinematics rows are masked from optimization and scoring.Source: D Regression and Tracking Protocols · p. 9
  • Regression interpolates features and starts from zero without a ground-truth initial pose, while Tracking causally repeats features and starts from the ground-truth scoring-boundary pose.Both decode 20 joint angles with a two-layer, 512-hidden-width LSTM; Tracking has 40 ms token-to-pose latency.Source: D Regression and Tracking Protocols · p. 9
  • Regression combines masked joint-angle L1 with fingertip distance, whereas Tracking adds velocity loss weighted by 4.Full adaptation trains the RVQ-input mapping, intra-frame encoder, temporal GPT blocks, embeddings, and final backbone LayerNorm.Source: D Regression and Tracking Protocols · p. 9

E.1 Tracking Results

Table 3 reports Tracking test results for the setting where the boundary pose is given.

E.1 Tracking Results

  • Table 3 reports Tracking test results with the boundary pose given; its vEMG2Pose values are from [Salter et al., 2024].Source: Table 3 · p. 10

E.2 Matching Evaluation Protocol with vEMG2Pose

In the locally matched evaluation, held-out-user Regression angular errors are equal after rounding, while vEMG2Pose has lower mean error in the other Regression comparisons and all Tracking comparisons. The comparison retains each model’s native frontend and is descriptive rather than a benchmark-superiority claim.

E.2 Matching Evaluation Protocol with vEMG2Pose

  • The local comparison uses identical target-index hashes, valid-value counts, metric implementation, and equal-user aggregation, but retains each model’s native frontend.It is not a reproduction of the published protocol, and the authors make no significance or benchmark-superiority claims.Source: §E.2 · p. 10
  • Table 4 reports locally matched checkpoint results as mean ± sample SD across users within each generalization condition.Bold marks lower means, with rounded ties bolded together.Source: Table 4 · p. 10
  • Held-out-user Regression angular errors are equal after rounding to two decimals; vEMG2Pose has lower mean error in remaining Regression comparisons and all Tracking comparisons.Source: §E.2 · p. 10, Table 4 · p. 10

E.3 Qualitative Results

The qualitative examples use validation clips selected by user and stage error percentiles, with windows chosen nearest the corresponding aggregate error. Regression comparisons verify target equality across aligned segments.

E.3 Qualitative Results

  • Figures 3–7 show selected Tracking clips for the median user and stage, lower-error stage, higher-error stage, and lower- or higher-error users.The examples compare motion-capture targets with EMG-GPT Tracking and a locally evaluated vEMG2Pose reference on the same clips.Source: §E.3 · p. 11, Figure 3 · p. 11, Figure 4 · p. 11
  • Figure 3 presents six FingerFreeform poses across a five-second validation clip, with motion capture above EMG-GPT Tracking and vEMG2Pose.Source: Figure 3 · p. 11
  • Figures 8–10 present regression examples for a median held-out user at median, lower-error, and higher-error held-out stages; their rows and target-aligned temporal layouts follow Figure 8.Figure 8 shows motion-capture targets, EMG-GPT Regression, and vEMG2Pose Regression across six poses in a five-second validation clip.Source: Figure 8 · p. 12, Figure 9 · p. 13, Figure 10 · p. 13
Loading 2610.05235v1…