PaperlayerView exact revision on arXiv

Source-linked AI summary

SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

Matea Tashkovska, Naël Ouerghemi, Haozhe Qi, Alexander Mathis

arXiv:2610.36384v1cs.CVcs.AIcs.CLq-bio.NC

TL;DR

MLLMs are mainly evaluated on third-person activity understanding, leaving fine-grained body-motion reasoning from egocentric video poorly assessed. SOMA benchmarks this capability across 12 tasks, and its evaluation finds that the strongest model trails human accuracy.

  • Problem

    MLLM evaluations largely emphasize third-person activity understanding, leaving fine-grained body-motion reasoning from the egocentric perspective poorly assessed.

    Source: §1 · p. 2

  • Method

    SOMA evaluates egocentric body-motion understanding with 1,632 human-verified questions across 12 tasks and three capability axes.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    50.0% zero-shot accuracy for Qwen3.6-27B trails the 68.5% human mean accuracy on SOMA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Perceiving the moving body from a first-person view remains an open problem across the evaluated model types.

    Source: §6 · p. 9

Abstract

from arXiv · show

Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone (68.5% accuracy), yet clearly outperform the strongest MLLM (Qwen3.6-27B), which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. Full benchmark, training data, and code will be released at: https://cnai.epfl.ch/soma

1 INTRODUCTION

SOMA benchmarks fine-grained body-motion understanding from egocentric video, where partial visibility and temporal cues make motion difficult to infer. Its evaluation finds a substantial gap between the best MLLM and human accuracy.

Introduction

  • Existing egocentric benchmarks target other dimensions of video understanding, leaving unmeasured whether models infer the moving body’s configuration and how its actions unfold over time.This gap makes it unclear whether progress reflects understanding human motion or recognition of the objects and scenes in which motion occurs.Source: §1 · p. 2
  • Egocentric body-motion understanding is challenging because the body is often partially visible, fine-grained dynamics are hard to perceive, and blur and viewpoint changes require integrating cues across frames.The viewpoint is relevant to augmented-reality interfaces, wearable assistants, and embodied agents.Source: §1 · p. 2
  • SOMA comprises 1,632 human-verified multiple-choice questions across 12 tasks in Body Configuration, Motion Dynamics, and Temporal Reasoning, and releases aligned video, body-motion, and language training data.The benchmark draws on Nymeria and Ego-Exo4D; SOMA-Train supplies a combination of aligned supervision absent from existing egocentric video benchmarks.Source: §1 · p. 2
  • 50.0% zero-shot accuracy for Qwen3.6-27B trails humans’ 68.5% mean accuracy across SOMA’s 1,632 questions, despite limited gains from denser sampling and test-time reasoning.The evaluation covered 37 closed-source, open-source, and specialized egocentric MLLMs; five human evaluators answered all questions.Source: §1 · p. 2
  • Error analysis identifies misreading the wearer’s body and difficulty localizing events in time as common failures; aligned video-and-motion fine-tuning yields modest gains, suggesting complementary information.The authors test explicit body-motion signals by fine-tuning on SOMA-Train.Source: §1 · p. 2

2 RELATED WORK

Prior work studies motion through pose perception, action recognition, generation, and video QA, but fine-grained body-motion perception from egocentric video remains underexplored. SOMA evaluates this capability with body-centric questions grounded in motion annotations.

Human Motion Understanding

  • SOMA evaluates body-motion perception from first-person video, complementing benchmarks based on third-person video or motion-capture input alone.Prior work spans 3D pose reconstruction, skeleton-based action recognition and segmentation, motion generation, and MLLMs that unify motion comprehension and generation; models explicitly combining video and body motion remain uncommon.Source: §2 · p. 3

MLLMs for Video Understanding

  • Existing egocentric MLLMs target activity- and scene-level understanding, leaving fine-grained body-motion perception largely unexplored; SOMA evaluates it and provides a training split.EgoThinker (Pei et al., 2025) uses chain-of-thought rationales, while EgoGPT (Yang et al., 2025b) is fine-tuned for egocentric captioning and QA.Source: §2 · p. 3

Egocentric VQA Benchmarks

  • Existing egocentric VQA benchmarks do not directly evaluate moving-body perception, which SOMA probes with body-centric questions grounded in fine-grained motion annotations, video, and 3D body pose.Prior benchmark tasks—including activity recognition, task planning, and temporal grounding—focus on objects and the surrounding environment rather than the body itself.Source: §2 · p. 3

3 THE SOMA BENCHMARK

SOMA is a benchmark for fine-grained egocentric body-motion understanding, organized around body configuration, motion dynamics, and temporal reasoning; it also includes a training split with aligned video, motion, and language supervision.

3.1 TASK SUITE

  • The benchmark spans three axes—Body Configuration, Motion Dynamics, and Temporal Reasoning—that comprise 12 tasks for understanding moving bodies from first-person video.Tasks cover hand use, bimanual coordination, posture, concurrent body-part state, repetition counting, movement direction, opposite-motion discrimination, adjacent actions, chronological ordering, missing-event inference, next-action prediction, and repeated-event reasoning.Source: §3.1 · pp. 3–4

3.2 DATA CURATION PIPELINE

  • SOMA’s curation combines egocentric video annotations, task-specific question generation, blind-language filtering, and human review to produce visually grounded questions.Videos come from Nymeria and Ego-Exo4D; an LLM selects task-relevant segments before generating multiple-choice questions, while repetition-count questions are created manually. Blind filtering removed 45.9% of candidate questions, and reviewers verified or edited the remainder using synchronized video and 3D body-pose renderings.Source: §3.2 · pp. 4–5

3.3 DATASET STATISTICS

  • 1,632 questions across 12 tasks draw on 397 videos and 1,611 clips, with balanced answer options and longer clips concentrated in Temporal Reasoning.The average clip is 35.6 seconds, with most lasting 10–60 seconds; Temporal Reasoning clips extend into the long tail.Source: §3.3 · p. 5

3.4 SOMA-TRAIN

  • SOMA-Train provides 28,596 question-answer pairs over 110.2 hours of aligned egocentric video, body motion, and language supervision.Training data is disjoint from the evaluation set at participant and scene level; 3D body motion is reconstructed at 20 fps from multi-view recordings.Source: §3.4 · p. 5

4 EXPERIMENTAL SETUP

The evaluation measures human accuracy on all 1,632 questions and compares 37 MLLMs under a standardized zero-shot protocol. It reports per-task and overall accuracy to support consistent evaluation and future comparisons.

Human Performance

  • Five compensated evaluators independently answered all 1,632 questions, totaling over 450 evaluation hours, and their mean accuracy estimates human-level performance.The evaluators had no prior familiarity with the benchmark and did not participate in data annotation.Source: §4 · p. 5

Evaluation Setup

  • The study evaluates 37 closed-source, open-source, and specialized egocentric MLLMs, reporting per-task and overall accuracy using lmms-eval (Zhang et al., 2025b).The framework is used to ensure consistent evaluation and facilitate future comparisons.Source: §4 · p. 6
  • Unless otherwise specified, models are evaluated zero-shot at temperature 0 using 16 uniformly sampled frames per segment to balance computational cost and comparability across context lengths.The study also ablates key evaluation settings in Section 5.2.Source: §4 · p. 6

5 EXPERIMENTS

Across SOMA experiments, MLLMs trail humans overall and struggle to integrate body-motion evidence over time; sampling and test-time reasoning offer limited gains, while aligned body-motion input provides modest, task-dependent benefits.

5.1 MLLM AND HUMAN PERFORMANCE ON SOMA

  • Best models reach 74.6% on Hand Use and 71.4% on Opposite Motion Discrimination, while most score below 40% on Chronological Ordering and Repeated Event Reasoning.Performance is stronger when evidence is concentrated in a few frames than when motion must be integrated across a clip.Source: §5.1 · p. 6
  • Egocentric specialization does not outperform similarly sized general models, while scaling improves accuracy but does not close the human gap.Qwen3-VL gains 13.1 points from 2B to 32B and Qwen3.5 gains 17.8 points from 0.8B to 27B; InternVL3.5 improves by 8.1 points from 1B to 38B, non-monotonically.Source: §5.1 · p. 6

5.2 IMPACT OF TEMPORAL SAMPLING AND TEST-TIME REASONING

  • Across models, increasing sampled frames from 1 to 16 raises accuracy by 5.4 points on average, while additional frames contribute at most 0.5 points.The 1-to-16-frame gain ranges from 1.2 points for VideoLLaMA3-7B to 10.3 points for Qwen3.5-9B, suggesting sampling density is not the principal limitation.Source: §5.2 · p. 6, Figure 4 · p. 7

5.1 MLLM AND HUMAN PERFORMANCE ON SOMA

  • Qwen3.6-27B reaches 50.0% average accuracy and Gemini 3.5 Flash 47.0%, both below the 68.5% human average.The human–model gap is largest on Repetition Counting (45.3 points), Repeated Event Reasoning (25.4), and Bimanual Coordination (25.3), but closes or reverses on Concurrent Body-Part State (−1.1) and Next Action Prediction (−10.0).Source: §5.1 · p. 6, Table 2 · p. 7

5.2 IMPACT OF TEMPORAL SAMPLING AND TEST-TIME REASONING

  • Enabling test-time reasoning yields gains of 1.0 to 10.2 points for five Qwen3.5–3.8 models, while seven other models show negligible change or decline.The effect is model-dependent and does not close the human–model gap.Source: Figure 4 · p. 7, §5.2 · p. 8

5.3 QUANTITATIVE AND QUALITATIVE ERROR ANALYSIS

  • Models show a right-hand bias and often under-count repetitions, with missed events between sampled frames contributing to downward counting errors.Across 37 models, handedness accuracy is 78.8% for right and 44.3% for left; 24 models lean toward under-counting, while human repetition-counting accuracy is 88.9% versus 43.6% for the best model.Source: §5.3 · p. 8
  • Misreading the wearer’s body dominates errors on Body Configuration and Motion Dynamics, while temporal localization is also difficult and visual grounding or prior-driven inference account for smaller shares.Body-perception errors include misjudging posture, the limb in use, or movement direction.Source: §5.3 · pp. 8–9

5.4 TRAINING BODY-AWARE MLLMS

  • Fine-tuned video models improve over zero-shot backbones, and adding aligned motion yields modest, task-dependent gains; motion alone is informative but often insufficient.Motion slightly improves Body Configuration and Temporal Reasoning for both backbones; on Motion Dynamics it helps Qwen3-VL-8B but slightly hurts InternVL3.5-8B.Source: §5.4 · p. 9

6 CONCLUSION … C EVALUATION DETAILS

SOMA evaluates fine-grained egocentric body-motion understanding across 12 tasks and three capability axes, revealing a substantial gap between current MLLMs and humans. The appendices document benchmark construction, examples, ethical safeguards, and reproducibility details.

6 CONCLUSION

  • SOMA spans 12 egocentric body-motion tasks across three capability axes, and current MLLMs remain far below human performance.The conclusion frames perceiving the moving body from a first-person view as an open problem.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assisted with manuscript editing and candidate question-answer generation, but authors developed the research and manually verified edits and generated questions.All generated questions were human-verified.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT; REPRODUCIBILITY STATEMENT; C EVALUATION DETAILS

  • SOMA uses consented public datasets and anonymized human-verified annotations; its five paid human evaluators consented, and the study notes both assistive benefits and surveillance or profiling concerns.The work evaluates public MLLM systems on aggregate benchmark performance, not real-world decision-making deployment. Reproducibility materials include construction and evaluation prompts, model lists, training data and fine-tuning details; code and data are slated for release upon acceptance.Source: §6 · p. 9, ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL SOMA EXAMPLES

B.1 SELECTOR AND GENERATOR PROMPT EXAMPLES; Selector Prompt: Repeated Event Reasoning; Generator Prompt: Repeated Event Reasoning

B.2 MITIGATING GENERATOR BIAS

  • Gemini 2.5 Flash generator bias is mitigated with fixed task-specific templates and blind filtering that removes questions it and another solver answer correctly without video.Correct answers and distractors are derived from ground-truth annotations under predefined rules, limiting the generator’s stylistic freedom.Source: §B.2 · p. 32

B.3 MANUAL ANNOTATION OF REPETITION COUNTING

  • 101 Repetition Counting questions were manually annotated using synchronized video and 3D pose, excluding partial, occluded, or unclear repetitions.A repetition counts only when one complete instance of the named action is clearly visible.Source: §B.3 · p. 32

B.4 FILTERING AND REVIEW STATISTICS; Blind Solver Prompt

  • 45.9% of 3,928 generated candidates were removed by blind filtering, and the final benchmark contains 1,632 questions after human review and manual additions.Filtering removed 1,801 questions; 945 were accepted as-is, 586 modified, and 596 deleted during review. Removal was highest for Body Posture (62.4%) and lowest for Repeated Event Reasoning (30.8%).Source: Blind Solver Prompt · p. 33, §B.4 · p. 33

C.1 DETAILS ON HUMAN EVALUATION · C.2 EVALUATED MODELS

The sections describe how human accuracy was evaluated and summarize the breadth of closed-source, open-source, and specialized egocentric models assessed on SOMA.

C.1 DETAILS ON HUMAN EVALUATION

  • Five uninvolved subjects independently answered all 1,632 video-based questions, choosing one of four options and using indirect visual cues when the wearer was not visible.Subjects had unlimited time and were instructed to rely on cues such as camera motion, viewpoint changes, and hand-object interactions.Source: §C.1 · p. 33
  • The evaluation took an average of 93 hours per annotator and 465 hours total; annotators were compensated for their time.Individual completion times ranged from 89 to 97 hours.Source: §C.1 · p. 33
  • Human accuracy was most consistent on Hand Use and varied most on Chronological Ordering and Missing Event Inference.Table 4 reports accuracy as mean ± standard deviation across five annotators for each SOMA task.Source: §C.1 · p. 33, Table 4 · p. 33

C.2 EVALUATED MODELS

  • Closed-source evaluation covered Gemini 2.5 Flash and Pro, Gemini 3.5 Flash, GPT-5.1 and GPT-5.6 Luna, and Claude Sonnet 5 and Opus 4.8.The cited models are attributed to Comanici et al. (2025), Google DeepMind (2026a), OpenAI (2025, 2026), and Anthropic (2026a, 2026b), respectively.Source: §C.2 · p. 34
  • Open-source evaluation spanned Qwen, InternVL, Gemma, GLM, Molmo, Kimi-VL, MiniCPM, LLaVA-OneVision, and VideoLLaMA3 model families.Examples include Qwen3-VL (2B–32B), Qwen3.5 (0.8B–27B), InternVL3.5 (1B–38B), and Gemma-4 (12B, 31B).Source: §C.2 · p. 34
  • Specialized egocentric models included EgoGPT-7B, EgoVLM-3B, and EgoThinker-7B.The passage attributes these models to Yang et al. (2025b), Vinod et al. (2025), and Pei et al. (2025), respectively.Source: §C.2 · p. 34

C.3 IMPLEMENTATION DETAILS … D.3 PERFORMANCE BY QUESTION DIFFICULTY

The evaluation standardizes video sampling, prompts, and answer parsing, then examines how chain-of-thought prompting, frame density, and question difficulty affect performance. Reasoning gains depend on the capability axis, while accuracy falls with difficulty and humans remain ahead of models.

C.3 IMPLEMENTATION DETAILS

  • All models used 16 uniformly sampled frames per video segment and lmms-eval; open-source experiments ran on four H100 or four A100 GPUs.The sampling choice balanced computational cost and comparability across models with different context lengths.Source: §C.3 · p. 34

C.4 ZERO-SHOT EVALUATION PROMPT TEMPLATE / Evaluation Prompt Template

  • The zero-shot template supplied the video, question, and four benchmark options, required a single-letter answer, and used standard lmms-eval parsing to score responses.Options appeared in their benchmark order, shuffled once during benchmark creation.Source: §C.4 · p. 34, Evaluation Prompt Template · p. 34

C.5 CHAIN-OF-THOUGHT EVALUATION PROMPT TEMPLATE / Chain-of-Thought Prompt Template

  • The CoT prompt kept the original question and options but replaced the final instruction with a step-by-step request to identify the question, describe relevant observations, evaluate options, and end with “Answer: X”.The instruction asks models to use the video and explain why each option is or is not supported.Source: §C.5 · p. 34, Chain-of-Thought Prompt Template · p. 35

D.1 EFFECT OF CHAIN-OF-THOUGHT PROMPTING

  • CoT did not systematically help: only Qwen3.8-27B gained 3.9 points and improved on 11 of 12 tasks, while other models were neutral or declined.Its largest gains were in temporal reasoning and motion dynamics; the result mirrors the thinking-mode finding.Source: §D.1 · p. 35, Table 5 · p. 35

D.2.1 EFFECT OF TEMPORAL SAMPLING DENSITY

  • Accuracy on Body Configuration saturates early with denser sampling, Motion Dynamics improves for most models but only marginally beyond 16 frames, and Temporal Reasoning remains weakest at every density.The authors suggest Body Configuration evidence is concentrated in a few frames, whereas Motion Dynamics is difficult to answer from only a few frames.Source: §D.2.1 · p. 36, Figure 28 · p. 36

D.2.2 EFFECT OF TEST-TIME REASONING

  • Test-time reasoning improves Motion Dynamics for most models, changes Body Configuration negligibly or negatively for most, and yields modest changes on Temporal Reasoning.Reasoning can help once relevant evidence has been perceived, but does not compensate for failure to perceive it.Source: §D.2.2 · p. 36, Figure 29 · p. 36

D.3 PERFORMANCE BY QUESTION DIFFICULTY

  • Accuracy falls as question difficulty rises for humans and every model, with humans ahead at every level and the largest gap on the easiest questions.Gemini 2.5 Pro rated question difficulty from 1 (very easy) to 4 (hard), based on how difficult it is to distinguish between options.Source: §D.3 · p. 37

D.4 ERROR ANALYSIS … D.4.2 RIGHT-HAND PRIOR IN LATERALITY QUESTIONS

SOMA’s error analysis defines four failure types and uses independently applied LLM judgments with manual verification to examine model errors. It finds a widespread right-hand advantage in laterality questions and illustrates the error categories with representative cases.

D.4 ERROR ANALYSIS

  • The analysis assigns each incorrect answer its most direct error type: visual grounding, motion or body-state perception, temporal localization, or prior-driven inference.The categories respectively cover misidentifying who or what is involved, misperceiving the wearer’s body or action, misplacing recognized events in time, and answering from expectation rather than evidence.Source: §D.4 · p. 37
  • Two judges independently labeled all 400 reasoning traces and agreed on 80.0% of error types (Cohen’s κ = 0.69, 95% bootstrap CI 0.63–0.75).LLM judges compared each trace and answer with ground-truth annotations; a sample was manually checked against the video and annotations.Source: §D.4 · p. 37

System prompt

  • The difficulty-rating system prompt asks judges to score how fine-grained an observation is needed to distinguish the correct answer from similar distractors, on a 1–4 scale.The prompt directs consideration of semantic similarity, action specificity, distractor confusability, and whether the answer is visible from an egocentric view or requires inference.Source: System prompt · p. 38

User message

  • The user-message template presents four answer options and a ground-truth answer, then asks the respondent to rate the difficulty of distinguishing the options on a 1–4 scale.A separate instruction says to respond with only a single number (1–4) representing the difficulty score.Source: User message · p. 38, System prompt · p. 38

D.4.1 QUALITATIVE EXAMPLES

  • Representative cases show object misidentification, inverted movement direction, incorrect event timing, and overriding observed hand use with an expectation.The examples depict failure to locate a remote control, misreading clothes being thrown onto a sofa, placing a plate grab before a cabinet door closes, and selecting the manipulating hand despite observing both hands act.Source: Figure 31 · p. 39, Figure 32 · p. 39, Figure 33 · p. 40

D.4.2 RIGHT-HAND PRIOR IN LATERALITY QUESTIONS

  • All but Gemini 2.5 Pro and EgoThinker-7B are more accurate on right-hand than left-hand questions, with left-hand accuracy ranging from 9.2% to 80.6%.The asymmetry tends to be smaller for the strongest models, but does not disappear entirely; most of the spread in accuracy comes from left-hand performance.Source: §D.4.2 · p. 41

D.4.3 COUNTING ERROR ASYMMETRY PER MODEL

Models show a predominantly under-counting bias on repetition-counting errors, but its direction and strength vary substantially by model.

D.4.3 COUNTING ERROR ASYMMETRY PER MODEL

  • Qwen3-VL-8B under-counts on 89.7% of its incorrect predictions, while Gemini 2.5 Flash over-counts on 84.4%.The error-direction bias is highly model-dependent beyond the shared sampling bottleneck discussed in Sec. 5.2.Source: §D.4.3 · p. 41
  • 24 of 37 models lean toward under-counting, while 12 lean toward over-counting among incorrect repetition-counting predictions.Shares are computed relative to each model’s total incorrect predictions on this task.Source: Figure 36 · p. 41, §D.4.3 · p. 41

D.5 EVALUATION OF MOTION-ONLY BASELINES … E.1.2 DEBIASING VALIDATION

Motion-only baselines perform near chance, while SOMA-Train pairs egocentric video and body motion with bias controls; text-only transfer remains near random, supporting gains from perceptual input.

D.5 EVALUATION OF MOTION-ONLY BASELINES

  • 24.9%–26.8% average accuracy across five motion-only baselines was close to the 25% random baseline.Four models were scored by ranking options under their likelihoods, while Motion-SMD decoded autoregressively; their motion inputs were also out of distribution relative to SOMA.Source: §D.5 · p. 41, Table 6 · p. 42

E SOMA-TRAIN

  • The SOMA-Train work covers dataset construction, reconstruction of Ego-Exo4D body motion, multimodal baseline architecture and training, and additional results.The dataset is designed to investigate whether body-motion supervision helps close gaps exposed by SOMA.Source: E SOMA-TRAIN · p. 41

E.1 QA GENERATION

  • SOMA-Train contains 28,596 questions from 1,779 takes, covering 110.2 hours of aligned egocentric video and body motion.The takes include 570 from Nymeria and 1,209 from Ego-Exo4D; the dataset follows the benchmark taxonomy and draws on those same sources.Source: §E.1 · p. 42

E.1.1 CONSTRUCTION

  • SOMA-Train questions use randomly selected task-specific templates and reproducible regex-based extractors instead of the benchmark’s LLM selector.The extractors use task-specific rejection rules to handle identified failure cases and reduce sampling biases from LLM-based selection.Source: §E.1.1 · p. 42, Table 7 · p. 43
  • Task-specific distractor strategies balance answer options using fixed sets, product combinations, within-context rotations, exchangeable options, or adversarial filtering.Adversarial filtering is used for Chronological Ordering, where balance cannot be guaranteed by construction.Source: §E.1.1 · p. 44
  • Generated questions were checked for grammatical and wording artifacts that could create shortcuts or change answer meaning, then corrected or removed when inconsistent with human annotations.Gemini 3.8 Flash was used to preserve semantic content while avoiding new answer-specific biases and to verify questions against the original annotations.Source: §E.1.1 · p. 44

E.1.2 DEBIASING VALIDATION

  • A text-only model stayed near the random baseline on SOMA, while the video+motion variant outperformed it by +16.65 points.This supports the finding that fine-tuning gains rely on perceptual input rather than language understanding and training-distribution shortcuts alone.Source: §E.1.2 · p. 44, Table 8 · p. 44

E.2 EGO-EXO4D REFITTING · E.3 TRAINING DETAILS

The authors refit Ego-Exo4D to provide denser SMPL-X motion aligned with Nymeria’s representation and frame rate, then train video-only, motion-only, and video-plus-motion models using staged motion-language supervision.

E.2 EGO-EXO4D REFITTING

  • The refitting pipeline replaced single-view pose estimation with SAM 3D Body, then used calibrated multi-view triangulation and joint SMPL-X fitting across views.It selected the camera wearer by matching detections to the projected Aria trajectory, required at least two valid views, and optimized reprojection, per-view prediction agreement, temporal smoothness, and pose and shape regularization. Rock Climbing was excluded; the pipeline fitted 3,475 of the remaining 3,634 takes, while 159 lacked source data.Source: §E.2 · p. 45
  • 36.7 mm PA-MPJPE on validation for the new fits beat EE4D-Motion’s 40.3 mm, after both methods received per-joint convention corrections.Without correction, the respective errors were 42.6 and 44.2 mm; the evaluation used the intersection of takes and take-averaged PA-MPJPE.Source: §E.2 · p. 45, Table 9 · p. 46
  • 93.1% of training video and 96.2% of validation carried fitted motion, versus 70.5% and 73.1% for EE4D-Motion.Coverage was measured on the original 30 fps video timeline, accounting for each motion source’s native sampling rate.Source: §E.2 · p. 45, Table 10 · p. 46
  • At 20 fps, fitted motion covered 96.6% of each SOMA segment versus 37.9% for EE4D-Motion, and 96.3% of SOMA-Train segments versus 48.1%.Interpolating EE4D-Motion to 20 fps still left the new pipeline with a 20.8-point mean SOMA coverage advantage; SOMA-Train coverage was similar because its questions use takes EE4D-Motion already covered.Source: §E.2 · p. 46, Figure 37 · p. 46

E.3 TRAINING DETAILS

  • The motion pathway encodes 263-dimensional body-motion features into continuous tokens projected into the language-model embedding space.The frozen VQ-VAE convolutional encoder was trained on 156,659 clips as 64-frame windows at 20 fps, producing one pre-quantization feature per four frames.Source: §E.3 · p. 46
  • Video-only, motion-only, and video-plus-motion variants were compared; motion variants first aligned a projector on motion narrations, then fine-tuned on SOMA-Train.Stage 1 used 24,549 body-part-description samples from 8,183 clips, trained the projector alone for one epoch, and kept the language model frozen; video-only skipped this stage.Source: §E.3 · p. 47
  • All variants received two epochs of SOMA-Train question-answering fine-tuning with LoRA on the language model.For motion variants, the Stage 1 projector was initialized and frozen; training used rank 128, α = 256, dropout 0.05, and eight H100 GPUs.Source: §E.3 · p. 47

E.4 ADDITIONAL RESULTS · F LICENSING INFORMATION

Motion input helps selectively: its value varies by task, while both backbones rely more on video overall. The paper also reports that artifacts follow their licenses and benchmark questions will be released for non-commercial research use.

E.4 ADDITIONAL RESULTS

  • With separately fine-tuned motion input, both backbones improve on Body Posture, Opposite Motion Discrimination, Missing Event Inference, and Next Action Prediction, but worsen on Hand Use and Adjacent Action Identification.The reported pattern suggests motion helps when body state or movement is difficult to recover from video, but may add redundant or noisy signals when relevant evidence is already visible.Source: §E.4 · p. 47, Table 11 · p. 47
  • Removing motion lowers Body Posture accuracy from 51.8 to 40.9 for Qwen3-VL-8B and from 50.0 to 33.6 for InternVL3.5-8B.These are test-time ablations of the same Video+Mot. checkpoint.Source: §E.4 · pp. 47–48
  • Motion increases Hand Use accuracy by 3.3 and 9.8 points, respectively; it helps when body state is hard to recover visually but may hurt when evidence is already visible.Source: §E.4 · p. 48
  • Removing video lowers average accuracy by 13.3 points for Qwen3-VL-8B and 14.0 for InternVL3.5-8B, while removing motion changes it by −1.4 and +0.8 points, respectively.All comparisons use the same jointly fine-tuned Video+Mot. checkpoint at test time.Source: §E.4 · p. 47, Table 12 · p. 48

F LICENSING INFORMATION

  • Third-party artifacts are used under their respective licenses and for research purposes consistent with their intended use.Table 13 inventories the third-party artifacts and their licenses.Source: F LICENSING INFORMATION · p. 48, Table 13 · p. 48
  • SOMA benchmark questions will be released for non-commercial research use, consistent with the source data licenses.Source: F LICENSING INFORMATION · p. 48

Related papers

Loading 2610.36384v1…