EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning

1 INTRODUCTION

• Egocentric video offers scalable, diverse human behavior, but prior hand-engineered representations inherit extraction-pipeline biases and errors, while embodiment differences limit direct action transfer.Robot demonstrations remain costly and narrow relative to human activity.Source: §1 · pp. 1–2
• EgoLAP represents what motion to produce with structured language actions and why it is appropriate with rationales grounded in contact, geometry, affordances, and intended effects.Together these targets form a physically grounded action chain-of-thought that connects to low-level actions while transferring across embodiments.Source: §1 · p. 2
• EgoLAP achieves 44.9 points more real-world performance than the strongest action representation, and motion-level reasoning contributes 44.7 points more real-world task progress than baselines.The framework co-trains on egocentric human and robot trajectories.Source: §1 · p. 2
• At inference, EgoLAP uses a lightweight action expert, with textual targets providing auxiliary supervision without autoregressive generation.The contribution statement also reports effective reinforcement-learning adaptation over language actions and motion-level reasoning.Source: §1 · p. 2

Learning Robot Policies from Egocentric Human Data

• Hand-designed interfaces require modular processing and inherit pipeline biases and errors; direct policy-learning studies have used only about 10 minutes of task-specific video, 100 human videos, or 250–350 demonstrations per task.The cited comparison notes 3,000 hours of egocentric video available in Ego4D.Source: §2 · p. 2
• LAP represents motion in natural language for zero-shot transfer among robots, and EgoLAP extends language targets to human–robot co-training with motion-level reasoning.Other direct-training approaches often retain embodiment-specific pose targets or non-linguistic action tokens that impede cross-embodiment transfer.Source: §2 · p. 2

Action Representations for VLAs

• Action tokenizers optimize compression or token structure, whereas language actions use the VLM’s pretrained semantic space; motion-level reasoning yields larger gains with language actions than with non-linguistic tokens.The comparison includes FAST, OAT, ActionCodec, and X-Tokenizer.Source: §2 · pp. 2–3

Reasoning for Visuomotor Control

• Prior reasoning work leaves unresolved which content best supports cross-embodiment transfer, motivating motion-level rationales that capture embodiment-invariant physical relations without multi-stage distillation or inference-time text generation.Figure 2 describes reasoning that relates current state to the motion needed for its intended effect.Source: §2 · p. 3, Figure 2 · p. 3

3.1 Problem Formulation

• EgoLAP maps human-hand and robot-end-effector trajectories to a shared language-based interface, with language actions and motion rationales alongside continuous control targets.Human hand poses are represented in the head-camera frame; their relative motion defines continuous action chunks.Source: §3.1 · pp. 3–4, §3 · p. 3

3.3 Language Actions

• Language actions deterministically describe one second of hand or end-effector motion in an egocentric frame, using pose changes and hand or gripper-state transitions.For bimanual samples, left and right trajectories are processed independently while retaining arm identity; deterministic, parseable targets also support GRPO post-training of the language backbone.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Model Architecture

• The vision–language backbone predicts language actions and available rationales on human and robot data, while a continuous action expert learns action chunks through flow matching.Action-loss gradients stop at the backbone boundary; at inference, only the action expert is rolled out, without decoding text.Source: §3.1 · p. 4, §3.2 · pp. 4–5, Figure 3 · p. 4

3.4 Motion-Level Reasoning

• Motion rationales connect a concrete end-effector–object relation, contact or affordance evidence, and the immediate physical effect of the labeled motion.This local, physics-focused reasoning sits below task or subtask reasoning and directly connects to low-level actions while describing motion intent shared across embodiments.Source: §3.4 · pp. 5–6

4.1 Experimental Design

• The zero-shot evaluation covers five real-world behaviors and 14 simulation tasks, with no models trained on simulation data.Real-world results report mean task progress; simulation results report mean success rate for each task group.Source: §4.1 · p. 6
• 80.1% mean progress on real-world tasks gives EgoLAP a 2.3× gain over alternative action representations; simulation success is 38.3%, 1.8× ABC-VLA [57].Figure 4 reports the zero-shot evaluation on bimanual YAM robots.Source: Figure 4 · p. 7

4.2 Language Actions Transfer Human Experience Most Effectively

• Adding human data lifts simulation success from 7.2% to 16.4% for the same language-action policy, a 2.3× gain.Neither policy had seen the simulator; the comparison holds the representation fixed.Source: §4.2 · p. 7, Figure 5 · p. 7
• Without reasoning, EgoLAP reaches 16.4% simulation success and 44.5% real-world progress, versus EgoFAST’s 1.6% and 35.2% and EgoOAT’s 2.7% and 18.0%, respectively.Source: §4.2 · p. 7

4.3 Motion-Level Reasoning and Language Actions Are Mutually Reinforcing

• Motion-level reasoning raises EgoLAP simulation success by 21.9 points; human data adds 19.9 points with reasoning versus 9.2 points without it.The reasoning gains hold across all five real-world tasks and every simulation task family; the authors describe the two components as a textual action chain-of-thought.Source: §4.3 · p. 8

4.4 Motion-Level Reasoning Outperforms Established Reasoning Formats

• 21.9 points: motion-level reasoning adds this to language actions, versus 9.1 to ECoT composite and 7.1 to FAST; ECoT alone reaches 5.3%.Stacking ECoT before motion-level reasoning reaches 14.4%, less than half the 38.3% from motion-level reasoning alone.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 Adapting EgoLAP to Downstream Tasks

• Post-training with GRPO improves simulation success in all three variants, but sampling reasoning during RL yields +12.7% versus +19.4% when sampling language actions only.The authors attribute the weaker result to out-of-distribution rationales not seen during training.Source: §4.5 · p. 9, Figure 7 · p. 9

4.6 Reasoning Aligns Skill Structure Across Embodiments

• Robot-to-human R@1 rises from 51% to 76% (+25 points), while reverse-direction R@1 rises from 42% to 63% (+21 points) with reasoning.The two-skill split improves from 69% to 92% and MRR from 61% to 80%; the embodiments remain linearly separable.Source: Table 2 · p. 9, §4.6 · p. 9

5 CONCLUSION

• EgoLAP transfers egocentric human experience through a shared language of motion, pairing language actions with physically grounded reasoning about why a motion is appropriate.Its textual targets remain training-only, adding no generation cost during control.Source: §5 · p. 9
• 80.1% real-world task progress and 38.3% simulation success were achieved on bimanual manipulation, substantially outperforming alternative action representations and reasoning formats.Motion-level reasoning also improved cross-embodiment skill retrieval by up to 25 points.Source: §5 · p. 9
• The evaluation focuses on bimanual tabletop tasks, and reinforcement learning is limited to simulation with a motion-matching proxy rather than task reward.The authors propose jointly post-training motion-level reasoning and the action expert using task-level feedback.Source: §5 · p. 10

AI Use Statement

• Large language models were used to edit and polish the manuscript.Source: AI Use Statement · p. 10
• Offline motion-level reasoning labels were generated with Gemini 3.5 Flash for DROID and Gemini 3.7 Flash for MECKA.Appendix A.5 documents the inputs and annotation procedure, while Appendix D provides prompts and representative outputs.Source: AI Use Statement · p. 10
• The authors designed and checked the labeling pipeline, reviewed AI-assisted manuscript edits, and take responsibility for the work and its AI-assisted artifacts.Source: AI Use Statement · p. 10

Reproducibility Statement

• The paper directs readers to the model, objectives, shared motion representation, and reasoning formulation, with appendices covering architecture, prompts, training, evaluation, and annotation materials.Appendix B specifies evaluation tasks and episode counts, and Appendix D includes Gemini prompts and representative annotations.Source: Reproducibility Statement · p. 10

A.1 Model Architecture and Tensor Shapes

• The model uses a shared SigLIP encoder, a 2B VLM branch for reasoning and language actions, and a 300M action expert for action-chunk flow velocity.The VLM and expert have separate parameters but share masked attention; action tokens do not attend to textual targets, and gradients through VLM keys and values are stopped.Source: Figure 8 · p. 16

A.2 Prompt and Language-Action Format

• Human hands and robot arms use the same bimanual language-action template, with left/right identifiers and open/close gripper tokens encoding hand aperture.The ordered clauses describe motion components; translation is rounded to integer centimeters, rotation to 5-degree increments, and zero-rounded components are omitted.Source: §A.2 · p. 17

A.3 Flow-Matching Objective

• The action expert learns to recover the velocity of continuous action chunks, while inference integrates its learned velocity field from noise at τ=1 to actions at τ=0.The multimodal prefix representation is stop-gradient for the continuous-action objective, insulating the VLM representation.Source: §A.3 · p. 18

A.4 Stochastic supervision

• Rationale presence and rationale-token loss are varied independently, so labeled samples train direct action prediction, rationale-conditioned prediction, or both rationale conditioning and supervision.With pctx=psup=0.5, the proportions are one half, one quarter, and one quarter, respectively; discrete action tokens are supervised in every case.Source: §A.4 · p. 18

A.5 Motion-Level Reasoning Annotation

• Motion-reasoning labels cover MECKA human data and DROID robot data, together accounting for 32% of each training batch; other datasets contribute actions without rationales.Gemini receives instructions, ordered action anchors, and target language actions; only the main motion-level rationale is used for training.Source: §A.5 · pp. 18–19
• DROID anchors use adaptive action segments and camera images, gripper metadata, segment actions, and action history; MECKA anchors use ordered recording subsequences, egocentric images, instruction and subtask context, and action history.The prompts distinguish the visible state at an anchor from the action over the following segment; MECKA maps shared action vocabulary to hands, fingers, contact, support, and release.Source: §A.5 · p. 19

A.6 Training Hyperparameters

• All policy variants were trained for 40,000 steps with shared hyperparameter settings, including variants with and without reasoning and a robot-data-only ablation.The language-action loss weight is below 1.0 because its discrete-token objective converges faster, helping prevent early backbone overfitting.Source: §A.6 · p. 19
• The shared configuration sets language-action and flow-matching loss weights to 0.8 and 1.0, respectively, with 30-step action chunks and 10 inference integration steps.It also specifies Adam settings, weight decay, gradient clipping, normalization bounds, and camera masking and dropout probabilities.Source: §A.6 · p. 19, Table 4 · p. 19
• Training images use aspect-ratio-preserving padding to 224×224, random crops and rotations up to 5 degrees, and brightness, contrast, and saturation jitter of 0.2.Source: §A.6 · pp. 19–20

A.7 Data Mixture

• The mixture contains ABC robot data (30%), EgoVerse human data (25%), additional bimanual robot data (25%), and single-arm OXE data (20%), totaling around 10k raw hours.Only MECKA (20%) and DROID (12%) carry motion-level reasoning labels.Source: §A.7 · p. 20, Figure 9 · p. 20

A.8 Baseline Implementation Details

• FAST uses a fixed released tokenizer, while OAT uses a separately trained, frozen codec with up to 16 ordered tokens per action chunk.The OAT codec is trained on chunks sampled from the same mixture used for policy pretraining.Source: §A.8 · pp. 20–21
• The composite ECoT prefix combines subtask context, visible-object boxes, and projected visual traces, with motion-level rationale excluded to isolate what/where supervision.Reasoning variants include rationale targets with probability 0.5, while action-target tokens receive loss in every case.Source: §A.8 · p. 21
• The robot-only baseline removes EgoVerse data and renormalizes the remaining robot-dataset weights, leaving architecture, hyperparameters, and compute budget unchanged.Source: §A.8 · p. 21
• All variants use the same offline held-out-split validation procedure, with a 40k-step budget except EgoLAP, trained for 30k steps because it converges faster.The selected checkpoint has the lowest offline validation loss on the held-out split.Source: §A.8 · p. 21

• The three post-training methods share reward, objective, optimizer, data pipeline, and compute budget, differing only in SFT starting checkpoint and rollout sampling.Source: §A.9 · p. 21
• The validation episodes are disjoint from RL training episodes, and each eligible ABC YAM episode contributes exactly one observation per run.Table 5 lists settings shared by all three runs.Source: §A.9 · pp. 21–22, Table 5 · p. 22
• The reward sums arm-wise translation, rotation, and gripper costs: a 2 cm single-axis error, 10° relative orientation error, or wrong gripper each costs one unit, while an exact match scores zero.Rotation uses an SO(3) residual, and gripper mismatch is a discrete penalty.Source: §A.9 · p. 22
• Each update uses 32 observations and 64 completions per observation, computes group-relative advantages, and applies the clipped objective with a non-negative k3 penalty against a frozen SFT reference.All 2,048 completions enter the gradient; only the VLM backbone, excluding the vision encoder, is updated.Source: §A.9 · p. 22
• Every 20 updates, validation evaluates 224 observations from 28 held-out ABC episodes using a language-action reward metric and continuous-action normalized MSE, and checkpoint selection roughly minimizes both.Source: §A.9 · pp. 22–23

B Evaluation Details

• The 14-task simulation suite covers picking, placement, color selection, and cloth manipulation, with 100 episodes per task per checkpoint.A complete evaluation totals 1,400 episodes.Source: B Evaluation Details · p. 23
• The five real-world tasks run on a bimanual YAM platform and are scored by the fraction of ordered task stages completed, averaged over trials.Object and container poses are randomized between trials, with identical initial configurations across policies.Source: B Evaluation Details · p. 23
• The real-world suite includes pick-and-place, pants folding, handover-and-place, and wiping; wiping appears in egocentric human data but not robot pre-training corpora.The complex pick-and-place and handover tasks include bimanual workspace constraints.Source: B Evaluation Details · p. 23

C.1 Should the Action Expert Learn from Human Actions?

• The masked variant is roughly on par for picking and placing and slightly better on cloth folding, whose dynamics are least like those in the human corpus.Source: §C.1 · p. 24
• Masking human action-expert loss reduces overall simulation success by 4.4 points and lowers language-conditioned task success from 26.0% to 10.0%.The comparison uses the matched variants at the 25k-step checkpoint.Source: Figure 10 · p. 24, §C.1 · p. 24

• EgoLAP with reasoning outperforms its no-reasoning counterpart on 13 of 14 tasks, with large gains on cluttered fruit picking, Lego placement, colored-Lego selection, and towel folding.Source: §C.2 · p. 24, Figure 11 · p. 25
• ABC-VLA is strongest on placing fruit in a bin and on a tray, while EgoLAP leads on most remaining placement tasks.Source: §C.2 · p. 24, Figure 11 · p. 25
• EgoOAT reaches 3.3% mean success with reasoning and 2.7% without; on towel folding it scores 15.6% versus 2.1%.Source: §C.2 · p. 24, Figure 11 · p. 25

C.3 Do language actions benefit more from human data?

• With motion-level reasoning, language actions rise from 18.4% to 38.3% (19.9 points), versus FAST's 1.4% to 8.7% (7.3 points), widening their advantage under human-data co-training.The gap grows from 17.0 to 29.6 points; language actions gain 12.6 points more from human data.Source: Table 6 · p. 25, §C.3 · p. 25
• Within language-action policies, human-data gains are larger with reasoning than without it (19.9 versus 9.2 points).Source: §C.3 · p. 25

D Generated Motion-Level Reasoning Examples and Prompt

• The appendix pairs sampled reasoning labels from DROID robot and MECKA human trajectories with language actions as the backbone's complete textual supervision targets.Source: D Generated Motion-Level Reasoning… · p. 26

D.1 DROID and MECKA Examples

• One example reasons from a left hand above an open bin to descending into it for a component while the right hand maintains its fixture grip.Source: §D.1 · p. 28

D.2 Annotation Prompt

• The annotation prompts require reasoning to connect visible progress and hand–object geometry to the next motion's physical purpose, without copying action magnitudes or merely paraphrasing commands.Source: §D.2 · pp. 29–30