KODA: Kinematics--Observation Data Alignment for Humanoid Learning from Human Egocentric Demonstration

1 Introduction

• Whole-body VLA training requires robot observations, whole-body states, and actions compatible with the deployment controller, but collecting target-humanoid demonstrations is costly.Egocentric datasets offer scalable images, camera poses, and sparse bilateral hand-position references.Source: §1 · p. 1
• Prior approaches either still require target-humanoid demonstrations for adaptation or provide controller-tied supervision without aligned observations for generated robot motion.KODA targets augmenting an existing controller-specific humanoid dataset with egocentric data.Source: §1 · p. 2
• KODA turns egocentric RGB demonstrations into a canonical record of simulated whole-body motion, trajectory-derived onboard views, and task instruction, then generates controller-specific state–action supervision for co-training.Switching deployment controllers regenerates the supervision while retaining the canonical motion and views.Source: Figure 1 · p. 2, §1 · p. 3
• With 50 robot demonstrations per task fixed, 100 converted human demonstrations per task increase macro-average success on human-covered, robot-absent conditions by 45.0 points for SONIC and 46.25 points for HEFT.The metric trends upward across evaluated human-data budgets under both controllers.Source: §1 · p. 3

2.1 Humanoid Loco-Manipulation

• Humanoid teleoperation, imitation, tracking, and VLA approaches provide control and learning foundations, but represent supervision in target-robot interfaces.Examples include OmniH2O, HOMIE, HumanPlus, TWIST2, OmniRetarget, SONIC, WholeBodyVLA, OpenHLM, and MotionWAM.Source: §2.1 · pp. 3–4

2.2 Egocentric Data and Cross-Embodiment Alignment

• Egocentric datasets supply scalable RGB, camera, and hand-motion supervision, and related methods combine human video with robot learning, synthesis, or viewpoint transfer.Examples include UMI, EgoDex, EgoVerse, Ψ0, EgoVLA, EMMA, EgoEngine, and Ego2Robot.Source: §2.2 · p. 4
• Unlike EgoHumanoid’s fixed-offset view alignment and HALOMI’s dedicated-controller adaptation, KODA derives onboard views from generated whole-body motion and reuses canonical records through controller-specific command generation and state recording.This supports augmenting manipulation datasets for compatible whole-body controllers.Source: §2.2 · p. 4

3 System Framework

• The system framework has three stages: generate simulator-checked whole-body motion from sparse references, synthesize aligned onboard observations, and track the motion with the selected controller to derive state–action supervision.The canonical motion and observations are reused across interfaces; controller-specific labels support co-training with matching robot demonstrations.Source: §3 · p. 4

3.1 Problem Setup

• Human episodes include egocentric RGB, observed or estimated depth, source-camera poses, bilateral 3-D hand keypoints, and task instructions; depth, poses, and hand references must share a metric coordinate system.When source depth is unavailable, KODA estimates it from RGB.Source: §3.1 · p. 4
• The canonical record contains the target humanoid’s whole-body configuration, including floating-base pose and body and hand joint positions, and serves as the shared reference for view synthesis and tracking.The record is constructed without controller-specific states or actions.Source: §3.1 · p. 4

3.2 Whole-Body Motion Generation

• Whole-Body Motion Generation produces simulator-checked canonical motion from the episode’s camera poses and hand references.The supplied passage identifies the stage’s inputs and output but does not detail its procedure.Source: §3.2 · p. 5

Contact-Aware Hand Retargeting

• Contact-aware hand refinement uses reliable hand–object contact frames to adjust wrist and finger poses while preserving demonstrated hand–object geometry.Without reliable contact, the initial retargeting is retained; contact frames are identified using proximity, depth, surface consistency, and temporal filtering.Source: Contact-Aware Hand Retargeting · p. 5
• CEM-based sampling-based command optimization searches sparse torso-and-wrist pose residuals through simulated rollouts, rejecting kinematic or physics violations and executing only each best rollout’s prefix before replanning.This receding-horizon procedure aims to improve reference tracking while remaining simulation-feasible.Source: Contact-Aware Hand Retargeting · pp. 5–6, Figure 3 · p. 6
• Temporal Mink IK refines only the arm chain while keeping contact-aware fingers and the optimized floating base, torso, and lower body fixed; replay shows comparable controller trackability after refinement.The refinement enforces joint limits, displacement and velocity bounds, and self-collision constraints.Source: Contact-Aware Hand Retargeting · pp. 6–7

3.3 Robot Onboard View Generation

• Robot-view synthesis uses the generated body–camera trajectory to reproject the scene, render the robot, composite by depth, and inpaint residual holes; the aligned images, motion, and instruction form the canonical record.Depth may be observed or estimated from RGB.Source: §3.3 · p. 7

3.4 Control Command Generation and VLA Training

• For each deployment controller, KODA optimizes commands to track the fixed canonical trajectory and records compatible state–action supervision for co-training with matching robot demonstrations.The supplied passage describes a shared rollout cost for the two controllers, combining joint-position, link-pose, reference-deformation, PD-target-change, and terminal tracking penalties.Source: §3.4 · p. 7
• SONIC and HEFT use different body-action interfaces, but each converted episode records a shared 44-dimensional state vector and retains canonical images and the instruction.The excerpt specifies 76-dimensional SONIC and 50-dimensional HEFT actions, including separately sent hand targets.Source: §3.4 · p. 8

4 Evaluation

• The evaluation tests data augmentation with SONIC and HEFT; SONIC ablations isolate observation alignment and controller-specific supervision, while EgoHumanoid and HALOMI do not directly supply compatible whole-body state–action data.Source: §4 · p. 8, §4.1 · p. 8

4.1 Experimental Setup

• The study uses four Unitree G1 tasks, 50 robot demonstrations per task, up to 100 human egocentric demonstrations, and 20 real-robot trials per task and condition.Robot-OOD tests altered conditions represented in human training clips but absent from robot demonstrations, rather than conditions unseen in all training data.Source: §4.1 · pp. 8–9

4.2 Extending Robot Demonstration Coverage

• SONIC Robot-OOD success rises from 3.75% to 48.75% (+45 percentage points), while HEFT rises from 5.00% to 51.25% (+46.25 points) with 100 converted demonstrations.Robot-OOD conditions are covered by human demonstrations but absent from robot demonstrations; macro-average success trends upward over tested human-data budgets for both controllers.Source: §4.2 · p. 9

4.3 Observation Alignment

• Full alignment reaches 48.75% Robot-OOD success versus 23.75% with the adapted fixed-offset baseline and 18.75% without alignment; Robot-ID estimates are 63.75%/66.25%/72.5% for fixed offset, motion-based reprojection, and full alignment.Motion-based reprojection reaches 37.5% Robot-OOD success.Source: §4.3 · p. 9, Table 2 · p. 10

4.4 Controller-Consistent State–Action Supervision

• Optimized command generation raises macro-average success from 48.75% to 72.5% Robot-ID and from 7.5% to 48.75% Robot-OOD, with gains on all four tasks.The comparison retains identical canonical trajectories and synthesized observations while omitting optimization only during Control Command Generation in the no-SCO variant.Source: §4.4 · p. 10, Table 3 · p. 10

4.5 Motion Fidelity and Controller Trackability

• Replay errors remain comparable: pelvis-relative non-arm/wrist positions 2.60/1.59→2.73/1.75 cm, torso orientation 0.0588→0.0579 rad, and non-arm joint RMSE 0.0750→0.0784 rad.Controller-specific SCO improves wrist/camera tracking except SONIC camera orientation (+0.0024 rad).Source: §4.5 · p. 10, Table 4 · p. 11

• KODA does not yet fully reproduce fine-grained hand–object interactions, limiting their fidelity and physical feasibility.Source: §5 · p. 11

6 Conclusion

• Reusing canonical records with controller-specific supervision improves VLA task success under SONIC and HEFT on four G1 tasks, especially in conditions covered only by human demonstrations.The conclusion also reports reduced tracking errors and co-training gains despite residual mismatch.Source: §6 · p. 11

Appendix

• Appendix A expands contact-aware retargeting, sparse command optimization, and arm refinement, while later appendices cover coordinate conventions, controllers, experiments, and remaining task comparisons.Appendix E adds complete-clip demonstration timelines.Source: Appendix · p. 15

A Whole-Body Motion Generation Details

• Motion generation combines retargeted wrist and finger tracks with torso and wrist pose references to produce realized whole-body motion and a canonical configuration sequence.The appendix states that arm refinement retains hand references and non-arm motion.Source: A Whole-Body Motion Generation Details · p. 15

A.1 Contact-aware hand retargeting

• Contact-aware hand retargeting refines wrist and finger initialization using hand–object geometry, while reliable depth-based contact evidence identifies supported frames and absent reliable contact preserves the initialization.The interaction mesh pairs corresponding human and robot-hand landmarks with shared object samples; correction weights anchor updates and penalize temporal changes.Source: §A.1 · pp. 15–16

A.2 Sparse pose-command optimization

• Sparse command optimization perturbs torso and bilateral-wrist pose references, then scores tracker-realized motion against the unchanged target in simulation rather than searching robot joint configurations directly.The search uses receding-horizon rollouts, carrying deviations forward from the resulting simulator state.Source: §A.2 · pp. 17–19

A.3 Temporal smoothing and constrained arm refinement

• The final motion refines only the arm chain to reduce residual wrist-tracking error, while preserving the optimized floating base, torso, lower body, and contact-aware finger trajectory.Joint limits, displacement and velocity bounds, self-collision constraints, and posture penalties restrict arm updates.Source: §A.3 · p. 20

B.1 Reference frames and the canonical record

• The rendered robot camera trajectory comes from the canonical configuration’s forward-kinematic torso pose, so it can differ from the source camera path when tracking constraints prevent exact realization.The canonical record retains the synthesized observations and task instruction when the deployment-controller state–action mapping changes.Source: §B.1 · p. 21

B.2 Metric back-projection and target-camera visibility

• Only unmasked source pixels with finite positive depth are projected, and target visibility uses the smallest valid target-camera depth.Projection assumes distortion-free pinhole coordinates, zero-skew intrinsics, and registered RGB and depth.Source: §B.2 · p. 21

B.3 Robot rendering, compositing, and restricted completion

• Scene and rendered-robot pixels are depth-composited; restricted completion fills only residual holes and leaves geometry, camera pose, and action labels unchanged.The completed observation is stored with the canonical configuration and instruction.Source: §B.3 · p. 22

C Controller-Specific Command and State Interfaces

• For each deployment controller, KODA optimizes a body reference against the fixed canonical trajectory, evaluates realized motion, and records state–action supervision.The appendix distinguishes search representations from the quantities exported for VLA training.Source: C Controller-Specific Command and… · p. 22

C.1 Shared Body-Reference Search

• CEM samples 350 normalized correction coordinates in a shared physical reference space, then candidates are encoded for the selected controller and evaluated in simulation.The search uses ten knots of pelvis-translation, rotation-vector, and body-joint corrections; hand trajectories are handled separately.Source: §C.1 · p. 23

C.2 Tracking Cost and Receding-Horizon Settings

• The shared tracking objective compares realized motion with the canonical target and penalizes command deformation and changes in body-joint PD targets.Control Command Generation uses an 0.80 s horizon, retains a 0.40 s prefix, and evaluates four randomized physics scenarios per candidate.Source: §C.2 · pp. 24–25

C.3 SONIC and HEFT Body-Action Labels

• SONIC exports a 64-dimensional motion token, while HEFT exports 29 joint targets and a nine-dimensional pelvis code that reconstructs pelvis poses from current and successor poses.The realized rollout, not the selected reference, supplies the resulting motion description.Source: §C.3 · pp. 25–26

C.4 Hand Targets and Recorded State

• Both action formats append 12 hand targets, and both controllers use a 44-dimensional policy state assembled from rollout body joints, projected gravity, and retargeted hand angles.The state retains estimated hand geometry even when the action requests a calibrated grasp-closure target.Source: §C.4 · p. 26

C.5 Episode Export and Observation Reuse

• Episode export associates controller-specific states and actions with retained canonical images and instructions, but image reuse follows the reference while body state follows the realized rollout.Their correspondence is approximate when tracking is imperfect; changing controllers repeats command optimization and state recording while retaining canonical motion generation and view synthesis.Source: §C.5 · p. 27

D Experimental Protocol and Metric Definitions

• The appendix locates task-success results in Tables 1–3 and motion and rollout errors in Table 4.It describes the experimental protocol and metric definitions rather than presenting those results here.Source: D Experimental Protocol and Metric… · p. 27

D.1 Demonstration Sources and Evaluation Conditions

• The platform is a Unitree G1 with BrainCo hands, and each task-controller pair has 50 robot demonstrations collected through its corresponding teleoperation pipeline.The human pool has 100 chest-mounted-iPhone demonstrations per task, with all 400 clips including phone-recorded depth.Source: §D.1 · p. 27
• Robot-OOD changes object appearance or pose beyond the robot demonstrations, while each task’s human pool contains 80 shifted and 20 robot-matched clips.The protocol tests coverage represented in human training data but absent from the robot data; it does not test conditions omitted from both sources.Source: §D.1 · p. 27

D.2 Training Budgets and Success-Rate Aggregation

• Training combines 50 fixed robot demonstrations with 0, 25, 50, 75, or 100 converted-human demonstrations per task, with independently sampled, non-nested subsets.Each configuration is trained once and evaluated in 20 real-robot trials per task under each condition; success is averaged unweighted across four tasks.Source: §D.2 · pp. 27–28

D.3 Controlled Component Comparisons

• The observation study holds data, state–action supervision, architecture, training budget, and evaluation fixed while changing human-derived observation construction.Full alignment changes arm removal and robot rendering/compositing together, so their separate effects are not identified.Source: §D.3 · p. 28, Table D.2 · p. 28
• The command-generation comparison changes both action labels and body-state features, so it evaluates state–action supervision jointly rather than isolating actions.Both variants retain the same canonical trajectories, observations, demonstrations, and policy-training settings.Source: §D.3 · p. 28

D.4 Motion References, Units, and Refinement Replay

• Table 4 separates source-reference motion-generation errors from controller-rollout errors against the fixed canonical trajectory; lower position error need not mean lower orientation error.SONIC camera-orientation error is 0.0784 rad for Direct and 0.0808 rad for SCO.Source: §D.4 · pp. 28–29
• Motion-generation aggregate errors average LAFAN and the 400 egocentric clips, but do not report separate results for the two sources.LAFAN is included as an auxiliary check on highly dynamic motion.Source: §D.4 · p. 29
• SONIC replay errors before and after Mink refinement show small changes, indicating comparable trackability on evaluated trajectories rather than a downstream policy-success effect.Replay errors are measured against the corresponding input reference, and arm-reference changes can affect realized non-arm motion.Source: §D.4 · p. 29, Table D.3 · p. 29

E Qualitative Demonstration Timelines

• Each frame retains the human view, synthesized robot view, and offline MuJoCo kinematic replay; these are illustrations, not additional real-robot trials.Timelines include the first and last frames and use timestamps from the original 50 Hz overlay video.Source: E Qualitative Demonstration Timelines · p. 30
• The box-transfer timeline follows reaching, lifting and carrying a box, placing it on a table, and withdrawing the hands across eight selected moments.The clip lasts 13.28 seconds, and frames 1–8 preserve chronological order.Source: Figure E.1 · p. 30
• The sequential-manipulation timeline shows moving a cup to a basket, a parcel to a shelf, and a bottle into a refrigerator before closing its door.Eight selected moments span 30.76 seconds; time advances left to right, then top to bottom.Source: Figure E.2 · p. 31

F Additional Task Demonstrations and Deployments

• F.1–F.3 show Robot Data in Robot-ID, converted KODA Data, and successful SONIC deployment in Robot-OOD, with four keyframes per row.Rows are independent recordings, not temporally synchronized; KODA rows retain the human view, synthesized robot view, and offline kinematic replay.Source: F Additional Task Demonstrations and… · p. 32
• In cup-to-basket, KODA converts the brown-bottle condition, while Robot Data shows the green bottle and deployment shows successful placement into the basket.Keyframes progress from grasping through transfer to placement.Source: Figure F.1 · p. 32
• In parcel-to-shelf, the purple Robot Data parcel is replaced by a white parcel in KODA and Robot-OOD examples, with keyframes through shelf placement.The shown action progresses through reaching, lifting, and transferring before placement.Source: Figure F.2 · p. 32
• In basket-to-shelf, Robot-OOD changes the basket-handle tilt by approximately 30° relative to Robot-ID; KODA supplies a corresponding converted demonstration.The deployment sequence shows grasping the handle, lifting the basket, and placing it on the shelf.Source: Figure F.3 · p. 33