PaperlayerView exact revision on arXiv

Source-linked AI summary

LocoManiLab: A Unified Testbed for Data, Policies, Controllers, and Deployment in Humanoid Loco-Manipulation

Dewei Wang, Jinrui Han, Jiyuan Shi, Chenyun Zhang, Xianbo Lu, Jiabao Wu, Jiqin Feng, Jipeng Kong, Ping Luo, Chenjia Bai, Xuelong Li

arXiv:2610.30876v1cs.RO

TL;DR

Humanoid loco-manipulation is difficult to assess across policies, controllers, and task settings. LocoManiLab unifies tasks, demonstrations, policy and controller evaluation, and deployment studies; results show sharp success drops on unseen tasks and scenes, while simulation and hardware yield the same average-success ranking.

  • Problem

    Evaluating humanoid loco-manipulation across policy–controller pairings requires diverse humanoid-specific tasks and standardized protocols that account for execution quality and generalization.

    Source: §1 · pp. 1–2

  • Method

    LocoManiLab connects a 234-task household benchmark and multiple demonstration sources with policy training, controller evaluation, and simulation and hardware testing.

    Source: §3.1 · p. 4

  • Results

    Policy rankings largely persist on unseen tasks and scenes, but absolute success drops sharply, and simulation and hardware yield the same ranking by average task success.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The testbed supports systematic comparisons of humanoid loco-manipulation systems and the data and execution choices underlying their performance.

    Source: §6 · p. 15

    Evaluation primarily covers indoor household and office tasks, and the hardware study tests only three policies on three tasks with ten trials per policy and task.

    Source: §5 · p. 15

Abstract

from arXiv · show

Humanoid robots must coordinate locomotion and manipulation in everyday environments, yet progress is difficult to assess across differing policies, controllers, and task settings. LocoManiLab provides an integrated testbed spanning data collection and generation, policy training, evaluation, and physical deployment. Its task suite covers 234 atomic and compositional tasks across five household scene categories, supported by 51.5 hours of simulated teleoperation. The evaluation covers twelve policies with a shared controller across task and scene splits and perturbation levels, two policies across five controllers, and studies of demonstration scale and data-source mixtures. Success rates fall on unseen tasks and scenes and generally decline as visual perturbations increase, with differences in generalization and robustness across policies. For data generation, a humanoid motion generation model is fine-tuned for loco-manipulation to study the alignment of its outputs with task execution requirements and its integration into the data generation pipeline. Three simulated loco-manipulation tasks closely match real-world setups, enabling evaluation of three policies in both domains. Similar policy rankings support the reliability of simulation-based evaluation.

1 Introduction

Humanoid loco-manipulation evaluation must account for policy–controller pairings, command interfaces, training data, and performance beyond familiar tasks. LocoManiLab combines a diverse household task suite with standardized protocols to study these factors across simulation and hardware.

Motivation

  • Existing efforts address task coverage, demonstration generation, physical deployment, or controller transfer, but leave a need for unified evaluation of tasks, execution quality, policies, controllers, and demonstrations.Similar task-success rates can conceal differences in stability and effort, while familiar-task performance alone does not establish generalization.Source: §1 · pp. 1–2

Unified testbed

  • LocoManiLab provides a shared episode format and controller-specific adapters connecting data collection and generation, policy training, controller execution, and evaluation across simulation and hardware.Its workflow is summarized in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2

Policy and controller

  • The benchmark evaluates twelve policies under a shared whole-body controller and two VLA models across five controllers, assessing task completion and execution quality.The evaluation protocols distinguish familiar tasks, unseen task definitions, unseen scenes, and visual randomization.Source: §1 · pp. 2–3

Tasks and demonstrations

  • The suite covers 234 atomic and compositional tasks across five household scene categories, supported by 51.5 hours of simulated teleoperation.Motion synthesis and trajectory augmentation add demonstration sources whose utility is evaluated separately from generator adaptation.Source: §1 · p. 2

Results

  • Policy rankings largely persist on unseen tasks and scenes, but absolute success drops sharply; rankings can also shift with perturbation level and controller choice.Higher task success does not necessarily imply better motion quality, and benefits from additional demonstrations or mixed data vary by policy and task.Source: §1 · p. 3

2 Related Work

Related work spans whole-body controllers, humanoid loco-manipulation policies, and manipulation benchmarks. LocoManiLab brings policy coverage, controller-dependent execution, and demonstration choices into a shared evaluation framework.

2.1 Humanoid Whole-Body Controllers

  • Velocity commands delegate gait control to the controller, whereas reference-motion control requires denser signals; comparisons therefore need command representations alongside task success and motion quality.LocoManiLab supports policy evaluation with five whole-body controllers.Source: §2.1 · p. 3

2.2 Humanoid Whole-Body Loco-Manipulation

  • VLA and WAM approaches have advanced humanoid loco-manipulation, but comprehensive evaluation of these systems remains limited.LocoManiLab evaluates policies under shared task protocols for success, generalization, and execution quality across controller and training-data configurations.Source: §2.2 · p. 3

2.3 Benchmarks for Manipulation and Loco-Manipulation

  • Manipulation benchmarks and humanoid platforms establish task suites and evaluation protocols, while audits show rankings can depend on action representation and environment.LocoManiLab combines policy coverage, controller-dependent execution, and demonstration choices in one framework.Source: §2.3 · p. 4

3 LocoManiLab

LocoManiLab connects household tasks, diverse demonstrations, policies, controllers, and evaluation across simulation and real-world settings. Its benchmark includes atomic and compositional tasks, multiple data-generation pathways, and studies of generalization, execution quality, and deployment.

3.1 Overview

  • The benchmark connects demonstration acquisition, policy learning, controller execution, and evaluation, with studies spanning generalization, controller choice, data utility, motion generation, and simulation–real-world comparisons.Its environments combine diverse room layouts and interactive objects.Source: §3.1 · p. 4

3.2 Policies and Controllers

  • Twelve policies spanning VLAs, WAMs, and imitation learning are evaluated, with a complementary study of two VLA policies across five controllers.Adapters handle observation processing, action representations, and normalization for policy–controller pairings.Source: §3.2 · p. 4

3.3 Task Design

  • The task suite contains 234 instructed tasks across 13 scene instances: 102 atomic skills and 132 compositional tasks combining two to four primitives.Checkpoints are evaluated on seen tasks, unseen tasks in training scenes, and unseen tasks in held-out scenes.Source: §3.3 · p. 5

3.4 Dataset

  • The dataset combines simulated and real-robot teleoperation with synthesized and trajectory-augmented demonstrations.Teleoperated data includes 51.5 simulated hours across five controllers and approximately 9.0 real-robot hours.Source: §3.4 · p. 5

3.4.1 Data Collection

  • VR operators teleoperate the robot through its whole-body controller, recording commands, measured states, and egocentric RGB images.Simulated episodes end on task-specific success conditions; real-robot episodes end when the operator judges completion.Source: §3.4.1 · p. 5

3.4.2 Data Generation

  • Fine-tuned Kimodo generates controller-executed motion chunks conditioned on text, motion prefixes, and spatial targets; its adaptation uses PPO with execution-based rewards.Scene-level generation alternates motion generation and controller execution, while trajectory augmentation retains candidates that pass task-completion and physical-validity checks.Source: §3.4.2 · p. 6

3.4.3 Data Statistics

  • The simulated teleoperation corpus contains 14,160 episodes and 51.5 hours across five controllers; compositional demonstrations more often include walking, turning, and bimanual manipulation.With 50 demonstrations per task on ten shared bathroom tasks, shorter recorded durations indicate more efficient data collection; duration costs differ most for compositional tasks.Source: §3.4.3 · pp. 6–7

3.5 Randomization and Real-Robot Validation

  • Visual randomization varies appearance while retaining task definitions, complementing unseen-task and unseen-scene tests; real-world validation compares three matched tasks on a Unitree G1.Different scene geometry and success criteria limit direct comparison of absolute success rates across simulation and hardware.Source: §3.5 · p. 7

4 Experiments

The experiments assess policy generalization, robustness, controller and data choices, motion generation, and simulation-to-hardware agreement. Success declines on unseen tasks and scenes and under stronger visual perturbations, while outcomes also vary with policy, controller, and data configuration.

4 Experiments

  • The benchmark trains twelve policies on separate atomic and compositional task lists, evaluates seen and unseen task/scene splits, and measures both success and successful-episode execution quality.The benchmark covers 102 atomic and 132 compositional tasks; the main comparison uses SONIC without domain randomization, while robustness is evaluated separately across perturbation levels.Source: §4 · p. 8, §4.1.1 · p. 8, §4.1.2 · p. 8

4.2.1 Seen and unseen evaluation

  • XR-1 drops from 80.2% on seen tasks to 25.2% on unseen tasks, while unseen-scene success remains low across policies and peaks at 15.5% for DM0.5.Unseen scenes combine new task definitions with scenes excluded from training; Diffusion Policy records no successes in either category.Source: §4.2.1 · p. 9, Table 2 · p. 9

4.2.2 Domain randomization on seen tasks

  • Average success falls at each stronger perturbation level, and DM0.5 ranks first at Mid and High despite XR-1 leading both categories at Low.On atomic tasks, π0.5 declines less than XR-1 and surpasses it at High.Source: §4.2.2 · pp. 9–10, Table 3 · p. 10

4.3 Multi-Controller Evaluation

  • ALMI achieves the highest SR for GR00T N1.7 on atomic tasks, but has more falls among successful episodes than SONIC, which uses the least mechanical output power.Controller rankings reflect policy–controller pairs trained with controller-specific demonstrations and action targets; higher task success does not necessarily mean better motion quality.Source: Figure 3 · p. 10, §4.3 · p. 11

4.4 Demonstration Scale Study

  • GR00T N1.7’s mean atomic SR rises from 33.0% to 69.8% between 10 and 50 demonstrations, but gains vary across policies, controllers, and task categories.GR00T N1.7’s compositional SR levels off from 30 to 50 demonstrations, while π0.5 improves on average in both categories; its 50-to-100 extension gains on atomic tasks but slightly decreases compositional success.Source: Figure 4 · p. 11, §4.4 · p. 12

4.5.1 Data-source mixtures

  • Teleoperation is the strongest individual source for both models, while augmentation raises GR00T N1.7’s compositional SR from 56.2% to 68.8% but slightly reduces its atomic SR.No tested mixture improves π0.5 over teleoperation alone, and combining all three sources does not yield consistent improvement.Source: §4.5.1 · p. 12

4.5.2 Kimodo: pretrained, full fine-tuning, and LoRA

  • Full Kimodo fine-tuning raises overall SR from 39.7% to 90.7%, outperforms LoRA, and achieves the lowest mean error in four of five reported measures.Both fine-tuning methods improve success across the three task groups, with the largest gains on Approach and compositional Approach; under combined locomotion and manipulation, Full reduces root and palm errors.Source: §4.5.2 · p. 13, Table 5 · p. 13

4.6 Simulation and Real-Robot Evaluation

  • XR-1 has the highest average success in simulation and hardware, followed by GR00T N1.7 and Fast-WAM in both, although task-level policy ordering can differ across environments.GR00T N1.7 and Fast-WAM reverse order on simulated versus hardware cup transport; all three policies have their lowest simulation success on conveyor handling.Source: §4.6 · p. 14, Table 6 · p. 14, Figure 5 · p. 14

5 Discussion and Limitations

Unseen tasks and scenes remain challenging, and task success does not capture every difference in execution quality. Demonstration gains vary, while the evaluation remains limited in coverage and deployment conditions.

5 Discussion and Limitations

  • Similar atomic-task performance can diverge on compositional tasks, while unseen tasks and scenes leave substantial room for improvement.Controller choice also affects task completion and motion quality.Source: §5 · p. 14
  • Task success generally improves with larger demonstration budgets, but gains vary across policies, controllers, and task categories.The benefits of mixed data sources also depend on the policy and task.Source: §5 · p. 15
  • The benchmark focuses on indoor household and office tasks; hardware evaluation covers three policies on three tasks with ten trials per policy and task.The authors identify broader model, activity, and environment coverage, repeated training, and more physical trials as useful for testing how consistently trends extend.Source: §5 · p. 15

6 Conclusion

LocoManiLab unifies household tasks, demonstration sources, policy training, and evaluation in simulation and the real world. Its results show challenges in generalization and execution quality, while simulated and hardware evaluations yield the same average policy ordering on the evaluated tasks.

6 Conclusion

  • LocoManiLab connects household tasks, teleoperated and generated demonstrations, policy training, and simulation and real-world evaluation in one testbed.Source: §6 · p. 15
  • Most policies perform well on familiar tasks, but substantial room for improvement remains on unseen tasks and scenes, and controller choice affects task completion and motion quality.Source: §6 · p. 15
  • Simulation and hardware yield the same average policy ordering on the evaluated tasks, supporting simulated tasks as a guide for physical policy selection.Source: §6 · p. 15

A Task Suite and Evaluation Protocol

The protocol evaluates a broad household task suite with distinct atomic and compositional objectives, defined evaluation splits, and progressively stronger visual perturbations. Demonstrations and scene assets illustrate the tasks and environments used.

A Task Suite and Evaluation Protocol

  • The suite contains 102 atomic and 132 compositional tasks across five household scene categories, with 14,160 simulated teleoperation episodes totaling 51.5 hours.The main benchmark uses separate 50-task atomic and compositional training lists.Source: A Task Suite and Evaluation Protocol · p. 20
  • Atomic objectives can span several motion phases, whereas compositional objectives require ordered stages whose later actions depend on earlier state changes.Success is determined by task-specific physical predicates and any configured hold interval.Source: A Task Suite and Evaluation Protocol · p. 20
  • Evaluation uses a Unitree G1, head RGB observations, policy-specific proprioception, randomized episode seeds, and rollout limits based on task demonstration lengths.Episodes end on success or timeout, without fall-based early termination.Source: A Task Suite and Evaluation Protocol · p. 20

Domain Randomization Presets

  • Visual perturbations progress from texture changes and lighting variation to added material, camera, distortion, image, and distractor changes at the high level.High-level distractors appeared in 38 of 100 tasks.Source: Domain Randomization Presets · p. 20

A Task Suite and Evaluation Protocol

  • Representative demonstrations show atomic and compositional task stages, while the environment overview depicts 13 household scenes, two Lab layouts, and 32 task assets.Demonstrations use six chronological frames with nonuniform intervals highlighting task stages.Source: Figure 6 · p. 21, Figure 7 · p. 22

Main-Benchmark Configuration

The benchmark adapts diverse policies to common inputs and SONIC outputs, while controller-specific comparisons use distinct command and state interfaces. Training, normalization, and inference configurations remain model- and controller-dependent.

Main-Benchmark Configuration

  • Most policies use a 320 × 240 head RGB image and 43 joint-state values, with three additional gravity values for GR00T N1.7 and no proprioception for StarVLA-α.Adapters output 64 SONIC motion-token values and 14 absolute finger targets.Source: Main-Benchmark Configuration · p. 22
  • Training budgets correspond to approximately two nominal passes over model-specific sampling units and do not imply equal compute budgets.Normalization uses Q for percentile mapping, M for mean and standard deviation, R for min/max scaling, and I for identity; statistics are fitted on each training selection.Source: Main-Benchmark Configuration · p. 22
  • Model implementations use different adaptation and inference setups, including diffusion-policy denoising, Fast-WAM video-and-action generation, and GR00T N1.7 flow inference.The configurations also specify model-specific training updates and image preprocessing.Source: Main-Benchmark Configuration · p. 23

Controller-Specific Policy Interfaces

  • The five controller interfaces expose 36 AMO, 78 SONIC, 50 HEFT, 52 ALMI, and 50 ScaleBFM action values at 50 Hz.The controller comparison evaluates GR00T N1.7 and π0.5 trained on each controller's command layout and associated data; their state inputs differ by controller.Source: Controller-Specific Policy Interfaces · p. 24

C Metrics and Aggregation

The section defines task-level success aggregation and the conventions for reporting execution metrics. It also specifies normalization for the radar visualization and cautions against reading polygon area as an aggregate score.

C Metrics and Aggregation

  • Success rate averages task-level success rates rather than pooling episodes.For task t, the rate uses n_t successes among N_t trials.Source: C Metrics and Aggregation · p. 24
  • The DR aggregate weights 100 tasks and three levels equally; Fig. 3 SR uses all 100 episodes per profile, while other metrics use successful episodes only.Power, slip, action rate, falls, and relative time are averaged equally across successful episodes, not task-level means.Source: C Metrics and Aggregation · p. 24
  • Mechanical output energy integrates absolute motor power across 43 simulated motors, and reported power divides that energy by episode duration.Power omits electrical losses; the controller studies use a 2 ms physics step.Source: C Metrics and Aggregation · p. 24
  • Foot slip is mean horizontal foot-body speed during contact, action rate is the mean change in 29 joint targets per nominal interval, and falls flag root-height drops over 0.30 m.The action-rate interval is 20 ms.Source: C Metrics and Aggregation · pp. 24–25
  • Relative time normalizes each successful episode by the median successful duration for that policy and task, so it does not compare absolute speed between models.Its reference pool combines successful episodes across five controllers and specified demonstration settings; unsuccessful episodes are excluded from relative time, power, and energy.Source: C Metrics and Aggregation · p. 25
  • Radar metrics are scaled using each metric's minimum and maximum across 20 profiles, but polygon area is not an aggregate score.Success rate and falls use identity scaling; other metrics use log scaling, with radius reversed for metrics other than success rate.Source: C Metrics and Aggregation · p. 25

D Auxiliary Data and Generator Studies

The auxiliary studies examine demonstration scale, data-source mixtures, and motion-generator adaptation. They report no consistent benefit from extending SONIC to 100 demonstrations, while full Kimodo fine-tuning shows stronger late reward and higher selected-model task success than LoRA.

Controller and Demonstration Scale Protocol

  • The scale study uses nested 10-, 30-, and 50-demonstration subsets, with 100 available for SONIC, and increases optimization budget alongside data.It covers five atomic and five compositional bathroom tasks across three scenes, with 20 trials per task; this is not a fixed-compute comparison.Source: Controller and Demonstration Scale… · p. 25
  • At 100 demonstrations, SONIC reaches 54/18% atomic/compositional SR for GR00T N1.7 and 78/29% for π0.5, but only π0.5 atomic performance improves over 50 demonstrations.The extension provides no consistent gain across policies and task categories and is excluded from five-controller averages.Source: Controller and Demonstration Scale… · p. 25

D-B Data-Source Mixtures

  • The source-mixture study trains GR00T N1.7 and π0.5 jointly on 16 atomic and four compositional tasks in six scenes, without count matching or source-balanced sampling.Its six conditions combine teleoperation, motion synthesis, and augmentation in listed mixtures.Source: D-B Data-Source Mixtures · p. 25

Motion-Generator Adaptation

  • Kimodo generates three-second, 30 Hz motion references that frozen SONIC executes at 50 Hz; training uses task-derived and random spatial constraints.The setup uses six denoising steps, with four stochastic transitions for a clipped, group-normalized policy-gradient update.Source: Motion-Generator Adaptation · p. 25
  • Over the last 100 shown iterations, Full averages 0.908 reward versus 0.818 for LoRA, consistent with higher overall task success for the selected Full model.LoRA improves faster initially but later fluctuates around a plateau; full fine-tuning continues to improve.Source: Motion-Generator Adaptation · p. 26, Figure 9 · p. 26
  • Only successful task-completing rollouts from fully fine-tuned Kimodo models executed by frozen SONIC enter the synthesis corpus.This defines the motion-synthesis demonstrations used in source-mixture comparisons.Source: Motion-Generator Adaptation · p. 26
  • Trajectory augmentation varies timing and initial conditions while preserving motion order and hand-body synchronization, retaining 1,000 accepted episodes from 1,606 attempts.Candidates are executed through SONIC and screened for task success and physical validity.Source: Motion-Generator Adaptation · p. 26

Kimodo Evaluation Conditions

The Kimodo scene-task evaluation and goal-reaching evaluation use separate condition sets. The latter is explicitly in-distribution because its prompt and target ranges overlap training support.

Kimodo Evaluation Conditions

  • Scene-task evaluation covers 30 task-scene pairs, with ten episodes per pair and model; 13 pairs overlap the task-derived fine-tuning set.The pairs include ten Approach, five compositional Approach, and fifteen Close tasks, weighted equally in overall success.Source: Kimodo Evaluation Conditions · p. 26
  • Goal-reaching evaluation uses 198 fixed conditions per model, and its prompt and target ranges overlap training support, making it in-distribution.The conditions comprise 54 locomotion, 72 stationary manipulation, and 72 combined locomotion/reaching cases; reported means and standard deviations summarize conditions, not independent training runs.Source: Kimodo Evaluation Conditions · p. 26

E Simulation and Hardware Evaluation

The real-to-sim evaluation trains three policies separately on three simulated tasks and evaluates corresponding hardware tasks. Hardware outcomes are human-judged, and the conveyor success predicates differ between simulation and hardware.

E Simulation and Hardware Evaluation

  • The study covers fridge opening, cup transport, and conveyor-box moving, using 50 simulated teleoperation episodes per task and separately trained policy-task models.Each model receives 10k updates; the fixed budget differs from the main experiment's two-pass planning rule.Source: E Simulation and Hardware Evaluation · p. 27
  • Simulation and hardware scenes are approximate counterparts rather than geometrically paired, and hardware policies use the head-mounted ZED Mini's left RGB image.Evaluation uses one 320 × 240 head camera, identical initial states across models per seed, and no robot/object reset randomization; Fig. 10 shows the camera views.Source: E Simulation and Hardware Evaluation · p. 27, Figure 10 · p. 27
  • The real-robot teleoperation dataset contains 567 episodes totaling approximately 9.0 hours across four tasks, with 50 demonstrations per evaluated task for hardware training.Real demonstrations replace simulated data while each model follows its corresponding simulation configuration.Source: E Simulation and Hardware Evaluation · p. 27
  • Hardware success counts are GR00T N1.7 5/6/5, Fast-WAM 4/5/5, and XR-1 8/7/6 for fridge, cup, and conveyor tasks, respectively.Each model-task cell has ten trials; simulation requires all three conveyor boxes on the desk, while hardware requires picking a box from the moving belt.Source: E Simulation and Hardware Evaluation · p. 27

Related papers

Loading 2610.30876v1…