PaperlayerView exact revision on arXiv

Source-linked AI summary

BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation

Zexi Zhang, Zecheng Zhu, Zidong Chen, Zulkhuu Tuya, Stephen James

arXiv:2610.07594v1cs.ROcs.LG

TL;DR

Humanoid household manipulation must coordinate arm actions with balance, stepping, and posture, while existing evidence does not combine native whole-body demonstrations and shared evaluation of learned and agent-developed policies. BiGym 2.0 adapts a 20-task G1 benchmark and compares these approaches under common observations and whole-body control. VLA fine-tuning has the highest nine-task mean, while coding-agent programs lead on bimanual reaching; several transport and stacking challenges remain difficult or open.

  • Problem

    Household manipulation couples arm actions to walking and balance, and existing benchmarks do not combine native whole-body demonstrations with common evaluation of learned and agent-developed policies.

    Source: §I · p. 1, A. Loco-manipulation challenges and… · p. 3

  • Method

    BiGym 2.0 adapts 20 household tasks to G1 and compares VLA fine-tuning, imitation learning, demo-driven reinforcement learning, and cold-start coding agents under matched interfaces.

    Source: §I · p. 2

  • Results

    VLA fine-tuning attains the highest nine-task mean, while agent-developed programs lead on bimanual reaching; cross-workspace stacking remains open and multi-object transport remains difficult for several policy families.

    Source: §I · p. 2, §VII · p. 7

  • Takeaways & Limitations

    The benchmark provides a cold-start reference for agent policy development, while cumulative reuse of verified programs across tasks remains untested.

    Source: §I · p. 2, §VI · p. 7

    Evaluations use simulation with one frozen whole-body controller per robot, so performance is characterized for that policy-controller stack rather than across alternative controllers.

    Source: §VI · p. 7

Abstract

from arXiv · show

Humanoid household manipulation requires the arms to act while the body balances, steps and changes posture. We present BiGym 2.0, an adaptation of BiGym for the Unitree G1 across 20 household tasks using a unified whole-body controller for demonstration and evaluation. The suite provides 60 native human virtual-reality demonstrations per task with synchronised multi-camera views and full-body execution records. We benchmark vision-language-action fine-tuning, imitation learning, demo-driven reinforcement learning, and cold-start coding agents given the interaction budget of online reinforcement learning. With the same onboard views, proprioception and whole-body controller for every method, vision-language-action fine-tuning has the highest nine-task mean, and agent-developed programs outperform every demo-driven reinforcement learning baseline on this mean and lead on bimanual reaching. Cross-workspace stacking remains open, $\pi_{0.5}$ stays low on pick-box, and multi-object transport is hard for imitation learning, demo-driven reinforcement learning and coding agents. All environments, human demonstrations, and evaluation traces are open-sourced at https://github.com/swirl-uk/BiGym2.

I. INTRODUCTION

Humanoid household manipulation couples arm actions to balance, stepping, and posture, but existing benchmarks do not combine native whole-body demonstrations with shared evaluation of learned and agent-developed policies. BiGym 2.0 adapts BiGym to G1 and compares these routes under common execution constraints.

I. INTRODUCTION

  • No current benchmark combines household tasks, native human demonstrations collected through active whole-body control, deterministic controller-state restoration, and common evaluation of learned and agent-developed policies.The shared execution stack matters because walking gaits lag, oscillate, and perturb the torso, unlike a wheeled base that executes velocity commands directly.Source: §I · p. 2
  • Coding agents receive three onboard 84×84 views and proprioception, develop programs from one demonstration without calibration or inverse-kinematics tools, and serve as a cold-start baseline for later skill accumulation.The setting matches learned policies' per-task training scope; it does not test reusing verified programs across tasks.Source: §I · p. 2
  • BiGym 2.0 adapts 20 household tasks to G1, uses frozen GR00T-WBC policies, and collects 60 native human VR demonstrations per task through the evaluation controller.It compares VLA fine-tuning, imitation learning, and demo-driven RL on nine tasks, and audits coding agents under matched observations and actions.Source: §I · p. 2

II. RELATED WORK

  • BiGym 2.0 focuses on G1 household contact and transport tasks with native demonstrations and comparisons across VLA fine-tuning, imitation learning, demo-driven RL, and agent-developed programs.Related work includes humanoid environments, teleoperation, humanoid data generation, and code-policy development; this benchmark combines those policy comparisons in household loco-manipulation.Source: §II · p. 3

III. BIGYM 2.0

BiGym 2.0 organizes 20 G1 tasks around gait-perturbed interaction, multi-object transport, and cross-paradigm reliability. Its scenes cover reaching, tabletop manipulation, dishwasher tasks, and kitchen-counter tasks.

A. Loco-manipulation challenges and tasks

  • The benchmark targets gait-perturbed interaction, coordinated multi-object transport, and reliability comparisons between learned policies and agent-written programs under shared observations, actions, and controller.The physical challenges include footfall shock and base lag, plus spatial tracking across changing viewpoints during transport.Source: A. Loco-manipulation challenges and… · p. 3
  • Reaching tasks distinguish single-hand selection from simultaneous bimanual coordination, while tabletop tasks cover plate transport, object reorientation, and cross-workspace block stacking.The suite has 20 tasks across four scene families.Source: A. Loco-manipulation challenges and… · p. 3
  • Dishwasher tasks combine door and rack handling with constrained placement of cups, cutlery, or plates.Source: A. Loco-manipulation challenges and… · p. 3
  • Kitchen-counter tasks include fixture manipulation, box pickup, cabinet storage, saucepan transfer, and spatula-based sandwich transfer.Source: A. Loco-manipulation challenges and… · p. 3
  • Task layouts adapt BiGym contact geometry to G1, and success predicates require specified scene states to persist for one second.Examples include released plate contact at a target pose and a released three-block contact chain satisfying height and planar-alignment constraints.Source: A. Loco-manipulation challenges and… · pp. 3–4

B. Whole-body execution

A frozen whole-body controller executes G1 body commands while maintaining balance during locomotion and posture changes. The benchmark records VR demonstrations with synchronized observations and execution data across near-stationary manipulation and cross-workspace transport.

B. Whole-body execution

  • Frozen GR00T-WBC policies convert high-level body commands into lower-body joint targets while maintaining balance during locomotion, height changes, and torso pitch.The 29-DoF G1 uses body-motion commands alongside arm targets and gripper commands; the interface runs at 50 Hz.Source: B. Whole-body execution · p. 4

C. Human demonstrations

  • VR operators provide hand targets through inverse kinematics and body commands through thumbsticks, while the controller maintains balance throughout demonstration collection.Collection requires success to hold for 3 seconds, with the first second retained in the training view to match evaluation.Source: C. Human demonstrations · p. 4
  • Each episode stores synchronized RGB, proprioception, normalized actions, rewards, terminal indicators, and simulator and controller snapshots for replay and re-rendering.Source: C. Human demonstrations · p. 4
  • The suite provides 60 successful demonstrations for each of 20 tasks, spanning near-stationary manipulation and cross-workspace transport with task-wise median planar base travel from 0.06 to 8.10 m.Demonstrations also include working-height changes and torso posture changes; G1 pitches forward during plate loading.Source: C. Human demonstrations · p. 4, Figure 4 · p. 4

D. Observations and evaluation

Policies share onboard visual and proprioceptive inputs and the same whole-body controller, while evaluation uses restored controller state and a defined checkpoint protocol. The benchmark compares VLA fine-tuning, imitation learning, demo-driven reinforcement learning, and cold-start coding agents.

D. Observations and evaluation

  • Learned policies and strict-interface coding agents receive the same three 84×84 onboard views and proprioception and issue commands to the same controller, without camera calibration or inverse-kinematics tools.A separate tool-augmented agent interface adds those tools.Source: D. Observations and evaluation · p. 4
  • Restoring controller state makes seeded reset trajectories bit-identical on the released substrate build; without restoration, used-environment resets differ from fresh ones.The main comparison averages each run's last five checkpoints and reports mean and standard error across three independent runs.Source: D. Observations and evaluation · pp. 4–5

E. Policy families

  • π0.5 is fine-tuned on converted demonstrations and task instructions, while ACT and Diffusion Policy learn action sequences by imitation.π0.5 uses relative-delta actions and executes 16 actions per prediction; ACT and DP each train for 101k updates.Source: E. Policy families · p. 5
  • CQN-AS and DrQ-v2+ learn online with demonstrations, whereas DEAS learns offline from a fixed dataset using action sequences.Online methods use 101k environment interactions; DEAS uses 101k updates, so their budgets are counted in different units.Source: E. Policy families · p. 5
  • GPT-6 Astra and Claude Opus 5.5 develop task-specific Python policies from a cold start, with up to 101,000 environment steps and three sessions per task.Programs run without LLM calls on unseen evaluation seeds, and ten episodes are repeated to check deterministic outcomes.Source: E. Policy families · p. 5

IV. CAPABILITY AUDIT

Under the strict interface, π0.5 has the highest nine-task mean, while coding agents outperform every demo-driven RL method on that mean. The task-level results distinguish the methods, checkpoint selection changes some rankings, and cross-workspace stacking remains a challenge.

IV. CAPABILITY AUDIT

  • Under the strict interface, π0.5 has the highest nine-task mean, while both coding agents beat every demo-driven RL mean.π0.5 leads plate and cup transport; coding agents lead on dual reaching, and cross-workspace stacking was not evaluated for π0.5.Source: §IV · p. 5

A. Dual reaching and transport separate the methods

  • Both coding agents stay above every learned policy on dual reaching, while adding a second plate lowers success for ACT, DP, CQN-AS and DEAS, but not π0.5.Every non-zero learned policy drops sharply from multi-modal reaching, where either hand suffices, to reaching with both hands; agents succeed on few plate-transport or cup-loading episodes.Source: A. Dual reaching and transport… · p. 5

B. Checkpoint selection changes the ranking

  • On single-plate transport, ACT has the highest mean peak checkpoint among three-run methods, but DP has the highest last-five mean; π0.5’s last-five score exceeds DP’s.Selecting the best of 20 checkpoints, each estimated from 100 episodes, takes a maximum over noisy estimates.Source: B. Checkpoint selection changes the… · p. 5

C. The wider suite presents further challenges

  • ACT or CQN-AS reaches at least 10% on each of the remaining eleven tasks except cross-workspace stacking, which neither solves in more than 1% of episodes.The paper examines program policies and development budget for stacking in Section V.Source: C. The wider suite presents further… · p. 5

V. CODING-AGENT POLICIES

  • The coding-agent section examines how programs behave and what classical robotics tools add, using the strict-interface protocol reported in Table II.The section reports both agents under the protocol established for the capability audit.Source: §V · p. 5

A. Harnesses and interfaces

The harnesses constrain coding agents to a strict interface, while a separate tool-augmented evaluation tests the contribution of classical robotics tools. Under the strict interface, agents outperform demo-driven RL on the nine-task mean, but their task outcomes vary across development sessions.

A. Harnesses and interfaces

  • The strict interface withholds tools rather than model prior knowledge: Opus 5.5 solved IK numerically in 24 of 27 programs and recalled G1 link offsets in 22, whereas Astra did neither.The agents had similar nine-task means, so recalled kinematics did not remove their contact-rich-task failures.Source: A. Harnesses and interfaces · p. 5
  • Coding-agent development starts from one human demonstration and onboard camera access; frozen programs are evaluated on 100 hidden seeds, with ten episodes repeated to check determinism.The development process uses the strict interface within a shared interaction budget.Source: Figure 5 · p. 6
  • Astra’s tool-augmented evaluation used a separate pre-release build and sessions developing on seeds 0–199; its strict sessions cost more than tool-augmented ones in that comparison.The cost pattern is consistent with more trial and error without kinematics solvers; the main comparison reports results from the strict interface.Source: A. Harnesses and interfaces · p. 6, Table IV · p. 7

B. Visual servoing under strict constraints, but contact vulnerability

  • Under the strict interface, both coding agents outperform demo-driven RL methods on the nine-task mean.Astra uses pixel-space visual servoing for reaching and an open-loop joint sequence for drawer closing, exploiting fixed contact geometry.Source: B. Visual servoing under strict… · p. 6
  • Both agents fail most plate-transport and cup-loading episodes, and outcomes vary widely across sessions, with tasks often solved in one session and failed in others.Thus, a three-session mean reflects how often development finds a working strategy.Source: B. Visual servoing under strict… · pp. 6–7

C. Impact of robotics tooling and cross-workspace stacking

Agent-tooling gains are uneven, and cross-workspace stacking remains difficult. The paper presents cold-start agent results as a reference, while noting that evaluation uses simulation with a frozen whole-body controller.

C. Impact of robotics tooling and cross-workspace stacking

  • The full robotics toolchain raises the nine-task mean most on drawer opening, dishwasher closing and cup loading, but barely changes reaching or drawer closing.The passage attributes the small changes on reaching and drawer closing to pixel-space servoing already sufficing for open-space alignment.Source: C. Impact of robotics tooling and… · p. 7
  • Under the tool-augmented interface, three block-stacking sessions achieve 19%, 0% and 0%, with only eleven episodes completing under the interaction budget.The strongest program stores world-frame block and pad coordinates and navigates using proprioceptive base pose, but grasp transitions remain time-indexed.Source: C. Impact of robotics tooling and… · p. 7

VI. DISCUSSION AND LIMITATIONS

  • Cold-start agent results are a reference rather than a ceiling because each task starts fresh, receives one demonstration, and does not reuse skills across tasks.Whether verified programs improve humanoid cross-task generalisation remains untested here.Source: §VI · p. 7
  • Evaluation uses simulation with one frozen whole-body controller per robot, so measured performance characterises the joint policy–controller stack.The authors note that alternative controllers would help assess controller-specific dynamics.Source: §VI · p. 7

VII. CONCLUSION

  • Vision-language-action fine-tuning has the highest nine-task mean; agent programs beat every demo-driven RL mean and lead on bimanual reaching.π0.5 leads plate and cup transport, while cross-workspace stacking remains open.Source: §VII · p. 7

APPENDIX A TASKS AND SUCCESS PREDICATES

The appendix specifies the benchmark’s task conditions, G1 embodiment and control interface, observation and demonstration setup, and simulation throughput. Its demonstrations span near-stationary manipulation through long cross-workspace transport, with substantial posture variation.

APPENDIX A TASKS AND SUCCESS PREDICATES

  • The 20-task benchmark reports action dimensions, episode budgets and success predicates; predicates must hold for 1 s, and selected placement tasks fail if an object touches the floor.Seventeen tasks randomise scene elements, while dishwasher closing and wall-cupboard tasks use fixed states.Source: Appendix A · p. 9, Table A1 · p. 9

A. Embodiment Specifications and Whole-Body Execution

  • The simulated G1 has 29 actuated degrees of freedom, while two passive pelvis hinges permit roll and pitch during demonstrations.With the hinges, a lateral command of 0.25 m/s yields +0.1725 m/s lateral and +0.014 m/s unintended forward velocity, versus +0.164 m/s lateral and +0.059 m/s forward with tilt locked.Source: A. Embodiment Specifications and… · pp. 9–10

B. Action Representation and Controller Contract

  • The action interface uses four initial body-command dimensions, absolute arm-joint targets and gripper commands; torso-pitch tasks add a pitch-reference dimension.Most policies predict absolute arm targets, whereas π0.5 predicts relative arm actions and keeps base and gripper channels absolute.Source: B. Action Representation and… · p. 10

C. Proprioceptive State and Multi-Camera Observations

D. Simulation Throughput and Training Compute

A. VR Teleoperation

  • VR demonstrations use tracked hand poses for arm targets and thumbsticks for base motion, yaw and height; the same frozen GR00T-WBC executes body commands during collection and evaluation.Demonstrations therefore include controller tracking lag and gait-induced torso motion.Source: A. VR Teleoperation · p. 12

B. Demonstration Motion Statistics

  • Demonstrations range from 0.06 m median planar base travel for wall-cupboard opening to 8.10 m for block stacking, while every task reaches 6.4°–27.9° peak pelvis lean.Only dishwasher plate loading commands torso pitch; 11 tasks never change the height command.Source: B. Demonstration Motion Statistics · pp. 12–13, Figure A3 · p. 13

C. Recording Format and Replay

The appendix documents demonstration storage and replay, baseline recipes, evaluation choices, and a gait-related grip-slip correction.

C. Recording Format and Replay

  • The release stores 1,200 successful episodes as compressed NumPy archives with JSON metadata, per-step observations and actions, full simulator states, controller signals, seeds, and engage snapshots.Each task has 60 demonstrations; the training view retains the first second of a three-second success hold.Source: C. Recording Format and Replay · p. 13
  • Snapshot replay is step-identical, whereas seed-only reset begins with 10^-6 rad joint-angle differences and ends with plate positions at most 0.73 mm apart; all 40 replays succeed.The figure compares the largest deviation from the first replay across 29 actuated joints and plate position.Source: C. Recording Format and Replay · p. 13, Figure A4 · p. 14

A. Imitation Learning: ACT and Diffusion Policy

  • ACT predicts 32 actions and replans each step with temporal ensembling, while Diffusion Policy predicts 16 and executes eight before replanning.Both use ResNet-18 image encoders and were trained for 101k updates; ACT uses a transformer with a conditional VAE, while DP uses a temporal U-Net.Source: A. Imitation Learning: ACT and… · p. 14

B. Demo-Driven Reinforcement Learning: CQN-AS, DrQ-v2+ and DEAS

  • CQN-AS is a critic-only action-sequence method, DrQ-v2+ is a single-step actor–critic, and DEAS learns offline from demonstrations.CQN-AS and DrQ-v2+ sample 256 online and 256 demonstration transitions per update; DEAS predicts 16 actions and executes the full chunk.Source: B. Demo-Driven Reinforcement Learning:… · p. 14, Table A4 · p. 15

C. Vision-Language-Action (VLA) Fine-Tuning: π0.5

D. Shared Training Augmentation

  • Non-VLA baselines share random image shifts and, with probability 0.2, zeroing of non-base proprioception during training; neither augmentation is used at evaluation.Images shift by up to eight pixels with edge-replication padding, and pelvis entries are retained.Source: D. Shared Training Augmentation · p. 15

APPENDIX E EVALUATION PROTOCOL AND CHECKPOINT SELECTION

  • Main-table scores use the last-five checkpoint mean rather than the best checkpoint, avoiding selection on evaluation seeds and upward bias from maximizing noisy estimates.For non-VLA runs, checkpoints are evaluated on 100 episodes and the last-five mean is aggregated across three runs; π0.5 pools 50 episodes at each of its last five checkpoints from one run.Source: Appendix E · p. 16, Table A7 · p. 16

A. Gait-Induced Grip Slip

  • With gait disturbance amplified threefold, increasing pad torsional friction from 0.01 to 0.05 reduced six-second modeled slip from 0.745 mm to 0.002 mm; matching plate contact parameters removed the remainder.The change addressed grip rotation during walking-induced foot-contact impacts; no weld or attachment constraint was used.Source: A. Gait-Induced Grip Slip · pp. 16–17

B. Workspace Departure of DrQ-v2+

DrQ-v2+ failed broadly on the main tasks, and the accompanying coding-agent protocol used a bounded cold-start setup.

B. Workspace Departure of DrQ-v2+

  • DrQ-v2+ scores 0% on eight of the nine main tasks; in 20 reach-target episodes, it walked away from the workspace and timed out every time.Pelvis displacement ranged from 1.38 to 7.13 m, with a mean of 4.20 m; the rollouts do not isolate lack of action chunking as the cause.Source: B. Workspace Departure of DrQ-v2+ · p. 17

A. Cold-Start Development Protocol and Sandbox

  • Coding-agent sessions received a task prompt, environment API documentation, and one human demonstration as three synchronized 84 × 84 video streams, with 101,000 environment steps budgeted per session.The prompt forbade learned components, but that rule was not automatically enforced.Source: A. Cold-Start Development Protocol and… · p. 17

B. Strict Interface versus Robotics Tooling

The comparison contrasts a restricted, learned-policy-matched coding-agent interface with a tool-augmented reference and reports session-level costs and stacking outcomes.

B. Strict Interface versus Robotics Tooling

  • Under the strict interface, agents used three 84 × 84 views and 50- or 56-dimensional proprioception, without camera calibration, depth, inverse kinematics, or rate limiting.Each interface had 27 development sessions: three independent sessions for each of nine tasks.Source: B. Strict Interface versus Robotics… · p. 17
  • The tool-augmented interface additionally supplied higher-resolution images, camera calibration, pixel-to-ray projection, world-frame positions, differential inverse kinematics, and command rate limiting.Agents developed on seeds 0–199 under this reference interface.Source: B. Strict Interface versus Robotics… · p. 17
  • On block stacking, ACT reached 0.13% and tool-augmented Astra 6.3% across three sessions, with Astra’s best session at 19%.The most expensive single strict-interface session cost $31.23; 27 tool-augmented sessions used 161.5 million input tokens and cost $235.07.Source: Table A8 · p. 17, B. Strict Interface versus Robotics… · p. 17

C. Recalled Kinematics under the Strict Interface

Under the strict interface, Opus reconstructed arm kinematics from recalled robot dimensions, while Astra used a simpler visual-control route; outcomes varied across tasks and sessions.

C. Recalled Kinematics under the Strict Interface

  • In 22 of 27 Opus scored programs, its arm-chain offsets matched the original public G1 description to five significant figures, although the simulator used a later revision.All seven offsets matched that original description; 24 programs solved inverse kinematics numerically, while no Astra program used the recalled values or solved inverse kinematics.Source: C. Recalled Kinematics under the… · p. 18
  • On multi-modal reaching, Astra thresholded target color and moved shoulders by pixel offset, while Opus reconstructed a camera and arm model and used numerical inverse kinematics; both solved all 100 hidden seeds.Opus lifted the target to 3D, walked to it, and reached through forward kinematics using recalled link offsets.Source: C. Recalled Kinematics under the… · p. 18
  • Opus 5.5 leads on multi-modal reaching and pick-box, while Astra does better on dual reaching and dishwasher closing; both agents fail most plate-transport and cup-loading episodes.The three-session means reflect how often a session finds a working strategy, as task outcomes vary substantially across sessions.Source: C. Recalled Kinematics under the… · p. 18, Table A10 · p. 18

D. Agent Prompt, Task Sentences, API Document and an Example Program

This appendix specifies the coding-agent prompt and interfaces, and illustrates strict-interface policies with a multi-modal reaching program. The example uses arm inverse kinematics with a geometric forward model and bounded, multi-start least-squares search.

D. Agent Prompt, Task Sentences, API Document and an Example Program

  • The strict-interface prompt gives agents a task-specific sentence, a 101,000-step interaction budget, and hidden-seed success-rate scoring, while prohibiting learned components.Falls count as failures, and evaluation uses 100 hidden seeds.Source: D. Agent Prompt, Task Sentences, API… · pp. 18–19
  • The API pairs a frozen whole-body controller with body and arm commands; tool-augmented sessions add inverse kinematics, camera calibration, pixel-to-ray projection, and world-frame positions.For 21-dimensional tasks, the API adds torso-pitch action and waist-joint proprioception.Source: D. Agent Prompt, Task Sentences, API… · pp. 18–19
  • Agents perceive objects through the robot’s three 84×84 cameras because observations contain proprioception but no object positions; demonstrations are development-only, not available at evaluation.The strict interface matches learned policies’ camera views and body-state vector, with no camera calibration or kinematic model.Source: D. Agent Prompt, Task Sentences, API… · pp. 20–21
  • Two strict-interface multi-modal-reaching programs each scored 100/100 hidden seeds: GPT-6 Astra used color-based visual servoing, while Claude Opus 5.5 used calibrated geometry and analytic/numerical inverse kinematics.The Astra program selects a hand from the head view and controls the selected wrist view with proprioceptive feedback; the Opus excerpt reconstructs sphere position and sweeps the hand to locate contact.Source: D. Agent Prompt, Task Sentences, API… · pp. 21–22

Example Program

  • The forward model computes the tool point and wrist rotation from seven arm joints, using torso and tool offsets and a sequence of rotations and translations.The IK uses five selected joints: shoulder pitch, roll and yaw, elbow, and wrist pitch.Source: D. Agent Prompt, Task Sentences, API… · p. 23
  • Joint limits define the solver’s working ranges, with roll and yaw mirrored for the right arm and initial configurations clipped within those bounds.The solver generates multiple starts and applies the same joint limits during optimization.Source: D. Agent Prompt, Task Sentences, API… · p. 23
  • The IK solver minimizes tool-position error, approach-direction error, and a small joint-value penalty using bounded least squares.It selects the lowest-cost result among multiple starting configurations and returns the resulting joint vector and tool-position error.Source: D. Agent Prompt, Task Sentences, API… · p. 23

Related papers

Loading 2610.07594v1…