HuGo: LLMs as Whole-Body Policy Code Designers for Humanoid Loco-Manipulation

I. INTRODUCTION

• Existing RL and demonstration-based approaches require reward design or suitable demonstrations, with those costs recurring for each new task.Demonstrations may require teleoperation or video extraction and, for human motions, retargeting to the robot.Source: §I · pp. 1–2
• Across five tasks and two low-level policies, HuGo substantially outperformed a hand-rewarded RL baseline and approached a demonstration-based baseline; hardware refinement raised one transferred policy from 20% to 90%.The physical-robot refinement used three depths.Source: §I · p. 2
• HuGo has an LLM generate closed-loop high-level policy code atop a frozen whole-body policy, then refine that code from rollout trajectories and selected video frames.The approach uses no task-specific training, expert demonstrations, reference motions, or reward design.Source: §I · p. 2

A. LLM-Guided Optimization

• Prior LLM-guided optimization has generated rewards, curricula, or coordination feedback, while still relying on costly task-specific RL training.HuGo instead updates policy code directly using rollout evaluations –.Source: A. LLM-Guided Optimization · p. 2

B. LLM-Generated Programmatic Robot Policies

• Unlike prior generated programs assembled from supplied skills or modules, HuGo gives the LLM only observation and command-interface descriptions and evaluates its code as a closed-loop policy at every timestep.The code must decide where to stand, where to reach, and when to transition between stages.Source: B. LLM-Generated Programmatic Robot… · p. 2

C. Humanoid Loco-Manipulation

• Humanoid loco-manipulation methods commonly obtain high-level policies through RL, demonstrations, video priors, or task-specific optimization, whereas HuGo derives them from task descriptions over an unmodified low-level policy.The alternatives carry task-specific costs such as reward design, demonstration collection and retargeting, or optimization.Source: C. Humanoid Loco-Manipulation · p. 2

III. PROBLEM FORMULATION

• A task is defined by its natural-language description, humanoid-and-task state space, randomized initial-state distribution, and binary success function.The randomized poses include the robot, objects, and goal.Source: §III · pp. 2–3
• The high-level policy receives timestep observations and outputs commands consistent with the low-level policy’s interface and coordinate frames.Observations include robot state and task-relevant object or goal information, and are numerical quantities available at each timestep rather than raw sensor data.Source: §III · p. 3
• The objective is to choose a high-level policy that maximizes expected task success over rollouts from the initial-state distribution, while the frozen low-level policy converts its commands into joint-position targets.The rollout executes the high-level policy with the frozen low-level policy from an initial state.Source: §III · pp. 2–3

IV. METHOD

• Executable policy code can run at the control rate even though the LLM cannot be queried that quickly.The generated policy may maintain internal state across timesteps.Source: §IV · p. 3
• The policy’s internal state retains task-relevant information across timesteps.For button pressing, it could track phases from approaching through aligning to pressing, or retain prior commands for smoothing.Source: §IV · p. 3
• The internal state is defined by the generated code and initialized at the beginning of each episode.Its form depends on the task logic encoded by that code.Source: §IV · p. 3

A. Zero-shot Policy Generation

• The LLM receives task, robot, observation, and command specifications, but no demonstrations, example policies, action sequences, or reference implementations.It generates a zero-shot policy that selects commands from current observations, such as deciding when to stop and how to position the end-effector.Source: A. Zero-shot Policy Generation · p. 3
• The prompt alone may not capture command interactions with the low-level policy, robot, and environment, so rollout behavior is examined for failures.Potential execution issues include tracking errors, contact dynamics, and poorly chosen transition conditions.Source: A. Zero-shot Policy Generation · p. 3

B. Rollout Evaluation and Feedback

• The system evaluates success across initial states, then selects the first unsuccessful rollout when available to expose flaws in policy logic.A rollout contains timestep-indexed observations, commands, and video frames, ending at termination.Source: B. Rollout Evaluation and Feedback · p. 3
• A rollout-specific executable analysis function produces a numerical summary of at most 200 words and task-relevant keyframe indices.Keyframes are combined with evenly spaced frames under a fixed frame budget, balancing brief events against broader rollout progression.Source: B. Rollout Evaluation and Feedback · pp. 3–4
• The LLM uses the prompt, policy code, summary, trajectory samples, and video frames to describe execution and likely causes of success or failure.This feedback comes from one rollout and does not have access to the policy's scalar success rate.Source: B. Rollout Evaluation and Feedback · p. 4

C. Iterative Policy Code Refinement

• The LLM proposes a localized code diff conditioned on the current policy and feedback, preserving working logic while targeting identified faults.The modification is not guaranteed to improve performance, so attempted changes and their evaluations inform subsequent rounds while relevant.Source: C. Iterative Policy Code Refinement · p. 4
• Each refinement depth allows up to three rounds; a candidate is accepted only if its success rate is higher, otherwise its diff, score, and feedback are added to the history.A successful candidate becomes the current policy, clears the history, and starts the next depth.Source: C. Iterative Policy Code Refinement · p. 4, D. Best-of-N Selection · p. 4
• Refinement stops at 100% success, the maximum depth, or three failed rounds in one depth, and returns the current best-performing policy.The authors note that policy logic cannot always be repaired through incremental changes.Source: C. Iterative Policy Code Refinement · p. 4

D. Best-of-N Selection

• Because stochastic generation yields different initial task logic and local refinement rarely replaces it, HuGo runs the pipeline from N independent policies and keeps the highest-success-rate one.The authors report that repeating the pipeline costs on the order of ten dollars and a few hours, without human intervention.Source: D. Best-of-N Selection · p. 4

A. Experiment Setup

• The five-task suite includes gate passing, button pressing, box pushing, and lifting, testing navigation, whole-body configuration, and object interaction.For box pushing, the robot must place the box within 10 cm of a target and hold it for one second; for lifting, it must raise a box above 1.0 m.Source: A. Experiment Setup · p. 4
• Randomized initial poses alter navigation paths, approach poses, and command timing, while indoor walls constrain routes and require obstacle-aware navigation.Gate tasks emphasize navigation and body configuration; the other tasks coordinate navigation with precise object interaction.Source: A. Experiment Setup · p. 4, Figure 2 · p. 5
• HuGo is evaluated with two frozen low-level policies that accept different command specifications; SONIC is excluded from low-gate passing because it cannot change height while walking.The policies also use different frames for end-effector targets and different definitions of torso or pelvis height.Source: A. Experiment Setup · pp. 4–5
• The success-rate tables report best-of-10 results, with the five-task suite evaluated on 100 held-out initial states shared across methods and excluded from refinement and selection.The separate HDMI table covers two tasks and reports HuGo with best-of-10 selection; the supplied passages do not include table cell values.Source: Table I · p. 5, Table II · p. 5
• The high-level RL baseline uses PPO for 384M environment steps with hand-designed dense rewards, while HDMI is evaluated on two tasks using human-video-derived reference motions.The HDMI tasks are pushing a door by hand and carrying and placing a bread box; their initial humanoid-to-object poses are fixed.Source: A. Experiment Setup · p. 5

• Across a five-task suite, HuGo with rl isaac achieves near or above 80% success on four tasks, including near-perfect success on both gate-passing tasks, without demonstrations, motion references, or reward functions.On PDH, HuGo matches HDMI, with both succeeding in essentially all episodes; on CPB, HuGo reaches 81.25% versus HDMI's 98.4%.Source: B. Main Results · pp. 5–6
• Refinement improves success by up to 40 percentage points and reduces run-to-run variation for most tasks.Most gains arrive within the first two depths, while harder LBX continues improving through the final depth.Source: B. Main Results · p. 5
• LBX exposes contact limitations: sonic's 4 cm tracking error pushes the box off the podium, and both low-level policies struggle to maintain grasp force during lifting.A geometry hint raises sonic's best-of-10 success from 24.9% to 72.3%, but the force limitation remains.Source: B. Main Results · pp. 5–6
• Across tasks, high-level RL acts on objects but does not discover the staged approach-align-act-hold structure HuGo code specifies.Source: B. Main Results · p. 6
• On PDH HuGo matches HDMI, while on CPB it reaches 81.25% success versus 98.4% for HDMI's retargeted demonstration.CPB failures are linked to the low-level policies' difficulty maintaining force on a grasped object during carrying.Source: B. Main Results · p. 6

• A single generated policy averages 43.9% success across the five tasks and both low-level policies, with a 23.4-point standard deviation.Source: C. Variation of Performance · p. 6
• Different generated task logics can perform unevenly; on PBX, one strategy avoids rotating the box but rarely finishes within the episode limit.Source: C. Variation of Performance · p. 6
• Providing no examples or solution hints produces strategy diversity, trading consistency across runs for the ability to discover and select solutions.The best strategy may depend on the environment setup and may not be known in advance.Source: C. Variation of Performance · p. 6

D. Cost of Reliability

• 79.1% mean success at N = 10 versus 43.9% at N = 1, while standard deviation falls from 23.4 to 6.6 across tasks and low-level policies.Source: D. Cost of Reliability · p. 7
• A best-of-10 policy costs approximately $11; API cost rises monotonically but flattens beyond N ≈8.The selected operating point is N = 10.Source: D. Cost of Reliability · p. 7
• Obtaining a policy takes a few hours, whereas one high-level RL training run takes more than two days and requires a hand-designed reward for each task.Source: D. Cost of Reliability · p. 7

E. Ablation Studies

• Removing video frames cuts average success by more than 40%, indicating that numerical trajectories alone miss humanoid positioning at object contact.The ablation starts from selected near-zero-success policies, except for LGP, whose zero-shot policies already succeed.Source: E. Ablation Studies · p. 7
• Evenly spaced frames alone and selected keyframes alone both perform worse than combining them, supporting broad temporal coverage plus task-relevant moments.Source: E. Ablation Studies · p. 7

A. Hardware Setup

• On a Unitree G1, the hardware observation comes from motion capture, and matching simulation observation and command specifications lets generated code run without modification.Source: A. Hardware Setup · p. 7

B. Zero-Shot Transfer of Simulation-Generated Policies

• 93.3% on Push Button and 66.7% on Lift Box: the best policy for each task transfers well zero-shot, close to its simulation success rate.Some policies nevertheless lose part of their success on hardware.Source: B. Zero-Shot Transfer of… · p. 7
• Transfer losses are primarily associated with low-level tracking and contact-dynamics mismatches; on Lift Box, boxes slip from the end-effectors more often during hardware lifts.A low-level policy better suited to contact-rich interaction could address this limitation without changing the high-level policy-generation framework.Source: B. Zero-Shot Transfer of… · p. 7

C. Refinement on Hardware

• Hardware refinement uses the same rollout-based pipeline as simulation, without changing prompts, evaluation, or refinement logic.The recovery uses hardware rollouts without real-world expert demonstrations or policy retraining.Source: C. Refinement on Hardware · pp. 7–8
• 20% to 90%: hardware refinement restores the worst-transferring Push Button policy to its simulation success rate after three refinement depths.The success rate rose through 50% and 60%; reaching 90% required four refinement rounds in total.Source: Figure 7 · p. 8, C. Refinement on Hardware · p. 8
• The LLM diagnoses hardware-specific button undershoot and adds an inward target margin, then enlarges it and holds the press longer when some trials still fall short.Simulation does not exhibit this undershoot, so simulation refinement would not have produced this correction.Source: C. Refinement on Hardware · p. 8

VII. CONCLUSION

• HuGo generates closed-loop high-level policy code over a frozen whole-body policy without expert demonstrations, reference motions, or reward design.Across five tasks and two low-level policies, the generated policies adapt to randomized initial conditions and support zero-shot hardware transfer.Source: §VII · p. 8
• Hardware-rollout refinement corrected sim-to-real discrepancies, raising Push Button policy success from 20% to 90%; policies were generated atop a frozen whole-body policy without expert demonstrations, reference motions, or reward design.At refinement depth 1, feedback attributed failure to end-effector under-reaching and the code change added an explicit inward margin.Source: §VII · p. 8, Figure 8 · p. 8
• HuGo inherits the limits of its low-level policy and LLM: object holding depends on contact force or dexterous hands, and collision-free approaches may exceed what the LLM infers.Source: §VII · p. 8
• The authors identify cross-task memory as a future direction and suggest using generated policies to supervise vision-based policies by distillation or seed an RL replay buffer.Source: §VII · p. 8