PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflex: Visually Grounded Reactive Humanoid Control

Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, Ranjay Krishna

arXiv:2610.12772v1cs.RO

TL;DR

Humanoid whole-body reactions to moving objects remain challenging because they require coordinating the body, predicting object dynamics, and grounding perception in egocentric vision. Reflex learns these capabilities in three stages linked by an object-dynamics latent, achieving 85.7% simulated catch success from RGB-D versus 87.9% with privileged box observations, and catching human-thrown boxes on a real G1.

  • Problem

    Visually reactive whole-body control remains challenging for humanoids, which must coordinate their body, react to object dynamics, and ground perception in egocentric vision.

    Source: §I · p. 1

  • Method

    Reflex trains whole-body catching, delayed-observation object dynamics, and visual recovery of the dynamics latent in three stages, keeping the controller fixed during visual learning.

    Source: §I · p. 2

  • Results

    85.7% catch success from egocentric RGB-D in simulation compares with 87.9% using privileged box observations; on a real G1, success was 65% at 2.5 m and 40% at 5 m.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results suggest a path toward humanoids reacting to dynamic events from onboard vision, with catching as one instance of broader visually reactive whole-body skills.

    Source: §I · p. 2

    Boxes outside the reference size are often intercepted but not retained.

    Source: §VI · p. 11

Abstract

from arXiv · show

Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.

I. INTRODUCTION

Reflex targets visually reactive whole-body catching, where a humanoid must predict a thrown box’s motion and coordinate its body before contact. It combines capabilities often treated separately and evaluates the approach in simulation and on a real humanoid.

I. INTRODUCTION

  • Humanoids must coordinate whole-body movement, react to fast object dynamics, and ground perception in egocentric vision, where self-motion and near-contact occlusion complicate observation.A thrown box reaches the robot roughly one second after release, while detector-based camera-to-control latency has a 69 ms median.Source: §I · p. 1
  • Thrown-box catching tests prediction and whole-body response together: the robot must form forearm-and-torso support, absorb momentum, and retain an uninstrumented box.Three human catchers succeeded on 28.3% of throws from 7 m.Source: §I · p. 2

II. RELATED WORK

  • Prior catching systems include fixed-base manipulators and humanoids using motion capture or retroreflective objects, with the three target capabilities rarely combined on a real humanoid.Related work spans fixed-base catching, learned catching motions, whole-body control, and onboard sensing, but most cited systems rely on motion capture or external cameras.Source: §I · p. 2, A. Catching thrown objects · p. 2

I. INTRODUCTION

  • Reflex learns whole-body catching, box dynamics from delayed and incomplete observations, then visual recovery of the same dynamics latent while keeping the controller fixed.The decomposition gives each capability a direct training signal and isolates visual learning from reinforcement learning.Source: §I · p. 2
  • 85.7% simulated catch success from egocentric RGB-D compares with 87.9% using privileged box observations; real G1 success is 65% at 2.5 m and 40% at 5 m.Those real-robot rates reach 76% and 59% of human catchers’ success at the same distances.Source: §I · p. 2
  • The paper presents catching as one instance of visually reactive whole-body skills such as interception, avoidance, handoffs, and physical assistance.These tasks involve predicting how the world will change and coordinating the body before contact.Source: §I · p. 2

B. Humanoid whole-body control and loco-manipulation

The paper situates Reflex among humanoid whole-body control and privileged visual learning, then defines a thrown-box task for a 29-DoF Unitree G1. Deployment relies on onboard RGB-D and robot state rather than box-state or external-camera measurements.

B. Humanoid whole-body control and loco-manipulation

  • Unlike prior loco-manipulation with stationary objects, catching requires the body to be positioned before the box arrives; Reflex trains a dedicated controller for interception and retention.The cited prior systems use human motion foundations, but their objects are at rest when interaction begins.Source: B. Humanoid whole-body control and… · p. 2

C. Privileged Learning and Visual Sim-to-Real

  • Figure 2’s three stages progress from PPO-trained state control to privileged dynamics learning and visual latent prediction, with the decoder unchanged during visual learning.Stage 3 also uses box-state supervision and DAgger rollouts; the pretrained CNN and heatmap head stay fixed during GRU training.Source: Figure 2 · p. 3
  • Following privileged-learning approaches, Reflex trains a visual model to recover a privileged latent from RGB-D history and camera ego-motion while keeping the action decoder fixed.The cited prior visual students generally perceive static terrain or slow-moving objects.Source: C. Privileged Learning and Visual… · p. 3

A. Visually Grounded Reactive Catching

  • The task requires a 29-DoF G1 to intercept a human-thrown box on its forearms against the torso, dissipate momentum, retain it, and avoid falling.Unlike quasi-static loco-manipulation, the robot cannot choose when contact occurs.Source: A. Visually Grounded Reactive Catching · p. 3
  • At deployment, the torso-mounted camera supplies RGB-D and the controller receives robot stabilization state, but no box pose, velocity, or external-camera measurement.The camera moves with the robot and the box becomes increasingly occluded near contact; each 20 ms step produces joint-position targets for 29 actuated DoF.Source: A. Visually Grounded Reactive Catching · p. 3

B. Success Criteria

The paper distinguishes interception from sustained catch success and separately measures post-catch stability. Its learning design separates visual inference from whole-body control to avoid coupling expensive pixel-based learning with motor-skill discovery.

B. Success Criteria

  • Catch success requires sustained, low-speed chest support with the wrists on opposite sides of the box, unlike interception, which only requires slowing the box near the torso.The catch criterion excludes support from the head or shoulders and requires 0.6 s of retention.Source: B. Success Criteria · p. 3
  • Torso excursion and falls measure post-catch stability separately, distinguishing retention from a settled posture and perception/interception failures from stabilization failures.Torso excursion records whether tilt reaches 38°; the box-size sweep uses a geometry-agnostic sustained-hold criterion.Source: B. Success Criteria · pp. 3–4

IV. LEARNING EGOCENTRIC WHOLE-BODY CATCHING

  • The visual function predicts an eight-dimensional object-dynamics latent from egocentric RGB-D history and camera ego-motion, which the controller combines with robot/reference observations to produce normalized 29-DoF actions.The latent interface separates what object information vision infers from how that information drives the body.Source: §IV · p. 4
  • Joint pixel-based reinforcement learning is computationally expensive and difficult to optimize because it couples visual representation learning with exploration across a 29-dimensional action space.Delayed catch-and-retention rewards, camera motion, occlusion, and policy-induced viewpoint changes further couple perception errors to visited states.Source: §IV · p. 4
  • The three-stage decomposition learns whole-body control, object dynamics, and visual grounding separately, with the controller fixed during visual grounding.A downstream carrying and handover module follows the catching controller.Source: §IV · p. 4

A. Stage 1: Learning Whole-Body Control

Stage 1 trains a state-based catching teacher with randomized PPO and human-motion guidance. Stage 2 transfers that control interface to a latent representation of delayed box observations, then refines it with task-specific PPO.

A. Stage 1: Learning Whole-Body Control

  • Stage 1 uses PPO on box-state channels to learn body placement, receiving posture, impact absorption, and retention in rendering-free simulation.The state-policy observation combines robot/reference inputs with box position, velocity, observation age, and measurement-quality indicators.Source: A. Stage 1: Learning Whole-Body Control · p. 4
  • Approach, forearm support, and sustained-catch rewards guide interception and retention, while retargeted human motion and an adversarial motion prior encourage coordinated movement.Throws, contact properties, and robot dynamics are randomized; held-box starts and perturbations train retention and recovery.Source: A. Stage 1: Learning Whole-Body Control · p. 4
  • After state-based PPO, the frozen teacher supplies action labels and decoder weights to initialize the latent controller.The experimental ground-truth state-policy reference instead uses ideal complete, noise-free, zero-delay box measurements through the policy’s Kalman filter.Source: A. Stage 1: Learning Whole-Body Control · p. 4

B. Stage 2: Learning Object Dynamics

  • Stage 2 maps delayed box observations to a bounded dynamics latent; the privileged encoder receives position, velocity, size, orientation, angular velocity, visibility, and observation age.Its single delayed sample includes velocity, while inference from observation history is left to Stage 3.Source: B. Stage 2: Learning Object Dynamics · pp. 4–5
  • Action matching initializes the latent pathway, then DAgger aggregation adds states visited by latent control and repeats matching against fixed-teacher action labels.The teacher labels are distinct from human motion examples used for reference tracking and the motion prior.Source: B. Stage 2: Learning Object Dynamics · p. 5
  • Task-specific PPO jointly optimizes the privileged encoder and full decoder, with the actor using delayed-observation latents and the asymmetric critic retaining privileged state.In a single-seed comparison, training with latency randomization was 5–11 points higher at every tested delay; Stage 2 samples 20–60 ms latency per episode.Source: B. Stage 2: Learning Object Dynamics · p. 5

C. Stage 3: Learning Visual Grounding

Stage 3 learns to recover the dynamics latent from egocentric vision while keeping the whole-body controller fixed. Its visual pipeline combines localization, geometric features, and temporal ego-motion to predict object dynamics.

C. Stage 3: Learning Visual Grounding

  • Stage 3 trains an egocentric visual encoder to recover the dynamics latent, with rendering used only in this stage and the controller unchanged.The visual model uses RGB-D history and camera ego-motion.Source: C. Stage 3: Learning Visual Grounding · p. 5
  • A CNN localizes the box, a geometric layer forms features, and a GRU uses feature and ego-motion histories to predict the dynamics latent and auxiliary position and velocity.Only the latent, not the auxiliary predictions, drives the decoder.Source: C. Stage 3: Learning Visual Grounding · p. 5
  • Visual training first supervises the RGB-D localization heatmap, then freezes the CNN and heatmap head while training the GRU to predict latent and auxiliary box-state targets.The heatmap uses Gaussian targets at projected box centers and binary cross-entropy supervision.Source: C. Stage 3: Learning Visual Grounding · p. 5
  • Training histories use fixed privileged-encoder latent targets and auxiliary targets for standardized current camera-frame box position and velocity.The latent targets use a fixed two-step (40 ms) latency with no dropout.Source: C. Stage 3: Learning Visual Grounding · p. 5
  • Visual-policy data aggregation labels states visited by the visual controller with the fixed privileged encoder, then refits the GRU and output heads with the localizer and controller fixed.Windows stay within episodes; validation and final evaluation use separate episodes and throws.Source: C. Stage 3: Learning Visual Grounding · pp. 5–6
  • Appearance augmentation includes color and image and depth perturbations, but motion blur, auto-exposure dynamics, and structured depth artifacts are not modeled.The temporal model is fit on noise-perturbed features with the color-augmented localizer frozen.Source: C. Stage 3: Learning Visual Grounding · p. 6

D. Downstream Carrying and Handover

The downstream demonstration connects the catching policy to walking with the retained box and a handover. Success declines across the carry, handover, and return stages.

D. Downstream Carrying and Handover

  • The demonstration connects the catching policy to the pretrained SONIC motion controller for walking while retaining the box and handing it over.Visual human detection identifies the handover recipient.Source: D. Downstream Carrying and Handover · p. 6
  • A supervisor transfers control after persistent near-chest box evidence and low leg velocity, torso tilt, and base speed indicate a settled hold.It then establishes the navigation frame, switches controller observations and gains, and holds posture for 2 s before transitioning to a standing cradle.Source: D. Downstream Carrying and Handover · p. 6
  • 65% of trials succeeded after catching, 35% after carrying, 30% after handover, and 25% after return.These are cumulative success rates from a 20-trial downstream task evaluation.Source: D. Downstream Carrying and Handover · p. 6

V. EXPERIMENTS

The experiments are organized around four questions: overall catching performance, whole-body control and payload conditions, latent learning under offsets and delay, and visual grounding across inputs and conditions.

V. EXPERIMENTS

  • The experiments ask how well Reflex catches in simulation and on hardware relative to privileged input and human catchers.This is the first of the four stated experimental questions.Source: §V · p. 6
  • The study asks how catching objectives and motion priors shape whole-body control, and how latent learning handles lateral offsets and perception delay.It also examines box-size and payload variation.Source: §V · p. 6
  • The visual-grounding experiments examine inputs and supervision alongside appearance, distance, and detector-based perception; table colors distinguish proposed, reference, and diagnostic configurations.Pink marks proposed configurations, gray marks references or auxiliary diagnostic rates, and bold marks the highest catch rate in each comparison, excluding gray references.Source: §V · p. 6

A. Experimental Setup

The setup defines randomized training and evaluation conditions, catch criteria, and comparisons between state, privileged-latent, and visual-latent controllers. Several tested conditions extend beyond training ranges.

A. Experimental Setup

  • In Stage 2 joint PPO, release distances span 2.5–5.0 m, while box size is randomized among three dimensions: 25×19×13, 34×26×22, and 42×32×27 cm.Source: A. Experimental Setup · p. 6
  • Evaluation uses one 3.0–3.5 m throw with a 34×26×22 cm box per environment, starts without a held box, and ends after 640 steps.Rethrows, deliberately unreachable targets, robot pushes, and held-box kicks are disabled.Source: A. Experimental Setup · p. 6
  • A catch requires chest-region support, wrists flanking the box, head and shoulder clearance, and low box speed for 0.6 s.Torso excursion is separately recorded when tilt reaches 38°; it is distinct from catch success and falls.Source: A. Experimental Setup · p. 6
  • The setup compares a state-policy reference with privileged and visual latent controllers, using six controllers per latent configuration and held-out throws for evaluation.The privileged mask and rendered pinhole image have different fields of view, so privileged-input results serve as a reference for visual substitution.Source: A. Experimental Setup · p. 6
  • In the tables and figures, the state policy is Stage 1, privileged z is the Stage 2 latent controller with privileged observations, and visual bz is the full Stage 3 system, Reflex.The first-trained controller supplies the reported benchmark configurations described in the setup.Source: A. Experimental Setup · pp. 6–7
  • The condition benchmark varies size, payload, distance, and color; 0.5–2 kg payloads and 7 m throws are outside training conditions.The 7 m tests include throws faster than any training throw, and size columns use a different success criterion than weight, distance, and color columns.Source: A. Experimental Setup · p. 7, Figure 3 · p. 7

B. Overall Performance

Reflex achieves strong simulated catching and transfers to a real G1, while whole-body performance depends on retaining the catching skill and handling box geometry.

B. Overall Performance

  • 85.7% catch success from egocentric RGB-D compares with 87.9% from privileged latent input; the G1 reaches 65%, 40%, and 15% for regular boxes at 2.5, 5, and 7 m.At 2.5 and 5 m, those G1 rates are 76% and 59% of human success, respectively.Source: B. Overall Performance · p. 7, Table II · p. 8

C. Whole-Body Control

  • Removing all task reward groups reduces success to 2.7%; removing hold shaping lowers offset success to 50.8%, while removing tracking or AMP alone costs 4–5 points.Without Stage 1, the Stage 2 recipe reaches only 6.7% and 0.2% after 1,000 iterations.Source: C. Whole-Body Control · p. 7, Table III · p. 8
  • Across six visual controllers, torso excursion ranges from 1% to 72%, distinguishing successful retention from a settled whole-body posture.The pelvis never drops below 0.52 m, and 3 of 3,840 episodes end early.Source: C. Whole-Body Control · p. 7
  • Visual control catches 0.5–2 kg payloads at 77.7–91.2%, but catches only 25.5% of small boxes and 39.7% of large boxes, often after interception.Stage 1 uses the regular box without a size input, and the latent does not encode size.Source: C. Whole-Body Control · p. 7
  • Within the visual training range, visual latent control exceeds detection plus filtering with the same detector.Table I reports catch and interception rates from one checkpoint per controller, with 640 throws per condition (320 for the detector).Source: Table I · p. 8

D. Object Dynamics

The learned latent represents delayed box dynamics and supports catching across varied lateral offsets and perception delays. Its performance depends on initialization and latency-aware training.

D. Object Dynamics

  • BC initialization improves visual catch success by 5.2 percentage points over position/velocity weight mapping, with a similar gain under privileged input.The learned latent predicts current position and velocity for samples at least 80 ms old with RMSE 0.33 m and 1.5 m/s.Source: D. Object Dynamics · p. 8
  • Visual latent input reaches 85.3%, 66.6%, and 35.6% at 0, 0.6, and 0.9 m offsets, exceeding the ground-truth state policy by about 12 points at the two larger offsets.YOLO detection lowers the state policy’s catch rate at every reported offset.Source: D. Object Dynamics · p. 9, Figure 4 · p. 9
  • With visual input, catch success is 88.4%, 87.2%, and 83.0% with 0, 20, and 40 ms added delay, and 72.7% at 60 ms.At every tested delay, the controller trained without latency randomization is 5–11 points lower; the 60 and 80 ms columns bracket the pipeline’s 69 ms median latency.Source: D. Object Dynamics · p. 9, Table IV · p. 9

E. Visual Grounding

Visual grounding benefits from temporal information, camera ego-motion, and color augmentation, while its performance depends on the visual training range.

E. Visual Grounding

  • Removing temporal history reduces catch success by 35.6 and 39.8 percentage points on centered and 0.6 m-offset throws; removing camera ego-motion costs 10.4 and 22.6 points.Removing heatmap supervision costs 7.4 and 10.9 points.Source: E. Visual Grounding · p. 9, Table V · p. 9
  • The same visual latent controller succeeds at every tested real-robot distance, box size, payload, and lateral offset.The figure shows 2.5, 5, and 7 m throws, three box sizes, payloads of 0.5, 1, and 2 kg, and offsets of 0.9 m right and 0.7 m left.Source: Figure 5 · p. 10
  • The Stage 2 initialization table reports action matching averaging 5 points higher across six controllers.Source: Table VII · p. 10
  • With the decoder fixed, color augmentation raises success on blue and black boxes from 60.2% and 61.2% to 88.0% and 88.8%.Across seven tested colors, success is 87.0–88.9% in this evaluation run.Source: E. Visual Grounding · pp. 10–11, Table VI · p. 10
  • Within its visual training range, the visual latent outperforms detection-plus-filtering baselines in most reported conditions, while at 5 m visual input reaches 43.0% versus 79.7% with privileged input.Adding 2.5–7 m throws to Stage 3 raises visual success to 75.8% at 5 m and 36.7% at 7 m.Source: E. Visual Grounding · p. 11

F. Real-Robot Evaluation

The conclusion reports strong simulated and real-robot catching results, while the discussion identifies long-range visual grounding as the main limitation. Hardware trials also include side-aimed throws and measurements of the detector-based pipeline’s latency.

F. Real-Robot Evaluation

  • 8 of 20 throws aimed to either side were caught, but human throwers could not place lateral offsets precisely.Source: F. Real-Robot Evaluation · p. 11
  • The detector-based pipeline’s 95th-percentile camera-to-control latency was 90.2 ms, and its 50 Hz control period exceeded 25 ms on 0.015% of ticks.Source: F. Real-Robot Evaluation · p. 11

VI. DISCUSSION AND LIMITATIONS

  • Keeping the controller fixed and rendering out of reinforcement learning reduces visual-learning demand to about 1.6M rendered steps.Source: §VI · p. 11
  • Beyond its 3.0–3.5 m Stage 3 range, 5 m visual success is 43.0% versus 79.7% privileged; adding far throws raises simulated 5 m success to 75.8%, untested on hardware.Source: §VI · p. 11

VII. CONCLUSION

  • 85.7% catch success from visual input versus 87.9% with privileged input in simulation; on hardware, Reflex reaches 76% and 59% of human success at 2.5 and 5 m.Source: §VII · p. 11
Loading 2610.12772v1…