PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates

Ryan Gao, Ashvin Anilkumar, Nicholas Corrado, William Cong, Josiah Hanna

arXiv:2610.14042v1cs.ROcs.AI

TL;DR

Real-robot RL is challenging when state must be inferred from onboard sensing that provides noisy, partial observations. The paper introduces RSSM–RLPD, using an observation-gated predictive belief for soccer dribbling. On hardware it outscored feed-forward and behavior-cloning baselines, and in simulation it improved learning efficiency over a GRU baseline.

  • Problem

    Real-robot RL often relies on noisy, partial onboard state estimates, especially outside highly observable or instrumented settings.

    Source: §I · p. 1

  • Method

    RSSM–RLPD learns a recurrent latent belief that updates on fresh ball observations and predicts between observations, with predictive and critic-based training signals.

    Source: §I · p. 1

  • Results

    RSSM–RLPD scored nine goals versus four for GRU–RLPD and zero for FF-RLPD and behavior cloning on hardware, while its normalized 0–60k success-curve area exceeded GRU–RLPD by 12.45 percentage points.

    Source: A. On-Robot Learning: Protocol and… · p. 5, C. Simulation Empirical Results · p. 7

  • Takeaways & Limitations

    In this setting, maintaining a predictive belief over an imperfect onboard estimate can make learning from physical experience tractable.

    Source: §I · p. 2

    Learning may depend on the quality and coverage of the fixed demonstration dataset, and the existing estimator’s errors can affect observations and rewards.

    Source: §VI · p. 7

Abstract

from arXiv · show

A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.

I. INTRODUCTION

The paper studies on-robot soccer dribbling when onboard sensing provides an uncertain, partial view of the world. It introduces an observation-gated predictive belief and evaluates it in hardware and simulation.

I. INTRODUCTION

  • Onboard estimates can be noisy when objects leave view, detections drop out, or robot motion perturbs sensing, limiting real-world RL beyond highly observable or instrumented settings.The task is a walking robot dribbling a ball into a goal.Source: §I · p. 1
  • RSSM–RLPD updates its latent belief when a fresh ball observation arrives and otherwise advances a predictive prior, training the representation with predictive loss and critic gradients.The actor and critics use belief statistics and an observation-gated innovation feature.Source: §I · p. 1
  • On hardware, the robot scored its first goal in 10 minutes and scored from multiple positions within 30 minutes without external instrumentation; simulation results attribute improvement to the learned belief.The simulation study distinguishes the learned belief from added capacity or demonstration data as the source of improvement.Source: §I · p. 2

A. Sample-Efficient On-Robot Learning

  • Prior hardware RL achieved quadruped forward walking in 20 minutes, combined demonstrations with online replay, integrated efficient learning with robot control and human interventions,, and improved manipulation through real-world interaction.This work studies ball contact under intermittent visual observations.Source: A. Sample-Efficient On-Robot Learning · p. 2

B. Memory and Predictive Representations

C. Robot Soccer

  • Prior robot-soccer work includes simulation-trained soccer policies, active perception [21], structured ball-and-goal observations, and modular perception and walking.This study uses predictive memory over estimator outputs for goal-directed dribbling from limited physical interaction.Source: C. Robot Soccer · p. 2

III. TASK AND SYSTEM

The task is formulated as partially observable control, with the learner relying on the NAO robot’s onboard state estimates. The system augments that estimation with an RSSM while preserving the existing walking controller.

A. Task Formulation

  • The dribbling task is a POMDP in which a policy acts from partial observations and seeks to maximize discounted return.The formulation specifies state, action, observation, transition, reward, initial-state distribution, and discount factor.Source: A. Task Formulation · p. 2

B. Robot Platform

  • A NAOv6 streams onboard-estimator observations to an external workstation, which returns actions while asynchronously updating the policy, without additional instrumentation.The interaction loop augments onboard estimation with the RSSM and leaves command execution to the existing walking engine.Source: B. Robot Platform · p. 2, Figure 2 · p. 3

C. Observations

  • The system decomposes the policy observation into ball-related and robot-related quantities, while the RSSM augments onboard estimation.The supplied description identifies this decomposition and the RSSM’s role in the physical interaction loop.Source: C. Observations · p. 3, Figure 2 · p. 3
  • The policy observation includes estimated ball information and visibility-related signals; when the ball is not visible, its position becomes a sentinel and its velocity is zero.The observation also includes the age of the latest ball observation and reliability measures derived from visibility and that age.Source: C. Observations · pp. 2–3

D. Actions

The policy issues high-level body-velocity commands, which are post-processed and safety-filtered before reaching the locomotion controller.

D. Actions

  • The policy outputs desired forward, lateral, and rotational body velocities at a nominal 30 Hz.Source: D. Actions · p. 3
  • The resulting rotational action is post-processed by clipping its magnitude to a maximum of 0.6 and smoothing it against the previous executed action.Source: D. Actions · p. 3
  • A safety filter suppresses movement toward field and goal keep-out boundaries before the resulting action is sent to the locomotion controller and recorded.Source: D. Actions · p. 3

E. Reward

The reward uses estimated state while gating progress shaping on fresh ball tracking to reduce the influence of unreliable estimates.

E. Reward

  • Progress reward terms are applied only when the ball is freshly tracked throughout the transition without loss or reacquisition.The freshness indicator is one for continuous observation and zero otherwise.Source: E. Reward · p. 3
  • The reward includes reductions in robot-to-ball and ball-to-goal distance, facing the ball, and an approach term adjusted to avoid penalizing successful forward dribbling.The approach term is suppressed when the ball advances toward the goal while robot-to-ball distance increases.Source: E. Reward · p. 3
  • When the ball is not freshly observed, a separate penalty grows with observation age and saturates at 0.5 seconds; the reward also includes action and per-step costs, a +10 goal-crossing bonus, and terminal reward.Because estimated ball position can produce false positives, physical goal crossings are evaluated separately from the training reward.Source: E. Reward · p. 3

F. Termination & Episode Resets

Episodes end on scoring, leaving field boundaries, extended ball-visibility loss, step limits, or specified robot and communication failures. RSSM–RLPD augments RLPD with an observation-gated recurrent state-space model.

F. Termination & Episode Resets

  • Episodes terminate after a goal, field-boundary violation, extended ball-visibility loss, the step limit, invalid telemetry, communication failure, or a fall.An operator manually resets the robot and ball and starts the next episode; no transitions are recorded during reset.Source: F. Termination & Episode Resets · p. 3
  • After termination, an operator resets the robot and ball, and no transitions are recorded during that interval.The operator starts a new episode after the manual reset.Source: F. Termination & Episode Resets · p. 3

IV. RSSM–RLPD FOR ON-ROBOT LEARNING

  • RSSM–RLPD augments RLPD with an observation-gated recurrent state-space model to address partial ball visibility.The actor and critics share a representation trained with prediction losses and the critic objective to retain ball-motion information relevant to control.Source: §IV · p. 3

A. Observation-Gated Belief

RSSM–RLPD maintains a recurrent latent belief, using ball observations for posterior updates and a prior when the ball is not observed. Predictive supervision trains the representation from replay, including multi-step predictions.

A. Observation-Gated Belief

  • The RSSM deterministic state is updated by a GRU using a learned input projection, elapsed time, and encoded robot-pose and ball-visibility context.The latent state has deterministic and stochastic components.Source: A. Observation-Gated Belief · p. 3
  • When the ball is observed, the belief uses a posterior; otherwise it uses a prior, with robot pose and ball visibility encoded as context.The context helps keep the prior consistent with robot motion while the ball is out of view.Source: A. Observation-Gated Belief · pp. 3–4
  • The actor and critics receive belief features; training samples the stochastic state, while actor and critic updates use its mean.The actor samples actions for exploration.Source: A. Observation-Gated Belief · p. 4

B. Predictive Supervision from Replay

  • Replay supervision combines reconstruction and transition losses with current and future ball-position and prior-posterior consistency losses.The described losses cover recorded observations, reward, termination, freshness, weak contact labels, and ball-position estimates.Source: B. Predictive Supervision from Replay · p. 4
  • Multi-step predictions use horizons of 1, 3, 5, 10, 20, and 40 steps to score future ball estimates and compare priors with filtering posteriors.Predictions advance from a sampled filtering state using recorded actions, intervals, context, and freshness, and train the representation from replay.Source: B. Predictive Supervision from Replay · p. 4

C. Critic-Guided Representation Learning

The critic objective trains the RSSM representation to retain task-relevant information. Target successor features and target critics define the TD target, while replay positions supply the sampled training data.

C. Critic-Guided Representation Learning

  • Two action-value critics train the RSSM to retain task-relevant information, while the actor selects actions from the belief features.The target filter computes successor features for the TD target.Source: C. Critic-Guided Representation Learning · p. 4
  • The critic loss is evaluated over sampled replay positions; TD updates train the critics and RSSM, while predictive updates train the core and decoders.Actor and temperature updates treat the belief features as fixed.Source: C. Critic-Guided Representation Learning · p. 4

D. Replay Masking and the Simulation Schedule

Replay masking hides spans of fresh ball evidence while retaining recorded supervision. In simulation, the actor–critic sampling schedule tapers augmented sequences, and training uses recurrent windows drawn from online and demonstration replay.

D. Replay Masking and the Simulation Schedule

  • Replay masking hides a contiguous 6–36-step span of fresh ball evidence by replacing ball position with a sentinel, zeroing velocity, and clearing visibility.Recorded supervision, actions, rewards, termination masks, and age-derived features are retained.Source: D. Replay Masking and the Simulation… · p. 4
  • In simulation, actor–critic sampling uses masked and event-centered sequences before switching to ordinary replay from 30k transitions.Predictive training keeps both augmented sequence fractions at 1/2; hardware trials below 20k use the initial regime.Source: D. Replay Masking and the Simulation… · pp. 4–5
  • Each replay pool contains 16 online and 16 demonstration windows of 64 observations, with the first 16 positions providing recurrent context.A learner block performs eight critic updates, one actor/temperature update, and one model update.Source: D. Replay Masking and the Simulation… · p. 5

V. EXPERIMENTS

The experiments test soccer-goal dribbling through complementary real-robot and simulation studies. On hardware, RSSM-RLPD leads the baselines in training and five-goal validation.

V. EXPERIMENTS

  • Hardware captures real sensing noise and contact dynamics, while simulation enables multi-seed replication and evaluation across more failure conditions.Source: §V · p. 5

A. On-Robot Learning: Protocol and Results

B. Simulation Empirical Setup

The simulation setup compares recurrent and feed-forward learning methods across controlled training and validation conditions. It evaluates episode success, early success-curve performance, and uncertainty across five trials.

  • Five trials compare RSSM-RLPD, GRU-RLPD, FF-RLPD, and behavior cloning; online methods train for 100k transitions with equal online and demonstration sampling.The simulator models sensing, state estimation, walking, and ball contact, using a shared archive of 50 real-robot episodes.Source: B. Simulation Empirical Setup · p. 6
  • Training runs at 30 Hz with 1,800-step episode caps, stationary-ball resets, and center–left–right placements; moving-ball validation starts at 0.20 m/s.Source: B. Simulation Empirical Setup · p. 6, Table III · p. 6
  • FF-RLPD uses only the current estimator-derived observation, while GRU-RLPD tests TD-trained recurrence against predictive memory at matched representation width; parameter counts differ.Behavior cloning fits the shared archive with 20,000 Adam updates.Source: B. Simulation Empirical Setup · p. 6
  • Validation crosses placements with nominal sensing and five perturbations, yielding 33 scenes and 66 attempts per trial, or 330 per method and checkpoint.Source: B. Simulation Empirical Setup · p. 6
  • Validation disables action filters and goal-keepout penalties for frozen policies, while retaining field and ball-out checks, 10-second ball-loss termination, and scoring.Source: B. Simulation Empirical Setup · p. 6
  • Metrics include episode success and cumulative 20/40/60-second rates; early performance is normalized 0–60k AUC, with 95% bootstrap intervals across five trials conditional on fixed scenes and sensors.Source: B. Simulation Empirical Setup · pp. 6–7

C. Simulation Empirical Results

In simulation, recurrent learners outperform feed-forward RLPD and behavior cloning, while RSSM–RLPD improves over GRU in the intermediate training interval. The paper also identifies boundaries involving demonstration data, state estimation, and command granularity.

C. Simulation Empirical Results

  • Both recurrent methods substantially outperform FF-RLPD and BC in simulation.Fig. 7 reports success within 60 seconds over 330 attempts per method and checkpoint.Source: C. Simulation Empirical Results · p. 7, Figure 7 · p. 7
  • 12.45 percentage points is the paired 0–60k success-curve AUC gain for RSSM–RLPD over GRU (95% interval [3.61,21.54]).RSSM–RLPD has higher mean success at every checkpoint from 20k to 90k, but GRU leads at 10k and mean success is similar at 100k.Source: C. Simulation Empirical Results · p. 7
  • RSSM–RLPD makes contact more consistently and loses the ball less often than the other methods, but times out more often than GRU.The reported pattern separates contact acquisition from timely completion.Source: C. Simulation Empirical Results · p. 7

VI. LIMITATIONS AND FUTURE WORK

  • The study used a fixed robot-collected demonstration dataset, so learning may depend on its quality and coverage.The authors propose measuring sensitivity to demonstration quality and quantity and incorporating targeted human corrections.Source: §VI · p. 7
  • The existing state estimator's errors can affect both observations and rewards.The authors suggest jointly improving estimation and predictive memory as future work.Source: §VI · p. 7
  • The policy's three high-level walking commands leave foot placement and contact execution to the walking controller, limiting fine ball manipulation.Richer footstep or contact commands could support more precise dribbling but would require more precise state estimation.Source: §VI · p. 7

VII. CONCLUSION

  • On hardware, RSSM–RLPD scored on all evaluation configurations within four minutes, compared with none for basic RLPD under the same training budget.The robot learned to score across multiple starting positions within 30 minutes of training.Source: §VII · p. 7
Loading 2610.14042v1…