PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning

Archit Sharma, Ahmed M. Ahmed, Rehaan Ahmad, Chelsea Finn

arXiv:2303.01488v1cs.ROcs.AIcs.LG

TL;DR

Robots need to improve through autonomous interaction because human supervision, reward design, and repeated environment resets limit scalable training. MEDAL++ learns forward and backward behavior and infers rewards end-to-end from visual inputs using a small set of demonstrations. It improves data efficiency and benchmark performance, while increasing real-robot success rates over behavior cloning on expert data.

  • Problem

    Human supervision, reward-function design, and repeated environment resets limit autonomous robot learning and scalable data collection.

    Source: §I · p. 1

  • Method

    MEDAL++ learns forward and backward policies, infers rewards from demonstrations, and trains visual policies end-to-end without explicit state estimation.

    Source: §I · p. 2, §VI · p. 8

  • Results

    MEDAL++ is more data efficient and achieves up to 30% better EARL performance, while real-robot autonomous training improves success rates by 30-70% over behavior cloning.

    Source: §I · p. 2, B. Real Robot Evaluations · p. 7

  • Takeaways & Limitations

    The results indicate that autonomous self-improvement can train visual manipulation policies with minimal oversight and without engineered rewards.

    Source: §I · p. 2, §VI · p. 8

    Real-robot data collection remains slow because training updates and collection steps are performed sequentially, yielding approximately 3.5 Hz data collection at 10 Hz control.

    Source: §VI · p. 9

Abstract

from arXiv · show

In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. An aspirational goal is to construct self-improving robots: robots that can learn and improve on their own, from autonomous interaction with minimal human supervision or oversight. Such robots could collect and train on much larger datasets, and thus learn more robust and performant policies. While reinforcement learning offers a framework for such autonomous learning via trial-and-error, practical realizations end up requiring extensive human supervision for reward function design and repeated resetting of the environment between episodes of interactions. In this work, we propose MEDAL++, a novel design for self-improving robotic systems: given a small set of expert demonstrations at the start, the robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations. The policy and reward function are learned end-to-end from high-dimensional visual inputs, bypassing the need for explicit state estimation or task-specific pre-training for visual encoders used in prior work. We first evaluate our proposed algorithm on a simulated non-episodic benchmark EARL, finding that MEDAL++ is both more data efficient and gets up to 30% better final performance compared to state-of-the-art vision-based methods. Our real-robot experiments show that MEDAL++ can be applied to manipulation problems in larger environments than those considered in prior work, and autonomous self-improvement can improve the success rate by 30-70% over behavior cloning on just the expert data. Code, training and evaluation videos along with a brief overview is available at: https://architsharma97.github.io/self-improving-robots/

I. INTRODUCTION

The paper targets self-improving robots that learn autonomously from visual interaction with minimal human supervision, avoiding engineered rewards, explicit state estimation, and repeated resets. MEDAL++ combines forward and backward task practice with online reward inference and end-to-end visual learning, improving simulated efficiency and real-robot success over expert-data behavior cloning.

I. INTRODUCTION

  • The system uses a forward policy to perform the task and a backward policy to undo it, allowing autonomous practice from a small, front-loaded set of expert demonstrations.The demonstrations help address exploration challenges as the robot reaches harder-to-recover states without frequent resets.Source: §I · pp. 1–2, Figure 1 · p. 1
  • MEDAL++ learns end-to-end from high-dimensional visual inputs while inferring rewards from demonstrations, avoiding explicit state estimation and task-specific visual-encoder pretraining.Its efficiency improvements also include Q-function ensembles, more updates per collected sample, behavior-cloning regularization, and demonstration oversampling.Source: §I · p. 2
  • 30-70% higher success rates than behavior cloning on expert data were observed after autonomous MEDAL++ training across four real-robot manipulation tasks.The evaluations used RGB observations on a Franka Panda robot arm in tasks including cloth hanging, bowl covering, and peg insertion.Source: §I · p. 2, §II · p. 2
  • Up to 30% better final performance and greater data efficiency than competitive vision-based methods were achieved on the pixel-based, non-episodic EARL benchmark.This evaluation addresses autonomous learning without relying on repeated episode resets.Source: §I · p. 2

II. RELATED WORK

  • Prior autonomous-learning systems typically require low-dimensional state inputs, engineered rewards, frozen visual encoders, or restricted environments, whereas MEDAL++ targets larger real-robot environments without repeated resets.Related work includes reset-free methods based on backward policies, unsupervised skill discovery, state novelty, or intervention learning, but their real-world scope remains constrained.Source: §I · p. 1, §II · p. 2

III. PRELIMINARIES

The paper studies non-episodic autonomous reinforcement learning with unknown rewards and small forward and backward expert demonstrations. MEDAL trains forward and backward policies so the robot can practice and reset through interaction.

Problem Setting

  • The objective is to learn a policy maximizing discounted reward from an initial-state distribution in an MDP with potentially high-dimensional observations.Source: §III · p. 2
  • Training is non-episodic, so the environment does not periodically reset to its initial-state distribution after a fixed number of steps.Source: §III · p. 2
  • Rewards are unavailable during training; instead, the robot receives small forward demonstrations of the task and backward demonstrations that undo it.Source: §III · p. 3

Autonomous Reinforcement Learning via MEDAL

  • MEDAL alternates a forward policy that solves the task with a backward policy that returns the environment toward the expert state distribution, reducing intervention-based resets.Source: §III · p. 3

Learning Reward Functions with VICE

  • VICE learns a forward reward from goal states by classifying goal images against online policy states, continually updating negative examples as interaction proceeds.Source: §III · p. 3

IV. MEDAL++: PRACTICAL AND EFFICIENT AUTONOMOUS REINFORCEMENT LEARNING

MEDAL++ adapts MEDAL for real-world autonomous learning by combining end-to-end visual encoding, learned rewards, and more efficient Q-value and policy training. Visual observations are augmented and encoded directly without explicit state estimation.

IV. MEDAL++: PRACTICAL AND EFFICIENT AUTONOMOUS REINFORCEMENT LEARNING

  • MEDAL++ is designed to learn from small forward and backward demonstrations without reward labels while minimizing real-world task-specific engineering.Source: §IV · p. 3
  • MEDAL++ combines learned VICE rewards, Q-value ensembles, increased update steps, behavior-cloning regularization, and oversampling of demonstration transitions for efficiency.Source: §IV · p. 3

A. Encoding Visual Inputs

  • The method learns an image encoder end-to-end and uses random crop and shift augmentations to regularize Q-value learning without explicit representation objectives.Source: A. Encoding Visual Inputs · p. 3
  • Q-value training uses an augmented-image transition loss with a TD target computed from slow-moving target networks and the current policy.Source: A. Encoding Visual Inputs · p. 3

B. Learning the Reward Function

MEDAL++ infers forward rewards from demonstration-derived goal states rather than requiring separately collected goal data. It matches policy states to successful outcomes while allowing actions to differ from teleoperated demonstrations.

B. Learning the Reward Function

  • The goal set is formed from the last K states of each forward demonstration trajectory, avoiding separate goal-state collection.Source: B. Learning the Reward Function · p. 3
  • VICE trains a classifier to distinguish demonstration-derived goal states from online policy states, and its reward encourages reaching states classified as goals.Source: B. Learning the Reward Function · p. 3
  • Spectral normalization, mixup, and image augmentations regularize the classifier because accurate discrimination can otherwise make the reward sparse.Source: B. Learning the Reward Function · p. 3
  • Unlike GAIL, VICE matches policy state distributions to goal states rather than matching expert state-action distributions, allowing actions to differ from teleoperation.Source: B. Learning the Reward Function · p. 4

C. Improving the Learning Efficiency

MEDAL++ improves learning efficiency through ensemble Q-value targets, expert-transition oversampling, and behavior-cloning regularization.

C. Improving the Learning Efficiency

  • MEDAL++ trains an ensemble of Q-value networks rather than a single Q-value estimator.Each ensemble member uses a target formed from a sampled subset of target networks.Source: C. Improving the Learning Efficiency · p. 4
  • Randomized subsets of target networks and minimum target values increase Q-learning sample efficiency without overfitting to one target.Target networks track online Q-networks through exponential moving averages.Source: C. Improving the Learning Efficiency · p. 4
  • Expert transitions are oversampled, while policy learning combines Q-value maximization with behavior-cloning regularization toward expert actions.The expert fraction is controlled by ρ, and λ controls the regularization strength.Source: C. Improving the Learning Efficiency · p. 4
  • The forward classifier uses the last K states of forward demonstrations as positive targets, while remaining forward states train the backward classifier.The visualized target-state partition distinguishes forward and backward classifier supervision.Source: Figure 2 · p. 4
  • The encoder is trained through the Q-value loss rather than the policy loss.Source: C. Improving the Learning Efficiency · p. 4

D. Putting it Together: MEDAL++

MEDAL++ combines forward and backward policies, classifier-derived rewards, alternating interaction, expert data, and behavior-cloning-regularized updates for autonomous visuomotor learning.

D. Putting it Together: MEDAL++

  • MEDAL++ trains a forward policy to solve the task and a backward policy to undo it toward the expert state distribution.The two policies alternate during continued interaction with the environment.Source: D. Putting it Together: MEDAL++ · p. 4
  • Forward and backward classifiers distinguish policy-visited states from target expert states, with their positive target sets defining the main difference between the policies.Forward positives are the last K states of forward demonstrations; backward positives are earlier forward states and optionally the last K backward-demonstration states.Source: D. Putting it Together: MEDAL++ · p. 4
  • Training alternates forward and backward control after fixed interaction intervals, repeating the cycle throughout autonomous learning.Real-world execution also permits intermittent human intervention before returning control to the forward policy.Source: D. Putting it Together: MEDAL++ · pp. 4–5
  • Figure 3 summarizes classifier discrimination, mixed expert and self-collected transitions, and behavior-cloning regularization.Source: Figure 3 · p. 5
  • The classifier is trained on online and expert-state batches, while Q-networks and the encoder use mixed replay and expert transitions.Policy updates use the behavior-cloning-regularized objective, and Q-networks are updated multiple times per environment step.Source: D. Putting it Together: MEDAL++ · p. 5

V. EXPERIMENTS

The experiments test whether MEDAL++ can provide practical self-improvement from high-dimensional observations in simulation and real-robot manipulation.

V. EXPERIMENTS

  • The evaluation covers pixel-based EARL benchmarking, four real-robot manipulation tasks, and component ablations.The real-robot tasks primarily involve soft-body objects, including hanging cloth on a hook and covering a bowl with cloth.Source: §V · p. 5

A. Benchmarking MEDAL++ on EARL

MEDAL++ is evaluated on non-episodic, vision-based EARL tasks requiring robots to learn both task completion and task reversal.

A. Benchmarking MEDAL++ on EARL

  • The benchmark uses 84×84 RGB observations, sparse rewards, and resets only every 25,000 interaction steps.The tasks require learning both how to perform and undo the manipulation behavior in a non-episodic setting.Source: A. Benchmarking MEDAL++ on EARL · pp. 5–6
  • R3L’s novelty-seeking controller can drift toward states farther from the goal in larger environments, slowing improvement from the initial-state distribution.Source: A. Benchmarking MEDAL++ on EARL · p. 6
  • MEDAL++ outperforms the compared methods on both Sawyer environments and is comparable to MEDAL on tabletop organization.Figure 4 reports evaluation performance against MEDAL, R3L, oracle RL, naïve RL, and behavior cloning.Source: A. Benchmarking MEDAL++ on EARL · p. 6, Figure 4 · p. 6
  • Behavior cloning reaches 45% success on tabletop organization and 70% and 80% on peg insertion and door closing, respectively.BC regularization can improve learning efficiency but may reduce final performance when the behavior-cloning policy has poor success.Source: A. Benchmarking MEDAL++ on EARL · p. 6

B. Real Robot Evaluations

MEDAL++ was evaluated on four real-robot manipulation tasks using visual and proprioceptive inputs, with autonomous forward-and-backward practice improving performance over behavior cloning.

Training setup

  • The system used two camera views and chained forward and backward policies so the robot could practice autonomously by inserting and then resetting the peg.Rewards were inferred from discriminator-based similarity to target states.Source: B. Real Robot Evaluations · p. 6, Figure 5 · p. 7

Real-robot results

  • 30-70%: MEDAL++ improved success rates over behavior cloning across four real-robot manipulation tasks evaluated over 50 trials.The tasks were cube grasping, cloth hanging, bowl covering, and soft peg insertion.Source: B. Real Robot Evaluations · p. 7, Figure 6 · p. 8

Cube grasping

  • 74%: autonomous training improved cube-grasping performance on out-of-distribution initial states, compared with 15% improvement on in-distribution states.The authors attribute this partly to practice from diverse states beyond the demonstrations.Source: B. Real Robot Evaluations · p. 7

Cloth on the hook

  • 36%: MEDAL++ improved cloth-on-hook success by learning to retry grasping rather than following a memorized trajectory.Autonomous improvement addressed failures involving grasping, drift, and recovery.Source: B. Real Robot Evaluations · p. 7

Bowl covering

  • 34%: MEDAL++ improved bowl-covering performance over behavior cloning, especially for grasping and retrying.Partial bowl coverage remained a failure mode despite better grasp planning.Source: B. Real Robot Evaluations · p. 7

Peg insertion

  • 48%: MEDAL++ improved soft peg-insertion performance over behavior cloning by repeatedly retrying imprecise insertions.Failures remained when the gripper was incorrectly oriented before grasping.Source: B. Real Robot Evaluations · p. 8

C. Ablations

Ablations show that ensemble Q-values and demonstration-focused training components contribute to MEDAL++ performance, particularly when rewards are learned rather than given.

Component ablations

  • BC regularization and oversampled expert transitions made the larger contribution to learning efficiency and final performance.The ablation also found a positive contribution from the ensemble of Q-value functions.Source: C. Ablations · p. 8, Figure 7 · p. 8

Learned versus true rewards

  • With learned rewards, BC regularization and expert-transition oversampling became more important because the learned reward signal was noisier.With true rewards, these components could instead hurt final performance.Source: C. Ablations · p. 8

VI. DISCUSSION

MEDAL++ enables autonomous improvement from visual observations and a small demonstration set, while real-world training remains constrained by slow data collection and occasional resets.

Discussion

  • MEDAL++ learns autonomously from high-dimensional images without engineered rewards or repeated human environment resets during normal training.It uses forward and backward demonstrations to practice the task and improve beyond behavior cloning.Source: §VI · pp. 8–9

Limitations and future work

  • Real-robot data collection occurs at approximately 3.5 Hz despite 10 Hz control, limiting training speed and learning efficiency.Asynchronous collection and training are proposed as ways to increase collected data.Source: §VI · p. 9
  • Intermittent human interventions may still be important because poor exploration can leave the robot stuck in a state and reduce replay-buffer diversity.Better exploration, offline pretraining, or more stable optimization could reduce these interventions.Source: §VI · p. 9

APPENDIX

The appendix describes the visual encoder, actor-critic architecture, discriminators, and training schedule used to implement MEDAL++ in simulation and on real robots.

A. Implementation Details and Practical Tips

  • The visual encoder uses four convolutional layers, produces a 50-dimensional normalized feature, and fuses visual features with end-effector position.For real robots, first-person and third-person views are concatenated before encoding.Source: A. Implementation Details and… · p. 12
  • Actor and critic networks use four fully connected layers with 1024 ReLU units, while the actor constrains actions to [−1, 1] and critics use an ensemble of 10.The actor parameterizes a Gaussian action distribution.Source: A. Implementation Details and… · p. 12
  • Forward and backward discriminators use shallower visual encoders with fully connected networks, mixup, and spectral normalization.Their visual embeddings are processed by two hidden layers with 256 ReLU units.Source: A. Implementation Details and… · p. 12
  • Training alternates 200 forward and 200 backward environment steps, oversamples demonstrations, and applies a decaying behavior-cloning regularizer.The real-robot update-to-data ratio is 1, and the regularizer decays from 1 to 0.1 over the first 50,000 steps.Source: A. Implementation Details and… · p. 12
  • Demonstrations are collected as paired forward and backward trajectories with randomized initial positions to increase state-space coverage.The wrist camera is supplied to both discriminators, while proprioception is additionally supplied to the MEDAL discriminator.Source: A. Implementation Details and… · p. 13

Related papers

Loading 2303.01488v1…