PaperlayerView exact revision on arXiv

Source-linked AI summary

RRL: Resnet as representation for Reinforcement Learning

Rutav Shah, Vikash Kumar

arXiv:2107.03380v3cs.RO

TL;DR

Uninstrumented settings make behavior learning from proprioceptive inputs challenging because observations are partial and high-dimensional. RRL combines pre-trained image-classification features with standard learning pipelines; on simulated high-dimensional dexterous tasks, it approaches state-based methods in performance and sample efficiency.

  • Problem

    Learning directly from visual inputs has high sample complexity, while uninstrumented environments require efficient learning without access to internal state.

    Source: §1 · p. 1

  • Method

    RRL uses pre-trained image-classification features as representations in standard imitation- and reinforcement-learning pipelines.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    On high-dimensional dexterous manipulation tasks where state-of-the-art methods struggle, RRL learns rich behaviors from visual inputs with performance and sample efficiency approaching state-based methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RRL demonstrates that Resnet features can be applied to complex ADROIT dexterous manipulation through a simple combination of representation learning, imitation learning, and reinforcement learning.

    Source: §6 · p. 9

    The paper does not investigate data mismatch between human-centric, primarily static image datasets used to train Resnet and robots' morphology, operation, and need for dynamic interactive features.

    Source: §6 · p. 9

Abstract

from arXiv · show

The ability to autonomously learn behaviors via direct interactions in uninstrumented environments can lead to generalist robots capable of enhancing productivity or providing care in unstructured settings like homes. Such uninstrumented settings warrant operations only using the robot's proprioceptive sensor such as onboard cameras, joint encoders, etc which can be challenging for policy learning owing to the high dimensionality and partial observability issues. We propose RRL: Resnet as representation for Reinforcement Learning -- a straightforward yet effective approach that can learn complex behaviors directly from proprioceptive inputs. RRL fuses features extracted from pre-trained Resnet into the standard reinforcement learning pipeline and delivers results comparable to learning directly from the state. In a simulated dexterous manipulation benchmark, where the state of the art methods fail to make significant progress, RRL delivers contact rich behaviors. The appeal of RRL lies in its simplicity in bringing together progress from the fields of Representation Learning, Imitation Learning, and Reinforcement Learning. Its effectiveness in learning behaviors directly from visual inputs with performance and sample efficiency matching learning directly from the state, even in complex high dimensional domains, is far from obvious.

1. Introduction

Learning from proprioceptive inputs in uninstrumented settings is difficult because observations are partial, high-dimensional, and contaminated by irrelevant detail. RRL uses broad-pretrained image-classification features in standard learning pipelines and reports robust, sample-efficient behavior learning, with performance and sample efficiency approaching state-based methods on high-dimensional dexterous tasks.

1. Introduction

  • Uninstrumented environments require decisions from proprioceptive sensors under partial observability, while high-dimensional inputs, noise, lighting, and clutter make policy learning challenging.Task-specific representations may require domain expertise or additional environment samples and can generalize poorly across tasks.Source: §1 · pp. 1–2
  • The method's premise is that representations trained on a sufficiently broad distribution can remain effective across distributions induced by task policies.The paper identifies generalization, robustness to visual distractors, and efficient policy learning from proprioceptive inputs as properties of image-classification features.Source: §1 · p. 2
  • The paper presents RRL as a simple approach bringing together representation learning, imitation learning, and reinforcement learning.Figure 2 visualizes Resnet Layer 4 with Grad-CAM and Guided Backpropagation and is described as showing relevant features in task images despite distribution shift.Source: §1 · p. 2, Figure 2 · p. 2
  • RRL uses pre-trained Resnet34 image-classification features as representations in standard RL pipelines, with minimal changes to many state-based RL and IL algorithms.The encoder is pre-trained on a broad distribution of real-world images and then frozen; Figure 1 depicts this design.Source: Figure 1 · p. 1, §1 · p. 2
  • RRL learns rich behaviors directly from visual inputs on high-dimensional dexterous manipulation tasks, approaching state-based methods in performance and sample efficiency where state-of-the-art methods struggle.The paper also reports a performance gap between RRL and state-of-the-art approaches on both simple low-dimensional and more realistic high-dimensional tasks.Source: §1 · p. 2

2. Related Work

Related work covers pixel-to-action learning, task-specific supervised features, unsupervised representations, and learning from demonstrations. The paper positions RRL's fixed, broad-pretrained representations against task-specific representations and compares its demonstration-based approach with DAPG and FERM.

2.1. Learning without explicit representation

  • End-to-end pixel-to-action methods have succeeded in several settings, but their high-dimensional inputs require many samples, while explicit representations can improve learning efficiency.The section contrasts these approaches with methods that use learned features as inputs to policy learning.Source: §2.1 · p. 3

2.2. Learning with supervised representations

  • Supervised keypoint methods support rich-scene and dexterous manipulation, but their task-specific features and expert supervision make transfer across tasks and scaling to task diversity harder.RRL is presented as using a single task-agnostic representation with better generalization capability.Source: §2.2 · p. 3

2.3. Learning with unsupervised representations

  • Unsupervised methods use approaches including contrastive learning, variational inference, and image augmentation, but often learn brittle task-specific features and face non-stationarity from distribution mismatch.The paper contrasts these with RRL's fixed representations trained on a wider-support distribution and describes them as robust to noise and irrelevant visual information.Source: §2.3 · p. 3

2.4. Learning with representations and demonstrations

  • DAPG combines imitation and reinforcement objectives in a state-based natural-gradient method, while RRL extends it to proprioceptive inputs with performance and sample efficiency comparable to state-DAPG.The paper also reports that RRL is more stable and robust to distractors, and convincingly outperforms off-policy FERM in its experiments.Source: §2.4 · p. 3

3. Background

The background frames policy learning as maximizing discounted reward in an MDP, then introduces policy gradients, demonstration-augmented learning, and compact representations as tools for complex tasks.

3.1. Preliminaries: MDP

  • An MDP specifies states, actions, rewards, transitions, initial-state distribution, and discount factor; the objective is a stochastic policy maximizing expected discounted return.Source: §3.1 · p. 4

3.2. Policy Gradient

  • Policy gradients optimize policy parameters by estimating the gradient of expected return, and the paper builds on Natural Policy Gradient for stability on complex problems.Source: §3.2 · p. 4

3.3. Demo Augmented Policy Gradient

  • DAPG combines task and imitation objectives, first behavior-cloning from 25 demonstrations to inform policy initialization and reduce early exploration demands.Source: §3.3 · pp. 4–5
  • The DAPG imitation objective decays over time, allowing the agent to learn behaviors that surpass the expert.Source: §3.3 · p. 5

3.4. Representation Learning

  • Representation learning maps high-dimensional inputs to compact features; the paper positions it as a substitute for state information when scaling DAPG to visual inputs.Source: §3.4 · p. 5

4. RRL: Resnet as Representation for RL

RRL replaces unavailable environment state with visual features and joint-encoder information, using a fixed pretrained representation within a policy-learning loop initialized from demonstrations.

4. RRL: Resnet as Representation for RL

  • The algorithm initializes from 25 human demonstrations using behavior cloning, then gathers sensor observations and updates the policy and value function until performance is satisfactory.Source: §4 · p. 5
  • RRL freezes a feature extractor pretrained on broad real-world scenarios, covering policy-induced distributions while avoiding representation non-stationarity during policy learning.Source: §4 · p. 5
  • The policy receives visual features concatenated with joint-encoder readings instead of exact state, and RRL builds on Natural Policy Gradients.Source: §4 · p. 5

5. Experimental Evaluations

The evaluation tests RRL on visually rich, contact-heavy ADROIT manipulation tasks; RRL performs well across tasks and often approaches the state-based DAPG oracle.

5. Experimental Evaluations

  • The evaluation asks whether pretrained representations support complex proprioceptive learning, how performance and efficiency compare, and how representation and design choices affect behavior.Source: §5 · p. 6

5.2. Implementation Details

  • RRL uses a frozen Resnet-34 pretrained on ImageNet, with its last layer removed to provide 512-dimensional features; results average three random seeds.Source: §5.2 · p. 6

5.1. Tasks

5.3. Results

  • RRL performs well across the ADROIT tasks and often approaches the strongest oracle, DAPG(State), while relocate is the hardest task.Source: §5.3 · p. 6
  • RRL outperforms FERM and is approximately five times more compute-efficient; FERM progresses slowly on pen and fails on relocate.Source: §5.3 · p. 6

5.4. Effects of Visual Distractors

RRL policies remain robust when visual distractors are introduced, while task-specific FERM features are more brittle. Figure 5 reports that RRL consistently outperforms FERM across the tested variations.

5.4. Effects of Visual Distractors

  • RRL policies show resilience to visual distractors, while task-specific features learned by FERM lead to larger performance degradation.The reported benefits also include robustness to scene clutter.Source: §5.4 · p. 6
  • RRL's pretrained features are associated with improved sample and time complexity as well as robustness to visual distractors.The passage attributes these properties to the resilience, robustness, and versatility of ResNet features.Source: §5.4 · p. 6
  • RRL consistently performs better than FERM across the tested lighting-condition and object-color variations.These comparisons are shown in Figure 5 center and right.Source: Figure 5 · p. 7

5.5. Effect of Representation

The representation comparisons favor broadly pretrained features over task-specific representations and show that effective feature extractors need not be ResNets. ResNet34 balances performance and computational overhead among the tested ResNet variants.

5.5. Effect of Representation

  • MobileNetV2, ShuffleNet, and a hierarchical VAE show little performance degradation as ImageNet-pretrained RRL feature extractors, supporting the role of pretraining data breadth over architecture choice.The figure caption likewise notes that sufficiently wide pretraining distributions can yield effective feature extractors.Source: §5.5 · p. 7, Figure 6 · p. 7
  • RRL using ResNet34 significantly outperforms task-specific VAE features, with both representations using a 512-dimensional latent space.The comparison is described as fair on latent dimension, though the model sizes differ.Source: §5.5 · p. 7
  • Among the tested ResNet variants, ResNet34 balances representation capacity and computational overhead, while NPG(ResNet34) shows that features alone are not enough to solve the suite.ResNet34 features without demonstration bootstrapping are the stated NPG comparison.Source: §5.5 · p. 8
  • RRL(VAE) is the cheapest compared method but has low performance, whereas RRL balances compute and efficiency.The comparison describes the VAE as using a narrower task-specific dataset than the broad distribution behind ResNet features.Source: §5.5 · p. 8

5.6. Effects of proprioception choices and sensor noise

RRL remains performant under noisy proprioception and when using only visual inputs. The reported performance changes are slight to none in both settings.

5.6. Effects of proprioception choices and sensor noise

  • The tested alternatives to standard proprioceptive signals are noisy joint-encoder information and vision-only input.Figure 8 identifies these variants as RRL(Noise) and RRL(Vision).Source: Figure 8 · p. 8, §5.6 · p. 8
  • RRL remains performant with 2% noise in joint-encoder information, with slight to no performance degradation.This condition is labeled RRL(Noise).Source: §5.6 · p. 8
  • RRL remains performant when only visual inputs are used as proprioceptive signals, with slight to no performance degradation.This condition is labeled RRL(Vision).Source: §5.6 · p. 8, Figure 8 · p. 8

5.7. Ablations and Analysis of Design Decisions

The design analyses find competitive performance with sparse rewards and resilience across policy sizes. They also report that ResNet34 performs better than ResNet18 and ResNet50 in the tested feature comparisons.

5.7. Ablations and Analysis of Design Decisions

  • RRL performance is described as stable across a wide range of policy-network sizes.Figure 9 right presents the policy-size comparison.Source: Figure 9 · p. 8
  • ResNet34 performs better than ResNet18, while ResNet50 degrades performance, likely from less useful features and increased samples needed to train the larger policy network.ResNet34 is more computationally demanding than ResNet18 but has improved representational capacity and feature expressivity.Source: §5.7 · p. 8
  • RRL with sparse rewards delivers competitive performance to a variant using well-shaped dense rewards, while using demonstrations to curb exploration challenges and lower domain expertise requirements.The passage states that sparse rewards indicate task completion.Source: §5.7 · p. 8

5.8. Rethinking benchmarking for visual RL

The paper argues that methods successful on simple visual-control benchmarks do not necessarily scale to realistic dexterous manipulation. RRL approaches state-based performance on Adroit, while the DMControl results and fixed-encoder comparison expose benchmark and representation limits.

5.8. Rethinking benchmarking for visual RL

  • On DMControl, RAD outperforms the baselines, while RRL performs worse at 100K and 500K environmental steps.The table caption attributes this pattern to the relative ease of learning task-specific representations in simple tasks.Source: Table 1 · p. 9
  • On Adroit, RRL delivers state-comparable results while FERM struggles, whereas RRL struggles on DMControl.The authors interpret this contrast as evidence of a large domain gap between DMControl and the real world.Source: §5.8 · pp. 8–9
  • A fixed RAD encoder trained on Cartpole degrades performance on every tested task except Cartpole, indicating narrow task-specific representations.The authors say task-specific representation learning becomes harder as tasks and scenes grow more complex.Source: §5.8 · p. 9

6. Strengths, Limitations & Opportunities

RRL combines pretrained Resnet features with reinforcement and imitation learning to support behavior learning from high-dimensional proprioceptive signals. The paper highlights broad applicability and stability, while acknowledging possible data mismatch and outlining shared-representation opportunities.

Strengths

  • RRL applies Resnet features to imitation and on-policy reinforcement learning, approaching state-based performance on complex ADROIT dexterous manipulation tasks.The authors present it as an instantiation of a simple insight that could apply to almost any RL or IL algorithm learning from high-dimensional proprioceptive signals.Source: §6 · p. 9
  • Frozen Resnet features pretrained on thousands of real-world images are described as more general and robust than features learned from task-distribution samples.The authors say frozen features also avoid non-stationarity from jointly optimizing representation and reinforcement objectives, with sample and compute gains from not training feature extractors.Source: §6 · p. 9

Limitations

  • The work does not investigate possible mismatch between human-centric, mostly static Resnet training data and robots’ morphology and dynamic interactions.The authors identify this as a limitation of the evidence for pretrained features.Source: §6 · p. 9

Opportunities

  • Using one pretrained representation across diverse tasks could support a shared representational space for robotics data.The authors identify possible downstream opportunities in feature compression, hardware acceleration, and low-latency, low-bandwidth transmission.Source: §6 · p. 9

7. Appendix

The appendix reports an MJRL comparison, implementation settings, alternative encoders, and evaluation procedures. It also describes compute-cost accounting and the project resources.

7.1. Project’s webpage; 7.2. Overview of all methods used in baselines and ablations

  • The project webpage provides video results and code, while Table 7.2 summarizes the environmental settings and feature extractors used across methods.Source: §7.1 · p. 13, §7.2 · p. 13

7.3. RRL(Ours)

  • RRL uses the same parameters across the manipulation tasks unless specified, with sparse rewards, 25 expert demonstrations, and DAPG-provided parameters.The appendix reports 200 trajectories per iteration for Hammer, Door, and Relocate, and 400 for Pen.Source: §7.3 · pp. 13–14, Table 2 · p. 13

7.4. Results on MJRL Environment

  • On Peg Insertion, RRL matches DAPG(State) and significantly outperforms FERM; on Reacher, DAPG(State) and FERM perform well while RRL initially struggles.The MJRL tasks use shaped rewards and 200 expert demonstrations; Figure 10 shows the results.Source: §7.4 · p. 14, Figure 10 · p. 14

7.5. Other variations of RRL

  • The RRL variations use frozen ImageNet-pretrained ShuffleNet or MobileNet features, or intermediate features from a pretrained hierarchical VAE.The respective feature dimensions are 1024, 1280, and 512.Source: §7.5 · p. 14

7.6. DMControl Experiment Details

  • The DMControl setup uses a fixed Resnet-34 extractor with three past output features for SAC+RRL and reports averages over three seeds for the listed comparisons.The appendix also summarizes the SAC hyperparameters and task-specific frame skips in Tables 3 and 4.Source: §7.6 · p. 14, Table 3 · p. 15, Table 4 · p. 15

7.7. RRL(VAE)

  • The VAE encoder is trained on one million 64 × 64 images spanning expert, noisy-expert, and random-action data, then frozen for feature extraction.It uses a reconstruction objective for 10 epochs and a latent size of 512; Figure 11 shows reconstructions for Hammer and Door.Source: §7.7 · pp. 14–15, Figure 11 · p. 16

7.8. Visual Distractor Evaluation details

  • Visual-distractor evaluation freezes parameters and averages performance over 75 rollouts under randomized lighting, object colors, and added objects.Figure 12 depicts the original scene and distractor types; added objects vary in color, position, size, and geometry.Source: §7.8 · p. 15, Figure 12 · p. 16

7.9. Compute Cost calculation

  • At equal sample complexity of 4 million steps, FERM is almost 5x as compute intensive as RRL(Ours), while RRL(Resnet-34) balances compute cost and performance.The accounting excludes simulation cost and includes VAE pretraining; no parallel processing was used.Source: §7.9 · p. 16
Loading 2107.03380v3…