PaperlayerView exact revision on arXiv

Source-linked AI summary

Echo: Scaling Visual Reinforcement Learning with Off-policy Experience

Gagan Khandate

arXiv:2610.37865v1cs.ROcs.AI

TL;DR

Visual RL is costly to scale because rendering and visual-policy inference add computation, and teacher–student distillation can introduce an imitation gap. Echo uses inverse distillation and shared-critic off-policy learning to use unrendered experience; preliminary evaluations report faster learning and higher peak returns than SDPG.

  • Problem

    Visual-policy rendering and inference make RL expensive to scale, while privileged teacher–student distillation can create an imitation gap.

    Source: §1 · p. 2

  • Method

    Echo trains a state-based imitator to track the visual policy and collect unrendered experience for off-policy value learning through a shared privileged critic.

    Source: §1 · pp. 2–3

  • Results

    Compared with SDPG, Echo reduces time to reference returns by averages of 60% across three locomotion tasks and 56% across both hand tasks, with 62% and 12% higher peak mean returns, respectively.

    Source: §7 · p. 12

  • Takeaways & Limitations

    These preliminary findings provide initial evidence that off-policy experience can be scaled without proportionally increasing visual computation.

    Source: §7 · p. 12

    The experiments do not isolate auxiliary experience from architectural and training differences, and the imitator may not reproduce the visual policy across all auxiliary-rollout states.

    Source: §6 · p. 11

Abstract

from arXiv · show

Scaling visual reinforcement learning (RL) with GPU-accelerated simulation is computationally expensive due to rendering and visual-policy inference costs. Consequently, two-stage teacher--student distillation is commonly used to train visual policies, but introduces an imitation gap. In this preliminary study, we introduce \echomethod{}, \textbf{a single-stage off-policy RL framework that scales experience collection without proportionally increasing visual computation}. Through \textbf{inverse distillation}, a state-based auxiliary policy imitates the evolving visual policy and collects inexpensive, unrendered experience. Echo leverages this experience for off-policy value learning, directly improving the visual policy through a shared privileged critic. We instantiate \echomethod{} with FastTD3 and evaluate it on three visual control tasks and two dexterous manipulation tasks in Genesis. Relative to \sdpgmethod{} under the evaluated configurations, Echo reduces time to reference returns by an average of \textbf{60\% across the three locomotion tasks} and \textbf{56\% across both hand tasks}, while improving peak mean returns by \textbf{62\% and 12\%}, respectively. Both hand tasks also achieve higher mean returns at matched simulator-transition budgets. These initial results suggest that scaling off-policy experience is a promising approach to efficient direct visual RL.

1 Introduction

Visual RL scales costly rendering and policy inference, while teacher–student distillation can create an imitation gap. Echo uses inverse distillation and shared-critic off-policy learning to scale unrendered experience; preliminary evaluations report faster learning and higher peak returns than SDPG.

1 Introduction

  • Visual RL remains more expensive to scale than state-based RL because rendering observations and running visual-policy inference add computational costs to parallel simulation.Those costs can grow substantially with model size.Source: §1 · p. 2
  • Teacher–student distillation uses a privileged state-based teacher to supervise a visual student, but information asymmetry and student-error distribution shift can create an imitation gap.Direct visual-policy optimization avoids that mismatch but restores the costs of visual experience collection.Source: §1 · p. 2
  • Direct visual RL methods motivate Echo: SDPG estimates policy improvements from unrendered perturbation rollouts, while off-policy methods reuse collected experience through repeated value and policy updates.The introduction notes that SDPG's on-policy formulation limits reuse of prior experience.Source: §1 · p. 2
  • Echo reverses conventional distillation: a state-based imitator tracks the evolving visual policy and collects inexpensive, unrendered experience that directly contributes to visual-policy improvement.The approach is presented as a preliminary, single-stage off-policy RL framework.Source: §1 · p. 2
  • The shared privileged critic learns from rendered and unrendered replay, while critic-guided first-order updates improve the visual policy using paired state–observation samples.The imitator also provides continuation actions for temporal-difference critic learning.Source: §1 · pp. 2–3
  • Echo is introduced to decouple experience-collection scale from rendering and visual-policy inference, enabling off-policy value learning from inexpensive unrendered transitions.The study instantiates Echo with FastTD3 on three visual control tasks and two dexterous manipulation tasks in Genesis.Source: §1 · p. 3
  • Echo reduces time to reference returns by an average of 60% across three locomotion tasks and 56% across both hand tasks, with 62% and 12% higher peak mean returns, respectively, than SDPG.Both hand tasks also achieve higher mean returns at matched simulator-transition budgets.Source: §1 · p. 3

2 Related Work

Related work covers privileged-policy distillation, scalable off-policy RL, and methods for reducing visual-RL computation. Echo combines inverse distillation with a shared critic trained on rendered and unrendered experience.

2 Related Work

  • Unlike privileged-teacher approaches, Echo trains a state-based auxiliary policy to imitate the evolving visual policy and collect inexpensive experience for off-policy value learning.Related approaches include feature-based auxiliary policies for vision-based control and shared-experience training of state-based and visual agents.Source: §2 · p. 3
  • FastTD3 and FlashSAC scale off-policy robotic RL using parallel collection, replay buffers, and efficient critic optimization, but direct visual-RL experience collection adds rendering and inference costs.Echo collects additional state-based experience for critic learning without visual observations at every simulation step.Source: §2 · p. 3
  • Squint reduces visual computation with low-resolution observations, while SDPG uses unrendered perturbation rollouts; Echo instead trains a shared critic on rendered and unrendered transitions.In Figure 2, gray arrows denote experience flow, and dashed purple denotes EMA continuation actions supplied without successor images.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 Problem Formulation

The problem formulation distinguishes inexpensive privileged simulator state from potentially costly observations available to the deployed policy. The visual policy is trained using observations, while auxiliary training components can use state.

3 Problem Formulation

  • The control problem is an MDP with state and continuous-action spaces, transition dynamics, reward, an initial-state distribution, and discount factor.The discount factor γ lies in [0, 1).Source: §3 · p. 4
  • Privileged state is accessible inexpensively during training, whereas observations may require costly rendering or sensor simulation and are the deployed policy's only inputs.Observations include camera images and other sensory inputs, together with permitted proprioception.Source: §3 · p. 4
  • Training uses privileged state in an auxiliary policy and differentiable state–action critic, but deploys only the visual policy.The notation sg[·] denotes stopped gradient, and bars denote slowly updated target parameters.Source: §3 · p. 4

4 Method

Echo pairs inverse distillation with off-policy actor–critic learning: a state-based imitator collects unrendered experience, and a shared privileged critic uses both streams to improve the visual policy directly.

4 Method

  • A shared privileged critic learns from rendered and unrendered experience, letting Echo improve the visual policy without rendering or evaluating it for every transition.The visual actor receives critic gradients using the smaller set of rendered state–observation pairs.Source: §4 · p. 4, §4.2 · pp. 6–7

Algorithm 1 Echo

  • Algorithm 1 splits environments between the visual policy and imitator, stores both streams in shared replay, and deploys only the visual policy.A separate rendered replay buffer retains paired state–observation samples for visual-policy updates.Source: Algorithm 1 Echo · p. 5, §4.1 · p. 5

4.1 Inverse Distillation for Rendering-Free Experience Collection

  • The state-based imitator tracks the evolving visual policy’s noise-free actions, enabling additional experience collection without rendering or visual-policy inference.It learns from fresh rendered state–observation pairs, while an EMA copy supplies continuation actions for critic learning.Source: §4.1 · pp. 5–6

4.2 Shared Critic Learning and Visual Policy Improvement

  • Critic targets for transitions from either stream use privileged successor states and imitator continuation actions, avoiding rendered successor images.The implementation uses FastTD3-style twin distributional critics, clipped double-Q learning, target-policy smoothing, and EMA target networks.Source: §4.2 · p. 6
  • The visual actor is optimized directly through first-order gradients of the learned action-value function, using paired states and observations from nominal replay.Unlike sampled trajectory perturbations, this update uses the shared critic’s differentiable action-value signal; its quality depends on critic accuracy.Source: §4.2 · p. 7

5 Experimental Results

The experiments compare Echo and SDPG on visual locomotion and dexterous manipulation, then examine auxiliary-policy averaging and learning dynamics. Echo shows faster learning in reported comparisons, while EMA effects vary by task and the imitator remains imperfect.

5 Experimental Results

  • The experiments ask how Echo compares with SDPG, how EMA affects learning, and how well the imitator tracks the visual policy and aligns critic gradients.The comparison concerns complete implementations rather than matched architectures or hyperparameters.Source: §5 · p. 7, §5.1 · p. 7

5.3 Training Performance

  • Echo reaches reference returns earlier in visual locomotion and dexterous manipulation; it also achieves higher mean returns at matched simulator-transition counts on both hand tasks.On Hopper, Echo reaches the SDPG reference return in approximately 21 minutes versus 60 minutes; on Allegro Hand, it takes approximately 96 versus 180 minutes and ends near 700 versus 600. On Shadow Hand, both methods end near 1100, with Echo reaching comparable performance earlier.Source: Figure 3 · p. 8, §5.3 · pp. 8–9, Figure 4 · p. 9

5.1 Compared Methods and Evaluation Protocol

  • Echo is evaluated against SDPG in three-seed runs, with returns measured against wall-clock time and total simulator transitions, including auxiliary interactions.Echo uses FastTD3 extended with a state-based imitator; the implementations are not matched in architecture or hyperparameters.Source: §5.1 · pp. 7–8

5.2 Tasks

  • The evaluation covers Hopper, Walker, and Ant locomotion plus Allegro Hand and Shadow Hand in-hand cube reorientation, with task-specific training budgets.Budgets range from one hour for Hopper to four hours for Ant; each hand task receives three hours.Source: §5.2 · p. 8

5.4 Design Choices and Ablations

  • EMA increases final mean returns on Hopper and Ant but lowers Walker's; its effects on performance and across-seed variation are task-dependent.Compared with no averaging, EMA raises final mean returns by 210 on Hopper and 369 on Ant, while removing EMA raises Walker's final mean return by 687.Source: §5.4 · p. 9

5.5 Learning Dynamics

  • Imitation errors remain nonzero, and positive critic action-gradient cosine similarity persists after the initial transient across all five tasks.The diagnostics do not establish accurate tracking on auxiliary-rollout states or critic accuracy.Source: §5.5 · p. 10

6 Limitations and Future Work

The experiments compare complete implementations and do not isolate the contribution of auxiliary experience. The authors also identify imperfect imitation as a limitation and present several extensions as directions for future experimental investigation.

6 Limitations and Future Work

  • The Echo–SDPG comparison does not isolate auxiliary experience from differences in architectures and training configurations.The authors identify a matched rendered-only FastTD3 baseline as useful for quantifying that contribution.Source: §6 · p. 11
  • The state-based imitator may not reproduce the visual policy across all auxiliary-rollout states, and diagnostics do not establish critic accuracy or accurate tracking there.The reported gradient alignment is positive despite imperfect imitation, but does not resolve these accuracy questions.Source: §6 · p. 11
  • Tactile sensing and larger VLA or WAM architectures are proposed extensions whose potential remains to be investigated experimentally.The authors also suggest Echo may help reduce the imitation gap associated with privileged teacher–student learning.Source: §6 · p. 11

7 Conclusion

Echo uses inverse distillation to scale direct visual RL with inexpensive unrendered experience, and preliminary comparisons with SDPG show faster attainment of reference returns and higher peak mean returns.

7 Conclusion

  • Echo trains a state-based auxiliary policy to imitate the evolving visual policy, collecting unrendered experience for off-policy learning through a shared privileged critic.The critic directly improves the visual policy through first-order gradients, decoupling experience-collection scale from visual rendering and inference costs.Source: §7 · p. 12
  • 60% lower time to reference returns across three locomotion tasks and 56% lower across both hand tasks, compared with SDPG.Peak mean returns were 62% higher for locomotion tasks and 12% higher for hand tasks, respectively.Source: §7 · p. 12
  • Both hand tasks achieved higher mean returns than SDPG at matched simulator-transition budgets.The authors describe these results as preliminary evidence for scaling off-policy experience without proportionally increasing visual computation.Source: §7 · p. 12

A Privileged Critic Interpretation

Averaging privileged-critic gradients over paired state–observation samples estimates the gradient of an induced observation-level objective under stated assumptions. This interpretation does not guarantee accurate critic values, unbiased true-return gradients, or accurate continuation actions throughout replay.

A Privileged Critic Interpretation

  • With the replay distribution and critic fixed, and differentiation interchangeable with conditional expectation, averaged critic gradients estimate the induced observation-level objective’s gradient.The interpretation explains how privileged state can guide an observation-based policy without becoming a deployment input.Source: A Privileged Critic Interpretation · p. 15
  • The gradient interpretation does not establish critic accuracy or unbiased estimates of the true return gradient.It describes gradients of the fitted critic on the rendered replay distribution.Source: A Privileged Critic Interpretation · p. 15
  • Auxiliary continuation actions can produce TD targets that differ from those using the visual policy’s actions, because the imitator need not reproduce them exactly.The imitator is supervised on rendered visual-policy experience, while critic replay also includes auxiliary-policy states; low nominal-state imitation error does not ensure accurate continuation actions throughout replay.Source: A Privileged Critic Interpretation · p. 15
  • In an idealized memoryless setting, a uniform continuation-action error bound plus suitable regularity assumptions can bound value discrepancy, but inverse distillation alone guarantees no such bound over full replay.Source: A Privileged Critic Interpretation · p. 15

B Gradient-Alignment Diagnostic

The diagnostic measures critic action-gradient alignment on rendered replay minibatches during training and reports gradient statistics across three seeds. Its scope is limited: the measurements do not establish critic accuracy or alignment on auxiliary-rollout states.

B Gradient-Alignment Diagnostic

  • Gradient alignment is measured every 1,000 visual actor updates on the corresponding rendered replay minibatch, with critic and policy parameters held fixed.Measurements are taken immediately before the optimizer step.Source: B Gradient-Alignment Diagnostic · p. 15
  • For paired privileged states and visual observations, the diagnostic uses either the online or EMA state-based imitator and the clipped twin-critic objective.Source: B Gradient-Alignment Diagnostic · p. 15
  • The diagnostic records minibatch means of gradient dot product, cosine similarity, gradient norms, valid-sample fraction, and positive-dot-product fraction.Cosine similarity is computed only when both gradient norms exceed 10^-8; all 536 records had a valid-sample fraction of one.Source: B Gradient-Alignment Diagnostic · p. 15
  • Figure 6 interpolates each seed’s minibatch statistics to 100 training-time points per task, with shaded bands showing sample standard deviation across three seeds.The bands are not confidence intervals over individual states.Source: B Gradient-Alignment Diagnostic · p. 15
  • Online and EMA gradient measurements come from the same EMA-enabled runs, so their similarity does not isolate EMA’s effect on learning performance.The diagnostics concern local fitted-critic gradients on rendered replay samples, not critic accuracy or gradient alignment on auxiliary-rollout states.Source: B Gradient-Alignment Diagnostic · p. 15

C Hyperparameters

Echo’s training hyperparameters and details are provided in Table 1.

C Hyperparameters

D Aggregate Performance Comparisons

The section defines how Echo and SDPG are compared using time to a reference return and peak mean-return gain, and identifies the task-level reporting dimensions.

D Aggregate Performance Comparisons

Loading 2610.37865v1…