PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward as Observation: Learning Reward-Based Policies for Rapid Adaptation

Morgan Byrd, Maks Sorokin, Robert Wright, Sehoon Ha

arXiv:2610.00729v1cs.LGcs.RO

TL;DR

Observation-based policies can struggle when a task’s observations change even though its underlying structure is preserved. The paper trains reward-and-action-history policies as transferable teachers; these policies achieve 66% to 95% of state-based performance across three environments and transfer zero-shot across drastic observation shifts.

  • Problem

    Deep RL policies are tightly coupled to their training observation spaces, while transfer may need to preserve task structure across arbitrarily different observations.

    Source: §I · p. 1

  • Method

    An LSTM reward-based policy processes reward-and-action histories, uses observation-based expert guidance, and serves as a zero-shot teacher for target-environment DAgger training.

    Source: §I · pp. 1–2, A. Reward-based MDP · p. 3

  • Results

    Across three environments, reward-based policies reach 66% to 95% of state-based performance and transfer zero-shot across drastic shifts, including 2D-to-3D rendering and Habitat-Sim navigation.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Reward estimation is substantially easier than state estimation under limited data, offering a practical advantage when target rewards are not directly available.

    Source: §I · p. 2, §VI · p. 6, D. Reward Estimation Succeeds Where… · p. 6

    Reward-based agents explore suboptimally: in 2D Pointmass, they need at least two information-gathering actions before localizing optimally, with overhead expected to worsen in higher-dimensional or more complex environments.

    Source: 4) Limitation: Suboptimal Exploration · p. 3

Abstract

from arXiv · show

This paper explores a reward-based policy to achieve zero-shot transfer between source and target environments with completely different observation spaces. While humans can demonstrate impressive adaptation capabilities, deep neural network policies often struggle to adapt to a new environment and require a considerable amount of samples for successful transfer. Instead, we propose a novel reward-based policy only conditioned on rewards and actions, enabling zero-shot adaptation to new environments with completely different observations. We discuss the challenges and feasibility of a reward-based policy and then propose a practical algorithm for training. We demonstrate that a reward policy can be trained within three different environments, Pointmass, Cartpole, and 2D Car Racing, and transferred to completely different observations, such as different color palettes or 3D rendering, or Stretch robot navigation in Habitat-Sim, in a zero-shot manner. We also demonstrate that a reward-based policy can further guide the training of an observation-based policy in the target environment.

I. INTRODUCTION

Observation shifts can leave task structure intact while defeating observation-bound policies; this paper trains reward-and-action-history policies as transferable teachers. Across three environments, the policies reach 66% to 95% of state-based performance and transfer zero-shot across drastic observation shifts.

I. INTRODUCTION

  • The transfer problem holds task structure fixed while allowing source and target observation spaces to differ arbitrarily, unlike observation-bound policies that may require retraining.The shared structure includes state, action, transition dynamics, and reward.Source: §I · p. 1
  • An LSTM policy processes reward-and-action history, receives guidance from an observation-based expert, and serves as a source-trained DAgger teacher in the target environment.The paper also proposes a practical training implementation for reward-based policies.Source: §I · pp. 1–2
  • Reward-based policies reach 66% to 95% of state-based performance across three environments and transfer zero-shot across 2D-to-3D rendering and Habitat-Sim Stretch navigation.The paper also reports substantially faster target-environment learning with a DAgger teacher and more tractable reward estimation than state estimation under limited data.Source: §I · pp. 1–2
  • Reward estimation is reported to be far more tractable than state estimation under limited data.The paper also presents reward-based policies as transferable teachers across observation shifts.Source: §I · p. 2
  • The paper defines a reward-based MDP and discusses its feasibility conditions and an optimality-gap bound.The contribution statement places the theoretical properties in the Appendix.Source: §I · p. 2

II. RELATED WORK

Prior transfer approaches include demonstrations, reusable representations, and policy distillation; this paper instead uses shared reward to avoid observation alignment. Its transfer setting fixes task structure while permitting arbitrary differences between observation spaces.

II. RELATED WORK

  • The reward-based teacher is closest to policy distillation but requires no observation alignment, unlike reward-conditioned policies that remain observation-based and use rewards to supervise suboptimal trajectories.The related-work discussion also contrasts demonstration-based transfer and representation-learning approaches.Source: §II · p. 2

III. PROBLEM DEFINITION: TRANSFER ACROSS DIFFERENT OBSERVATION SPACES

  • The source and target MDPs share state, action, transition, and reward structure, but their observation spaces may differ arbitrarily, so a source policy may not be directly reusable.The section frames the setting with POMDPs, whose policy maps observations to actions without access to the underlying state.Source: §III · p. 2
  • Table I reports Pointmass maximum reward as a percentage of optimal reward as a function of state-space dimension.The supplied caption gives the table's scope but not its cell values.Source: Table I · p. 3

IV. REWARD-BASED POLICY

The reward-based MDP replaces the original observation space with a history of reward/action pairs. Because source and target share reward and action spaces, a policy trained in this representation transfers without modification even when their observation spaces differ arbitrarily.

A. Reward-based MDP

  • The reward-based MDP replaces original observations with the most recent K reward/action pairs, and its policy maps that history to actions without access to original observations.K controls history length; effective control requires a dense reward signal.Source: A. Reward-based MDP · p. 3
  • Because the reward-based MDP is independent of the original observation space, source and target reward-based MDPs are identical when reward and action spaces match.A source policy can therefore be applied in the target without modification, unlike an observation-based policy when observation spaces differ arbitrarily.Source: A. Reward-based MDP · p. 3

B. Properties

Reward-only learning is harder because rewards provide limited state information, but reward/action history can support policy learning in some environments. Its feasibility depends on dimensionality and informative, dense rewards.

1) Difficulty: Limited Observability:

  • A scalar reward projects the generally D-dimensional state, making value and advantage estimation more difficult and reducing information available for exploration.The optimality gap is bounded linearly by reconstruction error, which is expected to grow exponentially with dimension.Source: 1) Difficulty: Limited Observability · p. 3

2) Feasibility: Low-dimensional Environments:

  • Takens’ theorem supports reconstructing a D-dimensional state from a scalar-observation history of length K ≥ 2D + 1 under a sufficiently smooth observation function.In Pointmass environments, reward changes following actions can help estimate position, but the time required grows nearly exponentially with dimension.Source: 2) Feasibility: Low-dimensional… · p. 3
  • 57% of maximum reward was achieved in the 6D environment with a fixed history length, as reward-based performance degraded sharply with increasing dimension.State-based policies maintained near-optimal performance across dimensions; reward-based policies remained feasible at least through 6D.Source: 2) Feasibility: Low-dimensional… · p. 3

3) Requirement: Dense Rewards:

  • Reward-based policy learning requires dense rewards; sparse goal-only feedback leaves most of the trajectory without useful information and makes learning infeasible in practice..Source: 3) Requirement: Dense Rewards · p. 3

4) Limitation: Suboptimal Exploration:

Reward-only policies incur an exploration cost because they must infer state from reward feedback. Training uses reward/action history and observation-based expert guidance, and the resulting policy can teach an observation-based target policy through DAgger.

4) Limitation: Suboptimal Exploration:

  • Even in 2D Pointmass, a reward-based agent needs at least two information-gathering actions before it can localize itself and act optimally.The overhead is inherent to the formulation and is expected to worsen in higher-dimensional or more complex environments.Source: 4) Limitation: Suboptimal Exploration · p. 3

C. Training Reward-Based Policies

  • An LSTM processes reward/action history for PPO policy and value networks, while a supervised guidance loss regresses actions toward a pretrained observation-based expert.The guidance loss is added to the PPO loss because temporal history alone is insufficient for effective learning in practice.Source: C. Training Reward-Based Policies · pp. 3–4, Figure 3 · p. 4

D. Training Observation-Based Policies in Target Environments

  • In target-environment DAgger, the reward-based policy labels actions on the observation-based policy’s trajectories, which then operates on target observations without rewards at inference.The reward-based policy is trained in the source environment and deployed directly in the target because reward and action spaces are shared.Source: D. Training Observation-Based Policies… · p. 4

V. RESULTS

Reward-based policies learn across Pointmass, Cartpole, and Car Racing, then transfer without retraining across observation changes ranging from palette swaps to 2D-to-3D rendering and Habitat-Sim robot navigation.

A. Reward-Based Policies Train Successfully Across Diverse Environments

  • Experiments span three environments selected with increasing task complexity, from a controlled Pointmass testbed to a maximally challenging observation-shift evaluation.Pointmass uses dense negative-exponential distance-to-goal reward; Cartpole uses dense, informative reward based on pole angle and regularizing terms; Car Racing uses a dense waypoint-distance and heading reward.Source: A. Reward-Based Policies Train… · p. 4
  • Reward-based policies reach 95%, 66%, and 70% of state-based performance in Pointmass, Cartpole, and Car Racing, respectively.State-based policies serve as the performance upper bound; the reward-based results suffice for the subsequent transfer experiments.Source: A. Reward-Based Policies Train… · p. 4, Figure 4 · p. 5

B. Reward-Based Policies Transfer Zero-Shot Under Drastic Observation Shifts

  • Palette-swapped observations yield nearly identical performance to the original environments, with no effect on reward-based policy behavior.The palette swap changes observation colors while keeping other environment aspects the same.Source: B. Reward-Based Policies Transfer… · pp. 4–5, Figure 5 · p. 5
  • A reward-based policy transfers directly from 2D Car Racing to matched-dynamics 3D rendering, producing driving behavior indistinguishable from the 2D expert reference.The 3D rendering preserves the source transition function to isolate the observation shift.Source: B. Reward-Based Policies Transfer… · p. 5
  • The same checkpoint succeeds without retraining in two photorealistic HM3D building scans while driving a Stretch robot.The scans use waypoint-based reward computed from pose, assumed available through localization during real deployments.Source: B. Reward-Based Policies Transfer… · p. 5

C. Reward-Based Policies Accelerate Observation-Based Policy Learning in the Target Environment via DAgger

The reward-based policy serves as a DAgger teacher in the 3D Car Racing target environment, where it substantially accelerates observation-based student learning.

C. Reward-Based Policies Accelerate Observation-Based Policy Learning in the Target Environment via DAgger

  • Learning the CNN-LSTM student from scratch converges slowly in the 3D target setting, even when viewed on a logarithmic timestep scale.The architecture processes pixels with a CNN, then passes features to an LSTM before the actor and critic networks.Source: C. Reward-Based Policies Accelerate… · p. 5
  • With the reward-based policy as a DAgger teacher, the student reaches near-optimal performance in 70,000 steps and reproduces consistent driving across 2D and 3D observations.The source 2D and target 3D environments are compared in a motion strip.Source: C. Reward-Based Policies Accelerate… · p. 5
  • The source-training cost can be amortized across target environments sharing reward and action spaces, especially when source simulation is cheaper per step.The paper contrasts lightweight, parallelizable source rollouts with potentially costly target environments rendered by commercial graphics engines.Source: C. Reward-Based Policies Accelerate… · p. 5

D. Reward Estimation Succeeds Where State Estimation Fails

With 100,000 target-image samples, reward estimation preserves substantial policy performance, whereas learned state estimation performs at the random-policy level.

D. Reward Estimation Succeeds Where State Estimation Fails

  • 735 reward-based performance with a learned reward estimator is close to the source-domain score of 836, while state-based performance with a learned estimator is 189 versus random baseline 192.Both estimators are trained with supervised learning from 100,000 target-image samples; the five-dimensional state is difficult to recover accurately from this data budget.Source: D. Reward Estimation Succeeds Where… · p. 6, Table III · p. 6
  • Reward estimation is more tractable under limited data because a reward is one-dimensional, while state estimation must recover higher-dimensional structure sensitive to estimation error.The comparison uses image observations in the DMC Cartpole target environment.Source: D. Reward Estimation Succeeds Where… · pp. 5–6

E. Expert Guidance Improves Training in Complex Environments

Expert guidance improves reward-policy training most in Car Racing, while the paper identifies identical source-target dynamics and untested physical deployment as scope boundaries.

E. Expert Guidance Improves Training in Complex Environments

  • Expert guidance gives small but consistent gains in Pointmass and Cartpole and a substantially larger benefit in Car Racing.In Car Racing, track deviation ends episodes and removes reward feedback; guidance prevents early termination so the policy can observe sufficient reward signal.Source: E. Expert Guidance Improves Training… · p. 6, Figure 9 · p. 7

VI. CONCLUSION

  • The framework combines an LSTM reward-and-action-history policy with expert guidance and uses it as a DAgger teacher for target observation-policy learning.The conclusion also reports zero-shot deployment in photorealistic Habitat-Sim scans with a Stretch robot and easier reward than state estimation from target observations.Source: §VI · p. 6, Figure 8 · p. 7
  • The work assumes identical source and target transition functions, leaving simultaneous observation and dynamics shifts for future extension.The authors suggest combining reward-based policies with domain randomization or system identification for that setting.Source: §VI · p. 6
  • Evaluation remains in simulation; deployment on physical robot hardware has not been tested.The paper calls for testing physical observation adaptations such as LiDAR-to-RGB-D changes or wider-field-of-view cameras.Source: §VI · pp. 6–7

APPENDIX

The appendix bounds the reward-based policy’s optimality gap under state reconstructibility and value-function smoothness. It also predicts that the gap can grow exponentially with state dimension when probing steps are fixed.

APPENDIX

  • Under ϵ-reconstructibility and Lipschitz continuity of the optimal value function, the reward-based policy’s optimality gap is bounded linearly by ϵ.The theorem states the bound; the preceding passages give its two conditions.Source: APPENDIX · p. 8
  • The proof uses a policy acting on reconstructed states, bounds its single-step value error, then accumulates that error over the infinite horizon.The reward-based policy’s optimality is used to relate the accumulated error to the gap.Source: APPENDIX · p. 8
  • Recovering a D-dimensional state from scalar rewards requires at least D + 1 independent probing steps, while sufficiently orthogonal measurements become exponentially less probable with D.For fixed K, the passage predicts ϵ(D) = O(cD) and an O(cD) optimality gap, consistent with Table I.Source: APPENDIX · p. 9

Related papers

Loading 2610.00729v1…