PaperlayerView exact revision on arXiv

Source-linked AI summary

Assisted Robust Reward Design

Jerry Zhi-Yang He, Anca D. Dragan

arXiv:2111.09884v1cs.ROcs.AIcs.LG

TL;DR

Reward design is iterative, yet robotic systems are often treated as if the designer’s reward were fixed, even when edge cases reveal failures. Assisted Reward Design models revisions as evidence and actively proposes environments for designers to investigate. In autonomous-driving experiments, it uncovered edge cases and sped reward design.

  • Problem

    Reward functions can fail in unforeseen edge-case environments, prompting repeated revisions during testing and even after deployment.

    Source: §1 · p. 1

  • Method

    The robot accounts for uncertainty in the current reward and proposes environments where the proxy may fail, treating subsequent reward revisions as evidence.

    Source: §1 · pp. 1–2

  • Results

    Assisted Reward Design uncovered difficult test scenarios and sped reward design; robotics experts confirmed that proposed environments triggered undesirable behavior.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors argue that robots should help engineers design more robust reward functions, rather than only optimize the rewards they specify.

    Source: §1 · p. 2

    The method assumes access to an environment distribution, and obtaining realistic distributions for simulation remains an open problem; combining it efficiently with RL also remains open.

    Source: §7 · p. 8

Abstract

from arXiv · show

Real-world robotic tasks require complex reward functions. When we define the problem the robot needs to solve, we pretend that a designer specifies this complex reward exactly, and it is set in stone from then on. In practice, however, reward design is an iterative process: the designer chooses a reward, eventually encounters an "edge-case" environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. What would it mean to rethink robotics problems to formally account for this iterative nature of reward design? We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations as future evidence. We contribute an Assisted Reward Design method that speeds up the design process by anticipating and influencing this future evidence: rather than letting the designer eventually encounter failure cases and revise the reward then, the method actively exposes the designer to such environments during the development phase. We test this method in a simplified autonomous driving task and find that it more quickly improves the car's behavior in held-out environments by proposing environments that are "edge cases" for the current reward.

1 Introduction

Reward design is iterative: designers revise proxy rewards as they discover environments where they fail. Assisted Reward Design formalizes that process by having the robot account for reward uncertainty and propose informative environments; experiments in autonomous driving find that this can speed reward design and uncover edge cases.

1 Introduction

  • Designers repeatedly revise rewards as testing reveals failures, and some systems continue encountering unforeseen edge cases after deployment.For autonomous cars, these cases can involve unseen road layouts or traffic situations.Source: §1 · p. 1
  • The robot treats reward revisions as evidence about the true reward and proposes environments likely to expose failures, potentially surfacing deployment-time edge cases before deployment.The problem is formalized as a Meta MDP with hidden state.Source: §1 · pp. 1–2
  • A particle-filtering algorithm approximates reward uncertainty because computing it over high-dimensional reward and environment spaces is intractable.The approximation is intended to enable efficient computation.Source: §1 · p. 2
  • The method uncovers difficult test scenarios and speeds reward design, with simulated designers and robotics experts confirming its practicality and effectiveness.Proposed environments tend to be edge cases where the current reward estimate fails.Source: §1 · p. 2
  • The paper frames reward specification as an iterative process, while noting that real-world systems need a simulator or learned environment model and further work.The authors also describe producing edge-case environments without hand-coded heuristics.Source: §1 · p. 2

2 Related Work

The paper situates Assisted Reward Design among reward learning, active reward learning, reward shaping, and safety work. It distinguishes its use of proxy rewards and active environment queries from approaches that rely on ground-truth rewards or explicit hard constraints.

2 Related Work

  • Earlier methods learn rewards from demonstrations or diverse human inputs [8] [11] [14] [15] [16], while this work uses proxy rewards [1].The paper describes its approach as extending the use of specified rewards into an iterative, active process.Source: §2 · pp. 2–3
  • Unlike active reward learning that queries trajectories, states, or synthesized environments, this method asks for proxy rewards designed to work in each newly queried environment.The authors argue that such queries can expose edge cases and aid reward revision.Source: §2 · p. 3
  • Reward shaping assumes access to a ground-truth reward that is difficult to optimize, whereas Assisted Reward Design uses designers’ best-guess proxy rewards.The paper notes that the ground-truth reward may not be obvious to specify in real-world tasks.Source: §2 · p. 3
  • Where safety methods often require hard constraints, this work targets settings where constraints are incomplete or difficult to specify and seeks edge cases without assuming access to them.The paper says its method can develop scenarios that implicitly violate constraints.Source: §2 · p. 3

3 Problem Statement

The problem is to design a reward that induces desired behavior across development and deployment environments, even though deployment environments and the ground-truth reward are unavailable. Assisted Reward Design models reward design as an iterative process in which the robot proposes environments and treats the designer’s resulting proxy rewards as evidence about the ground truth.

3.1 The Unassisted Reward Design Problem

  • Designing for deployment is difficult because the robot is tested on a large, non-exhaustive set of development environments but will encounter a different, unavailable set at deployment.Development environments may be scenarios drawn from data, parameterized configurations, or a generative model.Source: §3.1 · p. 3
  • The reward is parameterized by weights, and the inaccessible ground-truth weights are assumed to induce desired behavior across development and deployment environments.The reward family may use linear weights on predefined features or neural-network weights.Source: §3.1 · p. 3, Figure 2 · p. 4
  • The unassisted process repeatedly adds environments where the current reward fails and redesigns it, while implicitly treating each proxy reward as the ground truth.The process may continue into deployment if the reward still fails there.Source: §3.1 · p. 4

3.2 The Assisted Reward Design Problem

  • Assisted Reward Design formulates the process as a meta-MDP whose state includes the tested environments and hidden ground-truth reward, and whose actions add environments for the designer to assess.The designer’s proxy reward is treated as evidence about the ground truth, not as the ground truth itself.Source: §3.2 · p. 4
  • The meta-agent aims to minimize deployment regret by choosing environments that improve its belief about the ground-truth reward, although the direct objective cannot be optimized because deployment environments are unknown.Its deployment belief is based on evidence gathered across the sequence of tested environments.Source: §3.2 · pp. 4–5

4 Assisted Reward Design via Active Info-Gathering.

The method chooses development environments for the information they are expected to provide about the ground-truth reward. It updates its reward belief from designer feedback, with joint and independent feedback modes, and is compatible with adding features.

4 Assisted Reward Design via Active Info-Gathering.

  • The method proposes environments that maximize expected information gain about the ground-truth reward, using a one-step look-ahead for computational tractability.The paper notes that reasoning over multiple future iterations could benefit the meta-agent.Source: §4 · p. 5
  • In the information-gain calculation, the method estimates the expected reduction in posterior entropy after a new designer observation, and its implementation sets that observation to the ground-truth reward for tractability.The observation is otherwise sampled from the designer model.Source: §4 · p. 5

4.1 Updating Beliefs over 𝑤∗

  • Joint design updates the belief using feedback for all environments seen so far, whereas independent design uses feedback based only on the newly proposed environment.The joint-mode description assumes the new proxy reward is independent of previous observations.Source: §4.1 · p. 5

4.2 Adding New Features

  • The method is compatible with designers adding new features to the environment.Source: §4.2 · p. 5

5 Experiments with Simulated Designers

In a simplified autonomous-driving task, the authors evaluate the method with simulated designers using six random initial environments. The active method outperforms random and difficulty-based environment selection in both feedback modes, and its proposed environments expose high regret for the current reward estimate.

5 Experiments with Simulated Designers

  • The evaluation hypothesizes a ground-truth reward to simulate designer behavior; the method itself does not require access to a unique ground-truth reward.The paper separately describes an expert study without ground-truth access.Source: §5 · p. 5

5.1 Experiment Setup

  • The task requires an autonomous car to merge into the leftmost lane while deciding whether to overtake or slow down around human-driven vehicles and obstacles.The reward is a weighted sum of six features, including vehicle distance, progress, and control effort.Source: §5.1 · pp. 5–6

5.2 Experiment Results

  • The active method outperforms the proxy reward and random and difficult-environment baselines in both joint and independent modes across six random initial environments (p < 0.05).Figure 3 presents posterior-mean regret and proposed-environment efficacy for the two modes.Source: Figure 3 · p. 6, §5.2 · p. 6

5.1 Experiment Setup

  • The development and deployment environment sets have long-tailed difficulty distributions, so rare events may be useful for reward design.The difficulty metric increases when traffic and obstacles are closer to the autonomous vehicle.Source: §5.1 · p. 6

5.2 Experiment Results

  • The active method outperforms random selection and the difficulty heuristic; some useful environments are rated easy by heuristics and can subtly trigger undesirable behavior, such as a crash on a sparsely populated road.Source: Figure 3 · p. 6, §5.2 · p. 7
  • The method’s proposed environments have the highest relative regret, indicating that they better expose overall posterior regret than environments ranked by simple heuristics.Relative regret compares regret on the next environment with average regret in deployment.Source: §5.2 · p. 7

6 Experiments on Real Reward Design

In a study with 14 robotics experts, Assisted Reward Design was evaluated over repeated reward-design iterations for simplified autonomous driving. Maximal Information yielded fewer average violations than other acquisition functions and proposed environments that exposed edge cases beyond the predefined difficulty metrics.

6.1 Experiment Setup

  • Fourteen robotics experts designed reward functions over four iterations using three methods and three random seeds, for 36 designs in total.Methods were counterbalanced, and the experiments were conducted over Zoom.Source: §6 · p. 7, §6.1 · p. 7

6.2 Experiment Results

  • Maximal Information produced the fewest average violations on Mdeploy among the acquisition functions, and Figure 5 compares violation and proposed-environment values.The authors interpret fewer failed tests as a measurable proxy for better driving behavior.Source: Figure 5 · p. 7, §6.2 · p. 7

6.1 Experiment Setup

  • Because ground-truth rewards were unavailable, evaluation counted trajectory failures across seven criteria, including collision, driving off track, and stopping.The criteria were treated as unit tests rather than reward features, since some violations can be acceptable in edge cases and the tests omit behaviors such as tailgating.Source: §6.1 · p. 7

6.2 Experiment Results

  • Maximal Information outperformed the heuristic difficulty metric in finding edge-case environments containing more violations.Experts rated proposed trajectories on a normalized Likert scale, with the inverse of the behavior score used to indicate edge-caseness.Source: §6.2 · p. 7
  • Maximal Information proposed subtle failures that received high user ratings despite obeying predefined violation criteria, including narrowly avoided collisions.The authors argue that these cases can push designers to specify reward decision boundaries and gather preferences more quickly.Source: §6.2 · p. 8

7 Discussion

The discussion presents Assisted Reward Design as a way to expose edge cases and support reward design, while identifying limits on efficiency, guarantees, and environment-distribution access.

7 Discussion

  • Experiments with robotic experts confirmed that the proposed environments were edge cases leading to undesirable behaviors.The broader contribution is an active reward-learning method that generates environments and queries for reward functions as user inputs.Source: §7 · p. 8
  • The method applies to model predictive control with known dynamics and to reinforcement-learning problems when an environment simulator or learned model is available.The paper leaves efficient combination with reinforcement learning as an open question because naively applying it inside reward inference is intractable.Source: §7 · p. 8
  • The approach still needs a more efficient environment-proposal algorithm, formal analysis and guarantees, and access to an environment distribution.Obtaining realistic distributions for simulation remains an open problem.Source: §7 · p. 8

A Algorithm

Assisted Reward Design iteratively updates reward beliefs from designer input and selects new environments using an acquisition function. Its implementation represents beliefs with particles and evaluates candidate environments by planning across particles.

A Algorithm

  • At each iteration, the algorithm uses Maximal Information to select a new environment and query the reward designer, sampling candidate environments because exhaustive search is intractable.Uniform sampling is used for the candidate set, though other selection heuristics are possible.Source: A Algorithm · p. 11
  • The belief over the true reward is represented by particles that are sampled, importance-weighted using the posterior update, and resampled.This particle-filtering approximation supports computation in the high-dimensional reward and environment spaces.Source: A Algorithm · p. 11
  • The algorithm initializes a reward posterior from a prior, queries the designer on current environments, updates the posterior, scores sampled candidates, and adds the highest-scoring environment.The pseudocode specifies candidate and particle counts as inputs.Source: A Algorithm · p. 11
  • Planning across every particle and candidate requires O(N_p · N_cand) planning problems, and the batch-planning implementation is not feasible for general nondifferentiable dynamics or rewards.When a designer adds a feature, prior proxy rewards are incorporated and the posterior is recomputed in the augmented reward space using MCMC.Source: A Algorithm · p. 11

B Driving Environment Implementation

The driving implementation combines a point-mass vehicle model and feature-based rewards with sampled traffic environments and trajectory-level constraint evaluation. Its environment distribution is a coarse showcase, including unlikely cases where other vehicles run into obstacles.

Dynamics

  • A point-mass vehicle model represents forward dynamics, while human vehicles move at constant speed and traffic cones remain static.The vehicle state comprises position, heading, and speed; control inputs are steering and acceleration.Source: B Driving Environment Implementation · p. 12

Feature and Reward Functions

  • Driving behavior is represented by a linear weighted sum of differentiable features, each built from state or control inputs and a nonlinear transformation.The transformations are designed so maximizing a feature induces its desired behavior, such as moving to the target lane.Source: B Driving Environment Implementation · p. 12, Table 1 · p. 12

Violations

Planning Speed

  • Gradient-based planning uses Adam for 200 steps over a 10-timestep horizon, replanning every 5 timesteps.Planning generates 2.57 trajectories per second for one environment, or 157.72 trajectories per second when planning 500 in a batch.Source: B Driving Environment Implementation · pp. 12–13

Environment Distribution

  • The setup samples two human vehicles and two cones on a three-lane highway, filters initial collisions, and discretizes Mdevel and Mtest to 274k and 91k environments.Mdeploy uses a tighter feasible range for the car's x position, shifting the pseudo-difficulty distribution toward long-tail events.Source: B Driving Environment Implementation · p. 13
  • The environment distribution is a coarse showcase and includes relatively unlikely cases where other vehicles run into obstacles.The authors note that more realistic distributions could be defined through careful environment engineering or real-world driving datasets.Source: B Driving Environment Implementation · p. 13

C Experiment Details

The experiment details specify iterative comparisons among three environment-selection methods, a particle-based proposal procedure, and a held-out evaluation using relative regret. They also describe the reward-inference settings used for the designer and inverse model.

Reward Design

  • The simulated designer uses a rationality factor of β=0.1||M||, while the inverse model uses β=1.The authors divide the designer's factor by the number of proxy environments to maintain noisiness as the proxy reward approaches ground truth.Source: C Experiment Details · p. 13

Iterative Comparison

  • The study compares random, difficulty, and maximal-information methods for 9 iterations from the same initial environment, aggregating results over five random seeds.The supplied experiment details describe the comparison setup but do not report its outcomes.Source: C Experiment Details · p. 13

Environment Proposal

  • Environment proposals use 500 particles across 64 candidate environments and take 4 minutes on a c2-standard-4 Google Cloud instance.The implementation uses a vectorized car dynamics model with JAX and Ray for batch planning.Source: C Experiment Details · p. 13

Evaluation

  • Evaluation samples 500 deployment environments uniformly and uses relative regret normalized by each environment's reward range.The normalization accounts for different maximum and minimum possible rewards across environments.Source: C Experiment Details · p. 13

Related papers

Loading 2111.09884v1…