PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward (Mis)design for Autonomous Driving

W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, Peter Stone

arXiv:2104.13906v2cs.LG

TL;DR

Reward design is difficult but often underspecified, despite determining what sequential decision-making systems optimize. The paper develops 8 sanity checks and applies them to autonomous-driving reward functions, finding pervasive flaws while identifying design obstacles and directions for future work.

  • Problem

    Reward functions are rarely obvious, yet reinforcement-learning treatments often assume them fixed, leaving insufficient guidance for designing objectives that reflect AD stakeholders and outcomes.

  • Method

    The paper develops 8 sanity checks, applies them to published RL-for-AD reward functions, and examines reward attributes, safe shaping, learned rewards, and stakeholder-oriented design.

  • Results

    Published RL-for-AD reward functions show pervasive flaws: most or all fail the first 3 sanity checks, while the review also identifies obstacles in designing reward attributes.

  • Takeaways & Limitations

    The checks provide guidance for identifying reward and cost-function problems in AD and potentially other tasks with undefined reward functions.

  • Takeaways & Limitations

    Combining attributes is difficult because their units may not be convertible, linear weighting may be incorrect, and weights are often assigned by trial and error.

Abstract

from arXiv · show

This article considers the problem of diagnosing certain common errors in reward design. Its insights are also applicable to the design of cost functions and performance metrics more generally. To diagnose common errors, we develop 8 simple sanity checks for identifying flaws in reward functions. These sanity checks are applied to reward functions from past work on reinforcement learning (RL) for autonomous driving (AD), revealing near-universal flaws in reward design for AD that might also exist pervasively across reward design for other tasks. Lastly, we explore promising directions that may aid the design of reward functions for AD in subsequent research, following a process of inquiry that can be adapted to other domains.

1 Introduction

Reward functions are often treated as fixed even though the correct function is rarely clear and reward design receives little attention. Using autonomous driving, the article develops and applies sanity checks that expose widespread flaws while offering guidance rather than claiming to solve reward design.

  • Motivation: Reward design specifies the sequential decision-making problem, yet introductory reinforcement-learning treatments often assume the reward function is already given and fixed.The paper notes that standard treatments devote scant attention to designing rewards when no performance metric is known.
  • Approach and findings: The article applies 8 sanity checks to reward functions from reinforcement learning for autonomous driving, finding that most or all reviewed functions fail the first 3 checks.These checks are used to reveal pervasive issues in published AD reward functions.
  • Approach and findings: The paper examines obstacles to designing three AD reward attributes and discusses reward learning, multi-objective optimization, and financial currency as a reward unit.These investigations support directions for subsequent reward-design research.
  • Scope: The article aims to clarify common reward-design issues and guide their identification, not to condemn prior work or solve AD reward design.Its guidance is intended for autonomous driving and other tasks with undefined reward functions.

2 Background: objectives, utility functions, and reward functions

The paper distinguishes policy-level performance metrics, trajectory-level utility functions, and transition-level rewards within reinforcement learning. These definitions connect what algorithms optimize to the outcomes produced by policies and trajectories.

  • Performance metrics: A performance metric J maps each policy π to a real-valued score, ranking policies and identifying the set of optimal policies.Different definitions of J generally produce different policy rankings and optimal-policy sets.
  • Performance metrics: For episodic tasks, J(π) is the expected trajectory performance G(τ), estimated in practice by the mean G(τ) across trajectories generated by π.The expectation is taken over trajectories and the initial-state distribution.
  • Reward functions: In reinforcement learning, a reward function maps a transition’s state, action, and next state to a reward, whose undiscounted trajectory sum is the return G(τ).Policy-improvement algorithms seek to maximize expected return J(π).
  • Terminology: The article uses utility function for trajectory-level performance and focuses on it where feasible because its consequences are judged more analytically accessible than those of reward functions.The terminology distinguishes trajectory-level utility from the less common phrase return function.

3 The challenge of reward design for autonomous driving

Reward design for autonomous driving must represent many attributes, reconcile context-dependent stakeholder interests, and support rigorous evaluation. Delayed feedback further encourages naïve reward shaping, while the paper’s sanity-check framework distinguishes outright problems from warnings.

  • Multiattribute utility: AD utility functions must identify and quantify numerous outcomes, including progress, time, collisions, law compliance, consumption, vehicle wear, passenger experience, and external impacts.These attributes must ultimately be combined into a single real-valued utility output.
  • Stakeholders and context: Stakeholders and the weights assigned to their interests vary across vehicles and contexts, potentially requiring multiple utility functions or context-aware observations.Driving region is one example of a context that can change values, preferences, and driving culture.
  • Evaluation: The paper identifies a lack of rigorous methods for measuring how well one utility function conforms to stakeholders’ interests relative to another.It also reports no formal documentation of common evaluation practice for this question in the reviewed research.
  • Delayed feedback: Delayed feedback makes credit assignment difficult in drives lasting minutes or hours, making naïve reward shaping especially tempting and common.Much of a drive’s utility may be determined at its end, such as whether the destination was reached.

4 Sanity checks for reward functions

The paper proposes eight sanity checks for reward and cost functions, then applies them to 19 reinforcement-learning studies in autonomous driving. The review finds recurring problems involving unsafe shaping, incorrect trajectory preferences, learnable loopholes, trial-and-error design, and incomplete problem specifications.

  • The authors develop eight conceptually simple sanity checks for critiquing and improving reward or cost functions, many of which also apply to trajectory-level utility functions.
  • The review covers 19 publications on reinforcement learning for autonomous driving, including 10 designated focus papers.
  • 4.1 Identifying unsafe reward shaping: Among 13 publications with confidently identified reward shaping, none of the eight focus papers explicitly separated shaping rewards from true rewards or discussed policy-invariance guarantees.
  • 4.2 Comparing preference orderings: Only 2 of 9 evaluated reward functions correctly preferred leaving a vehicle idle to proceeding toward a certain collision, while 7 preferred the collision trajectory.
  • 4.3 Comparing indifference points: Preference tests can compare crashing, idling, and successful arrival to determine the permissible crash risk at which refusing to drive becomes preferable.
  • 4.4 Identifying learnable loopholes: A rigorous loophole check compares an undesirable observed trajectory with a minimally modified alternative and flags cases where the undesirable trajectory receives greater utility.
  • 4.5 Sanity checks for which failure is a warning: All 8 publications that disclosed their reward-design process reported manually tuning linear reward weights or attributes through trial and error.
  • 4.5 Sanity checks for which failure is a warning: Only 1 of 10 focus papers thoroughly described its reward function, discount factor, termination conditions, and time-step duration.

5 Exploring the design of an effective reward function

The paper explores how to design effective autonomous-driving reward functions by examining performance metrics, utility attributes, and alternatives to manual specification. It highlights unresolved difficulties in defining and combining attributes, and considers financial, learned, and multi-objective approaches.

  • 5.1 Performance metrics beyond RL: Performance metrics from regulatory, commercial, planning, and control communities may provide inspiration for autonomous-driving reward design.For example, distance per undesirable event captures both progress and avoidance of failure states, while planning and control costs could be converted into rewards by multiplying by −1.
  • 5.2 An exercise in designing utility function attributes: Progress to the destination is difficult to define because equal-distance increments can value stopping halfway as highly as reaching the destination on half of trips.The paper therefore leaves open whether progress should instead reflect passenger utility for different drop-off locations.
  • 5.2 An exercise in designing utility function attributes: Law adherence is hard to encode because penalties span incompatible units, may omit external costs, risk double-counting collisions, and require region-specific legal information.These penalties can include fines, time spent with law enforcement or courts, and possible incarceration.
  • 5.2 An exercise in designing utility function attributes: Passenger experience matters beyond collision metrics, but measuring it through end-of-ride satisfaction surveys risks biased self-report and disruptive data collection.The paper describes passenger fear from late braking as an example of an unpleasant experience that may be safe in hindsight.
  • 5.2 An exercise in designing utility function attributes: Linear reward combinations are difficult because attribute units may not be convertible, and their weights are often assigned by trial and error.The paper notes that linearity itself may be inappropriate when success requires a conjunction such as reaching the destination without collision.
  • 5.3–5.4 Alternative approaches: Potential alternatives include expressing outcomes in financial units, learning rewards from data, and using multi-objective optimization instead of a single utility function.Reward learning can avoid manually specifying some representations, but learned rewards remain difficult to evaluate when the true utility function is unknown, especially in safety-critical autonomous driving.

6 Conclusion

The article uses autonomous driving to expose widespread reward-design problems and proposes sanity checks and research directions for improving reward specification more broadly.

  • The article examines reward design for autonomous driving and evaluates reward functions from 10 papers.
  • The authors develop eight sanity checks and apply them to identify recurring issues in published autonomous-driving reward functions.
  • Further work includes constructing utility functions that pass the checks, evaluating them across tasks, and developing best practices for manual reward design.
  • Only one of the 10 focus papers thoroughly described its reward function, discount factor, termination conditions, and time-step duration.

A.1 LeTS-Drive: Driving in a Crowd by Learning from Tree Search [7]

The reward is an unweighted sum of efficiency, smoothness, and safety terms, with a severe collision penalty scaled by driving speed. Episodes use 100 ms steps, end at collisions or the goal, and are evaluated after a 120-second computational limit.

  • Reward function: The reward combines efficiency, smoothness, and safety as an unweighted sum of three attributes.Efficiency costs −0.1 at each non-terminal step, smoothness costs −0.1 for non-zero acceleration, and safety penalizes collisions.
  • Reward function: Collision with a pedestrian or static obstacle incurs −1000 × (v^2 + 0.5), where v is driving speed in m/s.
  • Time step duration: Each simulation step lasts 100 ms.
  • Episodic/continuing, time limit, and termination criteria: Episodes are computationally stopped after 120 seconds, or 1200 steps, for success-rate calculation.Trajectories stopped at the limit are excluded from value-function updates, making the evaluation somewhat optimistic.
  • Episodic/continuing, time limit, and termination criteria: Episodes terminate when the agent collides or reaches the goal.

A.2 Model-free Deep Reinforcement Learning for Urban Autonomous Driving [8]

Across the reviewed urban-driving setups, reward functions combine speed, steering, lane adherence, progress, collision, and other behavioral terms with varied weights and time scales. Tasks differ substantially in decision frequency, discounting, episode limits, and termination conditions.

  • Reward functions: One reward is an unweighted sum of speed preference, steering-angle penalty, lane-departure penalty, and a per-step goal-reaching incentive.The speed term is min(speed, 10 − speed), lane departure costs −1 beyond 2 m from the route polyline, and each step costs −0.1.
  • Time limits and termination: Episode limits range from 20–60 seconds to 50 seconds or a shortest-path travel time, with termination variously triggered by success, collisions, lane departure, timeouts, or other failures.Decision timing also varies: rewards may be received every 400 ms, steps may last 2 seconds, or actions may be frame-skipped.
  • Reward functions: Another reward uses r = (1)∆d + (0.05)∆v + (−0.00002)∆c + (−2)∆s + (−2)∆o, weighting progress, speed, collision damage, sidewalk overlap, and other-lane overlap.The passage defines ∆d, ∆c, ∆s, and ∆o as changes in distance, collision damage, sidewalk overlap, and other-lane overlap.
  • Reward functions: A separate setup rewards each step and penalizes lane changes by −0.01.Its task is continuing, with no time limit or termination criterion.
  • Reward functions: Other evaluated rewards include a sigmoid-shaped term based on forward velocity, road-center distance, heading error, and road width.The task uses 33 ms steps and terminates when the vehicle stops progressing, goes off-road, or travels in the wrong direction.
  • Reward functions: Another reward combines steering appropriateness, speed regulation, collision penalties, sidewalk overlap, and other-lane overlap.It assigns −100 for collisions with vehicles or pedestrians, −50 for other collisions, and −100 for sidewalk overlap; its discount factor is γ = 0.9.

A.8 Deep Distributional Reinforcement Learning Based High-Level Driving Policy Determination [31]

The reviewed policies use diverse reward constructions, including speed, overtaking, collision, and terminal-outcome terms, while operating at different temporal scales. One high-level policy derives non-terminal rewards from a motion planner’s trajectory costs and uses explicit terminal penalties and bonuses.

  • Reward function: One reward is an unweighted sum of speed within [40, 80] km/h, overtaking, and collision terms.The listed terms include a 0.5 overtaking reward and a −10 collision penalty.
  • Implementation and termination: Its time-step duration is unknown because the Unity frame-to-simulated-time correspondence was not reported.The discount factor is γ = 0.99, and episodes end on collision or after traveling the 2500-unit track length.
  • Terminal rewards: Another reward gives 100 for reaching the goal, −50 for collision or timeout, −10 for a red-light violation, and −1 for being in the wrong lane.The task is episodic and terminates at the goal or after collision, lane departure, red-light violation, or the time limit.
  • Non-terminal rewards: For non-terminal high-level actions, reward is the negative sum of costs from the best shorter-step trajectory found by the motion planner.The three attributes reward speeds near vref, distance traveled, and larger distances from nearby obstacles.
  • Non-terminal rewards: The high-level policy changes desired speed through speed_up and speed_down actions, while obstacle clearance uses longitudinal and lateral distances to the nearest vehicle within 20 meters.Distances are measured in meters and speed in m/s.
  • Implementation: The high-level agent chooses a new action after the previous planned trajectory completes, averaging 1 second per behavior-policy step and 100 ms per motion-planning iteration.The discount factor is γ = 0.99.

A.10 End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances [46]

The reviewed reward combines speed, path distance, and heading alignment, using a waypoint-generated path and context-dependent speed targets. Its supervisory context is unavailable during testing, and undesirable events terminate training episodes.

  • The reward is r = rspeed + 0.5 × rdist + 0.5 × rheading, bounded in [−1, 1].
  • Speed reward favors matching a desired speed, while distance and heading rewards penalize deviation from an optimal path.The distance term is normalized by dmax, and the heading term is clipped after scaling angular deviation.
  • The optimal path is generated through the waypoint API under simplifying assumptions, with turn directions randomly selected at intersections during training.
  • Desired speed is hard coded from traffic lights and obstacles using privileged training-only information, while dmax is half the lane width.
  • The reward appears supervisory because it teaches target speeds and path following, although the required context is unavailable to the agent during testing.
  • Training uses 100 ms time steps and terminates on path deviation beyond dmax, collisions, red-light violations, or certain zero-speed states.

B Reward shaping examples per paper

Reward shaping is common among the reviewed autonomous-driving papers, with many rewards explicitly discouraging or encouraging particular driving behaviors. Four reviewed papers did not include reward shaping, while two had potentially defensible cases.

  • 13 of 19 reviewed papers were judged highly likely to include reward shaping.
  • Negative reward discouraged lane-center deviation, opposite-lane overlap, boundary overlap, delayed intersection entry, steering deviation, drifting, getting close to vehicles, and turn-signal use.
  • Positive reward encouraged passing other vehicles and increasing distances from other vehicles.
  • Two papers had reward attributes considered somewhat defensible as non-shaping because their lower-level modules received lane commands.
  • Four papers were judged confidently not to include reward shaping.

C Calculation of trajectory returns

The appendix estimates trajectory returns to compare preference orderings and safety indifference points. It defines crash, idle, and successful trajectories, then solves for the success probability making idle behavior equally attractive to a crash-success lottery.

  • Return calculations support two sanity checks: comparing preference orderings and comparing indifference points.
  • The analysis compares τcrash, τidle, and τsucc: crashing halfway, remaining safely motionless, and successfully reaching the destination.
  • For τA ≺ τB ≺ τC, the indifference point solves G(τB) = pG(τC) + (1 −p)G(τA).
  • For the safety analysis, p is calculated from G(τidle) = pG(τsucc) + (1 −p)G(τcrash), then converted to kilometers per crash.The conversion uses ((p/(1 −p)) + 0.5) × path length.
  • With returns −10 for crashing, −5 for idling, and 10 for success, the indifference probability is p = 0.25, corresponding to 0.83 km per collision.
  • The quantitative analysis depends on stated assumptions, which readers can vary to test sensitivity; the authors expect no qualitative changes from those variations.

C.1 General methodology and assumptions

The return methodology estimates selected trajectories without reconstructing every state-action pair. It uses favorable assumptions for successful portions and reports units and term ordering explicitly in the calculations.

  • Exact state-action sequences are not needed to estimate returns for certain trajectories under the reviewed reward functions.
  • For each time step, the methodology estimates reward according to a specified calculation procedure.
  • Successful trajectory portions assume no unnecessary penalties, and positive reward attributes receive their maximum or experimentally supported favorable values.
  • Return equations explicitly retain units, use colors for time limits, path lengths, step durations, and speeds, and preserve attribute order for summed rewards.

C.2 Assumptions and calculations for each paper

The appendix makes explicit assumptions about trajectories, speeds, path lengths, time limits, and event rates before calculating returns and collision trade-offs across reviewed autonomous-driving reward functions.

  • Return calculations: For each reward function, returns are computed by combining per-step or per-event rewards with assumed trajectory duration, path length, and event frequency.The calculations include time-step rates, acceleration events, overtakes, lane changes, collisions, and other driving attributes.
  • Preference comparisons: The appendix compares successful, collision, and idle trajectories to identify indifference points between outcomes.One calculation uses G(τidle) = pG(τsucc) + (1 −p)G(τcrash), while related calculations convert the indifference point into kilometres per collision.
  • Scenario assumptions: The calculations standardize scenario assumptions, including path lengths, speeds, successful-trajectory durations, time limits, collision conditions, and lane or steering behavior.Examples include a 0.125 km roundabout path at 5 m/s, a 1 km CARLA path, and a 9.87 km racing track.
  • Reward components: Several examples assign explicit rewards and penalties for collision, leaving the lane, sidewalk overlap, steering deviation, reaching the goal, or time expiration.For the urban-driving example, collision receives −10, leaving the lane receives −1, and the constant reward accumulates over the 50 s time limit.
  • Scope conditions: The calculations are conditional on simplifying assumptions, and at least one reward function is excluded because collision appears impossible in its task.The Dynamic Input reward is therefore not analyzed for preference orderings or indifference points.
Loading 2104.13906v2…