PaperlayerView exact revision on arXiv

Source-linked AI summary

Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks

Andrea Censi, Konstantin Slutsky, Tichakorn Wongpiromsarn, Dmitry Yershov, Scott Pendleton, James Fu, Emilio Frazzoli

arXiv:1902.09355v2cs.AI

TL;DR

Self-driving behavior must reconcile conflicting requirements from law, ethics, culture, and other sources, while existing approaches do not provide a precise hierarchy for them. The paper formalizes behavior with prioritized rulebooks, which induce preferences over outcomes and support constraint-preserving operations. The authors demonstrate the formalism through self-driving examples, while noting that those examples do not represent the full development process.

  • Problem

    Self-driving behavior must accommodate conflicting legal, ethical, cultural, and practical objectives, but available formalisms do not combine these factors in one precise hierarchy.

    Source: §I · p. 1

  • Method

    Rulebooks represent behavior specifications as prioritized violation rules over observable outcomes, with manipulation operations for refining the specification.

    Source: §I · p. 1

  • Results

    Augmentation, refinement, and aggregation each produce a rulebook that is a strict refinement of the original, preserving its constraints.

    Source: 5) Properties preserved by the three… · p. 5

  • Takeaways & Limitations

    The formalism supports a shared high-level specification of behavior while leaving other rule and priority choices as implementation details that may differ across manufacturers.

    Source: §VI · p. 6

    The scenarios and specification examples are illustrative, and the described methodology omits the extensive verification and validation needed for safety-critical rules.

    Source: §I · p. 1

Abstract

from arXiv · show

The behavior of self-driving cars must be compatible with an enormous set of conflicting and ambiguous objectives, from law, from ethics, from the local culture, and so on. This paper describes a new way to conveniently define the desired behavior for autonomous agents, which we use on the self-driving cars developed at nuTonomy. We define a "rulebook" as a pre-ordered set of "rules", each akin to a violation metric on the possible outcomes ("realizations"). The rules are partially ordered by priority. The semantics of a rulebook imposes a pre-order on the set of realizations. We study the compositional properties of the rulebooks, and we derive which operations we can allow on the rulebooks to preserve previously-introduced constraints. While we demonstrate the application of these techniques in the self-driving domain, the methods are domain-independent.

I. INTRODUCTION

Self-driving behavior must reconcile legal, ethical, cultural, and practical constraints, yet existing formalisms do not combine these factors with a precise hierarchy. The paper introduces rulebooks: prioritized violation rules that order possible outcomes while leaving room for iterative refinement.

I. INTRODUCTION

  • The specification challenge spans road rules, implementation limits, natural behavior, local driving culture, and ethical concerns, without a formalism that unifies them precisely.Source: §I · p. 1
  • Rulebooks combine potentially conflicting behavior constraints by assigning priorities to rules that score violations across possible outcomes.Rules may be specified analytically or learned from data, and can derive from traffic laws, common sense, or ethical considerations.Source: §I · p. 1

2) Rules:

  • Rules are ordered by priority, but a rulebook induces only a pre-order on outcomes, preserving implementation freedom where priorities do not fully specify behavior.The rulebook is a pre-ordered set of rules; its rules can be arranged in a directed graph that represents priority relations.Source: §I · p. 1, 3) Rulebooks · p. 2

1) Realizations:

  • A realization is an objectively observable outcome; for a self-driving car, it is a world trajectory including every agent in the environment.Source: 1) Realizations · p. 2

2) Rules:

  • A rule scores how much a realization violates a requirement, and its score may be defined analytically or learned from data.The rulebook approach also emphasizes how behavior should be chosen when a rule must be violated.Source: 2) Rules · p. 2

4) Induced pre-order on realizations:

A rulebook induces a pre-order on realizations by letting higher-priority rules override lower-priority violations. This relation is reflexive and transitive, and equivalent realizations are exactly those with matching scores on every rule.

4) Induced pre-order on realizations:

III. EXAMPLES IN THE DRIVING DOMAIN

Driving examples show how prioritized rules resolve conflicts between collision avoidance, liability, road infractions, and other trajectory preferences. A partial rulebook can leave some outcomes incomparable, while learned parameters can adapt behavior to local culture under top-priority safety rules.

Safety vs. infractions

  • A higher-priority no-collision rule selects a trajectory across the double white line when collision would otherwise be unavoidable.The example assumes no agents outside the frame could trigger the no-collision rule.Source: §III · p. 3, Figure 3 · p. 3

Liability-aware specification

  • Prioritizing ego-fault collision over third-party-caused collision can favor colliding with an object in the ego vehicle’s lane rather than causing an at-fault accident.This example is identified as capturing the responsibility-sensitive safety model.Source: §III · p. 3, Figure 4 · p. 3

Partial priorities specification

  • With collision cost applied to any collision, swerving can be preferred because it provides more distance to decelerate and lowers expected impact speed.The example models blockage, lane keeping, obstacle clearance, and path length as rules over representative trajectories.Source: §III · p. 4
  • Obstacle-clearance parameters such as C0 can be learned from data to adapt behavior to local driving culture while top-priority safety rules remain in place.The authors state that this hierarchy preserves safety even with adversarial data.Source: §III · p. 4
  • A partial rulebook can leave trajectories incomparable, giving the implementation freedom over behavioral details that the specification does not prioritize.For the example ordering, b is incomparable with c and d.Source: §III · p. 4

IV. ITERATIVE SPECIFICATION REFINEMENT WITH RULEBOOKS MANIPULATION

Iterative specification refinement lets users add rules and priority relations until behavior is specified to their desire. A shared base rulebook can be particularized for different legislation rather than rebuilt from scratch.

IV. ITERATIVE SPECIFICATION REFINEMENT WITH RULEBOOKS MANIPULATION

  • Users can iteratively add rules and priority relations until the system’s behavior is specified to their desire.The process is presented as iterative specification refinement.Source: §IV · p. 4
  • Because regulations across states and countries often share similarities, a base rulebook can be tailored to specific legislation by adding rules or priority relations.The paper contrasts this with constructing each jurisdiction’s rulebook from scratch.Source: §IV · p. 4

1) Operations that refine rulebooks:

The paper identifies priority refinement, rule aggregation, and rule augmentation as operations for changing rulebooks. Their conditions are chosen to preserve the existing ordering while refining the specification.

1) Operations that refine rulebooks:

  • The three rulebook operations are priority refinement, aggregation of equally ranked rules, and augmentation with a lowest-priority rule.Priority refinement adds an edge between rules; aggregation collapses equi-ranked rules.Source: 1) Operations that refine rulebooks · p. 4

2) Priority refinement:

  • Priority refinement adds constraints to rule priorities and can resolve the incomparability between κ and λ in either direction.The example’s resulting trajectory order depends on which rule is given priority.Source: 2) Priority refinement · pp. 4–5

3) Rule aggregation:

  • Aggregation is allowed for two rules in the same equivalence class, using an embedding of their product pre-order into the nonnegative reals.Allowed choices include positive-coefficient linear combinations and functions strictly monotone in both arguments.Source: 3) Rule aggregation · p. 5

4) Rule augmentation:

  • Adding a rule can be destructive, so augmentation preserves the existing order only when the new rule is below every rule already in the rulebook.The operation adds the rule at the lowest priority level.Source: 4) Rule augmentation · p. 5

5) Properties preserved by the three operations:

Theorem 19 states that augmentation, refinement, and aggregation each produce a strict refinement of the original rulebook. In an unavoidable-collision experiment, the selected trajectory changes when fault-based collision priorities change.

5) Properties preserved by the three operations:

  • Applying augmentation, refinement, or aggregation creates a rulebook that is a strict refinement of the original.The paper places the proofs of this and ancillary results in the appendix.Source: 5) Properties preserved by the three… · p. 5

1) Unavoidable collision:

  • With collision cost applied to any collision, the planner swerves toward the opposite-lane vehicle to gain deceleration distance and reduce expected collision speed.The experiment makes collision unavoidable with two vehicles appearing about 12 m away, moving at 1.0 m/s, while the ego vehicle travels at 9.5 m/s and may decelerate by at most 3.5 m/s2.Source: 1) Unavoidable collision · p. 5
  • When ego-fault collision ranks above third-party-caused collision, the planner stays in lane and collides with the vehicle moving against traffic.Fig. 7 contrasts this green trajectory with the red trajectory chosen when the two collision types are treated equally.Source: 1) Unavoidable collision · p. 5, Figure 7 · p. 5

2) Clearance and lane keeping:

The overtaking examples show how rulebook priorities trade lane keeping against clearance, while the discussion addresses rulebook scale and who should set priorities.

2) Clearance and lane keeping:

  • With clearance prioritized, the vehicle crosses the lane boundary for sufficient clearance; with lane keeping prioritized, it stays within the lane and violates clearance.The stationary vehicle is the obstacle in this overtaking scenario.Source: 2) Clearance and lane keeping · p. 6, Figure 8 · p. 6

3) Lane change near intersection:

  • Near an intersection, a left turn requires reaching the left lane, but a stationary vehicle blocks the maneuver at an appropriate distance from the intersection.The figure presents optimal trajectories for cτ = 0 and cτ > 0.Source: 3) Lane change near intersection · p. 6, Figure 9 · p. 6

VI. DISCUSSION AND FUTURE WORK

  • The nuTonomy production code uses about 15 rules; full Massachusetts or Singapore coverage is estimated at about 200 rules in roughly a dozen priority groups.The estimate includes rare corner cases such as not scaring farm animals.Source: §VI · p. 6
  • Human safety is the stated exception to open priority choices; the authors argue that safety- and liability-related priorities belong in regulations shaped by informed public discourse.A shared high-level specification could set minimal constraints while other priorities remain implementation choices.Source: §VI · p. 6, Figure 10 · p. 6

APPENDIX I ORDERS AND PREORDERS

The appendix defines preorders and related notions used to describe rule priorities and comparisons.

APPENDIX I ORDERS AND PREORDERS

  • A preorder is a reflexive, transitive relation; it is total when every pair is comparable.The strict relation y < z means y ≲ z but not z ≲ y.Source: Appendix I · p. 8
  • A preorder induces equivalence when two elements relate in both directions; it is a partial order when that equivalence holds only for identical elements.The appendix uses ≤ for the strict relation or equality.Source: Appendix I · p. 8
  • An embedding preserves both the preorder relation and its strict version; refinement means the identity map embeds one preorder into another.Thus refinement concerns two preorders on the same set.Source: Appendix I · p. 8

APPENDIX II OPERATIONS ON RULEBOOKS

The appendix seeks conditions on rulebook operations that preserve existing comparisons between realizations while permitting new ones.

APPENDIX II OPERATIONS ON RULEBOOKS

  • Operations of interest preserve existing realization relations but may add comparisons, thereby refining the induced preorder.The paper frames this as a condition on a map between rulebooks.Source: APPENDIX II OPERATIONS ON RULEBOOKS · p. 8
  • The stated goal is to identify conditions on a map between two rulebooks that ensure the second induced preorder refines the first.The setup begins with a map collapsing several rules into one.Source: APPENDIX II OPERATIONS ON RULEBOOKS · p. 8
  • For the single-rule case, the question is when its total preorder on realizations refines the partial preorder induced by the original rules.Source: APPENDIX II OPERATIONS ON RULEBOOKS · p. 8

A. Aggregative maps

The appendix characterizes rulebook maps that preserve induced realization comparisons, then establishes composition and refinement results for those maps.

A. Aggregative maps

  • A collapse to one rule is admissible when that rule's value depends on the original rule values through an embedding of their partial orders.The new rule value cannot depend on the realization beyond the original rules' values.Source: A. Aggregative maps · p. 8
  • An aggregative rulebook map requires each target rule to be an admissible collapse of its preimage rules.Source: A. Aggregative maps · p. 8
  • Composition of surjective aggregative maps is aggregative.Source: A. Aggregative maps · p. 9
  • A rulebook embedding is an aggregative map that also embeds the rulebooks as partially preordered sets.Source: A. Aggregative maps · p. 9
  • The proof uses an ascending chain of mapped rules and finiteness of the rulebook to reach a rule that orders the realizations.The chain must stop because the rulebook is finite.Source: A. Aggregative maps · pp. 9–10
  • If a surjective embedding exists between rulebooks, the second induced preorder on realizations refines the first.Refinement preserves existing relations while possibly adding comparisons.Source: A. Aggregative maps · p. 10
  • Priority refinements and rule aggregation are identified as examples of surjective embeddings.Source: A. Aggregative maps · p. 10

APPENDIX III ADDING NEW RULES

Adding rules can change the preorder that a rulebook induces on realizations, so the effect depends on where a new rule is placed. Although adding rules below all existing ones preserves strict comparisons, it can split previously equivalent realizations.

APPENDIX III ADDING NEW RULES

  • A highest-priority new rule makes its ranking decisive: whenever r(x) < r(y), x <2 y regardless of the realizations’ prior relation under R1.Source: APPENDIX III ADDING NEW RULES · p. 10
  • A rule added in the middle can reverse an old comparison when the other old rules tie and the new rule favors the previously worse realization.Source: APPENDIX III ADDING NEW RULES · p. 10
  • An incomparable added rule can turn a previously ordered pair incomparable, and aggregating the rules can make every relation between that pair possible.Source: APPENDIX III ADDING NEW RULES · p. 10
  • The same preorder disruption can arise when the added rule is comparable to some existing rules but incomparable to others.Source: APPENDIX III ADDING NEW RULES · p. 10
  • Adding a new rule is not guaranteed to refine the realization preorder; adding it below all existing rules preserves strict comparisons but may split previously equivalent realizations.Source: APPENDIX III ADDING NEW RULES · p. 10
  • A dominant embedding, with new rules below every embedded old rule, preserves existing strict comparisons between realizations.Source: APPENDIX III ADDING NEW RULES · p. 10
Loading 1902.09355v2…