PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning an Urban Air Mobility Encounter Model from Expert Preferences

Sydney M. Katz, Anne-Claire Le Bihan, Mykel J. Kochenderfer

arXiv:1907.05575v1cs.AI

TL;DR

UAM encounter models are needed to simulate realistic aircraft behavior, but these systems lack the operational data used to build models for other aircraft. This paper learns a stochastic MDP encounter model from pairwise expert preferences, reporting accurate parameter estimates after a sensible number of queries while identifying feature selection and stochastic query presentation as open challenges.

  • Problem

    UAM systems are not yet deployed, leaving limited data on their airspace behavior for building representative encounter models.

    Source: §I · p. 1

  • Method

    The approach learns selected reward parameters in a stochastic MDP from pairwise expert comparisons of policies, while fixing parameters that encode prior knowledge such as landing.

    Source: §I · p. 1, §III · p. 3

  • Results

    After about 40 queries, the properly tuned multiobjective optimization method approaches the true reward, and the process takes roughly 15 minutes of expert time.

    Source: B. Overall performance · p. 6

  • Takeaways & Limitations

    Expert preferences can generate realistic UAM trajectories without a substantial dataset.

    Source: §VI · p. 7

    Effective feature selection remains difficult, and future work will examine queries that show stochastic policies rather than applying stochasticity after training.

    Source: §VI · p. 7

Abstract

from arXiv · show

Airspace models have played an important role in the development and evaluation of aircraft collision avoidance systems for both manned and unmanned aircraft. As Urban Air Mobility (UAM) systems are being developed, we need new encounter models that are representative of their operational environment. Developing such models is challenging due to the lack of data on UAM behavior in the airspace. While previous encounter models for other aircraft types rely on large datasets to produce realistic trajectories, this paper presents an approach to encounter modeling that instead relies on expert knowledge. In particular, recent advances in preference-based learning are extended to tune an encounter model from expert preferences. The model takes the form of a stochastic policy for a Markov decision process (MDP) in which the reward function is learned from pairwise queries of a domain expert. We evaluate the performance of two querying methods that seek to maximize the information obtained from each query. Ultimately, we demonstrate a method for generating realistic encounter trajectories with only a few minutes of an expert's time.

I. INTRODUCTION

UAM encounter models are needed for realistic airspace safety and performance simulations, but the lack of deployed UAM data motivates learning from expert preferences. The paper builds on data-driven encounter modeling and preference-based learning to develop this approach.

I. INTRODUCTION

  • Encounter models probabilistically represent typical aircraft behavior in close encounters and support simulations used to assess systems such as TCAS and ACAS X [2].Source: §I · p. 1
  • Existing encounter models use dynamic Bayesian networks trained on large datasets; the manned-aircraft model used nine months of radar data containing almost 400,000 encounters,.Source: §I · p. 1
  • Pairwise expert preferences tune a statistical UAM encounter model, addressing the lack of behavior data and generating realistic trajectories with only a few minutes of expert time.The paper elicits preferences by comparing candidate models.Source: §I · p. 1

II. RELATED WORK

  • Unlike inverse reinforcement learning, preference-based learning does not require demonstrations of optimal trajectories and has been applied in robotics –[12] and ACAS X objective balancing.It lets experts tune policies without specifying reward-parameter values [7].Source: §II · p. 1
  • Figure 1 compares rollouts of query policies from identical initial states, asking the expert to choose the policy that appears more realistic.Source: Figure 1 · p. 2
  • Prior preference-learning methods can learn policies directly or learn a reward first; reward learning can simplify parameter search and incorporate prior knowledge, but requires translating the reward into a policy.Source: §II · p. 2

III. APPROACH

The approach models low-altitude UAM trajectories with a stochastic policy whose reward parameters are learned from expert preferences. It compares policies in queries and fixes known reward parameters to incorporate prior knowledge.

III. APPROACH

  • The model generates UAM and intruder trajectories independently before combining them, reusing existing intruder models and focusing on low-altitude takeoff and landing procedures.Source: §III · p. 2
  • The method asks experts to compare policies rather than individual trajectories, with each query generated from policies solved under two sampled reward functions.Source: §III · p. 3
  • Known prior information, such as a high reward for landing, is encoded by fixing some reward parameters while leaving others for expert tuning.This avoids queries about behavior already specified by the task, such as policies that do not land.Source: §III · p. 3

A. Markov Decision Process with Preferences

  • The reward is parameterized, with some values fixed and others learned from the expert; restricting the learnable vector to an L1 norm of 1 imposes an implicit trade-off.Source: A. Markov Decision Process with… · p. 3
  • Value iteration computes optimal state-action values, and a softmax policy converts them into action probabilities, with precision λ controlling randomness.As λ approaches infinity, the policy approaches the maximizing-action policy; as λ approaches zero, actions become uniformly random.Source: A. Markov Decision Process with… · p. 3

B. Preference Model

The preference model maintains a Bayesian distribution over unknown reward parameters and updates it as expert query responses arrive. It allows for occasional expert errors and uses trajectory-set reward evaluations to compare preferences.

B. Preference Model

  • The Bayesian model maintains a distribution over unknown reward parameters and updates it using each expert response, starting from a uniform prior over the search space.Source: B. Preference Model · p. 3
  • A sigmoid likelihood models the probability of an expert response while allowing for occasional preference errors, following Sadigh et al..Source: B. Preference Model · p. 3
  • The trajectory-set reward evaluates a set under parameters w, and its feature trade-offs can affect algorithm performance; feature tuning may be needed.For stochastic query policies, the authors leave likelihood-based trajectory-set rewards for future work.Source: B. Preference Model · pp. 3–4
  • The posterior over reward parameters is estimated with MCMC, specifically the adaptive Metropolis algorithm, to generate samples at each iteration.Source: B. Preference Model · p. 4

C. Querying Methods

The querying methods seek informative comparisons while also addressing whether queries reflect realistic behavior and prior task knowledge. The proposed procedure samples reward parameters and rolls out policies obtained from the MDP.

C. Querying Methods

  • Continuous-input query optimization can maximize information gained about reward parameters but does not directly encourage realistic trajectories, making prior knowledge such as the need to land difficult to encode.Source: C. Querying Methods · p. 4
  • The method generates queries by rolling out two policies solved under different reward vectors sampled from MCMC estimates of the parameter distribution.Source: C. Querying Methods · p. 4
  • The sampled reward vectors provide candidate comparisons, and the querying heuristics select informative samples for the expert queries.Source: C. Querying Methods · p. 4

1) Multiobjective Optimization:

The first query-selection method uses a multiobjective optimization similar to query by disagreement. It balances plausible reward parameters against selecting sufficiently different samples for comparison.

1) Multiobjective Optimization:

  • The query-selection method uses a multiobjective optimization similar to Wilson et al.’s query by disagreement method.It operates on MCMC samples generated from each Bayesian update.Source: 1) Multiobjective Optimization · p. 4
  • One objective favors parameter samples likely under the current reward-function estimate, while another encourages enough difference to compare them.The sample probabilities are estimated with a Gaussian kernel density estimate over the M samples of w.Source: 1) Multiobjective Optimization · p. 4
  • The parameter µ controls the balance between the objectives, serving as a heuristic balance between exploration and exploitation.The samples are selected through the multiobjective optimization problem.Source: 1) Multiobjective Optimization · p. 4

2) Probabilistic Q-Eval:

Probabilistic Q-Eval approximates the permissible parameter region from samples and selects a query near a bisecting hyperplane that divides the samples evenly. The algorithm repeatedly generates policy comparisons from selected reward functions, obtains expert preferences, and updates the parameter distribution.

2) Probabilistic Q-Eval:

  • Probabilistic Q-Eval approximates the permissible region of reward parameters using samples, extending Q-Eval [15] to probabilistic preferences.In deterministic settings with consistent preferences, Q-Eval seeks queries that maximally reduce the permissible region.Source: 2) Probabilistic Q-Eval · p. 4
  • It estimates the region’s center with the sample mean, then considers bisecting hyperplanes close to that center.For the closest k hyperplanes, it estimates the volume ratio on each side from the number of samples there and selects the pair whose ratio is closest to 1.Source: 2) Probabilistic Q-Eval · p. 4

D. Algorithm Summary

  • Each iteration selects two reward-function samples, solves the MDP for their policies, and generates a query by rolling out both policies from sampled initial states.The expert’s preference updates the parameter distribution through MCMC sampling; iterations continue until a final reward-function estimate is obtained.Source: D. Algorithm Summary · p. 4

IV. UAM LANDING EXAMPLE

The landing example applies the developed methods to generate realistic UAM landing trajectories. Its state and action design aims to retain expressive modeling while keeping the MDP tractable.

IV. UAM LANDING EXAMPLE

  • The application uses the methods developed in the preceding section to generate realistic landing trajectories for UAM vehicles.An implementation is identified at the project repository.Source: §IV · p. 4

A. State and Action Space

  • State variables and discretizations were selected to be maximally expressive while keeping the MDP tractable.The passage points to Table I for the state variables, but does not provide its entries.Source: A. State and Action Space · p. 4, Table I · p. 5
  • Including the previous action in the current state allows the model to penalize acceleration changes (jerk) while preserving the Markov property.The vehicle chooses vertical and horizontal acceleration at each timestep; four possible values for each yield 16 joint actions, and transitions are deterministic from kinematics.Source: A. State and Action Space · p. 5

B. Reward Function

The landing reward uses five parameters: two fixed and three tuned from expert preferences. The design combines landing-related rewards and penalties, while using normalized features and average trajectory reward for comparing trajectories of differing lengths.

B. Reward Function

  • The reward function has five parameters that together promote a soft landing, with two fixed and three tuned by expert preferences.Table II is identified as outlining the parameters and their descriptions.Source: B. Reward Function · p. 5, Table II · p. 5
  • The fixed parameter ℓ ensures queried trajectories land, while parameter b penalizes slowing enough to move backward.The value of ℓ is kept large; the backward-motion penalty was added after that behavior appeared in initial tests.Source: B. Reward Function · p. 5
  • For this scenario, α and γ penalize jerk and acceleration, while β is intended to encourage a soft landing through a near-ground speed feature.The feature φβ is multiplied by β; its penalty magnitude increases as the vehicle approaches the ground, with hpen set to 50 ft.Source: B. Reward Function · p. 5
  • The reward parameters are constrained to have unit L1 norm and positive values, and the associated features are normalized to similar magnitudes.Average reward over trajectories is used because landing times differ; additional feature tuning normalizes average reward while preserving trade-offs among parameters.Source: B. Reward Function · p. 5

V. RESULTS

With μ tuned to 500, both querying methods approach the true reward function after about 40 queries, and multiobjective optimization approaches it faster. The parameter estimates and their distributions also move toward the true values.

V. RESULTS

  • The algorithm’s progress is measured by cosine similarity between estimated and true reward functions; an exact match scores 1.Higher values indicate closer estimates.Source: §V · p. 5

A. Parameter Tuning

B. Overall performance

  • After about 40 queries, both methods approach the true reward function, with tuned multiobjective optimization converging faster than probabilistic Q-Eval.Queries take under 15 seconds to generate; including expert response time, 40 queries take roughly 15 minutes.Source: B. Overall performance · p. 6, Figure 3 · p. 6
  • After 80 queries, the estimated distributions for individual parameters are centered near their true values.Figure 4 shows univariate Gaussian kernel density estimates, with vertical lines marking true values.Source: B. Overall performance · p. 6, Figure 4 · p. 6
  • The reward function is defined by α and β within a two-dimensional simplex, where the distribution shrinks toward the true values as preferences arrive.The initial distribution over the simplex is uniform.Source: B. Overall performance · p. 6

C. Effect of Human Error

The experiments model inconsistent expert answers by selecting the lower-reward policy with probability ϵ. The algorithm remains close to the true reward at ϵ = 0.2 after about 40 queries, but no longer approaches it at ϵ = 0.3.

C. Effect of Human Error

  • The experiment models an inconsistent answer by selecting the lower-reward policy according to wtrue with probability ϵ.The authors describe this random-error model as likely conservative because experts are more likely to err when policies are similar.Source: C. Effect of Human Error · p. 6
  • The modeled errors are randomly assigned, although experts may be more likely to answer inconsistently when the compared policies are similar.The authors contrast this with queries showing significantly different policies.Source: C. Effect of Human Error · p. 6
  • At ϵ = 0.3, the algorithm no longer approaches the true reward; at ϵ = 0.2, it remains close after approximately 40 queries.An error rate of ϵ = 0.1 causes minimal performance degradation.Source: C. Effect of Human Error · p. 6

D. Stochastic Model

The learned reward estimate is converted into a stochastic policy that can generate UAM encounter trajectories for simulation. The conclusion reports realistic trajectories without a large dataset, while identifying feature selection and stochastic query presentation as areas for future work.

D. Stochastic Model

  • The final reward estimate is used to solve for Q* and define a stochastic policy; rollouts use a precision parameter λ and can be sampled as needed for simulation.Varying λ changes the distribution of trajectories.Source: D. Stochastic Model · p. 6
  • Figure 5 tracks the bivariate density estimate of p(w) as preferences arrive, marking target parameters in red and showing the two query methods in separate rows.The top row is probabilistic Q-Eval; the bottom row is multiobjective optimization.Source: Figure 5 · p. 7
  • Figure 6 presents error analysis for both querying methods, with curves averaged over five trials.Source: Figure 6 · p. 7

VI. CONCLUSION

  • Expert preferences produce realistic UAM trajectories without a large collection of data, and the Bayesian preference model is robust to human error rates under 20%.The authors also report an accurate model-parameter estimate after a sensible number of expert queries.Source: §VI · p. 7
  • Effective feature selection remains difficult because features must encode experts’ internal trade-offs; future work will also examine queries that show stochastic policies.The current approach applies policy stochasticity after training.Source: §VI · p. 7
Loading 1907.05575v1…