Source-linked AI summary
Adaptive Shared Control with Online Bounded-Rational Human Behavior Estimation
Henry Ascencio Trejo, Roel Pieters, Gokhan Alcan
TL;DR
The paper addresses adaptive shared control when human behavior is boundedly rational and unknown rather than fully rational. It learns level-k candidate policies, estimates their probabilities online from state-transition residuals, and computes distribution-aware robot assistance. Simulations report decreasing KL divergence and lower robot accumulated running cost than alternative policy baselines.
Problem
The robot must adapt assistance to boundedly rational human behavior without reducing uncertainty to a single assumed human policy.
Method
The method builds a finite level-k policy bank with ADP, updates a probabilistic human model from state-transition residuals, and minimizes expected cooperative cost over the complete distribution.
Results
The simulations show decreasing KL divergence between estimated and simulated human behavior distributions and lower robot accumulated running cost than maximum-probability and probability-weighted alternatives.
Takeaways & Limitations
The proposed shared-control mechanism adapts assistance while accounting for the complete probabilistic model of boundedly rational human behavior.
Takeaways & Limitations
The approach assumes the task and human intention are known and remain fixed during interaction, while measurement noise and plant-model mismatch can reduce candidate distinguishability.
Abstract
from arXiv · showhide
This work considers adaptive shared human-robot control for nonlinear control-affine systems, where the assumption of a fully rational human is relaxed and the robot adapts its assistance to observed boundedly rational human behavior. We use a level-k bounded-rationality model of the two-player game to construct a finite bank of candidate human and robot policies through alternating best-response computations, with the associated value functions and policies approximated using adaptive dynamic programming. During the shared-control interaction, state-transition residuals compare the measured system evolution with the trajectories predicted by the candidate human policies. The residuals are accumulated using a forgetting factor and mapped to a probabilistic human-behavior model over the finite candidate bank. Rather than selecting a single candidate or averaging stored robot policies, the robot computes a distribution-aware one-step best response by minimizing an expected cooperative cost over the complete estimated human behavior distribution. For a quadratic terminal-value approximation and Euler state propagation, this response admits a closed-form solution expressed in terms of the expected human input. The proposed methods are evaluated in simulations of a benchmark nonlinear system stabilization task, and of a planar manipulator shared control setup. The reported results show decreasing Kullback-Leibler divergence between the estimated and simulated human behavior distributions, and a lower accumulated running cost for the robot agent over the shared control interaction period, than the maximum-probability and probability-weighted alternative policies baseline.
1 Introduction
Shared control must balance human agency and robot assistance while accounting for human behavior that may be boundedly rational rather than fully rational. This work estimates such behavior online and uses the resulting distribution to adapt robot assistance.
- Motivation: Bounded-rational human behavior can make fully rational assumptions inconsistent with observed decision-making and shared-control needs.
- Approach: The proposed framework estimates candidate level-k human behaviors online from observed state transitions and updates their probabilities during interaction.
- Evaluation: The paper evaluates the framework in simulation for nonlinear stabilization and a planar manipulator shared-control setup.
- Approach: Rather than selecting one likely human policy or averaging robot policies, the robot computes assistance using the complete estimated human-behavior distribution.
2 Problem Formulation
The problem is adaptive assistance for a nonlinear human-robot system in which the human policy is unknown, possibly time-varying, and boundedly rational. The formulation specifies admissible dynamics, cooperative objectives, behavior estimation, and causal robot assistance based on interaction history.
- 2.1 Human-Robot Cooperative System: The shared system is nonlinear and control-affine, with simultaneous human and robot inputs affecting the state dynamics.
- 2.1 Human-Robot Cooperative System: The dynamics and admissible policies are assumed to support forward-complete closed-loop trajectories with finite infinite-horizon cost.
- 2.1 Human-Robot Cooperative System: The cooperative game defines human and robot performance objectives and uses admissible policy sets for the two agents.
- 2.2 Shared Human-Robot Control: Unlike the fully rational equilibrium policy, the human may use an unknown admissible policy reflecting bounded reasoning, uncertainty, or changing assumptions about the robot.
- 2.2 Shared Human-Robot Control: The robot must infer assistance causally from the measurement and applied-input history because the human policy is not known a priori.
- 2.2 Shared Human-Robot Control: An adaptive shared-control policy uses an online human-behavior description produced by an estimation mechanism from the available history.
- 2.2 Shared Human-Robot Control: The resulting closed loop remains admissible, pursues the cooperative task, and seeks reduced performance cost; stabilization targets convergence to the origin with finite control effort.
3 Adaptive Shared Control Framework
The framework has two stages: learn finite candidate level-k human and robot behaviors, then adapt robot assistance using an estimated distribution over those behaviors.
- Framework: The first stage iteratively constructs candidate level-k human and robot policies for different bounded-rational behaviors using adaptive dynamic programming.
- Framework: The second stage uses estimated human behavior during shared control to determine the robot’s adaptive assistance.
4 Construction of the Candidate Level-k Policy Bank
The paper constructs a finite level-k policy bank by alternating human and robot best responses, then approximates the associated value functions and policies with data-driven adaptive dynamic programming.
- The candidate bank is built through alternating best responses, with each agent responding to the other agent’s previously computed policy.
- The level-0 human is non-strategic toward the robot and computes its policy without including the robot contribution in its internal dynamics model.
- The online bank retains paired human candidates and robot best-response policies under a common finite index set for later estimation and assistance.
- Each candidate value function is approximated offline with a critic neural network and updated using instantaneous and stored Hamiltonian-residual data.
- The ADP guarantees require assumptions such as an informative history stack, rank conditions, and bounded approximation errors.
5 Online Adaptive Shared-Control Interaction
During interaction, the robot infers boundedly rational human behavior from state-transition residuals and uses the resulting distribution to compute adaptive assistance. Under quadratic terminal values and Euler propagation, the distribution-aware response has a closed form.
- 5.1 State-transition Residuals: State-transition residuals compare measured system evolution with candidate human-policy predictions using the same initial state and robot action.
- 5.1 State-transition Residuals: Measurement noise and plant-model mismatch can reduce the distinguishability of candidate human behaviors inferred from state transitions.
- 5.2 Probabilistic Human-Behavior Model and Online Update: Accumulated residuals with forgetting are mapped by a softmax so candidates predicting smaller residuals receive larger human-behavior probabilities.
- 5.3 Distribution-Aware One-Step Best Response: The robot minimizes an expected cooperative cost over the complete estimated human-behavior distribution rather than selecting a single candidate.
- 5.4 Closed-Form Response for a Quadratic Terminal Cost: With a quadratic terminal-value approximation and Euler discretization, the one-step optimization admits a closed-form response when the objective is strictly convex in the robot input.
- 5.4 Closed-Form Response for a Quadratic Terminal Cost: The closed-form solution uses the expected human action and avoids numerical optimization at every interaction interval, although candidate forward simulations remain necessary for residual estimation.
6 Simulation Experiments
The evaluation uses simulations of a benchmark nonlinear system and a planar manipulator shared-control task to assess behavior estimation and adaptive response.
- The experiments evaluate the proposed methods on a benchmark nonlinear system and a planar robotic manipulator jointly controlled by a simulated human and robot.
- The procedures test whether probabilistic estimation describes observed human behavior relative to the finite level-k candidate bank.
- The procedures also test whether the distribution-aware one-step response reduces accumulated running cost against two baselines using the estimated human-behavior model.
6.1 Benchmark Nonlinear System
The benchmark constructs level-k human and robot policy banks, estimates unknown human behavior online, and evaluates distribution-aware robot responses during nonlinear stabilization. The proposed response tracks the human distribution while reducing running cost relative to alternative response rules.
- Learning the rationality levels model: The benchmark uses a four-dimensional nonlinear control-affine system with separate two-dimensional human and robot inputs and a cooperative stabilization objective.Training acquires up to five rationality levels for each agent through ten total thinking steps and repeated rollouts.
- Learning the rationality levels model: Complementary bounded-rational human and robot policy combinations accomplish stabilization while generating different state trajectories.The learned critic weights parameterize the policy bank used for the cooperative task.
- Performing the Shared Control Interaction with an unknown human behavior: The estimated PHM’s KL divergence decreases at the reported times for all three simulated human-behavior scenarios.Endpoint divergences are 0.042 and 0.036 for the level-∞ and level-5 scenarios, compared with 0.002 for the remaining scenario.
- Performing the Shared Control Interaction with an unknown human behavior: All robot-response alternatives stabilize the system, but they produce different state trajectories and control-effort distributions.The proposed response uses lower robot-input values overall and redistributes more effort toward the human.
- Performing the Shared Control Interaction with an unknown human behavior: The proposed response reduces accumulated running cost by 40.1% and 27.5% relative to the maximum-probability and probability-weighted baselines, respectively.The total human-plus-robot control effort is also lower, although effort shifts toward the human.
6.2 Planar Manipulator System
The planar manipulator study builds a level-k policy bank, estimates unknown human behavior online, and evaluates a distribution-aware robot response during shared control. The proposed response achieves the stabilization task with smoother, lower-input behavior and improving PHM estimates.
- System setup: The experiment models a two-link planar manipulator in which the human applies end-effector force and the robot controls joint torques toward a desired configuration.The system uses a four-dimensional state-space control-affine representation with separate human and robot input matrices.
- Policy-bank construction: Candidate human and robot behaviors are generated through level-k training up to level 5, producing a finite policy bank for the shared-control interaction.The associated value functions use six-neuron quadratic basis approximations, and the resulting policy bank is shown for the manipulator.
- Evaluation procedure: The interaction evaluates the robot’s response to an unknown human rationality using the estimated probabilistic human model derived from transition residuals.The manipulator experiment separately assesses PHM estimation and the robot response using the one-step distribution-aware solution.
- PHM estimation: KL divergence decreases in all three manipulator scenarios, although the level-3 case remains least accurate at 0.476 versus 0.156 and 0.248 at 60 s.The estimated PHM follows the ground-truth distributions’ overall shapes, while its predominant level matches the ground truth.
- Shared-control response: The distribution-aware robot response completes the task with smoother behavior, lower control inputs, and faster accomplishment than the alternative policy responses.The reported lower inputs and state values translate into more optimal robot performance in the manipulator interaction.
- Scope and limitation: The method assumes that the human’s task and intention are known and remain fixed throughout the interaction.Handling uncertain human intention would require additional computation and potentially retraining the candidate policies.
7 Conclusions
The paper proposes adaptive shared control that models boundedly rational human behavior online and uses the resulting distribution to select robot assistance. Across two nonlinear-system simulations, the methods estimate human behavior, achieve stabilization, and support an efficient closed-form robot response.
- Contributions: The method estimates boundedly rational human behavior online from a level-k policy bank and computes robot assistance over the complete estimated probabilistic human model.The policy bank is obtained through iterative adaptive-dynamic-programming training, while the human model is updated during interaction.
- Contributions: For quadratic basis functions and Euler discretization, the robot’s one-step best response has an efficient closed-form solution.This solution is derived for the terminal-cost computation used in the proposed response.
- Results: Simulations on two nonlinear systems show stabilization while accounting for boundedly rational human behavior and support the online human-model estimation procedure.The results also support defining the robot response from the complete probabilistic human model rather than a single estimated behavior.