Source-linked AI summary
Interactive Policy Learning through Confidence-Based Autonomy
Sonia Chernova, Manuela Veloso
TL;DR
Policy learning from demonstration needs an effective way to select informative demonstrations and handle mistakes during incremental practice. CBA combines confidence-based agent requests with teacher corrections, and in simulated driving its complete algorithm achieved the best overall learning performance while balancing demonstrations and incorrect actions.
Problem
Policy learning from demonstration requires methods for selecting useful demonstrations and correcting mistakes while an agent incrementally learns from a human teacher.
Method
CBA combines Confident Execution, which requests demonstrations in uncertain states, with Corrective Demonstration, which lets the teacher correct the agent’s mistakes.
Results
The complete CBA algorithm achieved the best overall learning performance in a complex simulated driving domain, reproducing teacher behavior while balancing demonstrations and incorrect actions.
Takeaways & Limitations
Confidence-based selection identified more informative demonstrations than human selection, while multiple adjustable thresholds required significantly fewer demonstrations than single-threshold selection.
Takeaways & Limitations
The reported results used a single teacher, so additional studies were proposed to evaluate usability and performance with non-technical users.
Abstract
from arXiv · showhide
We present Confidence-Based Autonomy (CBA), an interactive algorithm for policy learning from demonstration. The CBA algorithm consists of two components which take advantage of the complimentary abilities of humans and computer agents. The first component, Confident Execution, enables the agent to identify states in which demonstration is required, to request a demonstration from the human teacher and to learn a policy based on the acquired data. The algorithm selects demonstrations based on a measure of action selection confidence, and our results show that using Confident Execution the agent requires fewer demonstrations to learn the policy than when demonstrations are selected by a human teacher. The second algorithmic component, Corrective Demonstration, enables the teacher to correct any mistakes made by the agent through additional demonstrations in order to improve the policy and future task performance. CBA and its individual components are compared and evaluated in a complex simulated driving domain. The complete CBA algorithm results in the best overall learning performance, successfully reproducing the behavior of the teacher while balancing the tradeoff between number of demonstrations and number of incorrect actions during learning.
1. Introduction
Learning from demonstration programs autonomous agents by showing them desired behavior, while Confidence-Based Autonomy enables interactive policy learning with a human teacher. CBA selects demonstrations during task interaction and reduces demonstration requirements relative to human selection.
- Motivation: Learning from demonstration programs autonomous agents by recording demonstrated state-action pairs and learning a policy that reproduces observed behavior.Compared with exploration-based methods, it can reduce learning time and avoid defining a detailed reward function.
- Contribution: Confidence-Based Autonomy enables an agent with no initial knowledge to learn a policy incrementally through demonstrations acquired while practicing.Each demonstration provides the correct action for a particular state.
- Contribution: CBA gathers demonstrations through Confident Execution and Corrective Demonstration, combining agent-selected requests with teacher-provided corrections.Confident Execution uses automatically calculated confidence thresholds, while Corrective Demonstration improves the policy after mistakes.
- Results: In a complex domain, Confident Execution requires fewer demonstrations than demonstration selection performed by the human teacher.The evaluation compares five demonstration-selection techniques and finds teacher correction critical for optimizing policy performance.
2. Related Work
Prior work has used demonstrations for reinforcement learning, supervised policy learning, and active selection of informative examples. CBA is most closely related to confidence-based methods that request demonstrations when the agent is uncertain.
- Demonstration-Based Learning: Reinforcement-learning approaches use demonstrations to derive or modify reward functions and accelerate learning.The cited approaches treat demonstrations as reliable information for reinforcement learning.
- Demonstration-Based Learning: Supervised policy-learning approaches have applied regression, Bayesian networks, and k-nearest neighbors to skills, behaviors, games, and navigation.These methods span low-level skill acquisition and high-level behavior learning.
- Demonstration Selection: Active learning selects informative unlabeled examples for expert labeling to reduce the number of queries when labeling is expensive.The learner queries an expert rather than relying on a fixed demonstration set.
- Demonstration Selection: Related demonstration-selection systems request help when action choices are confused or identify situations where a robot should request a demonstration.CBA is most closely related to Dogged Learning, a confidence-based learning approach.
3. Confidence-Based Autonomy Overview
Confidence-Based Autonomy combines agent-initiated demonstration requests with teacher corrections in an interactive learning process. The agent uses classifier confidence to regulate autonomy, while the teacher supplies expert actions and corrects mistakes.
- Overview: Confidence-Based Autonomy consists of Confident Execution and Corrective Demonstration, which regulate autonomy and improve policies through supplementary demonstrations.The two components assign complementary roles to the learner and human teacher.
- Confident Execution: Confident Execution evaluates each state and chooses between autonomous execution and requesting a teacher demonstration.It targets unfamiliar states or states where the current policy action is uncertain.
- Policy Representation: The policy is learned with supervised learning under an MDP assumption and can use any classifier that provides classification confidence.The classifier returns an action, action-selection confidence, and a decision boundary for each query.
- Corrective Demonstration: Corrective Demonstration lets the teacher provide the action that should have been executed after an agent mistake.This addresses errors caused by overgeneralization or incomplete state-space data during incremental learning.
4. Confident Execution Algorithm
Confident Execution regulates autonomy by requesting demonstrations in unfamiliar or ambiguous states and acting autonomously when distance and classification-confidence thresholds are satisfied. It incrementally updates the policy from teacher demonstrations, with confidence and autonomy increasing as training data accumulate.
- Confident Execution: Confident Execution selects between autonomous action and teacher demonstration at each timestep using state familiarity and action-selection confidence.The agent evaluates its current state and requests help when the learned policy is insufficiently reliable.
- State regions: Unfamiliar states are distant from demonstrated states, whereas ambiguous states contain similar states associated with different actions.Additional demonstrations can help disambiguate overlapping action classes, while distance detects situations unlike previous demonstrations.
- Threshold criteria: The agent acts autonomously only when nearest-neighbor distance is below τdist and classification confidence is above τconf.These criteria divide the state space into high-confidence autonomy regions and low-confidence demonstration regions.
- Initialization: With no initial knowledge, τconf starts at infinity and τdist at 0, causing the agent to rely on demonstrations during initial learning.The algorithm begins with an empty training set and a classifier is unavailable at initialization.
- Interactive updates: The agent pauses to request a demonstration, adds the demonstrated state-action pair to its training set, and updates the classifier when the teacher responds.If the response is unavailable immediately, the non-blocking process continues at the next timestep while the agent remains aware of its surroundings.
- Learning progression: As demonstrations accumulate, fewer states lie far from training data, classification confidence and performance improve, and agent autonomy increases.Learning is complete when the agent repeatedly performs the desired behavior without requesting demonstrations.
5. Corrective Teacher Demonstration
Corrective Demonstration lets the teacher correct mistakes made during autonomous execution by attaching corrections to the state where the mistake occurred. Combined with Confident Execution, it forms the complete CBA algorithm.
- Motivation: Corrective Demonstration addresses high-confidence incorrect actions that Confident Execution cannot prevent through confidence thresholds alone.These errors typically arise from classifier over-generalization.
- Integration: Algorithm 2 combines Corrective Demonstration with Confident Execution into the complete Confidence-Based Autonomy algorithm.
- Correction process: During autonomous action execution, the algorithm records the preceding state and checks at every timestep for a teacher correction.A corrective demonstration adds a new training datapoint associated with the state that led to the autonomous action.
- Correction process: The teacher corrects an action relative to the agent’s previous state rather than simply specifying the next action.In the driving example, the teacher can correct an unsafe decision made while approaching another car.
6. Evaluation and Comparison
The simulated-driving evaluation compares teacher-guided selection, Confident Execution variants, Corrective Demonstration, and complete CBA. CBA combines confidence-based requests with corrective feedback to balance demonstrations against incorrect actions during learning.
- Domain and metrics: The driving domain represents state with lane number and noisy distances to the nearest cars in three lanes; policies are relearned after every 10 demonstrations.The distance features span [-25,25], and cars may be behind the agent.
- Domain and metrics: Lane preference measures similarity to the teacher’s driving style, while collision rate measures the percentage of timesteps spent contacting another car.The evaluation has no true reward function because the objective is imitation.
- Baseline: The teacher’s policy prefers the center and left lanes and avoids collisions entirely, establishing the target behavior for learned policies.All algorithms initially perform identically because learning starts from the same 300-demonstration dataset.
- Demonstration selection: After 1300 teacher-selected demonstrations, the learned policy closely matches expert lane preference but retains a 2.7% collision rate.Performance fluctuated substantially during training and stabilized after approximately 1100 demonstrations.
- Demonstration selection: CEM completed learning after 504 demonstrations, while its final lane preference remained similar to the expert with a 1.9% collision rate.Multiple adjustable thresholds focused requests on informative states and reduced redundant demonstrations.
- CBA: CBA learned collision-free highway navigation with 703 demonstrations by combining Confident Execution with Corrective Demonstration.Confident Execution supplied 83% of demonstrations, while Corrective Demonstration supplied 73% of the final 100 demonstrations and eliminated residual collisions.
7. Discussion
The discussion positions CBA as a fast, intuitive approach while identifying boundaries in teacher quality, interaction richness, and human attention. It also outlines extensions toward continued learning, richer dialogue, and multi-robot demonstration learning.
- 7. Discussion: CBA is presented as a fast, intuitive method for programming and adapting autonomous-agent behavior.Its general representation and classifier-independent design are described as suitable for a wide range of applications.
- 7.1 Evaluation with Non-Technical Users: The evaluation used a single teacher, so usability and performance remain to be studied across wider user groups, especially non-programmers.The authors specifically identify broader user studies as an additional direction.
- 7.3 Improvement Beyond Teacher Performance: Policy quality is inherently limited by the quality of the human demonstrations, although extending CBA with learning from the agent’s own experiences could improve performance beyond the teacher.Suggested extensions include high-level feedback or reward signals.
- 7.4 Continued Learning: Continued Confident Execution is proposed as a way to keep adapting the policy after the agent can perform repeatedly without further demonstrations or corrections.The standard alternative is to turn off learning and freeze the policy.
- 7.5 Richer Interaction: The current interaction is limited to agent requests and single recommended actions, motivating clarification questions, high-level advice, and replayable demonstrations.These richer interactions are proposed to speed learning and represent higher-level task goals.
- 7.6 Multi-Robot Learning: CBA also addresses limited human attention in multi-robot learning by preventing autonomous execution in low-confidence states.A separate multi-robot system uses individual CBA instances for each robot.
8. Conclusion
The conclusion presents CBA as interactive policy learning through demonstrations, combining confidence-based requests with corrective demonstrations during task practice. In simulated driving, the complete algorithm and Corrective Demonstration achieved the strongest final policy performance while balancing demonstrations against undesired actions during training.
- 8. Conclusion: CBA incrementally learns an action policy from demonstrations acquired while the agent practices the task.The algorithm uses shared decision making between the learner and teacher.
- 8. Conclusion: Confident Execution targets unfamiliar or uncertain states using confidence and distance thresholds, while Corrective Demonstration addresses specific mistakes after they occur.The teacher provides corrective demonstrations retroactively rather than anticipating errors.
- 8. Conclusion: In simulated driving, all confidence-based methods selected more informative demonstrations than the human teacher.The comparison included manual selection, fixed and adjustable confidence thresholds, corrective demonstration, and their combination.
- 8. Conclusion: The multiple adjustable-threshold approach required significantly fewer demonstrations by focusing on uncertainty and reducing redundant data points.This result compares the single-threshold and multiple-threshold confidence-based approaches.
- 8. Conclusion: Corrective Demonstration and complete CBA achieved the best final policy performance, matching the teacher’s lane preference without collisions.The methods jointly illustrate a tradeoff between demonstration count and undesired actions during training.