Source-linked AI summary
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Ben Shneiderman
TL;DR
The paper addresses the limitations and dangers of one-dimensional thinking about automation and human control. It presents the two-dimensional HCAI framework and argues that designs combining high human control with high automation can support RST systems and increase human performance.
Problem
Autonomy-centered design can create excessive monitoring demands and dangerous gaps in human coordination, training, or override mechanisms.
Method
The paper presents a two-dimensional HCAI framework that separates human control from computer automation and connects its design principles to RST systems.
Results
The framework shows how careful design can preserve human control while providing high levels of automation and supporting reliable, safe, and trustworthy systems.
Takeaways & Limitations
Successful HCAI designs can amplify, augment, enhance, and empower users while supporting responsibility, creativity, and human performance.
Takeaways & Limitations
Actionable guidelines for achieving responsibility, fairness, and explainability are beyond the scope of this paper.
Abstract
from arXiv · showhide
Well-designed technologies that offer high levels of human control and high levels of computer automation can increase human performance, leading to wider adoption. The Human-Centered Artificial Intelligence (HCAI) framework clarifies how to (1) design for high levels of human control and high levels of computer automation so as to increase human performance, (2) understand the situations in which full human control or full computer control are necessary, and (3) avoid the dangers of excessive human control or excessive computer control. The methods of HCAI are more likely to produce designs that are Reliable, Safe & Trustworthy (RST). Achieving these goals will dramatically increase human performance, while supporting human self-efficacy, mastery, creativity, and responsibility.
1. Introduction
The paper introduces HCAI as a two-dimensional framework separating human control from computer automation. It responds to problems with autonomy-centered design by seeking high levels of both control and automation.
- HCAI separates levels of human control from levels of automation or autonomy.
- Traditional one-dimensional autonomy models suggest that increasing automation requires lowering human control.
- Autonomous systems can increase human monitoring effort when users are unsure what the system will do, leading to inferior performance.
- Excessive autonomy has contributed to failures including Patriot missile incidents and Boeing 737 MAX crashes.
- Human-centered designs and creativity-support features are gaining attention among AI and creativity researchers.
2. Reliable, Safe & Trustworthy Systems
RST systems combine technical reliability practices, safety-oriented management, and independent oversight that supports trust. HCAI applies these structures while enabling highly automated technologies to preserve human control and support performance, responsibility, and creativity.
- RST systems are reliable through technical practices, safe through management strategies, and trustworthy through independent oversight.
- Reliability practices include audit trails, benchmark testing, data-quality and bias review, and analysis of past performance.
- Safety cultures rely on leadership commitment, internal problem reporting, and reviews of failures and near misses.
- Independent oversight structures build trust by reviewing the design, operation, and maintenance of products and services.
- HCAI designs let users steer, operate, and control highly automated devices, while automation can improve precision and reduce workload.
- The framework guides designs that promote human control alongside high automation and increase human performance.
3. The Human-Centered Artificial Intelligence for RST Systems
HCAI separates human control from computer automation, making high levels of both possible while identifying when full control, full automation, or safeguards against excess are appropriate. The framework applies across consumer, consequential, and life-critical systems to support Reliable, Safe & Trustworthy designs.
- Application scope: HCAI spans recommender systems, consequential applications, and life-critical systems, where harms can range from manipulated choices to irreversible consequences.Thoughtful design can improve user control in recommender systems, while life-critical systems such as self-driving cars and pacemakers require rapid action.
- Two-dimensional framework: The two-dimensional HCAI framework decouples human control from computer automation, enabling designs that achieve high levels of both.This contrasts with one-dimensional thinking that treats increased automation as requiring reduced user control.
- Design objectives: For complex tasks with varying contexts, high human control and high automation are needed, whereas predictable tasks can use high automation with less human control.Examples include creative decisions at the research frontier versus automatic transmission or skid control on normal highways.
- Design objectives: Some applications require rapid computer action without human intervention, while others prioritize human mastery, competence building, exploration, and creativity.Airbags, anti-lock brakes, pacemakers, and defibrillators illustrate rapid-action systems; bicycle riding, piano playing, and baking illustrate mastery-oriented activities.
- RST implementation: The framework’s practical challenge is developing effective, proven designs supported by trusted social structures, reliable practices, and safety cultures to gain RST recognition.A human-centered approach to cars adds appropriate controls while addressing unusual vehicles, construction zones, malicious interference, and emergency guidance.
- Failure boundaries: Designers must prevent excessive automation and excessive human control because both are identified as dangerous edge cases.The Boeing 737 MAX and Tesla Autopilot are presented as examples of excessive automation, while the framework also identifies excessive human control as a danger.
4. Prometheus Principles and Examples
The Prometheus Principles translate human-centered design into comprehensible, predictable, and controllable interfaces that support both human control and useful automation. Examples show how users express intent, receive feedback about machine state, and retain opportunities to monitor or adjust automated processes.
- Prometheus Principles: HCAI integrates artificial intelligence algorithms with user-interface designs to amplify and augment human abilities rather than treating automation and human control as alternatives.This reframes the AI-versus-intelligence-augmentation debate around their combination.
- Related Guidelines: Existing AI-human interaction guidelines emphasize user understanding and control, automation transparency, explanations of system behavior, and assistance with routine rather than higher-level cognitive tasks.The cited guidance spans initial use, normal use, problem handling, and changes over time.
- Prometheus Principles: The Prometheus Principles emphasize consistent interfaces, continuous displays, reversible actions, informative feedback, progress indicators, and completion reports.Together, these features help users express intent and understand system status throughout an interaction.
- Prometheus Principles: Successful designs are comprehensible, predictable, and controllable, supporting user self-efficacy and contributing to Reliable, Safe & Trustworthy systems.The paper links these qualities to validated theories, clear principles, actionable guidelines, and programming tools that preserve human control.
- Examples: Design decisions involve tradeoffs requiring careful study, creative design, rigorous testing, and continuous monitoring by domain experts.The paper presents these activities as necessary for refining implemented designs.
- Examples: Thermostats, vehicles, appliances, elevators, and cameras combine user-set intent with sensors, timers, displays, or automatic adjustments.These examples illustrate automation that responds to user choices while providing feedback, status information, or opportunities for intervention.
5. Summary, limitations, and conclusions
The HCAI framework separates human control from computer automation, supporting designs that provide high levels of both while clarifying when control should remain with humans or computers. It connects these designs to reliable, safe, and trustworthy systems, while identifying unresolved measurement, ethical, and human-factors challenges.
- Conclusions: HCAI separates human control from computer automation so well-designed systems can achieve high levels of both.Design choices should clarify machine state, user choices, mistake consequences, and reversibility.
- Conclusions: The framework guides control choices across consumer, consequential, and life-critical applications, including when rapid automated action or human mastery is paramount.It also identifies dangers associated with excessive automation or excessive human control.
- Limitations: Objective measures of control and autonomy tied to diverse tasks remain an important research direction for improving guidelines, evaluations, and theories.The paper presents these measures as a way to stimulate more meaningful design discussions.
- Limitations: Open challenges include de-skilling when automation fails, reduced vigilance when user actions become less frequent, and the lack of actionable guidelines for responsibility, fairness, and explainability.The HCAI framework lays a foundation for these goals, but actionable guidance is beyond the paper.
- Conclusions: Human-centered RST systems combine sound technical practices for reliability, open safety management, and independent oversight for trustworthiness.Monitoring failures and near misses supports continued system improvement.
- Conclusions: HCAI is intended to shift design from one-dimensional automation thinking toward technologies that provide appropriate user control alongside high automation.Successful systems are described as amplifying, augmenting, enhancing, and empowering users.