PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Situation Awareness Assessment Approaches in Aviation Environments

Thanh Nguyen, Chee Peng Lim, Ngoc Duy Nguyen, Lee Gordon-Brown, Saeid Nahavandi

arXiv:1803.08067v3cs.HC

TL;DR

Aviation SA assessment requires methods that capture pilots’ and other operators’ awareness across individual, team, and system contexts. This review synthesizes six assessment categories, theoretical models, quantitative and qualitative approaches, and unmanned-vehicle issues, concluding that combined measures are recommended despite method limitations.

  • Problem

    Aviation needs reliable ways to assess situation awareness because SA affects flight decisions and safety, while existing methods have important limitations.

  • Method

    The paper reviews six SA assessment categories alongside individual, team, shared, distributed, and system-level models, covering quantitative, qualitative, and unmanned-vehicle perspectives.

  • Results

    The review finds that no single assessment method has been discarded, and it recommends combining several measures to support concurrent validity.

  • Takeaways & Limitations

    Accurate SA assessment can support understanding pilot behavior and guide training intended to improve pilot performance and aviation safety.

Abstract

from arXiv · show

Situation awareness (SA) is an important constituent in human information processing and essential in pilots' decision-making processes. Acquiring and maintaining appropriate levels of SA is critical in aviation environments as it affects all decisions and actions taking place in flights and air traffic control. This paper provides an overview of recent measurement models and approaches to establishing and enhancing SA in aviation environments. Many aspects of SA are examined including the classification of SA techniques into six categories, and different theoretical SA models from individual, to shared or team, and to distributed or system levels. Quantitative and qualitative perspectives pertaining to SA methods and issues of SA for unmanned vehicles are also addressed. Furthermore, future research directions regarding SA assessment approaches are raised to deal with shortcomings of the existing state-of-the-art methods in the literature.

I. INTRODUCTION

Situation awareness supports aviation decision-making and safety, while its loss contributes to accidents and training challenges. This review surveys SA models and assessment methods, including their applicability and limitations.

  • Motivation: SA assessment matters because non-technical SA competencies support accident prevention, pilot training, and flight safety.The introduction identifies decision-making, crew cooperation, and systems management as related competencies.
  • SA foundations: Loss of SA occurs when a pilot’s mental model diverges from reality, requiring frantic reassessment after the discrepancy becomes apparent.The paper links this failure process to aviation accidents and incidents, including CFIT.
  • SA foundations: SA comprises perception, comprehension, and prediction of environmental events across three levels.Level 1 observes critical information, level 2 integrates and interprets it, and level 3 predicts possible events.
  • Review scope: The review organizes SA assessment into six categories: freeze-probe, real-time probe, post-trial self-rating, observer rating, process indices, and performance measures.It covers aviation settings including cockpits, air traffic control, and unmanned air vehicles.
  • Assessment methods: Freeze-probe methods directly compare responses at task freezes with the environment’s actual state, but disrupt tasks and remain difficult to use in real-world exercises.Validity concerns include measuring memory rather than SA and prompting recall of information the subject may not have noticed.

B. Real-time probe techniques

Real-time probes assess SA during task execution without freezing the task, whereas self-ratings are simpler and less intrusive but depend on imperfect post-trial recall. The methods therefore trade reduced disruption against intrusion, bias, and validity concerns.

  • B. Real-time probe techniques: Real-time probes administer SA queries during task execution, recording answers and response times for scoring.SPAM and SASHA_L are examples of this approach.
  • B. Real-time probe techniques: Real-time probes reduce disruption relative to freezes and can be used in the field, but queries may bias responses and burden experts in dynamic environments.Assessing team or shared SA is also difficult with this approach.
  • C. Post-trial self-rating techniques: Post-trial self-ratings are quick, inexpensive, non-intrusive, and potentially applicable to team SA through ratings from individual members.Examples include SART, SARS, C-SAS, CARS, MARS, and QUASA.
  • C. Post-trial self-rating techniques: Self-ratings can reflect performance selectively, omit poorly recalled periods, and inaccurately assess poor SA because participants may not recognize it.Post-trial questionnaires primarily measure awareness recalled at the task’s end.
  • Other assessment categories: Observer ratings may be biased or invalid because observers can misjudge SA, participants may change behavior, and multiple subject-matter experts are required.Performance measures likewise assume efficient performance corresponds to efficient SA, an assumption that can fail for experts.

D. Observer-rating techniques

Observer-rating techniques assess SA through subject-matter experts’ observations of behavior during task execution. They are non-intrusive but cannot precisely measure internal SA and require substantial observer access.

  • D. Observer-rating techniques: Observer-rating techniques use subject-matter experts to rate observable SA-related behavior during task execution.SABARS is a typical example using a five-point rating scale.
  • D. Observer-rating techniques: Their non-intrusive nature allows application to real-world activities without affecting the task being executed.
  • D. Observer-rating techniques: Observer ratings may not reflect internal SA because strong performance does not necessarily indicate good SA.
  • D. Observer-rating techniques: Observation can introduce behavioral bias and requires frequent access to multiple SMEs over long durations.

E. Performance measures

The supplied passages contrast performance-based and process-based approaches with broader SA models. Performance measures use task outcomes indirectly, while process indices examine how subjects perform tasks and allocate attention.

  • E. Performance measures: Performance measures rate task achievements through recorded performance characteristics as an indirect measure of SA.They are non-intrusive and often serve as backup measures for other techniques.
  • E. Performance measures: Process indices record and analyze task processes, including eye movements, to evaluate fixation and attention allocation.
  • E. Performance measures: Table I summarizes strengths, weaknesses, typical methods, and relevant papers across six SA assessment categories.
  • E. Performance measures: Individualistic SA models treat awareness as a cognitive phenomenon in operators’ heads and assume available ground truth and normative standards.
  • E. Performance measures: Team SA assessment shifts attention from individual awareness toward coordinated or shared awareness among multiple stakeholders.
  • E. Performance measures: CAST assesses team cognition through task-based interactions during performance and can provide diagnostic information about team-member connections.
  • E. Performance measures: Situated shared SA reduces working-memory demands by relying on environmental information, but forming a common picture remains difficult in dynamic environments.

C. System and distributed SA

System and distributed SA models analyze awareness across socio-technical systems rather than only individuals or teams. They emphasize interactions, information sharing, and model selection suited to the problem’s nature.

  • C. System and distributed SA: Socio-technical systems combine humans and technical elements in complex, non-deterministic, and often non-linear interactions.
  • C. System and distributed SA: The Event Analysis of Systemic Teamwork uses task, social, and SA networks to analyze goals, organization, communication, and information sharing.
  • C. System and distributed SA: No single SA model is universally superior because each addresses problems with different fundamental characteristics.The review describes a range from stable, normative individual problems to non-normative, non-stable socio-technical-system problems.
  • C. System and distributed SA: Distributed SA treats the entire socio-technical system as the unit of analysis, with cognitive processes distributed across human and technical agents.
  • C. System and distributed SA: Distributed SA requires the right information to reach the right agent at the right time, with compatible awareness maintained through system transactions.
  • C. System and distributed SA: Risk SA and risk DSA represent threats and vulnerabilities at individual and socio-technical-system levels, respectively.RiskSOAP numerically expresses a positive correlation between distributed SA and complex-system safety.

IV. SURVEY OF RECENT SA APPROACHES IN AVIATION

Aviation SA research focuses primarily on pilots and air-traffic controllers, while increasingly addressing unmanned air vehicles. Recent work includes quantitative and qualitative assessment methods.

  • IV. SURVEY OF RECENT SA APPROACHES IN AVIATION: Aviation SA studies mainly examine pilots and air-traffic controllers, with emerging research on unmanned air vehicles.
  • IV. SURVEY OF RECENT SA APPROACHES IN AVIATION: The review organizes recent aviation approaches into quantitative and qualitative SA methods.The qualitative overview is divided among pilots, air-traffic controllers, and unmanned-air-vehicle contexts.

A. Quantitative SA methods

Quantitative SA approaches model attention, perception, memory retrieval, rule selection, and cognitive levels, using measures ranging from psychophysiology to performance and eye movements. These models have been validated against multiple SA indicators and operational outcomes.

  • Psychophysiological measures are attractive because they are unobtrusive and continuous, and continuous EEG discriminated three SA levels with acceptable accuracy in twelve male participants.These measures connect operator cognitive activity with associated physiological changes.
  • The MIDAS-based approach measures actual SA relative to optimal SA as an SA ratio ranging from 0 to 1.Actual SA reflects perceived and comprehended required and desired situation elements, while optimal SA assumes all are understood.
  • The quantitative SA model was validated in a high-fidelity two-pilot landing simulation and was sensitive to differences in display designs and pilot responsibilities.The validation was presented as preliminary progress toward predicting multioperator SA from procedures and display designs.
  • The attention allocation model represents SA using situation-element importance, cognitive status, Bayesian conditional probabilities, and allocated attention resources.Attention allocation is computed from element-specific resources and normalized across situation elements.
  • The attention allocation model’s predicted SA was greatly correlated with operation performance, SAGAT and 3-D SART correct rates, and physiological features in experiments with 20 pilots.The reported physiological indicators included pupil diameter, blink frequency, and eyelid opening.
  • The ACT-R extension maps SA processing from selective visual attention and chunk retrieval to rule matching, comprehension, prediction, and motor execution.The model treats perception as SA level 1 and links comprehension and prediction through procedural and declarative memory processes.

1) SA for pilots:

Pilot SA research applies quantitative models, physiological and eye-movement measures, simulation, and training assessments to cockpit design and pilot performance. Findings also expose boundaries involving SA coverage, measurement validity, and the complexity of model-based approaches.

  • 1) SA for pilots:: The reviewed models have limitations including incomplete coverage of SA levels, complex or uncertain mental-model components, and possible divergence between correct rates and performance scores.One listed model focuses only on perception, while another is limited to levels 1 and 2; performance-based assumptions may also weaken correspondence with SA.
  • 1) SA for pilots:: The attention allocation model was reported as useful for predicting changing SA trends during task performance and correlated with SAGAT, 3-D SART, and physiological features.The summary table identifies these correlations as a principal result of the model.
  • 1) SA for pilots:: Quantitative SA approaches include situation-element models, attention allocation models, ACT-R models, concurrent verbal protocols, and performance or process indices.The reviewed approaches connect SA assessment with cognitive processing, task performance, eye movements, and cockpit simulation.
  • 1) SA for pilots:: ACT-R-based SA models combine attention allocation with cognitive-process analysis and can guide cockpit display designs intended to reduce pilot errors.The model was evaluated using SAGAT, 10D SART, operation scores, and eye-movement indices.
  • 1) SA for pilots:: Eye movements distinguish experienced from novice pilots, while fixation rates, dwell times, and scan entropy assess different SA levels during malfunction scenarios.The cited studies used fixation and dwell measures for level 1 SA and entropy for level 3 SA.
  • 1) SA for pilots:: Cockpit simulations use virtual instruments, flight visuals, control systems, SAGAT, and physiological data to evaluate display-interface designs objectively.The cited setup included a human-in-the-loop experiment measuring SA degrees and heart rate.

2) SA for ground air traffic controllers:

Ground-controller research examines SA in distributed air-ground operations, automation-supported environments, and high-workload interfaces. Studies address trust, visual salience, data integration, and the risks of monitoring automation.

  • 2) SA for ground air traffic controllers:: Distributed air-ground traffic-management simulations examine controller and pilot SA while teams coordinate weather avoidance, spacing, merging, and continuous-descent operations.The scenario involved eight pilots and controllers working with automation across en-route and approach phases.
  • 2) SA for ground air traffic controllers:: Lower trust in automation was associated with higher SPAM probe accuracy among student air traffic controllers during high-traffic conditions.The study followed twelve student controllers over a 16-week internship.
  • 2) SA for ground air traffic controllers:: Situated SA proposes that operators off-load information processing and storage to external tools rather than maintaining highly detailed internal representations.The approach is presented for individuals and teams in next-generation air transportation systems.
  • 2) SA for ground air traffic controllers:: Air-ground SA support can combine weather, airspace, airport, and aircraft data, but big-data solutions introduce unresolved problems.The review identifies machine learning, visual analytics, and text analytics as relevant processing developments while retaining cautions about their use.
  • 2) SA for ground air traffic controllers:: Screen-design research relates salience gaps to air traffic controllers’ task performance to develop policies for directing visual attention under high cognitive workload.The cited simulations examined visual attention as a high-impact perception for controller tasks.

3) SA for unmanned air vehicles:

Unmanned-vehicle situation awareness can complement human operators as autonomy increases, but responsibility for SA shifts between humans and vehicles. The review covers artificial SA capabilities, autonomy-linked allocation, and applications such as beyond-visual-line-of-sight operation.

  • 3) SA for unmanned air vehicles:: Unmanned-vehicle SA can support human SA and mission success, but it does not map one-to-one onto human SA.The review notes that artificial intelligence and perception technologies can give unmanned vehicles capabilities that exceed human SA in some respects.
  • 3) SA for unmanned air vehicles:: Unmanned vehicles can outperform humans in perception through persistent information storage, greater collection capacity, and sensing beyond human perceptual ranges.The review highlights monitoring in dull, dirty, and dangerous 3D environments, along with examples such as night vision and extended auditory perception.
  • 3) SA for unmanned air vehicles:: Unmanned-vehicle comprehension transfer can reduce novice-user effects that arise when inexperienced humans lack the mental models developed through training and experience.The proposed benefit concerns transferring comprehension between unmanned entities rather than reproducing human SA exactly.
  • 3) SA for unmanned air vehicles:: Unmanned systems may mitigate limits on human level-3 SA by processing more information and incorporating memory, decision-making, and mental models.The review frames prediction as cognitively demanding and affected by workload, mental capacity, and environmental stressors.
  • 3) SA for unmanned air vehicles:: Beyond visual line of sight, SA responsibility shifts between the human operator and the increasingly autonomous vehicle, requiring artificial SA comparable to the operator’s.The review also describes spatial projection of traffic vehicles for highly autonomous UAS operating in terminal areas.
  • 3) SA for unmanned air vehicles:: As autonomy rises, unmanned-vehicle SA increases while human SA decreases, shifting SA responsibility from the operator toward the vehicle.At low autonomy, the human retains control and responsibility; at high autonomy, the system must possess high SA for safe and successful operation.

V. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS

The discussion emphasizes that accurate SA assessment supports pilot training and flight safety while requiring multiple complementary measures. It also identifies human-autonomy teaming and adjustable autonomy as important directions for future research.

  • V. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS: Accurate SA assessment can inform guidelines and training programs aimed at improving pilot performance and flight safety.The review links many aviation SA errors to failures to monitor, discriminate, detect, perceive, or remember relevant information.
  • V. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS: The review covers six assessment categories and recommends combining several measures to support concurrent validity despite their individual limitations.The categories are freeze-probe, real-time probe, post-trial self-rating, observer rating, process-index, and performance-measure techniques.
  • V. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS: SA varies with task expertise, with novice pilots less proficient at anticipating future aircraft states than experienced pilots.The review attributes part of this difference to less flexible visual scanning and associated differences in perceiving and interpreting cockpit information.
  • V. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS: Future human-autonomy teaming research must address shared SA, transparency, communication, trust, timing, overconfidence, and machine ethics.The review identifies interactive collaboration between humans and machines as less studied than human teaming.
Loading 1803.08067v3…