PaperlayerView exact revision on arXiv

Source-linked AI summary

Systematic Literature Review of Validation Methods for AI Systems

Lalli Myllyaho, Mikko Raatikainen, Tomi Männistö, Tommi Mikkonen, Jukka K. Nurminen

arXiv:2107.12190v1cs.SE

TL;DR

Practical AI systems are difficult to validate because their behavior and contexts can change, while prior work often centers on ML models. This systematic review analyzes 90 papers and synthesizes four validation-method categories. Its taxonomy offers a basis for further synthesis, although most included studies concern ML.

  • Problem

    AI-system validation is challenging, while earlier reviews often focus on ML models or offline testing rather than practical, complete systems.

    Source: §2.1 · p. 2, §2.2 · p. 3

  • Method

    The authors systematically reviewed 90 papers and analyzed their systems, validation methods, and study quality.

    Source: §4.3 · p. 7, §10 · p. 18

  • Results

    The review identifies four validation-method categories: simulation, trial, model-centred validation, and expert opinion.

    Source: §10 · p. 18

  • Takeaways & Limitations

    The taxonomy provides a basis for further synthesis of AI-system validation methods and proposals for how to validate systems.

    Source: §10 · p. 18

    Because most included papers describe ML systems, the findings are safest to interpret as concerning ML literature rather than AI in general.

    Source: §9 · p. 17

Abstract

from arXiv · show

Context: Artificial intelligence (AI) has made its way into everyday activities, particularly through new techniques such as machine learning (ML). These techniques are implementable with little domain knowledge. This, combined with the difficulty of testing AI systems with traditional methods, has made system trustworthiness a pressing issue. Objective: This paper studies the methods used to validate practical AI systems reported in the literature. Our goal is to classify and describe the methods that are used in realistic settings to ensure the dependability of AI systems. Method: A systematic literature review resulted in 90 papers. Systems presented in the papers were analysed based on their domain, task, complexity, and applied validation methods. Results: The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions are methods used to continuously validate the systems after deployment. Conclusions: Our results clarify existing strategies applied to validation. They form a basis for the synthesization, assessment, and refinement of AI system validation in research and guidelines for validating individual systems in practice. While various validation strategies have all been relatively widely applied, only few studies report on continuous validation. Keywords: artificial intelligence, machine learning, validation, testing, V&V, systematic literature review.

1. Introduction

The paper addresses how practical AI systems can be trusted, since AI's adaptability and unpredictability complicate traditional software testing. It focuses on validating complete AI systems in realistic contexts, rather than only their ML models.

1. Introduction

  • AI's adaptability and unpredictability complicate traditional software expectations and make trustworthiness a pressing concern.AI tools such as TensorFlow and PyTorch also make these techniques implementable by people with little domain knowledge.Source: §1 · p. 1
  • Prior work largely emphasizes ML-model correctness and robustness, leaving validation of complete AI systems and applications less addressed.The paper defines its focus as practical systems: complete systems performing purposeful functions in realistic contexts.Source: §1 · p. 1
  • The review includes AI beyond ML and covers multiple domains and tasks, including studies that validate practical systems or apply validation proposals to them.The authors exclude theoretical settings and solution proposals not applied to a practical system.Source: §1 · p. 1
  • Validation means checking that a system works as intended in context, and continuous validation covers measures used to ensure it keeps doing so after deployment.The paper distinguishes simple, complete model-centred systems from systems such as autonomous robots and surveillance systems with autonomous access control.Source: §1 · p. 2
  • The authors use a systematic literature review because existing research on practical AI-system validation had not been synthesized.They searched four databases and selected papers based on title, keywords, and abstract before closer assessment.Source: §1 · pp. 1–2

2. Previous work: AI Systems and Validation

AI systems can adapt or change after deployment, making it difficult to specify requirements, determine desired outcomes, and decide when validation is needed. Earlier work has focused on ML and model validation, leaving a need to synthesize methods for validating practical, complete systems.

2.1. Background

  • The paper distinguishes system validation—whether the intended product meets stakeholder needs—from ML model validation, which often tests generalization ability.Verification and other V&V activities beyond validation are outside the paper's scope.Source: §2.1 · p. 2
  • Validating adaptive AI is difficult because requirements can be complex, desired outcomes may be unknown in advance, and changing environments can make models outdated.The paper frames validation as concerning both what to validate and when.Source: §2.1 · p. 2
  • Continuous validation refers to post-deployment observation and measures intended to ensure that systems work as intended in situations unseen during development or training.Prior literature uses differing terms, including online testing and monitoring testing.Source: §2.1 · p. 3

2.2. Related work

  • AI validation challenges include the scale and diversity of operating contexts; one cited estimate says demonstrating autonomous cars' lower fatal-accident risk would require 275 million test miles.Even simulation can face the enormous state space of AI systems if it is not applied purposefully.Source: §2.2 · p. 3
  • Prior work reports difficulty finding suitable validation tools, while this review focuses on methods those tools might support.Examples mentioned include Simulink, SOCRaTEs, and Dronology.Source: §2.2 · p. 3
  • Prior reviews and guidelines focus on ML and often model-centred or offline testing, while identifying a need for more knowledge about online testing of entire systems.The paper positions its review around validation methods for practical systems.Source: §2.2 · p. 3

3. Research questions

The paper asks what practical AI systems are represented in research and what validation methods are used with them. It also frames synthesis of validation methods, including continuous validation, as useful to researchers and system builders.

3. Research questions

  • RQ1 asks which practical AI systems and characteristics are presented and validated in the literature.The authors aim to identify what is covered and where research gaps may exist.Source: §3 · p. 3
  • RQ2 asks which validation methods are used with practical AI systems, including methods for continuous validation.The study aims to synthesize scattered information about the methods and their implementation.Source: §3 · p. 3
  • The authors state that knowledge of available and effective validation methods can inform development of AI-based systems and appropriate safety and reliability measures.Source: §3 · p. 4

4. Research method

The authors conducted a systematic literature review of practical AI systems, using a database search, staged selection, and structured extraction and assessment. They retained 90 papers and analyzed system characteristics, validation methods, and study quality.

4. Research method

  • The review searched four databases and analyzed the 90 papers remaining after three selection stages.The search began with 1,164 papers; the selection process is shown in Figure 1.Source: §4 · p. 4, §4.2 · p. 5

4.1. Search strategy

  • The search string combined AI terms, validation terms, and terms intended to identify systems with practical use or industrial relevance.Automatic searching was chosen to reach relevant application domains beyond computer science venues.Source: §4.1 · pp. 4–5

4.2. Paper selection

  • Figure 1 summarizes the staged screening: 1,164 initial papers were reduced to 372, then 142, and finally 90 papers.Source: Figure 1 · p. 4, §4.2 · p. 5

4.2.1. The first stage

  • The first-stage screen required papers to introduce or apply a validation method and describe a system with multiple components or clear usability.This excluded model-validation work unless the model was described as part of a system or as a product used in context.Source: §4.2.1 · p. 5

4.2.2. The second stage

  • The second-stage exclusions included papers that did not validate a system, lacked an accessible full text, or were short workshop or symposium publications.The authors accepted a risk of missing relevant grey literature in favor of papers expected to provide fuller descriptions.Source: §4.2.2 · pp. 5–6

4.2.3. The third stage

  • The final set included only papers rated L4, L5, or L6 for realism and relevance.These levels range from realistic but not real evaluation settings to real settings and, at L6, actual practical use.Source: §4.2.3 · p. 6

4.3. Information extraction

  • The extraction classified system domains, tasks, complexity, malfunction impact, ML use, validation methods, and continuous validation.Validation methods were grouped into a taxonomy by comparing similarities and differences across papers, then described qualitatively with exemplars.Source: §4.3 · pp. 6–7
  • Study quality assessment used relevance labels based on realism and a rigour score based on context description, design description, and validity discussion.The relevance assessment reused the L4–L6 labels applied during selection.Source: §4.3 · p. 7

5. Overview of the final papers

The 90-paper set grew sharply in 2017–2019, while most papers had moderate rigour scores and rarely discussed result validity. Included papers met a minimum relevance level of 4.

5. Overview of the final papers

  • 23 papers appeared in 2019, following 19 in 2018 and 10 in 2017; 2020 was excluded because data were incomplete.The review’s search covered only the beginning of 2020.Source: §5 · p. 7, Figure 2 · p. 8
  • The 90 papers came from 2001–2020, with a median publication year of 2017.5 and a rising publication trend.IEEE published 50 papers, Elsevier 15, Springer 11, ACM 5, and other publishers 9.Source: §5 · p. 7
  • 71 papers (79%) scored 1.5 or 2 out of 3 for rigour, while 76 did not discuss validity at all.Only seven papers scored at least 2.5; context and design were more often reported than result validity.Source: §5 · pp. 7–8, Figure 3 · p. 8
  • 72.2% of the included papers were relevance level 4, 24.4% level 5, and 3.3% level 6.Only papers at level 4 or higher were included, and synthesis comparisons therefore relied primarily on rigour scores.Source: §5 · p. 8

6. Characteristics of validated AI systems (RQ1)

The reviewed systems covered varied domains, tasks, complexity levels, and potential malfunction impacts. Robotics and recognition/classification were the most represented domain and task, respectively.

6. Characteristics of validated AI systems (RQ1)

  • 79 papers explicitly concerned ML or were reasonably interpreted as using it; six were clearly non-ML and five were classified as plausible.Non-ML papers remained in the results because the search was not designed to include only ML papers.Source: §6 · p. 8
  • 21 papers concerned robotics, the most frequent domain; commercial-use systems numbered 13, industrial systems 12, and cars 10.The review identified 13 domains, excluding unspecified ones.Source: §6 · p. 8, Table 2 · p. 8
  • Recognition and classification was the most common task, with 31 systems; transportation followed with nine, and each other task appeared fewer than seven times.Eighteen tasks were recognized, excluding unspecified ones.Source: §6 · p. 8, Table 3 · p. 9
  • The 90 systems were distributed across full systems (36), complex multi-component systems (31), and simple model-centred systems (21); two were unspecified.Source: §6 · p. 8
  • Economic damage was the most frequent probable malfunction impact, with 28 papers; lethal and nuisance impacts each appeared in 17.Eight impact categories were identified, excluding unspecified ones.Source: §6 · p. 8, Table 4 · p. 9
  • Robotics and car systems rose in number in recent years, while commercial, industrial, and medical systems appeared more consistently over time.Other domains appeared intermittently; the 2012–2013 peak comprised nearly equal numbers of systems across almost all domains.Source: §6 · pp. 8–9

7.1. Quantitative analysis of validation methods

The review classified validation into trial, simulation, model-centred validation, and expert opinion. Trials were most common, while continuous validation was reported in only a minority of papers.

7.1. Quantitative analysis of validation methods

  • 42 papers used trial as their only validation method, compared with 20 using simulation; expert opinion was the sole method in two papers.Thirteen papers combined listed methods.Source: §7.1 · p. 9, Table 5 · p. 9
  • When combined methods were partitioned by category, trial appeared 55 times, simulation 31, model-centred validation 16, and expert opinion three times.Every combination of validation methods included a trial.Source: §7.1 · p. 9, Table 6 · p. 9
  • 14 papers described continuous validation over the system life cycle, while 76 reported no continuous validation method.Safety channels appeared five times and failure monitors four times; other methods included input/output restrictions, redundancy, and voting.Source: §7.1 · pp. 9–10, Table 7 · p. 10
  • Trials and simulations increased sharply in recent years, whereas model-centred approaches and expert opinions remained comparatively consistent and low in total numbers.The authors relate simulation’s rising popularity to increased interest in cyber-physical domains such as cars and robots.Source: §7.1 · p. 10, Figure 5 · p. 10

7.2. Descriptive analysis on validation methods

The paper describes four main validation approaches and their variations. They differ in how closely the system and setting reflect deployment, and in the evidence and constraints each provides.

7.2. Descriptive analysis on validation methods

  • The four main approaches are trial, simulation, model-centred validation, and expert opinion, with variations described across the taxonomy.The taxonomy was derived bottom-up from the reviewed papers.Source: §7.2 · p. 10

7.2.1. Simulation

  • Simulation uses an artificial environment to mimic deployment; its variants include fully virtual, system-in-the-loop, and hardware-in-the-loop setups.Simulations can test dangerous scenarios safely and efficiently, but simulator validity is difficult to prove; HIL also requires physical components.Source: §7.2.1 · pp. 10–11

7.2.2. Trial

  • Trials deploy the system in a real or mock environment; real-environment trials expose system-wide issues but are laborious, while mock settings can narrow validity.Trials are planned around key scenarios; mock environments reduce the risk of damage or distraction in dangerous settings.Source: §7.2.2 · pp. 12–13

7.2.3. Model-centred validation

  • Model-centred validation checks the AI model, often using data-centred methods such as cross-validation or robustness tests, rather than deployment-style data collection.It is mostly used for simple model-centred systems and may precede benchmarks and trials.Source: §7.2.3 · p. 13

7.2.4. Expert opinion

  • Expert opinion draws on external application-domain expertise, including interviews about useful features and acceptable thresholds; short sessions can leave key-scenario problems hidden.It can also involve comparing system decisions with an expert or benchmark application; combining it with other methods is proposed to address coverage limits.Source: §7.2.4 · p. 13

7.3. Descriptive analysis of continuous validation methods (RQ2)

The section describes five methods for continuously validating deployed systems, from monitoring for malfunctions to restricting inputs and outputs. These methods differ in whether they alert people, switch to backups, combine component decisions, or constrain system behavior.

7.3. Descriptive analysis of continuous validation methods (RQ2)

  • Continuous validation methods receive less detailed synthesis because less research has been conducted on them.The section describes their basic principles and benefits and drawbacks to a lesser degree than earlier methods.Source: §7.3 · p. 13

7.3.1. Failure monitoring

  • Failure monitors observe systems, subsystems, and components, alerting users to malfunctions or suspicious behavior but taking no corrective action themselves.Human intervention is therefore required, which may be problematic or dangerous when rapid action is needed.Source: §7.3.1 · p. 14

7.3.2. Safety channel

  • Safety channels take over when primary components are compromised, helping systems continue functioning safely, usually with reduced functionality.A safety channel may eventually require intervention if it cannot keep the system safe for longer periods.Source: §7.3.2 · p. 14

7.3.3. Redundancy

  • Redundancy duplicates critical components so backups can replace failures and help maintain initial functionality, but increases costs and may complicate monitoring.Redundant components may also share workload when system performance drops; duplication expands the system’s state space.Source: §7.3.3 · p. 14

7.3.4. Voting

  • Voting lets multiple intelligent components perform the same task and vote on an action, reducing damage from a single failure when others remain functional.It requires multiple components capable of the same task, raising system costs.Source: §7.3.4 · p. 14

7.3.5. Output and input restrictions

  • Output restrictions impose hard limits on potentially dangerous or false actions, while input restrictions limit situations handled autonomously to those the system is designed or trained for.One example excludes input strings whose commonality in product reports decreases prediction accuracy.Source: §7.3.5 · p. 14

8. Discussion

The review finds varied practical AI systems and validation approaches, but reported validation is often narrowly scoped and continuous validation is uncommon. Its findings are bounded by a selection of empirically supported studies, and the authors identify gaps in reporting and coverage.

8.1. Overview of results

  • The review found 90 studies and five consecutive years of publication growth, but its 2012–2013 publication spike remained unexplained.The review included only studies with at least L4 empirical evidence, so its counts do not represent the research literature as a whole.Source: §8.1 · p. 15
  • Most papers did not discuss research validity, and realistic industrial studies at evidence levels 5 and 6 were scarce.More than half scored highly in context, design, or both; the authors note that industrial studies could increase validity and support research on continuous validation.Source: §8.1 · p. 15

8.2. Answer to RQ1: Characteristics of validated systems

  • The review identified 13 system classes, with machine learning present in at least 87% of studies, across a wide range of practical applications.The authors found no large common characteristics beyond ML's dominance across domains, complexity, and malfunction impact.Source: §8.2 · p. 15
  • Coverage was narrow within categories: recognition and classification dominated, many tasks were low-complexity, and malfunction impact ranged from nuisance to lethal.Robotics overshadowed agriculture and medical applications; privacy and discrimination concerns were scarce, and the authors argue that malfunction impact should inform system development.Source: §8.2 · pp. 15–16

8.3. Answer to RQ2: Validation methods

  • The taxonomy has four methods—simulation, trial, model-centred validation, and expert opinion—and 77 papers reported one method while 13 reported more than one.Simulation includes virtual, hardware-in-the-loop, and system-in-the-loop forms; trials include real and mock environments. Trial was the most-used method.Source: §8.3 · p. 16
  • Trials avoid simulator and data deficits but can be laborious or hazardous; complex cyber-physical systems often use simulations, whose fidelity can be difficult to establish.Some trials used very small samples, and many simulators were illustrated with only one or two cases.Source: §8.3 · p. 16
  • Model-centred validation is common for model-centred systems but can face data problems; expert opinion is rarely reported, and traditional software tests were often unmentioned.The authors suggest deterministic tests may be difficult to specify for nondeterministic behaviour, while corner cases could support regression testing for deterioration over time.Source: §8.3 · pp. 16–17
  • Only 14 papers reported continuous validation, using failure monitoring, safety channels, redundancy, voting, and input or output restrictions.The authors caution that unreported methods may still exist, and say their classifications emerged from the studies rather than assessing taxonomy completeness.Source: §8.3 · p. 17

9. Threats to validity

The review’s findings are bounded by its search and the composition of the included literature, especially the dominance of ML studies. The authors also note sparse, less detailed evidence on continuous validation and possible researcher and classification bias.

9. Threats to validity

  • The automatic search may have missed relevant papers, and search terms may have skewed domain coverage and favored more recent studies.The authors connect high occurrence rates to domains explicitly named in the search string, while noting that trendy domains may contribute to the recency skew.Source: §9 · p. 17
  • ML dominance means the findings are safest to interpret as applying to ML literature, not AI in general.The authors also caution that the scientific literature reviewed does not necessarily reflect industry practice or how common particular methods are there.Source: §9 · p. 17
  • Continuous-validation methods were described in fewer papers and with less detail, while the taxonomy analysis was largely conducted by one researcher.The authors say this leaves the taxonomy prone to bias.Source: §9 · pp. 17–18
  • Researcher interpretation could affect paper selection and analysis, including judgments about which domains and search terms to include.Other researchers evaluated unclear cases, but the authors cannot guarantee that cases judged clear by the first author would be unambiguous to others.Source: §9 · p. 18
  • Distinguishing methods such as system-in-the-loop simulation and trials in mock environments can be difficult or arbitrary, complicating precise categorization.Because no method description relied on a single paper, the authors judge that misclassifying one or two papers should not threaten the descriptions’ validity.Source: §9 · p. 18

10. Conclusions

The review synthesizes validation approaches from 90 empirical studies into a four-method taxonomy and a set of seven continuous-validation classes. The authors see these as a basis for further work, while noting that continuous-validation evidence is sparse and the taxonomy may be incomplete.

10. Conclusions

  • The review covered 90 studies across 14 domains and 18 tasks, with malfunction impacts ranging from nuisance to lethal.The selection emphasized solid empirical evidence, so the studies represent relatively mature practical applications; most systems used ML.Source: §10 · p. 18
  • Validation methods were synthesized into four classes: simulation, trial, model-centred validation, and expert opinion.The taxonomy and its common variations draw on the included studies and are offered as a basis for further synthesis and validation guidance.Source: §10 · p. 18
  • The taxonomy may be incomplete because it emerged from the included papers alone, so further assessment and refinement may be needed.The authors frame it as a basis for further synthesis and general ideas for system validation, rather than a finished account.Source: §10 · p. 18
  • Seven classes of continuous-validation methods were presented, but they appear in only 13 papers and are described in less detail.The authors call for assessment and fuller reporting, noting these methods can be vital to system safety and functionality over the life cycle.Source: §10 · p. 18
  • The authors call for more validation methods in academic studies, research in realistic industrial settings, and further work on continuous validation and high-quality simulators.They also recommend research on rarer and entirely new application domains; most reviewed papers described context and design well but lacked discussion of study validity.Source: §10 · p. 18
Loading 2107.12190v1…