Source-linked AI summary
A systematic review and taxonomy of explanations in decision support and recommender systems
Ingrid Nunes, Dietmar Jannach
TL;DR
As software increasingly supports or automates decisions, users need trustworthy advice, while complex machine-learning systems make useful explanations difficult. This paper systematically reviews explanations in advice-giving systems, synthesizes how they are generated, presented, and evaluated, and derives a comprehensive taxonomy while identifying unresolved evaluation and presentation issues.
Problem
Complex machine-learning decision functions make useful, understandable, and user-tailored explanations challenging, while explanation evaluation lacks standardized study designs and metrics.
Method
The paper conducts a systematic literature review of explanations in advice-giving systems, analyzing their generation, information, presentation, evaluation, and reported conclusions.
Results
The review derives a comprehensive taxonomy spanning explanation objective, responsiveness, content, and presentation, and identifies unresolved issues in presentation and evaluation.
Takeaways & Limitations
Designers can use the taxonomy to consider multiple explanation facets when developing current and future advice-giving systems.
Takeaways & Limitations
Because studies were selected from specified digital libraries, the review does not cover all existing work on explanations.
Abstract
from arXiv · showhide
With the recent advances in the field of artificial intelligence, an increasing number of decision-making tasks are delegated to software systems. A key requirement for the success and adoption of such systems is that users must trust system choices or even fully automated decisions. To achieve this, explanation facilities have been widely investigated as a means of establishing trust in these systems since the early years of expert systems. With today's increasingly sophisticated machine learning algorithms, new challenges in the context of explanations, accountability, and trust towards such systems constantly arise. In this work, we systematically review the literature on explanations in advice-giving systems. This is a family of systems that includes recommender systems, which is one of the most successful classes of advice-giving software in practice. We investigate the purposes of explanations as well as how they are generated, presented to users, and evaluated. As a result, we derive a novel comprehensive taxonomy of aspects to be considered when designing explanation facilities for current and future decision support systems. The taxonomy includes a variety of different facets, such as explanation objective, responsiveness, content and presentation. Moreover, we identified several challenges that remain unaddressed so far, for example related to fine-grained issues associated with the presentation of explanations and how explanation facilities are evaluated.
1 Introduction
As intelligent software increasingly supports or automates decisions, users need confidence in its recommendations and decisions, while modern machine-learning systems make explanations harder to generate and understand. This review systematically examines explanations in advice-giving systems and proposes a comprehensive taxonomy for designing them.
- Motivation: Intelligent software now ranges from decision support and recommender systems to fully autonomous decision-making applications.The spread of machine learning across software also raises accountability and ethics issues.
- Motivation: Users’ confidence in recommendations and automated decisions is a key requirement for system success and adoption.Automatically generated explanations have long been investigated as a mechanism for increasing trust in system suggestions.
- Challenges: Modern machine-learning models make useful and understandable explanations more challenging, especially when outputs depend on complex neural networks.User-tailored explanations for learned decision functions can require additional domain knowledge.
- Scope: Advice-giving systems encompass expert, knowledge-based, decision support, and recommender systems, but adequate explanations remain difficult to define.The literature lacks clear consensus on what constitutes a good explanation, and explanation types can affect users’ decisions differently.
- Contribution: The review analyzes explanation generation, information content, presentation, evaluation, and researchers’ conclusions across the literature.It derives a taxonomy covering explanation objective, responsiveness, content, and presentation, while identifying unresolved presentation and evaluation issues.
- Contribution: The review included 217 papers selected from 1209 initially retrieved through a structured process.This systematic selection was presented as addressing potential researcher bias compared with subjective overview papers.
2 Systematic Review Planning
The review was planned as a transparent, reproducible synthesis of explanations for end users making decisions among limited alternatives. It defined research questions, search sources, and inclusion and exclusion criteria to reduce selection bias and structure the analysis.
- Review rationale: A systematic review specifies its goals, research questions, search procedure, and inclusion and exclusion criteria before synthesizing results.This defined procedure is intended to minimize researcher bias and support reproducibility.
- Review rationale: The review seeks a comprehensive overview because earlier surveys used subjective selections, focused on particular aspects, or overlooked developments over time.The paper positions systematic selection as a response to these limitations.
- Research questions: The review addresses explanation characteristics, generation, evaluation, and conclusions from evaluation or foundational studies.These aims are expressed through research questions RQ-1 through RQ-4.
- Search strategy: Searches were conducted across selected digital databases, while Google Scholar and arXiv were excluded because they could duplicate studies and include non-peer-reviewed work.The review focused on peer-reviewed computer-science literature as an indication of study quality.
- Selection criteria: Eligible studies proposed explanation-generation techniques, described tools with explanation facilities, or evaluated or compared explanations.The review focused on explanations intended for end users responsible for final decisions.
- Selection criteria: The scope was limited to explanations tied to specific decisions in settings where systems select one or more options from a limited set of alternatives.Background knowledge displays, interface-interaction guidance, mathematically optimal solutions without alternatives, and simulation outcomes were excluded.
- Exclusion criteria: Studies were excluded when they claimed explanation potential without concretely describing how explanations were provided to end users.Examples included transforming neural networks into rules without explaining how those rules would be used.
3 Systematic Review Execution
The review executed a synonym-based search across four databases, retrieved 1209 papers, and applied a two-step screening and full-text assessment procedure. After protocol-based selection, 217 primary studies remained for analysis.
- 3.1 Search String Construction: The search string required terms for explanations and decision support systems, supplemented by synonyms including expert, knowledge-based, and recommender systems.The resulting Boolean query connected the two term groups with AND.
- 3.1 Search String Construction: Database-specific syntax was used, with searches performed in abstracts except for Springer Link, where keywords were searched because of API limitations.The search was adapted to each target database.
- 3.2 Selection of Primary Studies: 1209 papers were retrieved from four databases on August 12, 2016, excluding duplicates.The source-specific counts were reported in Table 4.
- 3.2 Selection of Primary Studies: The first screening step assessed titles and abstracts and provisionally classified papers by inclusion criterion.Papers showing possible eligibility were selected for detailed analysis.
- 3.2 Selection of Primary Studies: The second step retrieved and analyzed full texts when available, then checked exclusion criteria and reassessed inclusion status.Some papers were discarded or assigned to a different inclusion criterion.
- 3.2 Selection of Primary Studies: Borderline cases were reviewed by a second researcher, while each abstract and paper was otherwise analyzed by one researcher following the protocol.This procedure was intended to minimize potential researcher bias.
- 3.2 Selection of Primary Studies: 217 primary studies remained after the selection procedure.The final set and criterion statistics were reported in Tables 5 and 6.
4 Results
The authors analyse 217 selected primary studies to extract insights for developing a comprehensive taxonomy of explanation-facility design aspects.
- The analysis of 217 selected primary studies provides the basis for a comprehensive taxonomy of aspects to consider when designing future explanation facilities.
4.1 Historical Developments
Research on explanations has expanded over time, with recent work placing greater emphasis on empirical evaluation and new explanation techniques. The review also identifies diverse content and presentation strategies, including context-tailored, interactive, and unusual formats.
- Historical developments: The total number of published papers generally increases, with a renewed rise in recent years after stagnation in the 2000s.The authors attribute the earlier stagnation partly to the declining role of knowledge-based systems and the later shift toward ML-based approaches.
- Historical developments: Papers proposing new explanation techniques increased over time, while papers on tools with explanation facilities were more common in the past.
- Historical developments: Empirical evaluation received more attention recently, whereas papers on foundational explanation aspects remained very scarce.
- Explanation content: The review organised 26 content codes into information types and general observations, including context-tailoring and other content characteristics.The codes were merged when they represented the same underlying idea, such as combining Pros and Cons with Trade-off.
- Explanation content: Most explanations focus on a single recommended alternative, although some compare it with other alternatives or groups to communicate relative strengths and trade-offs.Some interfaces aim to educate users about trade-offs, including through interactive decision-making support such as dynamic critiquing.
- Explanation content: Sixteen studies tailored explanations to users’ expertise, interests, or situational context, often by varying the level of detail.
- Explanation content: Interactive explanations supported what-if, why, and why-not questions in 10, 37, and 13 studies, respectively.
- Explanation presentation: Natural language was the most frequent presentation format, alongside lists, visualisations, and four unusual forms grouped as Other.The Other group comprised audio, highlighting, query results, and OWL outputs.
4.3 RQ-2: How are explanations generated?
Explanations are generated mainly in close connection with the underlying decision-inference method, data, application domain, and intended purpose. The review finds that most studies provide limited generation detail, while method-independent generation remains uncommon.
- Generation process: Most studies provide few details about explanation generation because it is closely tied to the inference method and data producing the recommendation.Rule-based systems may verbalize triggered rules, whereas more complex methods require additional processing.
- Generation drivers: 18 of 101 explanation-generation techniques (17.8%) are domain-specific and exploit information available only in particular application domains.Such approaches are concentrated especially in Computing & Robotics and Media Recommendation.
- Generation drivers: The application domain, explanation purpose, and underlying decision-inference method are the three factors identified as strongly influencing which explanation is provided.These drivers jointly connect the explanation to the system’s context, objective, and reasoning process.
- Explanation purpose: The review classifies explanation purposes using prior purpose categories, extending them with additional categories identified during analysis.The taxonomy distinguishes labels such as transparency, understanding, trust, and acceptance according to the selected reference terminology.
- Explanation purpose: Transparency is the most common explanation purpose, while effectiveness is second and persuasiveness can conflict with effectiveness.Transparency exposes the system’s inference process; effectiveness helps users assess whether a recommendation is adequate for them.
- Method independence: Only 11 approaches describe explanation-generation methods independent of the underlying inference method.The review identifies this as requiring further investigation as decision-inference methods become more complex.
4.4 RQ-3: How are explanations evaluated?
Evaluation of explanation techniques is uncommon, and user studies are the predominant evaluation method. Studies vary widely in designs, measured outcomes, domains, and reported effectiveness, limiting straightforward comparison across findings.
- Evaluation prevalence: Only 21.5% of technique and tool studies include an evaluation beyond toy examples.Even from 2010 onward, almost two thirds of analysed studies lack a proper evaluation.
- Evaluation types: User studies account for 52.4% of evaluations because many scenarios lack a formal definition of a correct or best explanation.Researchers therefore measure users’ subjective perceptions or the explanations’ impact on behaviour.
- Evaluation types: Alternative evaluations include customised empirical measurements, computational-efficiency studies, pilot studies, acceptance tests, and case studies.Explanation coverage is one example of a customised measurement, defined as the fraction of preference features used in an explanation.
- Evaluation domains: Evaluations most commonly occur in Media Recommendation, followed by Health.Movie and music recommendation studies form part of the Media Recommendation category and have increased in recent years.
- User-study designs: Most user studies compare alternative explanation types or explanation presence, while fewer vary expertise levels, explanation properties, or interface components.The review distinguishes alternative explanations from alternative user interfaces and notes properties such as source, length, direction, and confidence.
- Measured outcomes: Transparency is among the main aspects measured in user studies, matching its status as the most frequently stated intended explanation purpose.The review notes that inconsistent study designs make combining reported results difficult.
- Evaluation conclusions: Effectiveness findings diverge most because studies examine different explanation styles and may contain confounding variables.Reported conclusions include both nonsignificant results and evidence favoring popularity-based or non-personalised decisive features over personalised ones.
5 Discussion
The review proposes a comprehensive taxonomy for designing explanation facilities in advice-giving systems and identifies unresolved challenges in objectives, content, presentation, responsiveness, and evaluation.
- Taxonomy: The taxonomy organizes explanation design around objectives, generation, content, and presentation, including general facets and user-interface components.It treats explanation components as elements such as natural-language justifications or histograms.
- Objectives: Explanation objectives should guide design, because adding explanations without clear goals can make design arbitrary.The review distinguishes stakeholder goals, user-perceived quality factors, and explanation purposes.
- General facets: The taxonomy distinguishes target, generality, responsiveness, and level of detail, including adaptation to user context and expertise.Multiple detail levels can address the trade-off between informativeness and users’ cognitive effort.
- Content and presentation: Content and presentation choices are partly orthogonal to general facets, but design decisions in one dimension can constrain choices in another.For example, selecting a baseline for a user-interface component depends on the explanation target.
- Future directions: Open research issues include selecting comparable explanation content and isolating fine-grained presentation effects such as length, vocabulary, and format.Existing user studies often compare explanation forms that differ across several aspects, making individual effects difficult to identify.
- Responsiveness: Only 16 explanation techniques and tools, or 8% of studies of these types, tailored explanations to context or user expertise.The review argues that appropriate explanation forms may depend on domain criticality, user background, and willingness to invest effort.
- Evaluation: Evaluation needs standardized protocols and objective measures alongside subjective questionnaires because current studies lack common designs and item sets.The review cites explanation exposure delta as one objective protocol, while noting that it assesses only certain aspects.
6 Summary
As software systems take on more advisory and autonomous decision-making, explanations become increasingly important. The review synthesizes prior work, develops a taxonomy, and identifies remaining scope and research challenges.
- Motivation: The review addresses explanations for software systems that support decisions or make them autonomously as such systems become more widespread.Examples include recommender systems and automated pricing algorithms.
- Method: The authors systematically review studies proposing explanation techniques, explanation-enabled tools, and evaluation or foundational research.The review aims to provide a comprehensive view of work in the field.
- Findings: Most existing explanations consist of inference traces from internal reasoning, often transformed into natural-language statements because rule-based reasoning predominated.The review links the frequent use of traces to the underlying inference methods.
- Contribution: The proposed taxonomy guides researchers in designing and evaluating future explanation approaches by emphasizing clear objectives and multiple explanation facets.It also identifies open challenges requiring further research.
- Scope: The review covers explanations for end users, while explanations that help data scientists understand complex or deep-learning outcomes are outside its scope.The latter topic is associated with explainable artificial intelligence and is identified as a future research area.