Source-linked AI summary
Addressing the Selection Problem in Explainable AI
Claire Vlases, Katelyn Morrison
TL;DR
The paper addresses the gap between users’ natural-language uncertainty and the explanation technique selected by conventional XAI interfaces. It formalizes this selection problem and proposes a conversational multi-agent LLM orchestrator that performs the mapping in the system. The paper presents this orchestration approach as a structural solution and calls for further investigation of the problem’s reliability and scope.
Problem
User studies find that XAI explanations are often ignored, misinterpreted, or used to confirm AI outputs, while conventional interfaces require users to select an appropriate technique.
Method
The paper formalizes the selection problem and instantiates a multi-agent LLM orchestrator that maps natural-language queries to XAI agents using function calling.
Results
The paper formalizes the selection problem as users’ potentially unreliable translation of uncertainty into technique choices and proposes an orchestrator that absorbs this mapping function.
Takeaways & Limitations
The paper encourages human-centered XAI researchers to consider technique selection when designing XAI systems.
Takeaways & Limitations
Further studies are needed to determine whether user-performed mapping is unreliable and how extensively the selection problem affects end-users versus AI system designers.
Abstract
from arXiv · showhide
Explainable AI (XAI) research has produced a plethora of explanation techniques, yet user studies repeatedly show that available explanations are not effective in practice. We argue that, given the siloed nature of conventional XAI, users are struggling to select the appropriate XAI technique. Viewing XAI through a philosophical lens, we offer a formalization of what we call the selection problem: the systematic failure of XAI interfaces to bridge the gap between a user's natural-language uncertainty and the explanation technique that resolves it. Following a logical premise-conclusion format, we show that conventional interfaces require users to translate their uncertainty into a technique selection, a challenging prerequisite to meet. We also propose a structural solution: a multi-agent LLM orchestration tool that translates the user's query to the proper XAI explanation technique. We provide an example of how this structural solution could be instantiated to address the selection problem.
1 Introduction
XAI offers many explanation techniques, but practical user studies find that explanations are often ineffective and difficult to use. The paper frames this as a selection problem and proposes conversational multi-agent orchestration to translate users’ natural-language uncertainty into suitable techniques.
- Motivation: XAI research has produced diverse explanation techniques, many of which are unimodal, non-interactive, and evaluated in isolation.Users often have limited control over explanations and follow-up requests, while studies commonly compare configurations of individual methods.
- Motivation: Real-user evaluations remain limited, and available explanations are frequently ignored, misinterpreted, or used to confirm AI outputs.These findings have led some researchers to attribute the challenges to a lack of human-centered techniques.
- Related work: Conversational and interactive XAI treats explanations as interactions rather than static outputs, motivating the paper’s formalization of interactive technique operationalization.Related work includes narrative-driven XAI with large language models and dialogue-based agentic interpretability.
- The selection problem: Conventional XAI interfaces require users to determine which explanation technique matches what they currently want to understand.The paper identifies this gap between natural-language uncertainty and technique selection as the selection problem.
- Proposed direction: A structural solution should accept natural-language input, provide adequate explanation techniques, and perform the translation from uncertainty to technique on the user’s behalf.The paper proposes conversational multi-agent orchestration of XAI to satisfy these requirements.
2 The Selection Problem in XAI
The selection problem arises because users must translate natural-language uncertainty into an explanation-technique choice, although only a subset of techniques can resolve any given epistemic state. This mapping is unreliable and can leave explanations functionally irrelevant to users’ needs.
- Formal structure: Only a subset of available explanation techniques can resolve a given user’s epistemic state.The adequate set is denoted E∗(ω) ⊆ E, and technique efficacy is defined relative to the user’s epistemic state.
- Interface mismatch: Users express uncertainty in natural language, while conventional XAI interfaces require a technique selection as input.The user’s query q represents uncertainty, but the interface requires a chosen technique ê ∈ E.
- Uncertainty representation: Users’ queries may incompletely represent their underlying uncertainty, and conventional interfaces cannot detect unresolved uncertainty signaled by an inadequate selection.This leaves mismatches between user needs and system outputs able to persist without model detection.
- User-side translation: Users must map their query to a technique without guaranteed technical knowledge or system support for detecting mapping failures.The system sees only the selected technique and cannot detect whether the user’s curiosity matches the output.
- Observed consequences: User-performed mapping produces recurring mismatch, counterintuitive, and biased-inference failures when explanations do not align with what users sought to understand.The resulting selection ê may fall outside the adequate set E∗(ω).
- Definition: The selection problem occurs when the technique chosen by the user falls outside the techniques capable of resolving their uncertainty.Formally, the user translates a natural-language query through f_user : L → E before receiving an explanation.
3 Structural Solution: Multi-Agent LLM Orchestration System
The structural solution shifts translation from natural-language uncertainty to adequate XAI techniques from the user to the system. It requires direct natural-language input, adequate technique routing, and system-assisted mapping.
- Structural solution: The selection problem is addressed by shifting the query-to-technique mapping from the user to the system.The proposed mapping is fsys: L → E∗(ω), where the system translates queries into techniques adequate for the user’s uncertainty.
- System requirements: A replacement system must accept natural-language queries directly rather than requiring technique-based input.This prevents users from translating their questions into the vocabulary of discrete explanation methods.
- System requirements: The system must route users to techniques in the adequate set E∗(ω) for their expressed uncertainty.Techniques outside this set may produce explanations that fail to address the user’s actual question.
- System requirements: The system should perform the translation between natural-language queries and explanation techniques, enabling collaborative multi-turn interaction.The user’s remaining contribution is to articulate the query, while the system can potentially elicit additional information.
4 Instantiating a Multi-Agent LLM Orchestrator for XAI
The paper illustrates its structural solution with a multi-agent LLM orchestrator that routes natural-language XAI questions to specialized agents. Its design uses query classification and function calling to match uncertainty types with suitable techniques.
- Example instantiation: The proposed instantiation demonstrates the structural solution with four XAI techniques in a medical-imaging domain.The authors present the domain-specific example as an illustration of a solution they believe can be domain-agnostic.
- Example instantiation: Figure 1 contrasts existing multi-agent LLM interfaces with a “Smart” mode that routes natural-language queries to specialized XAI agents.The bottom design uses function calling for the routing process.
- Orchestrator design: The orchestrator is designed to accept natural language, encode knowledge of uncertainty–technique matches, and route queries automatically.These correspond to the three system requirements for replacing user-performed mapping.
- Orchestrator design: A “Smart” orchestration tool matches users’ natural-language questions to question types addressed by known XAI techniques at query time.The routing decision is made when the user submits the query, so returned techniques are matched to the uncertainty structure.
- Orchestrator design: The orchestrator models fsys: L → E∗(ω) by classifying queries and routing them to the best-suited explanation agent.Users who already know their preferred technique can invoke individual modes directly; the instantiation uses Gemini 2.5 Flash.
- Orchestrator design: The LLM-based implementation uses natural-language understanding and function calling to absorb query interpretation, technique selection, and output delivery into one pipeline.This design is intended to satisfy all three structural requirements.
5 Future Work & Conclusion
The paper concludes that interface presentation contributes to XAI’s adoption gap and proposes an LLM-based multi-agent orchestrator as one solution. Future work should test the selection problem, compare orchestrated and non-orchestrated systems, and extend the design.
- Conclusion: The paper frames the XAI adoption gap partly as an interface-presentation problem involving unreliable user-performed technique mapping.It formalizes this issue as the selection problem and proposes a multi-agent orchestrator that absorbs the mapping function.
- Future work: Future studies should test whether user-performed mapping is unreliable and determine how the selection problem differs for end-users and AI system designers.The authors identify these questions as requiring further investigation of their premises and conclusions.
- Future work: Future work could compare multi-agent LLM orchestration with presenting XAI techniques directly to users.Researchers could also examine backend implementations using prompt engineering or multi-turn conversations.
- Conclusion: The authors hope the formalization, system requirements, and example implementation will encourage human-centered XAI researchers to consider the selection problem.The stated aim is to inspire discussion and design attention within the community.