Source-linked AI summary
Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Models
Rodion Krjutškov, Eduard Barbu, Nikos Sakkas, Sofia Yfanti
TL;DR
The paper addresses the difficulty of interpreting complex ML predictions through technically demanding, inflexible XAI interfaces and prior grammar-constrained conversational systems. It introduces a modular conversational XAI assistant using LLM function calling, achieving higher parsing accuracy and strong usability preferences in an expert evaluation. The authors identify small-sample validation and the need for broader, longer-term studies as important scope boundaries.
Problem
Complex ML predictions are difficult for energy-domain practitioners to interpret, while traditional dashboards and earlier conversational systems impose technical or grammar-based interaction limits.
Method
The Explainability Assistant uses a modular conversational architecture with LLM function calling to invoke explainability operations across ML tasks without task-specific fine-tuning.
Results
94% intent classification accuracy improves on prior grammar-based systems, while all three experts preferred the conversational interface and task accuracy remained 100% versus 93%.
Takeaways & Limitations
Conversational XAI can reduce usability barriers while preserving analytical task accuracy and exposing intended actions before execution.
Takeaways & Limitations
Expert validation used only three participants, limiting statistical power and generalizability; broader and longer-term studies are needed.
Abstract
from arXiv · showhide
Energy consumption forecasting relies on increasingly complex machine learning (ML) models, such as Genetic Programming-based symbolic regressors, whose predictions can be difficult for facility managers and building operators to interpret. Explainable Artificial Intelligence (XAI) techniques address this opacity, but traditional XAI dashboards require substantial technical expertise and provide limited flexibility for dynamic, context-aware inquiry. Conversational XAI systems offer a promising alternative; however, previous approaches, such as TalkToModel, were constrained by rigid custom grammars and achieved only 76.8% intent-parsing accuracy. This paper introduces the Explainability Assistant, an open-source conversational XAI system that leverages the function-calling capabilities of modern Large Language Models (LLMs) to overcome these limitations. The system achieves 94% intent-parsing accuracy, supports flexible natural language interaction, and adapts to different ML problem types without task-specific fine-tuning. We present the system's architecture and report results from a comparative evaluation conducted with energy domain specialists, contrasting the Explainability Assistant with a traditional XAI dashboard. The evaluation suggests improved usability and consistent task accuracy, with all experts unanimously preferring the conversational interface for practical use.
1 Introduction
The paper introduces the Explainability Assistant to make ML explanations more accessible and flexible for energy-management practitioners. It combines conversational interaction with LLM function calling and reports higher parsing accuracy than prior grammar-based approaches.
- 1 Introduction: Traditional XAI dashboards can require substantial technical expertise and provide limited support for flexible, iterative exploration by facility managers and engineers.Conversational dialogue is presented as an accessible alternative for context-aware model analysis.
- 1 Introduction: The Explainability Assistant uses modern LLM function calling to improve natural-language understanding, modularity, and applicability across use cases.It builds on TalkToModel while addressing limitations of rigid custom grammars.
- 1 Introduction: 94% parsing accuracy surpasses the 76.8% accuracy reported for previous grammar-based conversational XAI approaches.The evaluation also used automatic metrics and stakeholder feedback.
- 1 Introduction: The paper contributes a conversational LLM-based architecture, a lightweight modular design, and empirical evidence from domain-expert evaluation.The listed contributions target both system flexibility and practical model understanding.
2 Related Work
Prior explainability systems provide important techniques and interactive tools, but conversational XAI has remained constrained by technical requirements and custom grammars. The Explainability Assistant uses structured function calling to provide more predictable, adaptable interactions for applied domain users.
- 2 Related Work: Explainability research includes feature-importance, counterfactual, and gradient-based methods, while modern LLMs enable tool orchestration in conversational applications.The related work positions the paper at the intersection of established XAI techniques and newer LLM capabilities.
- 2 Related Work: Earlier tools combined explanation techniques through dashboards, while TalkToModel introduced dialogue but reported only 76.8% parsing accuracy with custom grammars.These approaches established relevant foundations but retained interaction or parsing limitations.
- 2 Related Work: The Explainability Assistant translates natural-language queries into explainability operations using structured LLM function calling rather than prompt chaining.This design is presented as providing more predictable tool invocation.
- 2 Related Work: The system differs from prior conversational XAI work by eliminating task-specific fine-tuning and focusing on prediction-level explanations for applied domain experts.The paper highlights energy management as a representative setting.
3 System Description
The Explainability Assistant separates its interface from computational logic and uses LLM function calling to select and parameterize explainability operations. Its modular design supports multiple explanation techniques, model types, and application domains without task-specific fine-tuning.
- 3 System Description: The modular architecture decouples presentation from computational logic, enabling maintainability, independent scaling, and integration with different frameworks and LLM providers.The front end manages conversational interaction, sessions, authentication, and model-provider selection.
- 3 System Description: The back end supplies user context and JSON function specifications to an LLM, which selects explainability functions and generates structured arguments.This replaces task-specific grammar fine-tuning with function-calling orchestration.
- 3 System Description: The function-calling design exceeds 90% intent-extraction accuracy for capable models and supports structured responses that expose planned actions.Gemini-2.5-Flash is reported at 93–94%, compared with 76.8% for TalkToModel’s grammar-based system.
- 3 System Description: Configuration changes allow the system to adapt to new datasets and model types without task-specific fine-tuning.The paper demonstrates this across energy-consumption regression and clinical-risk classification.
- 3 System Description: The system supports SHAP feature-importance analysis, DiCE counterfactual explanations, and user-directed what-if scenario analysis.These tools cover attribution, contrastive changes, and hypothetical feature modifications.
- 3 System Description: The demonstrated use cases include energy-demand forecasting with a Genetic Programming symbolic regressor and heart-disease classification with a Decision Tree.The latter illustrates extension beyond regression through updated function specifications.
4 Evaluation
The evaluation measures parsing reliability and practical utility through gold-standard LLM tests and a within-subjects comparison of the Explainability Assistant with a traditional dashboard. EA achieved higher parsing accuracy, maintained high task accuracy, and received stronger usability evaluations from energy specialists.
- 4.1 LLM Parsing Accuracy: 94% accuracy on A+B and 93% on A+C made Gemini-2.5-Flash the strongest parser, exceeding earlier grammar-based systems’ approximately 75% plateau.The comparison used exact-match accuracy on two 100-sample combined datasets.
- 4.2 Expert Validation Design: The expert study compared EA and ED on the same energy model through parallel prediction, explanation, what-if, accuracy, and error-analysis tasks.Three energy specialists completed five questions per interface and rated usability, trust, system understanding, and future use.
- 4.3 Expert Validation Results: 100% task accuracy with EA versus 93% with ED indicates that the conversational interface maintained or improved objective performance.The difference was not statistically significant given the sample size.
- 4.3 Expert Validation Results: All three experts rated EA’s ease of use and likelihood of future use at or near 5, while ED ratings varied more across experts.EA also received consistently higher and more uniform ratings overall.
- 4.3 Expert Validation Results: EA received unanimous maximum ratings of 5 for future use, compared with ED’s variable 2–4 ratings, while trust and system-understanding scores remained more measured.Participants also noted that ED’s visual overview could aid global orientation, suggesting complementary strengths.
- 4.4 Discussion: The evaluation links function-calling integration with improved parsing reliability and structured pre-execution action descriptions that increase transparency.The discussion frames these properties as supporting practical utility for domain experts.
5 Conclusion
The paper concludes that the Explainability Assistant combines LLM function-calling with a modular conversational design to improve parsing reliability and dashboard usability. Evaluation with energy specialists found unanimous preference for EA’s ease of use and future adoption while preserving high task accuracy, while larger and longer-term studies remain future work.
- 5 Conclusion: 94% intent classification accuracy represents a substantial improvement over prior grammar-based systems.The architecture uses LLM function-calling and structured responses.
- 5 Conclusion: All three experts preferred EA for ease of use and future adoption, with task accuracy of 100% versus 93% for ED.The evaluation also reports reduced deployment complexity and enhanced reasoning transparency.
- 5 Conclusion: Future work should test larger and more diverse populations, examine long-term adoption, and investigate hybrid conversational-visual interfaces.These directions follow participant feedback about combining conversational flexibility with visualization.
Review Materials
The review materials provide demonstrations, deployment links, source code, evaluation resources, and participant-facing study materials. The evaluation scenario concerns building energy forecasting, with participant background and survey instructions supplied alongside the materials.
- Review Materials: Readers can access a video demonstration, public demo, complete source code, gold parse datasets, evaluation scripts, and the expert questionnaire.The links are listed in Appendix B.
- Review Materials: The system website, demo, video, backend, frontend, and evaluation repositories are listed as direct resources.The resources include separate frontend and backend code repositories.
- Review Materials: The participant scenario uses a building energy model forecasting consumption from indoor temperature, outdoor temperature, and the previous hour’s consumption.The survey asks participants to answer questions about the model’s predictions.
- Review Materials: The survey asks about energy experience, ML familiarity, and current role before participants begin the study.The role question is optional.
C.2 Block A Task Questions
Block A asks participants to use the Explainer Dashboard to answer prediction, feature-attribution, counterfactual, error, and alternative-example questions.
- The dashboard is the designated interface for answering these questions, with an “I can’t tell” option when answers cannot be found.
- Block A asks for a prediction, its most influential feature, and the effect of changing indoor temperature for ID 33.
- Participants must also quantify the prediction error for ID 33 and identify another example where inputs and predictions differ.
C.3 Block A Post-Task Questions
After Block A, participants report their experience using the Explainer Dashboard, including usability, trust, understanding, and willingness to reuse it.
- Post-task questions assess whether the dashboard was easy to use, trustworthy, understandable, and worth using again.
C.4 Block B Task Questions
Block B repeats the task structure with the Explainability Assistant, using ID 55 and the same prediction, explanation, counterfactual, error, and alternative-example questions.
- The Explainability Assistant is the designated interface for answering these questions, with an “I can’t tell” option when answers cannot be found.
- Block B asks for a prediction, its most influential feature, and the effect of changing indoor temperature for ID 55.
- Participants must also quantify the prediction error for ID 55 and identify another example where inputs and predictions differ.
C.5 Block B Post-Task Questions
After Block B, participants evaluate the Explainability Assistant and may explore it further before providing feedback.
- Post-task questions assess whether the assistant was easy to use, trustworthy, understandable, and worth using again.
- Participants are invited to explore additional question types and toggle example questions in the assistant.
- Participants are asked to share feedback, ideas, and thoughts after using both interfaces.
D Limitations
The evaluation is limited by a small expert sample, asynchronous study procedures, fixed interface order, and testing confined to energy forecasting with symbolic regression.
- The evaluation used only three experts, limiting statistical power and the generalizability of its findings.The within-subjects design and unanimous usability preferences nevertheless provide initial evidence of practical utility.
- Asynchronous participation prevented observation of real-time interaction and think-aloud behavior during exploratory analysis.Such observations could have revealed how experts formulate queries and use each interface.
- Fixed presentation order may have inflated the Explainability Assistant’s performance or usability ratings through participant familiarization with the task and data.Future studies should counterbalance interface order to control for this effect.
- The evaluation covered only energy forecasting with a specific symbolic-regression model, so broader generalizability remains unestablished.Validation in other domains, particularly healthcare, is still needed.