PaperlayerView exact revision on arXiv

Source-linked AI summary

From Plots to Words: Model-Aware Multimodal Explanations as a Foundation for Accessible, Non-Visual Interaction

Nur Keleşoğlu, Łukasz Sobczak, Joanna Domańska

arXiv:2608.24910v1cs.HCcs.AI

TL;DR

The paper addresses the challenge of making multimodal time-series forecasts consistent, trustworthy, and accessible when forecasting outputs are primarily visual. It presents a context-aware multi-agent framework that converts numerical, visual, and model-derived information into structured textual explanations, with the explainable configuration improving overall quality by up to 32% over baseline. However, human-centered trustworthiness and usability remain empirically unestablished.

  • Problem

    Forecast explanations are often difficult to interpret and plot-centric interfaces are largely inaccessible to blind and low-vision users.

  • Method

    A context-aware multimodal multi-agent framework integrates forecasts, visualizations, SHAP attributions, temporal importance, uncertainty, and structured natural-language generation.

  • Results

    Approximately 32% overall improvement over the baseline was achieved, with explainable forecasting further improving model awareness and trustworthiness.

  • Takeaways & Limitations

    Model-aware textual explanations provide a foundation for non-visual interaction with forecasting outputs without depending on a visual channel.

  • Takeaways & Limitations

    The evaluation uses an LLM-based judge as an early-stage proxy, so real user trustworthiness and usability are not yet established empirically.

Abstract

from arXiv · show

Multimodal large language models are increasingly used in interactive systems, yet ensuring consistent, trustworthy reasoning across heterogeneous modalities remains challenging. We present a context-aware, multi-agent framework that integrates textual queries, numerical data, visual representations, and model-derived signals for explainable time-series forecasting. A distinctive feature is that it turns predominantly visual forecasting outputs (e.g., trend plots) into structured, model-aware textual explanations. We argue that this makes the approach a natural foundation for non-visual, accessible interaction of particular relevance to blind and visually impaired users, for whom plot-centric interfaces are largely inaccessible. The framework supports three progressively richer pipelines (baseline, interpretable, explainable), enabling systematic comparison of unimodal, perception-driven, and model-aware responses. In an exploratory evaluation using an LLM-based judge as an early-stage proxy for human assessment, the explainable configuration improves overall explanation quality by up to 32% over a numerical baseline, with notable gains in trustworthiness and model awareness. We position user-centered validation with target users, including screen-reader and speech-interface users, as the essential next step rather than a claim established here.

1 Introduction

The paper addresses the difficulty of producing faithful, trustworthy explanations for multimodal time-series forecasts, especially when users cannot access visual plots. It proposes a model-aware multimodal framework that communicates forecasting behavior through structured language.

  • Motivation: DevOps forecasting outputs are difficult to interpret and often lack user-centered, faithful explanations of why predictions are made.This limits operators’ insight into model behavior and can reduce trust in decision-making.
  • Motivation: Trend plots, dashboards, and charts are largely inaccessible to blind and low-vision users and in eyes-free settings.Screen readers convey little of the meaning in forecast curves or uncertainty bands.
  • Approach: The framework integrates textual queries, numerical forecasts, visualizations, SHAP attributions, temporal dependencies, and uncertainty estimates in an interactive reasoning pipeline.Its outputs are structured, uncertainty-explicit textual explanations rather than plot-dependent responses.
  • Approach: Three pipelines compare numerical-only baseline, visualization-augmented interpretable forecasting, and model-aware explainable forecasting.This progression supports systematic comparison of unimodal, perception-driven, and model-aware responses.
  • Evaluation: The paper evaluates the framework on a large-scale GPU cluster dataset to assess explanation quality, trustworthiness, and insightfulness.The evaluation compares progressively richer multimodal and explainability configurations.

2 Related Works

Prior work covers accessible non-visual interaction, language-based time-series analysis, and multimodal explainability, but the paper identifies a gap in model-aware forecasting explanations for operational settings.

  • Accessible non-visual interaction: Multimodal LLMs have been used as assistive agents for natural-language access to heterogeneous interfaces and data graphics.Existing systems include screen-reader augmentation, computer-vision support, conversational chart access, and multimodal authoring tools.
  • Gap and contribution: The paper’s framework combines SHAP attribution, temporal dependencies, forecast behavior, uncertainty estimation, visual representations, and LLM reasoning.These components support structured, uncertainty-explicit descriptions of model behavior.
  • LLM-based time-series analysis: Natural-language time-series methods support interpretation and question answering for trends, seasonality, similarity, and causal reasoning.These approaches improve accessibility and flexibility but rely primarily on language-based reasoning.
  • Multimodal explainability: LLM-driven multimodal systems commonly use textual rationales, report-oriented explanations, or related explanation strategies.Examples include clinician-style rationales and prediction–reflection–refinement loops.
  • Gap and contribution: The claimed distinction from prior work is comprehensive model-aware explainability for multimodal time-series forecasting.The approach contrasts with methods relying primarily on textual or knowledge-driven reasoning.

3 System Overview

The system is a context-aware, hierarchical multi-agent framework that plans and executes forecasting analyses, combines multimodal outputs, and produces structured explanations. Its progressive pipelines increase multimodal context and model awareness.

  • System architecture: The system processes user queries through coordinated agents and returns structured natural-language explanations rather than rendered charts.Its modality-agnostic interaction layer is designed to support non-visual access to forecasting outputs.
  • System architecture: The architecture contains Root, Planner, Plan Executor, Knowledge, Future Forecaster MCP Server, and Responder components.These components divide orchestration, planning, execution, contextual enrichment, analytical tooling, and response generation.
  • System workflow: Queries are transformed into execution plans, processed step by step, enriched with multimodal outputs, and translated into explainable responses.The workflow supports iterative refinement for conversational follow-up questions.
  • System workflow: The Planner Agent uses available tools, predefined DevOps templates, and query semantics to construct structured execution plans.Templates can be adapted for recurring scenarios such as anomaly investigation and forecasting-based diagnostics.
  • Analytical tools: The Future Forecaster MCP Server modularizes forecasting, visualization, interpretation, and explainability tools for dynamic invocation.Its tools combine historical data, forecasts, visual representations, and model-specific explanation signals.
  • Progressive pipelines: Progressive Forecasting Result Pipelines increase multimodal context and model awareness across baseline, interpretable, and explainable configurations.The baseline receives only numerical forecast values, whereas later configurations add richer inputs.
  • Model-aware grounding: Model-derived signals include SHAP-based feature attributions, temporal importance, and uncertainty estimates for grounding explanations in predictive behavior.The framework also addresses consistency across numerical, visual, and textual modalities.

4 Experimental Setup

The evaluation uses a GPU-demand forecasting task built from historical cluster data and a supervised lag-based model. Forecasts are generated recursively, while SHAP attribution and ensemble variation provide explanation and uncertainty signals.

  • Dataset and tasks: The experimental setup evaluates the framework on GPU-cluster monitoring, forecasting, anomaly detection, and capacity-planning use cases.The dataset includes heterogeneous GPU types and multi-organization competition relevant to operational resource management.
  • Dataset and tasks: The target metric is gpus_active_requested, representing actively requested GPUs over time.The raw dataset is filtered to extract this forecasting target.
  • Preprocessing: A sliding window converts the series into 24 lag features, using the previous 24 time steps to predict current GPU demand.This representation captures short-term temporal dependencies and workload dynamics.
  • Preprocessing: The final H observations are held out for evaluation, while the remaining observations are used for training.This is a hold-out split based on the forecast horizon.
  • Forecasting model: A Random Forest Regressor with 100 decision trees predicts GPU demand from the lag-based feature matrix.A fixed random seed is used for reproducibility, and the trained model is stored for inference.
  • Explainability: SHAP TreeExplainer estimates feature contributions from 100 training instances, aggregated into recent, mid-term, and long-term lag groups.Uncertainty is estimated from the standard deviation of ensemble-tree predictions at each forecast step.
  • Forecasting model: Recursive multi-step forecasting feeds each prediction back as input for the next step across the forecasting horizon.Intermediate prediction inputs are recorded for downstream explanation.

4.5 Agents and Large Language Model

The system uses LLMs within a modular multi-agent architecture to reason over numerical, visual, and model-derived forecasting information. Its design combines multimodal inputs with SHAP and uncertainty signals for grounded explanations.

  • Agents and Large Language Model: LLMs serve as the core reasoning engine within the multi-agent architecture.GPT-5-mini supports orchestration, while GPT-5.2 handles multimodal interpretation and explanation.
  • Agents and Large Language Model: The multimodal inputs include historical and forecasted time series, forecast plots, SHAP values, and uncertainty estimates.
  • Agents and Large Language Model: Structured prompts combine data-level information, visual evidence, and model-level explanations to align generated responses across modalities.The design is intended to reduce hallucination and improve interpretability through grounded reasoning.
  • Agents and Large Language Model: A modular containerized architecture separates the frontend, multi-agent reasoning layer, and backend analytical layer.The frontend supports natural-language interaction and visualization of forecasts and explanations.
  • Agents and Large Language Model: The analytical backend uses Random Forest forecasting, SHAP attribution, and standard numerical and plotting libraries.The containerized setup separates reasoning from computation and supports reproducibility.

4.7 User Queries

The evaluation uses representative user queries drawn from the system’s operational forecasting and monitoring setting.

  • User Queries: The study constructs 30 representative user queries reflecting the target interaction scenarios.

4.8 Evaluation

The evaluation compares three forecasting pipelines using an LLM-based judge and user-centric and model-centric explanation metrics. Scores are aggregated across the query set with extra weight on explainability-related dimensions.

  • Evaluation: The evaluation compares baseline forecasting (BF), interpretability-enhanced (IF), and explainability-enhanced (EF) pipelines on 30 queries.Each query is processed by all three pipelines under identical conditions.
  • Evaluation: An LLM-based evaluator receives the query, system response, and operational constraints to assess explanation quality.
  • Evaluation: Deterministic decoding and identical prompts are used across variants to reduce evaluation variance.The evaluator uses temperature = 0.
  • Evaluation: The metrics cover clarity, helpfulness, insightfulness, trustworthiness, model awareness, consistency, hallucination, actionability, and uncertainty awareness.Clarity, helpfulness, insightfulness, trustworthiness, model awareness, and consistency use specified bounded scoring ranges; hallucination is binary.
  • Evaluation: The overall score emphasizes insightfulness, trustworthiness, and model awareness because these dimensions are central to explainability.

5 Results

The explainability-enhanced configuration performs best overall, with gains concentrated in model awareness, insightfulness, trustworthiness, uncertainty awareness, and hallucination reduction. Clarity and consistency remain comparable across configurations.

  • 5 Results: 3.56 is the EF overall score, exceeding IF at 3.29 and BF at 2.69.IF improves over BF mainly in helpfulness, insightfulness, and model awareness; EF adds gains in model awareness, trustworthiness, and insightfulness.
  • 5 Results: The results show monotonic improvement from BF to IF and EF, with the largest gains in model awareness and insightfulness.
  • 5 Results: EF consistently outperforms or matches the baselines across most metrics while maintaining high clarity and consistency.
  • 5 Results: Multimodal grounding improves helpfulness and insightfulness over the baseline, while the largest gains appear after explicit explainability signals are introduced.
  • 5 Results: EF achieves the highest insightfulness, trustworthiness, and model awareness, while also improving uncertainty awareness and reducing hallucinations.
  • 5 Results: IF improves overall performance by approximately 22% over BF, while EF adds approximately 8% over IF.

7 Conclusion

The framework delivers forecasting model behavior through structured language rather than plots, positioning multimodal, model-aware explanations as a foundation for non-visual interaction. The explainable framework improves overall performance by approximately 32% over the baseline.

  • The framework conveys time-series forecasting model behavior entirely through language rather than plots and dashboards, supporting non-visual interaction.Its unified multi-agent architecture integrates forecasting, visualization, and model-aware reasoning into structured, uncertainty-explicit explanations.
  • The radar plot presents normalized system performance profiles across evaluation metrics, with all metrics scaled to [0,1] for comparability.
  • Approximately 32% improvement over the baseline demonstrates the cumulative impact of multimodal context and model-aware explanations.

6 Accessibility Perspective and Non-Visual Interaction

The framework treats accessible forecasting as a model-aware, multimodal interaction problem, translating visual outputs into structured textual explanations. It presents this as a promising foundation, while identifying human-centered validation as necessary to establish accessibility and usability.

  • Non-Visual Interaction: The Explainable pipeline grounds textual explanations in SHAP attributions, temporal importance, and uncertainty signals available independently of images.Unlike the perception-driven Interpretable pipeline, it can convey forecast behavior and confidence entirely in language.
  • Accessibility Perspective: Multimodal grounding improves clarity, helpfulness, and insightfulness, while model-aware signals further enhance trustworthiness and model awareness.
  • Limitations and Future Work: Participatory evaluations with blind and low-vision users are required before accessibility can be claimed empirically.The paper explicitly distinguishes a promising foundation from an established accessibility result.
  • Limitations and Future Work: The framework’s evaluation uses an LLM-based judge as an early-stage proxy, so real trustworthiness and usability remain unestablished empirically.Future work includes participatory evaluation and a speech-based, non-visual interaction loop.

Safe and Responsible Innovation Statement

The work studies multimodal LLM-based decision support for DevOps using system-level, non-personal telemetry and supporting human decision-making rather than full automation. Accessibility and user-experience claims remain outside the evidence because the evaluation includes no human participants and relies on an LLM-based judge.

  • The system uses system-level, non-personal telemetry data and supports human decision-making rather than full automation.
  • No empirical accessibility or user-experience claims are made because the evaluation involves no human participants and relies on an LLM-based judge.
  • Participatory design and evaluation with low-vision users are required before accessibility can be claimed.
Loading 2608.24910v1…