Source-linked AI summary
MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations
Yuyang Cheng, Raghav Kaushik Ravi, Srivarshinee Sridhar, Sriparna Saha, Akash Ghosh, Chirag Agarwal
TL;DR
High-stakes machine-learning models remain opaque, and using post-hoc explanations can require expertise that domain practitioners lack. MEA selects and executes explanation tools, then optimizes natural-language explanations against faithfulness; it outperforms several explanation baselines, with gains over the untrained backbone across tabular, text, and vision tasks.
Problem
Machine-learning opacity is a critical concern in high-stakes domains, while existing explanation systems lack learned adaptation and broad, principled evaluation.
Source: §1 · pp. 1–2Method
MEA uses Proposer and Actor agents to select, execute, and synthesize explanation tools, optimizing explanations with reinforcement learning and perturbation-based faithfulness scores.
Source: §1 · p. 2, §3.4 · p. 4Results
Takeaways & Limitations
The authors conclude that learning to explain is more principled than prompting to explain.
Source: §6 · p. 9MEA is scoped to classification tasks and a fixed tool library, and its perturbation-based protocol is sensitive to distributional shift.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Recent years have seen the employment of a plethora of machine learning (ML) models in high-stakes domains, but they remain largely opaque to the practitioners who act on their predictions. While post-hoc explanation methods offer a lens into this model behavior, wielding them effectively demands expertise most domain experts lack: navigating high-dimensional outputs, selecting the best explanations, and synthesizing evidence across disparate tools. To this end, we present MEA, a multi-agent framework that removes the explanation knowledge barrier entirely: a Proposer agent selects and configures explanation tools based on the question and modality, while an Actor agent is optimized end-to-end against faithfulness, transforming the outputs into natural language explanations grounded in model behavior across tabular, text, and vision modalities. Further, we introduce diverse question types spanning feature attribution, counterfactual reasoning, and spurious feature detection, each paired with a perturbation-based faithfulness metric. We find that frontier LLMs systematically produce unfaithful explanations. By optimizing against faithfulness rewards augmented with a modality-adaptive penalty, MEA consistently outperforms post hoc explainers, agentic, and closed-source baselines across six datasets, with reward-driven optimization yielding faithfulness gains of +28% (tabular), +21% (text), and +34% (vision) over the untrained backbone. More broadly, our findings suggest that AI agents themselves can serve as a scalable, adaptable interface to ML explainability, opening a path toward natural-language explainability that generalizes beyond the fixed, single-purpose tools that have long defined the field.
1 Introduction
MEA addresses the expertise and evaluation gaps in model explainability with a multi-agent framework trained for faithful natural-language explanations. Across modalities, the trained system outperforms post-hoc explainers, frontier models, and agentic baselines.
1 Introduction
- Opaque ML models are increasingly used in high-stakes domains, raising concerns about trustworthiness among practitioners, regulators, and end users.The introduction frames the gap between predictive performance and human interpretability as a central explainability challenge.Source: §1 · p. 1
- Post-hoc explainers produce fixed output types, while selecting tools, interpreting high-dimensional results, and combining evidence can require expertise domain practitioners lack.Examples of output types include attribution vectors, heatmaps, and super-pixels.Source: §1 · p. 1
- Conversational explanation frameworks reduce tool-selection demands, but are largely training-free, lack broad multimodal evaluation, and often lack quantitative verification against model behavior.The passage notes that prior evaluations may be limited to one explanation type or simple linear models.Source: §1 · p. 2
- MEA’s Proposer selects and configures tools for the question and modality, while its Actor executes them and synthesizes natural-language explanations optimized end-to-end for faithfulness.The framework adds a modality-adaptive reward pairing a universal tool-count penalty with modality-specific constraints.Source: §1 · p. 2
- MEA-BENCH covers tabular, text, and vision data, six model architectures, six datasets, and 13K training plus 3.5K test instances.The benchmark pairs diverse question types—including feature attribution, counterfactual reasoning, and spurious feature detection—with perturbation-based faithfulness metrics.Source: §1 · p. 2
- +28% (tabular), +21% (text), and +34% (vision) faithfulness gains over the untrained backbone accompany MEA’s reported wins over post-hoc, closed-source, and agentic baselines.The gains are reported in the introduction’s summary of results.Source: §1 · p. 3
2 Related Works
Related work spans fixed post-hoc explainers, conversational systems, and agentic approaches. The paper identifies a gap in adapting faithfulness evaluation into end-to-end training across modalities and question types.
2 Related Works
- Post-hoc explainers attribute predictions to input features, using local surrogates such as LIME [7], Shapley-value attributions such as SHAP [10], and pixel-level signals such as Grad-CAM.Gradient techniques [8] [12] also provide pixel-level attribution signals for vision.Source: §2 · p. 3
- Prior faithfulness evaluation is described as offline, with no prior work adapting it into an online training signal across comprehensive multi-question, multi-modal settings.The paper also identifies a gap in jointly optimizing a multi-agent pipeline end-to-end with RL across modalities and question types.Source: §2 · p. 3
- Conversational systems include TalkToModel [15] for natural-language exploration and ECHO for grounding answers in computed attributions; FAX combines agentic tool use with faithfulness evaluation.The passage distinguishes FAX as targeting open-ended questions and relying on an LLM-as-a-judge pipeline.Source: §2 · p. 3
3 Our Framework: MEA
MEA formulates cross-modal, prediction-level explanation generation as a Proposer–Actor pipeline and trains its components to produce question-matched, model-grounded explanations. It pairs this design with perturbation-based faithfulness metrics and end-to-end reward optimization.
3 Our Framework: MEA
- MEA generates explanations for classifier predictions across vision, text, and tabular inputs, using feature attribution, counterfactual, and spurious-feature questions.Q8 and Q9 are excluded from training because they apply only to misclassified instances, limiting eligible samples and destabilizing gradient estimation.Source: §3 · p. 3
3.1 MEA Overview
- A shared multimodal backbone first has the Proposer select tools and reasoning tasks, then has the Actor synthesize their outputs into an explanation.The Proposer’s selected tools are a subset of those available for the input modality.Source: §3.1 · pp. 3–4
3.2 Proposer Agent: Intent-Aware Strategy Formation
- The trainable Proposer forms an intent-aware strategy by choosing autonomous, tool-based, or hybrid reasoning, the tools to invoke, and Actor subtasks.An ablation finds tool use and autonomous reasoning independently load-bearing, with MEA combining them more effectively than the base agent.Source: §3.2 · p. 4
3.3 Actor Agent: Tool-Grounded Explanation Synthesis
- The Actor executes selected tools, converts their outputs into structured text summaries, and uses those summaries to generate evidence-grounded explanations.It standardizes tabular and text outputs to the top 25% of features or tokens and aggregates the top 1% of vision attribution coordinates into a bounding box.Source: §3.3 · p. 4
3.4 Perturbation-based Faithfulness Evaluation
- Faithfulness is evaluated by masking the explanation’s identified region, rerunning the classifier, and checking whether its behavior changes as the question predicts.Masking is modality-specific: zero-filling for tabular data, grey-filling for vision, and span deletion for text.Source: §3.4 · p. 4
- Attribution metrics test confidence or margin changes after masking, counterfactual metrics test predicted probability shifts, and spurious-feature metrics measure correct-class probability gain after masking.Q4 and Q10 also assess contrastiveness across differently predicted instances; Q9 averages Q8 over misclassified instances.Source: §3.4 · pp. 4–5
3.5 End-to-End Training
- GRPO optimizes both agents end-to-end using a terminal faithfulness reward, with the group-normalized advantage applied to both transitions.The Proposer and Actor are treated as jointly responsible because the Actor cannot recover from a misaligned Proposer strategy.Source: §3.5 · p. 5
- Modality-adaptive size and tool-count penalties discourage large vision masks and indiscriminate tool invocation while supporting informative tool selection and faithful grounding.The fixed top-25% output constraint makes an explicit size penalty unnecessary for tabular and text explanations.Source: §3.5 · p. 5
4 Experiments
The experiments evaluate MEA’s faithfulness across three modalities and diverse explanation questions, using a multimodal benchmark, perturbation-based metrics, and comparisons with prompting, post-hoc, and counterfactual baselines.
4 Experiments
- The experiments ask whether MEA is faithful across three modalities, outperforms post-hoc explainers, depends on prompting style and agentic design, and generalizes to OOD questions, datasets, and architectures.The benchmark includes feature attribution, counterfactual analysis, and spurious-feature identification questions.Source: §4 · p. 6, §4.1 · p. 6
4.1 Datasets and Experimental Setup
- The benchmark pairs Adult Census Income and Breast Cancer, SNLI and IMDb, and CUB-200-2011 and STL-10 with trained models, structured questions, and instance-level prediction information.Questions cover feature attribution, counterfactual analysis, and spurious-feature identification across tabular, text, and vision data.Source: §4.1 · p. 6
- Sequential training degraded earlier-modality performance and reduced behavioral diversity on the final modality, so the main results use interleaved Hybrid training.The setup fine-tunes Qwen3.6-35B-A3B for three epochs; a 4B-backbone ablation tests transfer to smaller models.Source: §4.1 · p. 6
- Baselines include Zero-Shot, CoT, ReAct, ToT, standard post-hoc explainers, and the specialized counterfactual generators DiCE and Growing Spheres.Prompting paradigms share model infrastructure, execution environments, and XAI toolsets.Source: §4.1 · p. 7
- Faithfulness is assessed by perturbing the explanation’s identified region, rerunning the target model, and scoring its behavior with a corresponding metric.The perturbation varies by modality, including gray-filled or inpainted vision regions, deleted or replaced text spans, and mean-filled or agent-specified tabular features.Source: §4.1 · p. 7
5 Results
MEA reports stronger faithfulness across modalities and question types, including comparisons with post-hoc explainers and specialized counterfactual generators. The results also examine prompt robustness, critic-based refinement, and generalization to unseen datasets and architectures.
RQ1: MEA improves explanation faithfulness across all modalities
- MEA reports the highest faithfulness across all three modalities, including 0.598 vs. GPT-5.4’s 0.328 on vision counterfactual questions and ceiling spurious-feature scores of 1.000 on text and vision.It also reports outperforming closed-source models and CoT, ToT, and ReAct across all question types and modalities.Source: §5 · p. 7
RQ2: MEA outperforms post-hoc explainers
- On feature attribution, MEA scores 0.195 vs. 0.150 on tabular and 0.265 vs. 0.237 on text, while matching the best vision method at 0.499 vs. 0.497.The vision comparison reports p = 0.944.Source: Figure 4 · p. 8, §5 · p. 8
- On tabular counterfactual questions, MEA achieves comparable flip-success rates to DiCE and Growing Spheres but higher faithfulness, with average target-class probability of 0.886–0.894 vs. 0.499–0.631.The comparison is restricted to tabular data because the specialized generators do not meaningfully search text spans or image regions.Source: Table 1 · p. 8, §5 · p. 8
RQ4: Generalization to unseen datasets and architectures
- MEA achieves the highest overall faithfulness on unseen datasets and architectures: 0.581 vs. 0.379 on tabular, 0.689 vs. 0.625 on text, and 0.576 vs. 0.448 on vision.It underperforms the base agent on text counterfactual questions (0.770 vs. 0.860) and vision feature attribution (0.583 vs. 0.621), while also outperforming the base agent by ∼18% in mean faithfulness on concept-level explanations.Source: Table 2 · p. 9, §5 · p. 9
RQ3: Prompt robustness and critic-augmented agents
- Paraphrases score within < 1% of original MEA results, while MEA exceeds Agent+Reflection by +36% on text and +30% on vision counterfactuals.Agent+Reflection also degrades performance relative to the untrained base agent on tabular spurious features; paraphrase tests find no significant degradation in eight of nine conditions.Source: §5 · p. 9
6 Conclusion
The conclusion presents MEA and MEA-BENCH for faithful prediction-level explanations across tabular, text, and vision. It reports gains from end-to-end RL and identifies the current task, tool-library, and evaluation-protocol boundaries.
6 Conclusion
- MEA combines multi-agent prediction-level explanations across tabular, text, and vision with MEA-BENCH, a model- and modality-agnostic benchmark of 16k instances paired with faithfulness metrics.The conclusion describes the benchmark instances as each paired with a faithfulness metric.Source: §6 · p. 9
- End-to-end RL with a modality-adaptive reward yields faithfulness gains of +28% on tabular, +21% on text, and +34% on vision over the untrained backbone.The conclusion says the trained agent consistently outperforms post-hoc explainers, frontier models, and advanced agentic frameworks.Source: §6 · p. 9
- The system is currently scoped to classification tasks and a fixed tool library, and its perturbation-based protocol is sensitive to distributional shift.The conclusion identifies extending to other tasks, tool libraries, and faithfulness signals as future work.Source: §6 · p. 9
A Appendix
The appendix collects supporting material on the question taxonomy, tools, metrics, training, evaluations, and implementation details.
A Appendix
- Appendix material covers the question taxonomy, XAI tools, faithfulness metrics, formal RL problem definition, and dataset statistics.These topics are listed in sections A.1–A.5.Source: A Appendix · p. 14
- Additional analyses address training procedures, OOD and concept-level generalization, prompt robustness, critic behavior, Proposer strategy modes, backbone-scale transfer, and a synthetic-shortcut sanity check.The appendix lists these topics across sections A.6–A.12.Source: A Appendix · p. 14
- The appendix also includes statistical significance analyses, computational-cost details, and Actor and Proposer prompt templates.These are listed in sections A.13–A.15.Source: A Appendix · p. 14
A.1 Question Taxonomy
The section points readers to Table 3 for detailed question definitions and categories.
A.1 Question Taxonomy
- Table 3 presents the detailed definitions and categories of the questions.The supplied passage does not specify the individual questions or categories.Source: §A.1 · p. 14
A.2 XAI Tools
The framework provides established explanation-attribution tools spanning all three modalities, including gradient-based and perturbation-based methods.
A.2 XAI Tools
- The explanation library includes established attribution methods spanning vision, text, and tabular modalities.It groups the listed approaches into gradient-based and perturbation-based methods.Source: §A.2 · p. 14
- Gradient-based tools include GradCAM, Integrated Gradients, Guided Backpropagation, SmoothGrad, and Sensitivity Analysis.GradCAM is vision-only; Sensitivity Analysis is limited to text and tabular inputs.Source: §A.2 · p. 14
- Perturbation-based tools include LIME and SHAP, with LIME using superpixel or token-level segmentation for vision and token-level perturbation for text.SHAP estimates Shapley additive explanations through marginal contributions.Source: §A.2 · p. 14
A.3 Faithfulness Metrics
The section summarizes faithfulness metrics for different explanation questions and states how to interpret their scores, including a modality-specific distinction for Q6.
A.3 Faithfulness Metrics
- The metric notation distinguishes original and perturbed top-class gaps, the ground-truth class, the post-perturbation prediction, prediction-probability drop, and attribution maps for correct and incorrect classifications.The passage names these quantities but does not provide their equations.Source: §A.3 · p. 14
- Table 4 summarizes evaluation metrics corresponding to different explanation-faithfulness questions.The supplied table caption does not provide the individual metric definitions or values.Source: Table 4 · p. 15
- Higher metric scores indicate more faithful explanations; binary metrics Q3, Q5, Q6, and Q7 score 1 when their condition holds and 0 otherwise.For Q6, tabular and text inputs use a specific target class, while vision checks for a change to any alternative class.Source: §A.3 · p. 15
A.4 Formal RL Problem Definition
MEA frames the Proposer and Actor as sequential decision points in a single-episode MDP with a shared backbone and a terminal faithfulness reward.
A.4 Formal RL Problem Definition
- MEA models the Proposer and Actor as two sequential decision points in a single-episode Markov Decision Process sharing backbone θ.The Proposer selects a structured strategy; the Actor produces the explanation.Source: §A.4 · p. 15
- The Proposer state contains the question, input, frozen-classifier prediction, and modality-compatible tools, while its structured action selects tools and automatic analyses.The strategy is a structured action because its tool and analysis selections are combinatorial subsets.Source: §A.4 · p. 15
- Selected tools run deterministically against the frozen classifier and input, producing summarized outputs that join the Proposer strategy in the Actor state.The only episode randomness comes from the agents’ sampling.Source: §A.4 · p. 15
- A terminal reward applies the question-specific faithfulness metric with modality-adaptive size and tool penalties; shared-backbone roles receive the same group-normalized advantage across their tokens.There is no intermediate reward at the Proposer transition.Source: §A.4 · pp. 15–16
A.5 Dataset Statistics
The appendices examine training schedules, out-of-distribution performance, prompt robustness, and agent-design ablations. They also test whether MEA follows a deliberately injected shortcut and report statistical comparisons across selected tasks.
A.6 Sequential and Hybrid Training
- Sequential training shows catastrophic forgetting: after text training, tabular Q3 falls from 0.326 to 0.264 and Q6 from 0.780 to 0.728; hybrid reward rises from 0.313 to 0.594.During sequential vision training, reward rises only from 0.512 to 0.578; the authors link this limited signal to reduced rollout diversity and a policy concentrated on tabular/text formats. Hybrid training interleaves all modalities, and its reward curve is shown in Figure 6.Source: Sequential Training · p. 16, Hybrid Training · p. 16, Figure 6 · p. 17
A.7 Ablation Study: OOD Generalization
- On held-out spurious-feature questions, MEA has higher overall faithfulness on tabular and text, but the base agent slightly outperforms it on vision.For the held-out concept-attribution question, MEA scores about 18% higher mean concept faithfulness than the base agent across 120 questions, despite no training on that question type.Source: §A.7 · pp. 17–18, Table 5 · p. 17, Table 7 · p. 18
- On unseen dataset–model combinations, MEA achieves the highest overall faithfulness across all three modalities, while trailing the base agent on text counterfactuals and vision feature attribution.The authors attribute those exceptions to a need for further hyperparameter tuning of modality-specific trade-offs.Source: §A.7 · p. 17, Table 6 · p. 18
A.9 Ablation Study: Critic Agent
- MEA outperforms Agent+Reflection on text and vision counterfactuals, while the critic step lowers the base agent’s tabular spurious-feature score from 0.677 to 0.514.MEA reaches 0.876 on text counterfactuals and 0.569 on vision counterfactuals; the figure compares the base agent, Agent+Reflection, and MEA.Source: §A.9 · pp. 18–19, Figure 8 · p. 19
A.10 Ablation Study: Proposer Strategy Mode
- For MEA, free Proposer choice exceeds the better forced single-mechanism mode in every modality, whereas the base agent’s free-choice score remains below tools-only.On Q6, MEA scores 0.822 with free choice, versus 0.766 for tool_only and 0.782 for autonomous_only.Source: §A.10 · pp. 20–21, Table 10 · p. 21
A.11 Ablation Study: Backbone Scale Transfer
- On the smaller Qwen3.5-4B backbone, training improves faithfulness in all modalities: tabular 0.419 → 0.671 (+60%), text 0.403 → 0.575 (+43%), and vision 0.142 → 0.508 (+258%).This comparison uses one training epoch and the same reward pipeline as the paper’s evaluation protocol.Source: §A.11 · p. 21, Table 11 · p. 23
A.12 Sanity Check: Synthetic Shortcut on a Biased Classifier
- On the injected-shortcut classifier, MEA wins or ties on all eight tested question types and names the dominant shortcut on 12/13 eligible Q1 rows, versus 9/13 for the base agent.It never incorrectly labels the dominant shortcut least responsible on the 20 Q2 rows; the authors scope the finding to this tabular bias-injection design.Source: §A.12 · p. 22, Table 12 · p. 23
A.13 Statistical Significance Tests
- MEA is statistically indistinguishable from three vision gradient-based explainers and significantly outperforms three others; on Q10, its ceiling scores leave only marginal gaps over the strongest baselines.For Q10, the reported gaps are +0.004 over Gemini-3.1 on text and +0.019 over the base agent on vision; 8 of 9 paraphrase conditions show no significant difference, with vision counterfactuals favoring paraphrase by 0.075.Source: §A.13 · pp. 23–24, Table 13 · p. 23, Table 14 · p. 24