PaperlayerView exact revision on arXiv

Source-linked AI summary

Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations

Antonin Poché, Fanny Jourdan, Nils Feldhus, Qianli Wang, Jing Yang, Simon Ostermann, Nicholas Asher, Philippe Muller, Vera Schmitt

arXiv:2609.08585v1cs.CLcs.LG

TL;DR

The paper asks whether automated simulatability reliably measures explanation usefulness when LLM simulators replace human evaluators. It replicates and extends ConSim across datasets, explanation families, and simulators, finding task-prior and hidden-label shortcuts that limit interpretation of the scores. The authors use these findings to recommend more robust evaluation protocols.

  • Problem

    Existing explanation metrics emphasize internal faithfulness or complexity, while human simulatability studies are costly to scale.

  • Method

    The paper qualitatively replicates and extends ConSim across datasets, explanation families, LLM simulators, and anonymized evaluation variants.

  • Results

    Across tested settings, simulator predictions are often dominated by task priors with explanations producing small changes, while anonymized rankings can reward hidden label leakage.

  • Takeaways & Limitations

    Automated simulatability should be designed and interpreted cautiously because high scores may reflect direct task solving or recovery of the hidden label mapping.

  • Takeaways & Limitations

    The findings use an LLM simulator with up to 15B parameters without reasoning and might not generalize to larger, better, or closed-source LLMs.

Abstract

from arXiv · show

Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs. Since human evaluation is costly, automated simulatability replaces human explainees with LLM simulators, as proposed in ConSim (Poché et al., 2025) for large-scale experiments. We qualitatively replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs, and identify two limitations. First, when class names are meaningful, simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations. Second, class anonymization can reward explanations for leaking the hidden label mapping, a limitation we expose with a new classes-as-concepts baseline. These results are consistent with a shortcut hypothesis: in the tested settings, simulator predictions mainly rely on task priors, while explanations produce small changes. We derive recommendations for more robust automated simulatability evaluations.

1 Introduction

The paper examines automated simulatability as a scalable way to evaluate whether explanations help users predict task-model behavior. It identifies shortcut risks in which simulators rely on task priors or recover hidden label mappings instead of using explanation content.

  • Faithfulness and complexity metrics assess explanations’ internal properties, not whether they are useful or plausible to an explainee.
  • Simulatability evaluates explanations end-to-end by measuring whether they help an explainee predict task-model behavior on held-out inputs.
  • The paper replicates and extends ConSim across additional explanation types, LLM simulators, and datasets.
  • In non-anonymized experiments, simulator predictions change little with explanations, consistent with a shortcut in which task priors dominate explanation content.
  • Under anonymization, the classes-as-concepts baseline outperforms every tested method, showing that rankings can reward hidden-label leakage rather than meaningful explanations.

2 Background

Simulatability shifts explanation evaluation from internal faithfulness toward downstream usefulness, but automated versions replace human explainees with LLM simulators. ConSim uses anonymization to encourage explanation use, while this paper studies the resulting evaluation risks.

  • Explanation types: Attributions quantify input-feature contributions, rationales provide free-form textual justifications, and concept-based explanations map task-model computations to human-understandable concepts.
  • Existing metrics mainly assess task-model behavior or generalization rather than how effectively explanations communicate to an evaluator.
  • Simulatability measures whether explanations help an evaluator reproduce a task model’s outputs on held-out inputs.
  • Automated simulatability: Automated simulatability replaces costly human explainees with LLM simulators, and ConSim adds anonymized experiments intended to force reliance on explanations.

3 Replication and extension of ConSim

The paper qualitatively reconstructs ConSim while extending its datasets, explanation coverage, simulators, and prompting setup. The reconstruction preserves the broad method ordering, while revised prompts improve scores and non-anonymized concept methods remain close to no-explanation baselines.

  • 3.1 Replication setup: The study compares old_consim with new_consim and uses new_consim as the common prompt format after protocol selection.
  • 3.1 Replication setup: The replication preserves ConSim’s simulatability logic, prompt-type families, anonymized variants, methods, TopK interpretation, and three original datasets.
  • 3.1 Replication setup: The extension adds AG News and GoEmotions, replaces Tweet Eval Emotion with Emotion, and modifies BIOS handling.
  • 3.1 Replication setup: Evaluation uses 50 seeds grouped into 10 disjoint groups of 5, with each grouped observation aggregating 100 predictions to reduce discreteness and variance.
  • 3.3 Reproduction and protocol selection: The reconstruction preserves the broad ConSim ordering: Vanilla SAE, ICA, and Semi-NMF rank above no explanation, while PCA and SVD remain near the bottom.
  • 3.3 Reproduction and protocol selection: new_consim yields significantly higher simulatability scores for 8/10 prompt types, while its rankings correlate with old_consim at Pearson r = .87 and Spearman ρ = .87.
  • 3.3 Reproduction and protocol selection: TopK yields significantly higher scores for 5/6 applicable concept prompt types and becomes the default concept interpretation.
  • 3.3 Reproduction and protocol selection: Non-anonymized score distributions overlap strongly between concept methods and matched no-explanation baselines, motivating analysis of this limitation.

4 Task-solving shortcuts in non-anonymized simulatability

In non-anonymized settings, explanations generally produce small simulator changes, consistent with task-prior shortcuts rather than strong reliance on explanation content. This pattern is broad but depends on task difficulty and simulator choice.

  • Experimental setup: The evaluation extends non-anonymized experiments across concepts, attributions, rationales, datasets, and LLM simulators while retaining fixed prompt structure.Representative methods were Vanilla SAE, LIME, and Qwen3.5-2B; comparisons included Qwen-3.5-9B, Llama-3.1-8B, Gemma-4-12B, and Phi-4.
  • Explanation gains: With Qwen-3.5-9B, concept and attribution prompts significantly exceed matched no-explanation baselines, but mean gains are small relative to experimental variability.Gemma-4-12B and Phi-4 show similar rankings and conclusions, with concept prompts showing the clearest improvements.
  • Explanation gains: Three of four retained user-LLMs show similar small gains, whereas Llama-3.1-8B reverses the pattern, demonstrating simulator dependence.Llama-3.1-8B’s lower agreement with the task model may contribute to this difference.
  • Task-solving shortcuts: B1, which provides class names and task description but no examples, predictions, or explanations, achieves simulatability above 0.7 across several datasets.Because balanced evaluation makes always predicting the gold label worth 0.5, a score of 0.7 requires agreement with at least 40% of task-model errors.
  • Task-solving shortcuts: Prompt predictions strongly agree, including B1, so adding examples or explanations often does not change simulator outputs; IMDB predictions match gold labels nearly 100% across prompt types.These patterns are consistent with task-prior reliance, especially when direct classification is easy.
  • Task-solving shortcuts: Explicitly instructing simulators to simulate the task model generally reduces simulatability rather than aligning predictions with task-model behavior.The comparison holds prompt content and experimental settings fixed apart from the framing instruction.
  • Interpretation: These observations support, but do not establish, a shortcut hypothesis in which explanations produce smaller changes when direct classification is easy.The authors note that explanations may be uninformative or may be ignored even when relevant.

5 Anonymization sensitivity to leakage

Class anonymization removes direct access to meaningful output names but can introduce a leakage shortcut: explanations may reveal the hidden mapping between anonymous labels and original classes. The classes-as-concepts baseline exposes this failure mode by achieving the highest anonymized scores through direct mapping disclosure.

  • Anonymization: Anonymization replaces meaningful class names with generic labels, requiring simulators to infer the label mapping from examples and explanations rather than class semantics.Without learning-phase information, the anonymized B1 baseline is expected to perform at chance.
  • Leakage mechanism: Anonymization hides output labels but not the semantic information in explanations, allowing explanations to reveal which original class each anonymous label represents.This creates a second shortcut distinct from directly solving the classification task from meaningful class names.
  • Classes-as-concepts: The classes-as-concepts baseline directly associates each anonymous label with a class name, revealing the hidden mapping in anonymized prompts.For example, it can associate Class_0 with “surgeon.”
  • Results: Classes-as-concepts obtain the highest scores across all anonymized prompt types and outperform every other tested concept method.This ranking reflects label-mapping disclosure rather than a better explanation of task-model behavior.
  • Implications: Anonymizing explanation strings alone would not fully prevent leakage because synonyms and other semantic descriptions could reveal the same mapping.The authors recommend including classes-as-concepts or a similar label-leakage baseline whenever anonymized prompts are used.

6 Conclusions and recommendations

The paper finds that non-anonymized explanations provide limited separation from no-explanation baselines, while anonymization can reward explanations that leak hidden label mappings. These limitations motivate recommendations for more robust automated simulatability evaluation.

  • Non-anonymized explanations provide limited separation from no-explanation baselines across concepts, attributions, rationales, and LLM simulators.Although gains are statistically significant, they remain small relative to their variability.
  • Simulator predictions are often dominated by direct task solving, with explanations producing only secondary changes.The paper presents several observations consistent with this shortcut hypothesis.
  • Anonymized evaluations can reward explanations that solely reveal the hidden label mapping.Class anonymization is intended to prevent direct task solving, but introduces this separate failure mode.
  • The classes-as-concepts baseline outperforms other concept methods under anonymization despite providing no information about the task model.
  • The authors recommend over-sampling task-model errors, including a label-leakage baseline, using multiple simulators, increasing seed counts, and selecting difficult tasks.These recommendations aim to make scores more interpretable, stable, and resistant to shortcuts.

Limitations

The findings are scoped to LLM simulators of up to 15B parameters without reasoning and to tasks likely encountered during training or solvable through semantic knowledge.

  • The experiments use an LLM simulator with up to 15B parameters and without reasoning.The authors caution that results may not generalize to larger, better, or closed-source LLMs.
  • The analysis focuses on use cases simulators likely encountered during training or could solve using sufficient semantic knowledge.The authors did not work around contamination.

Ethical statement

Coding and writing AI assistants supported debugging, experiment scaling, visualization, appendices, and simulated reviews, with all work human-reviewed at least once.

  • Coding AI assistants helped debug, scale cluster experiments, improve visualizations, and detail appendices.The project used 590 H100 hours.
  • Writing AI assistants mainly simulated reviews to improve coherence and narrative flow.Everything was human-reviewed at least once.

Ackowledgements

The paper defines explanation families, simulatability, and the ConSim protocol, then describes concept-explanation pipelines, datasets, sampling, prompt comparisons, and evaluation details.

  • Explanation types: Attributions quantify input-feature contributions, rationales provide textual justifications, and concept-based explanations map internal computations to human-understandable concepts.
  • Concept-based explanations: Concept-based explanations use a three-stage pipeline: learn concepts, interpret directions semantically, and weight concept importance toward predictions or classes.
  • Simulatability: Simulatability measures whether an explainee can predict task-model outputs from explanations on held-out inputs.It shifts evaluation from internal faithfulness toward downstream utility.
  • Automated simulatability: Automated simulatability replaces human explainees with LLM simulators, changing what the evaluation measures.ConSim introduces the simulator to a classification task, provides samples and predictions, and evaluates held-out predictions.
  • Datasets and methods: The retained datasets include BIOS, IMDB, Rotten Tomatoes, AG News, and GoEmotions, with concept methods including NMF, SAE, ICA, PCA, SVD, and neurons-as-concepts.The implementation uses one task model per dataset and SemiNMF where activations are not necessarily positive.
  • Experimental design: The evaluation uses fixed three-class subsets, balanced correct and incorrect predictions, multiple seeds, and controlled ConSim prompt specifications.Table 2 compares changes in prompt organization, importance rendering, evaluation batching, wording, and field names.
  • Class-subset construction: BIOS and GoEmotions subsets were selected from confusion matrices, prioritizing groups with large off-diagonal confusion relative to class frequency.Table 3 reports the resulting three-class subsets and their original dataset label indices.
  • Evaluation details: Less than 1% of predictions were invalid and therefore were not reported.

F Protocol comparison

The protocol comparison selects new_consim and TopK interpretations for subsequent experiments while qualitatively reproducing ConSim’s method ordering. Across formats and datasets, score shifts are often significant but remain broadly correlated, and non-anonymized explanation distributions overlap their baselines.

  • Prompt-format comparison: new_consim significantly outperforms old_consim for 8 of 10 prompt types, while remaining strongly correlated with the reconstructed format.C1 performs significantly worse and B1 shows no significant difference.
  • Concept interpretations: TopK interpretations significantly outperform LLM-generated interpretations for 5 of 6 applicable concept prompt types.The comparison applies to C1–C3 and AC1–AC3; AC1 shows no significant difference.
  • Method reproduction: Vanilla SAE ranks first in both reconstructed old-ConSim and new-ConSim comparisons, followed by ICA or Semi-NMF, while PCA and SVD remain lower-ranked.The new-ConSim comparison covers all six datasets and retains 17 of 20 pairwise differences after Holm correction.
  • Score distributions: Non-anonymized concept-method score distributions overlap strongly with matched no-explanation baselines, whereas anonymized prompts show greater separation.The distributions are shown for grouped-seed TopK configurations under old and new prompting.
  • Dataset effects: Prompt-format effects vary by dataset and prompt type: reorganizing prompts usually helps, especially for anonymized concepts, but does not create a uniform score shift.The plotted quantity is new_consim −old_consim, with positive values favoring the revised format.

I Explanation-method rankings with Qwen-3.5-9B

Within explanation families, Vanilla SAE and LIME are selected as representative methods, while rationale generators are statistically indistinguishable. Across simulators, explanation gains are generally small and unstable, with Llama showing the opposite pattern from the other judges.

  • Concepts: Vanilla SAE ranks first among six concept methods, winning 60–66% of comparisons with mean advantages of 0.013–0.018.All five comparisons against contenders remain significant after Holm correction.
  • Attributions: LIME ranks first among ten attribution methods with 55–62% win rates, but none of the 45 pairwise differences remains significant after Holm correction.It is retained as a deterministic representative rather than evidence of reliable superiority.
  • Rationales: Qwen3.5-2B wins 57% of rationale comparisons, but its mean advantage is 0.0017 and is not significant (p = .38).The two rationale generators are treated as statistically indistinguishable under this metric.
  • Cross-simulator results: Qwen, Gemma, and Phi show broad score overlap and generally small concept gains, while rationales do not reliably improve over B2 and attributions improve significantly only for Qwen.Pooled prompt-type means for these judges concentrate around 0.64–0.68.
  • Cross-simulator results: Llama is the main exception: B2 has the highest pooled mean, while A, C2, C3, and R are lower.The paper cautions that Llama’s distinct behavior may reflect judge performance or simulator dependence.

K Matched explanation–baseline differences across LLM simulators

Matched comparisons show that explanation effects depend on the simulator and are usually modest, while prediction agreement reveals substantial task-prior influence. Under anonymization, classes-as-concepts ranks first by leaking the hidden label mapping, exposing a major evaluation limitation.

  • Matched score differences: Qwen, Gemma, and Phi generally favor concept prompts, whereas Llama shows negative differences for C2, C3, A, and R.The cross-simulator pattern motivates using multiple judges and reporting effect sizes rather than significance alone.
  • Prediction agreement: Off-diagonal prompt-type exact agreement ranges from 77% to 90%, including the B1 baseline without examples or explanations.This indicates that many simulator predictions remain unchanged across prompt types.
  • Dataset-specific agreement: On IMDB, prompt–task-model agreement is only 45–53% for every judge, while B1 and C1 agree with gold labels above 90%.The simulators therefore mostly follow sentiment labels rather than the task model in this balanced sample.
  • Interpretation: The agreement patterns support a shortcut hypothesis in which task text and judge priors determine much of simulator prediction rather than explanation-specific information.The evidence is descriptive and does not by itself prove explanations are unused.
  • Anonymization: Under non-anonymization, classes-as-concepts offers no material advantage; under anonymization, it ranks first and beats every contender after correction.Its anonymized mean advantage ranges from 0.022 over Vanilla SAE to 0.138 over the no-explanation baseline, with 24 of 28 pairs significant.
  • Anonymization: Anonymized simulatability can reward label leakage because classes-as-concepts reveals the mapping from hidden Class_i identifiers to class names.This reverses its non-anonymized position and shows why anonymization alone is not a sufficient safeguard.
Loading 2609.08585v1…