PaperlayerView exact revision on arXiv

Source-linked AI summary

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri

arXiv:2609.05385v1cs.AI

TL;DR

LLM explanations used in agent oversight may name factors that do not match the component’s observable decision behaviour. This paper uses black-box interventions to test necessity and sufficiency across two synthetic tasks, finding useful but inconsistent agreement and unreliable identification of the strongest factors.

  • Problem

    The paper asks whether LLMs’ cited top-three factors reliably reflect and rank the factors influencing their outputs, because explanations may guide monitoring, diagnosis, and escalation.

  • Method

    The study compares fixed top-three explanations from eight Claude, GPT, and Gemini models with necessity and sufficiency scores obtained by controlled input interventions.

  • Results

    The cited explanations contain useful but inconsistent information, and their ordering and feature selection only partially agree with measured influence across models, tasks, and criteria.

  • Takeaways & Limitations

    Explanations used for oversight should be treated as testable behavioural claims, and black-box checks can expose mismatches at an individual LLM decision boundary.

  • Takeaways & Limitations

    The study uses synthetic advisor and prompt-monitoring inputs, eight models, and temperature-zero decoding, and does not assess recommendation accuracy, fairness, user understanding, or downstream decision effects.

Abstract

from arXiv · show

LLM decision components that can operate within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use the named factors to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes that the explanations agree with the component's observable decision behaviour. We test two interpretations of the named factors: necessity, meaning that changing a factor would change the output, and sufficiency, meaning that retaining it while removing other changeable information would preserve the output. We evaluate these interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Models return an output and the top three factors that most influenced it. Controlled black-box interventions estimate a necessity score for each factor by measuring how often changing it changes the output, and a sufficiency score by measuring how often retaining it preserves the output. Across eight models from the Claude, GPT, and Gemini families, the mean Spearman correlations between the cited ranking and the necessity and sufficiency scores are 0.349 and 0.354 for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. Furthermore, an uncited factor scores above the lowest-scoring cited factor in 57.6% of advisor responses under necessity and 58.1% under sufficiency; the corresponding prompt-monitoring rates are 25.8% and 8.9%. The cited top three contain useful information but do not reliably identify the three factors with the strongest measured influence under necessity or sufficiency. The framework provides a black-box reliability check for explanations used in agent oversight while remaining scoped to individual LLM decisions.

1 Introduction

The paper tests whether LLMs’ cited top-three factors correspond to observable decision influence under necessity and sufficiency interpretations. It proposes controlled interventions to compare cited rankings with measured influence and to detect stronger uncited factors.

  • LLM explanations may be used to diagnose errors, revise prompts, or decide when to escalate recommendations and judgements.
  • A cited feature is necessary when changing it changes the output, and sufficient when retaining it while removing other changeable information preserves the output.
  • The study asks whether cited rankings agree with necessity and sufficiency scores and whether stronger uncited features were omitted.
  • Controlled input interventions estimate necessity from output changes and sufficiency from output preservation, using fixed dataset-specific feature names for the top three citations.
  • Across eight models and two use cases, the framework evaluates whether explanations identify and rank influential factors without claiming to recover internal reasoning.

2 Related Work

Prior work evaluates explanations through similarity, feature matching, or free-form reasoning traces, but these approaches do not directly test necessity or sufficiency. This paper treats self-ranked features as testable claims against black-box interventions.

  • Explainable recommendation research has used necessity and sufficiency concepts, counterfactual recommendation methods, and intervention-based analyses of selected text.
  • Recommendation-explanation measures such as lexical overlap, semantic similarity, and feature matching assess writing or reference resemblance rather than whether named factors are necessary or sufficient.
  • Studies of LLM self-explanations find that plausible rationales may not reflect influential factors, with faithfulness varying across models and settings.
  • This paper addresses the gap by testing each elicited top-three feature against empirical necessity and sufficiency scores from black-box input interventions.
  • Prompt sensitivity and systematic disagreement in automated judging motivate caution when decisions and explanations are evaluated together.

3 Methodology

The methodology evaluates fixed top-three feature explanations from eight models using controlled black-box interventions in advisor recommendation and prompt-risk monitoring. It measures necessity, sufficiency, ranking agreement, and whether uncited features outperform cited ones.

  • Datasets and models: The advisor task uses 100 synthetic client profiles with 18 features and 13 advisors, while prompt monitoring uses 100 synthetic prompts containing three or four risk segments.
  • Datasets and models: Eight Claude, GPT, and Gemini models return decisions and top-three explanations using fixed dataset-specific feature names at temperature zero.
  • Necessity: Necessity interventions change only feature f and estimate PN from how often the model changes its advisor or integer risk score across three trials.
  • Sufficiency: Sufficiency interventions retain feature f while removing other changeable information and estimate PS from how often the original decision is reproduced.
  • Evaluation criteria: Spearman correlations compare the cited order with PN and PS, while uncited-feature analysis compares the highest-scoring eligible uncited feature with the lowest-scoring cited feature.
  • Evaluation criteria: The set-level analysis compares the first k cited features with the k highest-scoring uncited features for k=2 and k=3.
  • Prompt selection: Fixed feature names reduced mean positional instability from 0.929 to 0.235 for advisor recommendation and from 0.858 to 0.240 for prompt monitoring.
  • Experimental controls: Contrast substitution and blank removal affect absolute PN values but preserve the principal descending rank pattern across cited positions.

4 Results

The cited top-three ranking carries partial, context-dependent information about measured feature influence, but uncited features frequently outperform cited ones. Higher reasoning effort reduces sufficiency-based omissions in GPT-5.4, while broader rankings remain unstable beyond the top three.

  • 4.2 The Cited Ranking Carries Partial, Context-Dependent Signal: 0.903 and 0.808: citation frequency correlates strongly with necessity and sufficiency in advisor recommendation, versus 0.690 and 0.500 in prompt monitoring.The prompt-monitoring calculation is conditional on the risk feature being active, while all client features are active in every advisor profile.
  • 4.2 The Cited Ranking Carries Partial, Context-Dependent Signal: The cited top-three order has partial agreement with measured influence, with mean Spearman correlations of 0.349 and 0.354 in advisor recommendation and 0.431 and 0.580 in prompt monitoring.The paired values correspond to necessity and sufficiency, respectively; model-level means range from 0.014 to 0.755.
  • 4.3 Higher GPT-5.4 Reasoning Effort Reduces Sufficiency-Based Omissions: 76% to 40%: GPT-5.4 advisor responses omitting a higher-sufficiency uncited feature decline as reasoning increases from none to High, with prompt monitoring falling from 25% to 3%.The intermediate advisor rates are 60%, 48%, and 40%; prompt-monitoring rates are 12%, 9%, and 3% at Low, Medium, and High reasoning.
  • 4.3 Higher GPT-5.4 Reasoning Effort Reduces Sufficiency-Based Omissions: Higher reasoning does not steadily improve cited-feature ranking correlations: GPT-5.4 advisor necessity rises through Medium reasoning before falling at High, while sufficiency fluctuates.The necessity values are 0.194, 0.428, 0.452, and 0.353; sufficiency values are 0.288, 0.515, 0.450, and 0.498.
  • 4.4 Uncited Features Often Outscore the Weakest Cited Feature: 57.6% and 58.1%: an uncited feature exceeds the lowest-scoring cited feature under necessity and sufficiency in advisor recommendation, versus 25.8% and 8.9% in prompt monitoring.The comparison is made separately under each criterion, using the relevant lowest-scoring cited feature as comparator.
  • 4.4 Uncited Features Often Outscore the Weakest Cited Feature: In advisor recommendation, the highest-scoring uncited feature has mean PN 0.698 and PS 0.492, exceeding cited ranks 2 and 3 and all three cited positions under PS.In prompt monitoring, the highest-scoring uncited feature exceeds cited rank 3 under both PN and PS but remains below cited rank 1.
  • 4.5 Exploratory GPT-5.4 Full Rankings Show Instability Beyond the Top-Three Boundary: The exploratory full-ranking pilot found individual rank inversions despite aggregate concentration near early ranks, so it does not establish an optimal explanation length or extend the primary top-three findings.For advisor recommendation, rank 4 exceeds rank 3 in 7.7% of responses, and a below-top-three feature exceeds the cited-top-three minimum PN in 55.0%.

Limitations

The intervention scores capture observable dependence only under the specific changes tested, not change magnitude, practical importance, or internal computation. The primary evaluation focuses on cited top-three features and does not establish explanation completeness or generality beyond the synthetic settings studied.

  • Necessity and sufficiency scores record whether outputs change or are preserved, not the size or practical importance of those changes.They also do not recover the model’s internal computations, so values should be interpreted relative to the intervention used.
  • The primary evaluation tests only whether cited top-three features identify and rank the strongest features under necessity or sufficiency.It does not determine the best explanation length or the completeness of an open-ended explanation.
  • The full-ranking pilot examines lower ranks only for GPT-5.4 with reasoning disabled and remains separate from the primary evaluation.
  • The study uses synthetic inputs, eight Claude, GPT, and Gemini models, and temperature-zero decoding, so other settings may produce different patterns.It also does not assess recommendation or risk-score quality, explanation understandability, or downstream decision improvement.

5 Conclusion

The conclusion treats cited explanations as partial, variable evidence about observable feature influence rather than dependable rankings. For oversight, feature selection and ordering should be treated as distinct, testable reliability concerns.

  • 5 Conclusion: The cited ranking provides partial signal whose strength depends on the model, use case, and necessity or sufficiency criterion.Citation frequency follows measured influence in advisor recommendation, while prompt monitoring shows a salience gap.
  • 5 Conclusion: A cited feature set can exclude a higher-scoring feature even when within-set ordering correlates highly with intervention scores.
  • 5 Conclusion: Explanations used for monitoring, debugging, or escalation should therefore be treated as sets of testable claims about decision behaviour.
  • A.1 Selecting a Fixed-Feature Prompt: Fixed feature names reduced positional instability from 0.929 to 0.235 for advisor recommendation and from 0.858 to 0.240 for prompt monitoring.They support reproducible feature identification and consistent perturbation across repeated queries.
  • A.1 Selecting a Fixed-Feature Prompt: No explanation-prompt variation consistently improves alignment between cited rank and necessity scores across models and use cases.

B Reproducibility and Data Availability

The reproducibility materials describe synthetic advisor and prompt-monitoring inputs, fixed-feature prompts, and intervention-ready schemas. Exact synthetic records are withheld because they use proprietary schemas and include potentially sensitive or harmful prompt content.

  • Advisor-recommendation prompt: The advisor use case uses synthetic client records with 18 fields and a fixed pool of 13 advisors.Client and advisor attributes are sampled from predefined value pools and rendered in structured prompts.
  • Data availability: Exact synthetic records are not released because they use proprietary evaluation schemas and contain potentially sensitive or harmful prompt content.The paper instead reports schemas, representative value types, risk categories, prompts, and intervention procedures.
  • Advisor-recommendation prompt: The primary advisor prompt asks the model to recommend the best-matched advisor and rank exactly three feature keys by importance.The prompt supplies the client profile and complete advisor pool, then requires fixed dataset feature names.
  • Prompt-monitoring judge prompt: The prompt-monitoring use case asks models to rate risk from 1 (benign) to 5 (critical) and rank three named risk features.Risk categories include personally identifiable information, illegal advice, high-stakes contexts, jailbreaks, profanity, harmful requests, data extraction, and social engineering.

C Perturbation Design and Validation

The study validates intervention design by comparing contrast, blank-removal, and similar-value changes while defining sufficiency as retaining one feature and removing other changeable information. Rank patterns are broadly stable across intervention choices, but absolute necessity scores depend on change strength.

  • Perturbation design: Contrast substitution replaces a feature with a deliberately different valid value, while blank removal removes its information without introducing an alternative.Contrast substitution is primary for advisor recommendation; segment removal is primary for prompt monitoring.
  • Perturbation validation: 0.615, 0.438, and 0.322 are the advisor mean PN values across cited ranks under contrast substitution, versus 0.531, 0.335, and 0.254 under blank removal.Both interventions show the same descending rank pattern, while contrast substitution yields higher absolute scores.
  • Perturbation design: Similar-value substitution makes a small valid change intended to preserve most of the original feature information.Ordinal values move to adjacent intervals, 1–10 values move by one point, and binary values remain unchanged.
  • Perturbation validation: 0.706 to 0.311 is the decrease in prompt-monitoring mean rank-1 PN from contrast to similar-value substitution.Similar-value scores remain nearly constant across ranks, showing that measured necessity responds to input-change strength.
  • Sufficiency intervention: The sufficiency intervention retains the evaluated feature while removing the other changeable information.For advisors, the remaining 17 client features become [UNKNOWN]; for prompt monitoring, other risk segments are removed.

D Additional Necessity and Sufficiency Results

These additional results disaggregate citation calibration by feature and report the metrics underlying the ranking analysis. Figure 10 compares necessity, sufficiency, and citation frequency across the two use cases.

  • D Additional Necessity and Sufficiency Results: Figure 10 reports mean PN, mean PS, and their descriptive mean for each feature, alongside conditional citation frequency.Features are ordered by decreasing mean PN within each use case.

D.2 Agreement Between Necessity and Sufficiency Rankings

Necessity and sufficiency rankings are not consistently aligned within the cited top three. Their agreement is lower for advisor recommendation than for prompt monitoring.

  • D.2 Agreement Between Necessity and Sufficiency Rankings: Mean agreement ranges from 0.18 to 0.35 in advisor recommendation and from 0.38 to 0.87 in prompt monitoring.The correlation is undefined when all three PN or PS scores are equal, so included responses vary across models.
  • D.2 Agreement Between Necessity and Sufficiency Rankings: Figure 11 reports mean Spearman correlation between necessity and sufficiency rankings for the three cited features.The analysis excludes responses with constant scores separately for PN and PS.

D.3 Cited-Feature Rankings Across Models

Agreement between cited feature order and intervention scores varies substantially across models, use cases, and criteria. No model is consistently strongest across all four comparisons.

  • D.3 Cited-Feature Rankings Across Models: Mean necessity and sufficiency correlations range from 0.014 to 0.755 across models, use cases, and criteria.The highest reported correlations are 0.711 for prompt-monitoring necessity and 0.755 for prompt-monitoring sufficiency, both for Gemini 3.5 Flash.
  • D.3 Cited-Feature Rankings Across Models: Sonnet 5 has the highest advisor-recommendation necessity correlation at 0.571, while GPT-5.4 Low has the highest advisor-recommendation sufficiency correlation at 0.515.Sonnet 5’s sufficiency correlation differs between advisor recommendation and prompt monitoring, at 0.273 and 0.698 respectively.

D.4 Cross-Model Reproducibility

Models show greater feature-set overlap in prompt monitoring but greater ordering agreement in advisor recommendation. Set-level advisor comparisons also show only small differences between cited and uncited feature groups.

  • D.4 Cross-Model Reproducibility: Mean feature-set overlap is higher in prompt monitoring than advisor recommendation, at 0.729 versus 0.676.GPT-5.4 settings have a mean advisor-recommendation overlap of 0.827, while Claude models show the strongest reported within-family prompt-monitoring consistency.
  • D.4 Cross-Model Reproducibility: Mean ordering agreement is higher in advisor recommendation than prompt monitoring, at 0.873 versus 0.776.GPT-5.4 settings reach a mean advisor-recommendation ordering agreement of 0.936.
  • D.4 Cross-Model Reproducibility: The first two cited advisor features have mean PN 0.640 versus 0.617 for the two highest-scoring uncited features.At k=3, the corresponding PN values are 0.584 for cited features and 0.561 for uncited features; corresponding PS differences are also small.
Loading 2609.05385v1…