LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations

• Validity requires changing the prediction, whereas minimality requires making the smallest edit needed to change the outcome.Minimal counterfactuals are intended to isolate consequential features and can be more actionable in settings such as clinical decision-making.Source: §1 · p. 1
• The experiments evaluate SCE validity and minimality across several LLMs, datasets, and evaluation settings using tabular prediction tasks.Natural-language prompts are evaluated in structured tabular environments.Source: §1 · p. 2
• LLMs usually produce valid counterfactuals, but excessive input changes make them far from minimal and provide little insight into decision boundaries.This pattern is reported for frontier LLMs when simply asked to generate counterfactual explanations.Source: §1 · p. 2, Figure 1 · p. 2
• When instructed to make minimal counterfactuals, models generally make overly conservative edits that fail to flip predictions.Such invalid explanations can misrepresent the model’s behaviour.Source: §1 · p. 2
• Because SCEs can be valid but non-minimal or minimal but invalid, they may provide misleading explanations and risk incorrect downstream decisions in high-stakes settings.Source: §1 · pp. 1–2

• An SCE is an alternative input that the LLM itself generates to produce a specified alternative output.This differs from counterfactuals found by algorithms that iteratively perturb input keywords.Source: §2 · pp. 2–3
• Independent reevaluation in a new context window tests whether the SCE actually flips the prediction without the original prediction biasing the assessment.Source: §2 · p. 3

3 Methods

• The method uses discrete tabular datasets containing every feature combination, enabling the exact closest boundary-crossing counterfactual to be identified.The datasets are simplified versions of real-world datasets with two to four features.Source: §3.1 · p. 3, §3.2 · p. 3
• Numerical and ordinal tabular instances are converted into natural-language inputs, while SCEs are evaluated in the original tabular space.This restricted input space makes minimality measurable.Source: §3.1 · p. 3
• For each instance, the model first predicts a class, then revises the input toward the complementary class; predictions over the complete dataset define its decision boundary.Validity measures whether the revision crosses that boundary.Source: §3.1 · p. 3
• The experiments cover income, house-price, and heart-disease binary classification datasets, using two to four discrete input features.The datasets contain 1,920, 1,600, and 1,936 instances, respectively.Source: §3.2 · p. 3

3.4 Prompt settings

• Seven open-source and proprietary LLMs are evaluated under unconstrained and minimal prompting settings.The unconstrained setting asks for any prediction-flipping counterfactual, whereas the minimal setting explicitly requests the smallest necessary edit.Source: §3.3 · p. 3, §3.4 · p. 3

4.1 Valid but not minimal

• 100% validity across all three datasets for o3 still came with mean excess distances of 0.16, 0.42, and 0.50.These correspond to overshooting the decision boundary by an average of 16% of the dataset span on income, with larger excess distances on house prices and heart disease.Source: §4.1 · p. 4
• Only 2.89% average exact-match accuracy identified the minimal counterfactual in the unconstrained setting.Table 1 reports that models rarely identify the exact minimal valid counterfactual, although performance is higher under minimal prompting and no model exceeds 32%.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Minimal but rarely valid

• Validity fell from 97.14% to 36.30% on income under minimal prompting, while valid SCEs had much smaller excess distances of 0.02, 0.01, and 0.02 for o3.Across models and datasets, exact-match accuracy rose to 14.9%, indicating a trade-off rather than reliable joint satisfaction of validity and minimality.Source: §4.2 · p. 5

4.3 Where do models place SCEs?

• Unconstrained models often maximize or minimize every feature on monotonic tasks, while minimal prompting produces conservative edits that usually fail to cross the boundary.For house prices, o3 selected the maximum or minimum point in 46% of cases; in heart disease, this occurred in 34% of SCEs, whereas income showed region preferences instead.Source: §4.3 · p. 5

4.4 Robustness analysis

• The validity–minimality trade-off persisted across alternative distance functions, prompt variations, and temperature 1.0 decoding.None of the tested prompt variations produced SCEs that were both valid and minimal.Source: §4.4 · p. 6

4.5 What limits performance?

• Decision boundaries were stable except near the boundary, and 91.4% of originally invalid SCEs remained invalid across 50 perturbations.This suggests that boundary inconsistency does not explain the observed failures.Source: §4.5 · p. 7

Counterfactual explanations

• In NLP, counterfactuals are commonly generated through keyword perturbations or separate language models rather than by the predicting model itself.Source: §5 · p. 8
• Prior counterfactual-explanation research seeks actionable explanations by identifying the closest possible world, motivating minimality as a central property.Source: §5 · p. 8
• Complete tabular datasets restrict the input space enough to locate exact minimum counterfactuals and measure SCE minimality precisely.Source: §5 · p. 9

Self-explanations and SCEs

• SCEs are structured self-explanations in which an LLM generates a counterfactual input intended to explain its own prediction.Source: §5 · p. 9
• Earlier SCE studies primarily measured validity, whereas this work evaluates both validity and minimality and examines their trade-off.Source: §5 · p. 9

• SCEs trade off validity and minimality: unconstrained prompts yield trivially valid counterfactuals, whereas minimal prompts often produce edits that fail to flip predictions.Source: §6 · p. 9
• Because unreliable SCEs can mislead human decision-making and potentially cause harm, high-stakes LLM deployments should account for their impact.Source: §6 · p. 9
• Models do not spontaneously engage in self-prediction, and explicitly prompting self-prediction in reasoning traces does not significantly improve aggregate performance.Source: §6 · p. 9

Alternative minimality metrics

• Mean excess distance measures minimality for valid SCEs but does not characterize invalid SCEs or how far they lie from the decision boundary.Source: §7 · p. 10

Self-prediction capabilities

• The study does not analyze self-prediction accuracy in detail or establish whether LLMs are fundamentally capable of accurate self-prediction.Source: §7 · p. 10
• The mechanisms by which models might achieve self-prediction remain unexamined, leaving prompted or fine-tuned self-prediction as a future research question.Source: §7 · p. 10

Generalisation to real-world datasets

• Experiments use three synthetic datasets with only two to four input dimensions, so generalisation to more complex, higher-dimensional real-world data is unresolved.Source: §7 · p. 10

Datasets

• Each synthetic dataset contains every combination of its discrete feature values, completely representing the input space for exact boundary evaluation.Source: §A.1 · p. 12
• The tasks cover income, house prices, and heart disease using discrete tabular features and binary prediction targets.Source: §A.1 · pp. 12–13
• The heart-disease dataset contains 1,936 distinct instances formed from 11 × 2 × 8 × 11 feature-value combinations.Source: §A.1 · p. 13

Models and execution

• The experiments evaluate open and proprietary LLMs, with most models run at temperature 0 and o3 at its fixed temperature of 1.Source: §A.2 · p. 13

Experimental pipeline

• The pipeline first collects predictions for every dataset instance, then prompts each model to generate an SCE that should produce the complementary class.Source: §A.3 · p. 13

• Validity measures whether a revised input produces the intended target prediction when evaluated by a new model instance.Source: §A.4 · p. 13
• Minimality is evaluated with Gower’s Distance, which compares tabular instances feature by feature.Source: §A.4 · p. 13
• Gower’s Distance uses non-negative feature weights and supports binary, continuous, and ordinal variables.Source: §A.4 · p. 13
• Because the datasets use numerical, binary, or rank-converted ordinal features, Gower’s Distance simplifies to normalized feature differences.Source: §A.4 · p. 13
• The resulting distance averages per-feature differences normalized by feature range, ranging from 0 for identical instances to 1 for maximally different instances.Source: §A.4 · p. 16

• Table 6 reports average validity and excess distance for the main experiment at temperature 0.0.Source: §B.1 · p. 16, Table 6 · p. 17
• The analysis compares Llama 3.3 70B’s results under multiple distance functions, including L1, L2, and cosine distance.Source: §C.1 · pp. 16–17
• The cosine distance compares embeddings of templated tabular inputs using one minus cosine similarity.Source: §C.1 · p. 17
• The prompt-perturbation experiment generated 20 versions of the SCE-eliciting prompt and evaluated changes in Llama 3.3 70B’s behavior.Source: §C.2 · p. 17

• At temperature 1.0, aggregate results show minimal differences, consistent with an earlier experiment by Dehghanighobadi et al. (2025).Source: §C.3 · p. 17
• The decision-boundary consistency experiment generated 50 income-task prompt perturbations using OpenAI’s o3.Source: §D.1 · p. 17
• The Gower-distance validation baseline asks models to select the closest house-price instance among four alternatives.Source: §D.2 · p. 18
• Accuracy is measured over 1000 trials against the implementation of Gower’s Distance, with random-choice accuracy equal to 25%.Source: §D.2 · p. 18
• DeepSeek-R1 70B’s reasoning trace suggests that it misunderstands the self-explanation task and does not explicitly self-predict in its chain-of-thought.Source: §D.3 · p. 18

E Incentivising models to self-predict

• The six-step plan requires candidate generation, explicit self-prediction, validity and minimality assessment, revision, repetition across five candidates, and final selection.Source: E Incentivising models to self-predict · p. 18
• The plan frames the task as self-modelling and says an independent model instance will evaluate the output.Source: E Incentivising models to self-predict · p. 18
• DeepSeek-R1 70B generally follows the steps and considers candidate SCEs, but often continues making general predictions without explicitly recognizing its own decision process.Source: E Incentivising models to self-predict · p. 18
• Validity rises from 21.44% to 23.34%, while excess distance increases from 0.025 to 0.033 under the self-prediction prompt.Source: E Incentivising models to self-predict · p. 18
• The reasoning traces frequently contain incorrect self-predictions, and the authors leave whether this reflects a fundamental LLM limitation for future work.Source: E Incentivising models to self-predict · p. 18